Operation method and operation system

Abstract

An operation method suitable for a transformer model, the operation method includes the following steps. An input matrix corresponding to an input sequence is mapped to a query matrix according to a plurality of first learnable weights. The input matrix is mapped to a value matrix according to a plurality of second learnable weights. A decomposition matrix is generated by an incomplete Cholesky decomposition according to the query matrix and a transpose query matrix of the query matrix. An intermediate matrix is calculated according to a product of a transpose decomposition matrix of the decomposition matrix and the value matrix. An output matrix is calculated according to a product of the decomposition matrix and the intermediate matrix. An operation system is also disclosed. By using the incomplete Cholesky decomposition, the transformer model can process input sequences with longer lengths.

Description

Technical Field

[0001] This disclosure relates to a machine learning technique, and more particularly to a machine learning technique with data augmentation capabilities. Background Technology

[0002] Machine learning technology is applied in many fields, such as artificial intelligence, big data processing, and autonomous driving. Many different types of neural network architectures have been developed to handle various kinds of problems. Among these neural networks, the transformer model is a particularly popular one. Transformer models are commonly used to solve natural language processing problems.

[0003] A key component in the converter model is the self-attention mechanism, which enables the model to capture contextual information from the entire string sequence and has the flexibility to learn its representation from different types of data. Summary of the Invention

[0004] One embodiment of this disclosure discloses a computational method applicable to a converter model. The computational method includes: mapping an input matrix to a query matrix according to a plurality of first learnable weights, the input matrix corresponding to an input sequence containing a plurality of feature vectors, and the query matrix containing a plurality of query vectors; mapping the input matrix to a numerical matrix according to a plurality of second learnable weights, the numerical matrix containing a plurality of numerical vectors; generating a decomposition matrix by an incomplete Koleski decomposition of the query matrix and a transpose of the query matrix, wherein the dimension of the decomposition matrix is ​​smaller than the dimension of the query matrix; calculating an intermediate matrix by multiplying the transpose of the decomposition matrix and the numerical matrix; and calculating an output matrix by multiplying the decomposition matrix and the intermediate matrix.

[0005] In some embodiments, the dimension of the input matrix is ​​. , Let be the length of a sequence of the input sequence, and The dimension values ​​of these feature vectors in the input sequence.

[0006] In some embodiments, the incomplete Koleski decomposition is used to approximate the product of the decomposition matrix and the transpose of the decomposition matrix as an exponential function of a shared query key attention matrix, the dimension of which is... One parameter Corresponding to one iteration of the incomplete Koleski decomposition, the converter model uses the decomposition matrix and the transpose decomposition matrix to replace the shared query key attention matrix during the computation of the output matrix.

[0007] In some embodiments, the parameter is used in the approximation operation for the incomplete Colskii decomposition.

[0008] In some embodiments, the parameter is set to be equal to or less than .

[0009] In some embodiments, the parameter is set to be equal to or less than the rank of the shared query-key attention matrix.

[0010] In some embodiments, the shared query-key attention matrix is defined according to a product of the query matrix and a transpose query matrix of the query matrix, wherein the dimension of the intermediate matrix is less than the dimension of the shared query-key attention matrix.

[0011] In some embodiments, the dimension of the transpose query matrix is , the dimension of the shared query-key attention matrix is , and the dimension of the intermediate matrix is .

[0012] In some embodiments, the output matrix is calculated in the following manner:

[0013]

[0014] wherein represents the division between matrix elements, represents an all-one vector, H represents the decomposition matrix, H T represents the transpose decomposition matrix, H T V represents the intermediate matrix.

[0015] In some embodiments, the operation method further comprises: generating an output sequence using a fully connected layer according to the output matrix.

[0016] In some embodiments, the input sequence comprises a plurality of words or a plurality of phrases in a first language, the output sequence comprises a plurality of words or a plurality of phrases in a second language, and the transformer model is used to translate the input sequence into the output sequence.

[0017] In some embodiments, the input sequence and the output sequence each comprise a plurality of words or a plurality of phrases, the input sequence comprises an article or a file archive, and the output sequence comprises an abstract, a classification result, an answer to a question, or a title corresponding to the input sequence, and the transformer model is used to extract, determine, or generate the output sequence from the input sequence.

[0018] Another embodiment of the disclosure discloses a computing system including a memory and a processor. The memory is configured to store a plurality of computer-executable instructions. The processor is coupled to the memory and is configured to execute the computer-executable instructions to run a transformer model. The transformer model includes an attention layer configured to map an input matrix to a query matrix according to a plurality of first learnable weights, the input matrix corresponding to an input sequence including a plurality of feature vectors, the query matrix including a plurality of query vectors; map the input matrix to a value matrix according to a plurality of second learnable weights, the value matrix including a plurality of value vectors; generate a decomposition matrix from the query matrix and a transpose query matrix of the query matrix via an incomplete Cholesky decomposition, wherein a dimension of the decomposition matrix is smaller than a dimension of the query matrix; compute an intermediate matrix from a product of a transpose decomposition matrix of the decomposition matrix and the value matrix; and compute an output matrix from a product of the decomposition matrix and the intermediate matrix.

[0019] In some embodiments, a dimension of the input matrix is , is a sequence length of the input sequence, and is a dimension value of the feature vectors in the input sequence.

[0020] In some embodiments, the incomplete Cholesky decomposition is configured to cause a product of the decomposition matrix and the transpose decomposition matrix to approximate an exponential function of a shared query-key attention matrix, a dimension of the decomposition matrix is where a parameter corresponds to a number of iterations of the incomplete Cholesky decomposition, the decomposition matrix and the transpose decomposition matrix are utilized by the transformer model to replace the shared query-key attention matrix in a process of computing the output matrix.

[0021] In some embodiments, the shared query-key attention matrix is defined according to a product of the query matrix and a transpose query matrix of the query matrix, a dimension of the intermediate matrix is smaller than a dimension of the shared query-key attention matrix.

[0022] In some embodiments, a dimension of the transpose query matrix is a dimension of the shared query-key attention matrix is a dimension of the intermediate matrix is .

[0023] In some embodiments, the computing system further includes generating an output sequence from the output matrix using a fully connected layer.

[0024] In some embodiments, the input sequence comprises words or tokens in a first language, the output sequence comprises words or tokens in a second language, and the transformer model is configured to translate the input sequence into the output sequence.

[0025] In some embodiments, the input sequence and the output sequence each comprise words or tokens, the input sequence comprises an article or a file archive, and the output sequence comprises an abstract, a classification result, an answer to a question, or a title corresponding to the input sequence, and the transformer model is configured to extract, recognize, or generate the output sequence from the input sequence.

[0026] By employing the attention layer with the incomplete Collobert decomposition attention mechanism, the transformer model is capable of processing input sequences with longer length, which facilitates the use of the transformer model to translate a paper, a magazine, or a long article, because the transformer model is capable of obtaining the attention matrix of the entire input sequence and accordingly generating the output without the need to split the longer input sequence into several parts for segmented input.

[0027] It should be noted that the above description and the following detailed description are illustrative of the present application in an exemplary manner and are used to assist in explaining and understanding the content of the application as claimed. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to make the above and other objects, features and embodiments of the present disclosure more apparent, the following description of the accompanying drawings is given:

[0029] Figure 1 A schematic diagram of an operation system according to an embodiment of the present disclosure is shown;

[0030] Figure 2 A schematic diagram of a neural network architecture of a transformer model according to some embodiments of the present disclosure is shown;

[0031] Figure 3 A schematic diagram of an internal structure of an attention layer employing a self-attention mechanism in some embodiments is shown;

[0032] Figure 4 A schematic diagram of an internal structure of an attention layer employing an incomplete Collobert decomposition attention mechanism according to some embodiments of the present disclosure is shown; and

[0033] Figure 5 A flowchart of an operation method according to some embodiments of the present disclosure is shown.

[0034] SYMBOL DESCRIPTION:

[0035] 100: operation system

[0036] 120: processor

[0037] 140: memory

[0038] 160: interface

[0039] 200: converter model

[0040] 210: input embedding module

[0041] 220: encoder module

[0042] 221: attention layer

[0043] 222: feedforward layer

[0044] 230: output embedding module

[0045] 240: decoder module

[0046] 241: attention layer

[0047] 242: encoder-to-decoder attention layer

[0048] 243: feedforward layer

[0049] 250: fully connected layer

[0050] INseq: input sequence

[0051] OUTseq: output sequence

[0052] TAR: target sequence

[0053] 300: operation method

[0054] S310, S320, S330, S340, S350: steps

[0055] ALself: attention layer

[0056] L1: mapping layer

[0057] L2: softmax function layer

[0058] AL ICF : attention layer

[0059] L3: mapping layer

[0060] L4: incomplete Cholesky decomposition function layer

[0061] M IN : input matrix

[0062] M OUT : output matrix

[0063] W Q : learnable weight

[0064] W K Learnable weights

[0065] W V Learnable weights

[0066] Q: Query matrix

[0067] K: Key Matrix

[0068] V: Numerical matrix

[0069] QK T Attention matrix

[0070] H: Decomposition matrix

[0071] H T : Transpose decomposition matrix

[0072] H T V: Intermediate Matrix Detailed Implementation

[0073] The following disclosure provides numerous different embodiments or examples for implementing various features of this disclosure. Elements and configurations in the specific examples are used in the following discussion to simplify this disclosure. Any examples discussed are for illustrative purposes only and do not in any way limit the scope or meaning of this disclosure or its examples. Where appropriate, the same reference numerals are used in the accompanying drawings and corresponding text descriptions to represent the same or similar elements.

[0074] Please see Figure 1 The diagram illustrates a computing system 100 according to an embodiment of the present disclosure. The computing system 100 includes a processor 120, a memory 140, and an interface 160. The computing system 100 can compute, train, and use a transformer model 200.

[0075] In some embodiments, memory 140 is used to store computer-executable instructions, training data (during the training phase of converter model 200), learnable parameters of converter model 200, input data to be processed by converter model 200, and / or output data generated by converter model. In some embodiments, memory 140 may be implemented using random access memory modules, read-only memory modules, flash memory, hard disks, cache memory, static random access memory, dynamic random access memory, nontransitory memory, solid-state drives, optical storage media, or other equivalent storage components. In some embodiments, memory 140 stores instructions executable by processor 120 for performing the detailed steps described in subsequent paragraphs.

[0076] The processor 120 is coupled with the memory 140. The processor 120 is configured to execute computer-executable instructions to compute, train, or operate the converter model 200. In some embodiments, the converter model 200 is configured to perform various natural language tasks, such as answering questions, document classification, name extraction, discourse coherence parsing, natural language inference, document summarization, and translation, etc.

[0077] In some embodiments, the processor 120 can include a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a tensor processing unit (TPU), a digital signal processor (DSP), a single-instruction multiple-data (SIMD) processor, and / or any equivalent processing circuitry. Generally, such processors can accelerate various computing tasks related to neural network model estimation (e.g., training, prediction, pre-processing, etc.) compared to a general-purpose central processing unit (CPU), and the acceleration can be by an order of magnitude or more.

[0078] The interface 160 is coupled with the processor 120. The interface 160 can include a keyboard, a display, a network transceiver, a connection port (e.g., a USB connection port), a touch panel, or any equivalent input / output component through which a user or other external device can provide input to the converter model 200 or receive output generated by the converter model 200.

[0079] In one example, when the converter model 200 is configured to perform a translation task, a user can input an article in a first language, such as a financial report in Chinese, and the processor 120 is configured to transmit the financial report as an input sequence to the converter model 200. In some embodiments, the input sequence includes a plurality of characters or a plurality of words in the first language (e.g., Chinese). The converter model 200 is configured to translate the financial report in Chinese to another language and generate an output sequence. The output sequence includes a plurality of characters or a plurality of words in the second language (e.g., English). In this case, the converter model 200 is configured to translate the input sequence to the output sequence. The converter model 200 in the present disclosure is not limited to performing a translation task.

[0080] In some embodiments, the transformer model 200 is used to perform the task of extracting the topic of a document. In such an example, the input sequence comprises an article or a document archive, and the output sequence generated by the transformer model 200 contains a summary corresponding to the article or the document archive. For example, in the case where the corresponding input sequence contains a financial report, the output sequence generated by the transformer model 200 can be “inflation in the Asian region”. In other cases, the output sequence generated by the transformer model 200 can be a classification result, an answer to a question, or an article title, etc. corresponding to the input sequence. The transformer model 200 is used to extract, resolve, or generate the output sequence from the input sequence.

[0081] In some embodiments, in order to accomplish the above-mentioned tasks (translation, classification, or answering questions), the transformer model 200 in the present disclosure employs an incomplete Cholesky factorization (ICF) attention mechanism, thereby enabling the transformer model 200 to have flexibility in extracting information from the context of the entire input sequence and learning the expression of different types of data. The function achieved by the incomplete Cholesky factorization attention mechanism discussed in the present disclosure is similar to that of a self-attention mechanism. A self-attention mechanism is a full-vector-to-full-vector attention mechanism (i.e., all attention vectors are calculated with respect to all attention vectors). In this case, a transformer model with a self-attention mechanism will occupy a large memory storage space (for storing a large amount of vector data used in full-vector-to-full-vector attention). When the length of the string of the input sequence increases, the memory occupied by the self-attention mechanism will grow in a quadratic order. The quadratic order growth of the memory usage will become a resource bottleneck for the transformer model 200 in the training phase. As a result, the transformer model with a self-attention mechanism requires a large amount of memory resources and time in the pre-training phase, and also makes it difficult for the transformer model with a self-attention mechanism to process input sequences with a long length, such as high-resolution images and / or large files with a large number of words.

[0082] In some embodiments, the transformer model 200 in the present disclosure employs an incomplete Cholesky factorization attention mechanism, which can produce results similar to a self-attention mechanism. The transformer model 200 employing the incomplete Cholesky factorization attention mechanism achieves the effect of a full-vector-to-full-vector attention mechanism with linear growth in memory complexity. In some embodiments, the transformer model 200 employing the incomplete Cholesky factorization attention mechanism can maintain the flexibility of the transformer model 200 and solve the problem of quadratic order increase in memory usage mentioned earlier.

[0083] Please refer to further information. Figure 2 This illustrates a schematic diagram of the neural network architecture of a transformer model 200 according to some embodiments of this disclosure. For example... Figure 2 As shown, the transformer model 200 includes an input embedding module 210, an encoder module 220, and a decoder module 240.

[0084] The input embedding module 210 is used to convert the input sequence INseq into an input representation. In some embodiments, the input embedding module 210 is used to tokenize the input sequence (e.g., a string containing a series of words) and map each token to a vector representation in a multidimensional vector space. For example, a token corresponding to a word can be mapped to a 100-dimensional vector representation to correspond to the meaning of the word.

[0085] In an exemplary example, suppose the input sequence is "Today is a good day" with a sequence length of 5. The first word in the input sequence, "today," is mapped to a vector representation using a 100-dimensional vector representation (e.g., a 1x100 matrix). The second word in the input sequence, "day," is mapped to another vector representation, also using a 100-dimensional vector representation. In this case, the entire input sequence containing five words can be represented as an input representation of dimension 5x100. Similarly, if the input sequence contains 2048 words, the input sequence will be represented as an input representation of dimension 2048x100.

[0086] Encoder module 220 generates an encoded representation based on the input representation of the corresponding input sequence INseq. Decoder module 240 generates or predicts the output sequence OUTseq based on the encoded representation generated by encoder module 220.

[0087] In some embodiments, converter model 200 includes a fully connected layer 250 coupled to decoder module 240. In some embodiments, the fully connected layer 250 generates an output OUTseq based on the output matrix generated by decoder module 240.

[0088] In some embodiments, the converter model 200 may include an output embedding module 230 for generating an output representation based on the output sequence OUTseq or the target sequence TAR. Generally, the output embedding module 230 may perform an embedding operation based on the output sequence OUTseq, similar to the embedding operation performed by the input embedding module 210 based on the input sequence INseq described above.

[0089] like Figure 2 As shown, in some embodiments, encoder module 220 includes attention layer 221 and feedforward layer 222, and decoder module 240 includes attention layer 241, encoder-to-decoder attention layer 242, and feedforward layer 243. In these layers, attention layer 241 and encoder-to-decoder attention layer 242 are used to find attention relationships between different tokens (e.g., corresponding words in the input or output sequence), while feedforward layers 222 and 243 are used to generate learnable parameters used by converter model 200.

[0090] Further reading Figure 3 This illustrates a schematic diagram of the internal structure of an attention layer ALself employing a self-attention mechanism in some embodiments. In some practical cases, an attention layer ALself with a self-attention mechanism can be used in at least one (or each) of attention layer 241 and encoder-to-decoder attention layer 242. In some embodiments, the attention layer ALself is... Figure 1 The program instructions executed by the processor 120 shown are implemented.

[0091] like Figure 3 As shown, the attention layer ALself, which employs a self-attention mechanism, contains a mapping layer L1 and a softmax function layer L2.

[0092] The attention layer ALself, employing a self-attention mechanism, allows a label to focus on all other labels in the entire sequence and incorporate information from those other labels. The input matrix M corresponds to the input sequence INseq. IN The mapping layer L1 is obtained by applying three linear mappings (with three learnable weights W). Q W K and W V The process is used to send out the query vector, key vector, and value vector respectively.

[0093] In some embodiments, the input matrix M IN The dimension is ,in is the sequence length of the input sequence INseq, is the dimension value of the feature vector in the input sequence INseq. For example, when the input sequence INseq contains 2048 words, and each word in the input sequence INseq is mapped to a 100-dimensional vector representation, the input matrix M IN will be a 2048x100 matrix (n = 2048, d = 100). Due to the parallelization process, the query vectors generated by the mapping layer L1 will be packed into a query matrix Q with dimension Similarly, the key vectors and the value vectors are packed into a key matrix K with dimension and a value matrix V with dimension respectively.

[0094] The output matrix M OUT generated by the attention layer ALself with self-attention mechanism is defined as:

[0095] Equation (1)

[0096] In Equation (1), the attention matrix QK T is the product between the query matrix Q and the transpose key matrix K T of the key matrix K. The attention matrix QK T holds the attention values between all pairs of tokens and relates all tokens of the whole sequence to each other. The attention matrix QK T is divided by a scaling factor and passed to the softmax function layer L2. The output of the softmax function layer L2 is which represents all attention weights between the query vector and all key vectors. The output of the softmax function layer L2 is linearly combined with the value matrix V to generate the output matrix M OUT .

[0097] In the above calculation process of generating the output matrix M OUT , the softmax function layer L2 is calculated based on the attention matrix QK T . In this case, the query matrix Q is a matrix with dimension ; the transpose key matrix K T of the key matrix K is a matrix with dimension ; therefore the attention matrix QK T is a matrix with dimension . With the increase of the sequence length , the attention matrix QK TThe memory usage of the model will grow in quadratic order, which will become a resource bottleneck in the training process.

[0098] Further referring to Figure 4 , which shows an internal structure diagram of an attention layer ALICF employing an incomplete Krylov subspace decomposition attention mechanism according to some embodiments of the present disclosure. In some embodiments, the attention layer ALICF employing an incomplete Krylov subspace decomposition attention mechanism can be used in at least one (or each) of the attention layer 241, the attention layer 241, and the encoder-to-decoder attention layer 242. In some embodiments, the attention layer ALICF is implemented by program instructions executed by the processor 120 as shown in Figure 1

[0099] As shown in Figure 4 , the attention layer ALICF employing an incomplete Krylov subspace decomposition attention mechanism includes a mapping layer L3 and an incomplete Krylov subspace decomposition function layer L4. Please further refer to Figure 5 , which shows a flowchart of an operation method 300 that can be performed by the attention layer ALICF employing an incomplete Krylov subspace decomposition attention mechanism in Figure 4

[0100] As shown in Figure 4 and Figure 5 , in step S310 of the operation method 300, the mapping layer L3 maps the input matrix M Q corresponding to the input sequence INseq (see Figure 2 ) to a query matrix Q. The query matrix Q includes a plurality of query vectors corresponding to the tokens (e.g., words from the input sequence INseq) of all the feature vectors in the input matrix M IN . IN

[0101] In some embodiments, the input matrix M IN has a dimension of , where is the sequence length of the input sequence INseq, and is the dimension value of the feature vectors in the input sequence INseq. In some embodiments, the query vectors generated by the mapping layer L3 are packed into the query matrix Q with a dimension of . For example, when the input sequence INseq includes 2048 words, and each word in the input sequence INseq is mapped to a 100-dimensional vector representation, the input matrix M IN ​​​The matrix Q will be a 2048x100 matrix (n=2048, d=100), and the query matrix Q will also be a 2048x100 matrix.

[0102] like Figure 4 as well as Figure 5 As shown, in step S320 of the operation method 300, the mapping layer L3 calculates based on multiple second learnable weights W. V The corresponding input sequence INseq (see...) Figure 2 The input matrix M IN This is mapped to a numerical matrix V. The numerical matrix V contains multiple numerical vectors, each corresponding to an input matrix M. IN The labels of all feature vectors in the input sequence (e.g., words from the input sequence INseq).

[0103] In some embodiments, the numerical vectors generated by the mapping layer L3 are packaged into a... In a numerical matrix V of dimension M. For example, when the input matrix M... IN When the matrix is ​​a 2048x100 matrix (n=2048, d=100), the numerical matrix V will also be a 2048x100 matrix.

[0104] like Figure 4 as well as Figure 5 As shown, in step S330 of operation method 300, the incomplete Koleski decomposition function layer L4 is used to perform the query matrix Q (generated by the mapping layer L3) and the transpose of the query matrix Q, query matrix Q'. T The incomplete Koleski decomposition produces the decomposition matrix H. In some embodiments, the dimension of the decomposition matrix H is smaller than the dimension of the query matrix Q. Further details on how the decomposition matrix H is generated will be discussed in the following paragraphs.

[0105] In step S330, the incomplete Koleski decomposition function layer L4 is used to compute the decomposition matrix H and the transpose decomposition matrix H of the decomposition matrix H. T And make the decomposition matrix H and the transpose decomposition matrix H T The product between HH T It approximates a shared-QK attention matrix. T The exponential function.

[0106] Shared Query Key Attention Matrix QQ T Defined as the query matrix Q and the transpose of the query matrix Q. T The product between them. For example... Figure 4 As shown, attention layer AL ICF No key matrix generated (see reference) Figure 3The key matrix K) in the attention layer AL Figure 4 The key vector value is defined as equal to the query vector value in the shared query key attention matrix QQ ICF The shared query key attention matrix QQ T will be symmetric and positive semi-definite. In some embodiments, the effect of the shared query key attention in certain applications can be comparable to the non-shared approach, for example in sound source classification, frame-level sound source classification, and phoneme classification. It is noted that the shared query key attention matrix QQ T still has dimension If the shared query key attention matrix QQ T is directly used to compute the output matrix M OUT , it will still occupy a large amount of memory space similar to the self-attention mechanism.

[0107] In some embodiments, the incomplete Cholesky decomposition function layer L4 is used to compute the decomposition matrix H and the transpose decomposition matrix H T , and the incomplete Cholesky decomposition function can ensure that the product HH T approximates the exponential function of the shared query key attention matrix QQ T , so that the decomposition matrix H and the transpose decomposition matrix H T can be used as substitutes for the shared query key attention matrix QQ T .

[0108] To explain the principle of matrix approximation, the foregoing equation (1) is rewritten as and the transpose query matrix Q T is used instead of the transpose key matrix K T (due to the shared query key), and is expanded according to the definition of the softmax function as the following equation (2):

[0109]

[0110] … equation (2)

[0111] In equation (2), is the exponential function of the shared query key attention matrix QQ T ; represents the division between matrix elements; represents a full 1 vector; in equation (2), the scaling factor is omitted for simplicity.

[0112] It is noted that the exponential function is a symmetric and positive semi-definite matrix. The incomplete Cholesky decomposition can be used to generate a decomposition matrix H with dimension A smaller decomposition matrix H, such that it can approximate a matrix of dimension 1. A symmetric and positive definite matrix, where Due to the exponential function Meeting the requirements of being a symmetric and positive definite matrix, the incomplete Kolesky decomposition can produce a decomposition matrix H and a transpose decomposition matrix Ht. T This enables it to approximately share the query key attention matrix QQ. T The Hadamard exponential function.

[0113] parameter These are dynamic variables approximated by incomplete Kolesky decomposition. In some embodiments, the parameters... Set to equal to or less than . when Set to equal to Incomplete Koleski decomposition can effectively reduce the size of the decomposition matrix H and preserve most of the features of the original full vector in the full vector attention.

[0114] In some other embodiments, parameters Set to be equal to or less than the shared query key attention matrix QQ T The rank. If the parameter Greater than the shared query key attention matrix QQ T If the rank is zero, then the decomposition matrix H in this case will include a column of all zeros, which does not carry any valid information and will occupy unnecessary memory.

[0115] As mentioned above, the dimension of the query matrix Q is In some embodiments, the dimension of the decomposition matrix H generated by the incomplete Koleski decomposition function layer L4 is . ,and Where p is a parameter corresponding to the number of iterations in the incomplete Kolesky decomposition. In calculating the output matrix M... OUT At that time, converter model 200 utilizes the decomposition matrix H and the transpose decomposition matrix H T To replace the shared query key attention matrix QQ T .

[0116] The decomposition matrix H and the transpose decomposition matrix H are obtained by incomplete Koleski decomposition. T Then, the product of the two matrices (HH) T This can be used to replace the shared query key attention matrix QQ in the aforementioned equation (2). T The exponential function, and Figure 4 The attention layer AL shownICF the output matrix M generated OUT is defined as follows:

[0117] Equation (3)

[0118] In step S340, the attention layer AL executed by the processor ICF to calculate the intermediate matrix H T from the product of the transposed decomposition matrix H T and the numerical matrix V.

[0119] In step S350, the attention layer AL executed by the processor ICF to calculate the output matrix M OUT from Equation (3) above. In other words, the calculation of the output matrix M OUT is performed from the product of the decomposition matrix H T and the intermediate matrix H T V.

[0120] In some embodiments, the dimension of the intermediate matrix H T V is smaller than the dimension of the shared query-key attention matrix QQ T As mentioned above, the dimension of the query matrix Q is ; the dimension of the transposed query matrix Q T is ; the dimension of the query-key attention matrix QQ T is ; the dimension of the transposed decomposition matrix H T is ; the dimension of the numerical matrix V is ; and the dimension of the intermediate matrix H T V is .

[0121] In most cases when the input sequence contains a large document (i.e. the sequence length is relatively large), the sequence length will be significantly larger than and i.e. and . For example, when the large document contains three thousand words or more, will reach 3000 or more. Therefore, the dimension of the intermediate matrix H T V will be significantly smaller than the dimension of the shared query-key attention matrix QQ T .

[0122] In the embodiment of the attention layer ALself employing the self-attention mechanism shown in Figure 3 , the output matrix M OUTIt is based on the attention matrix QK T exponential function The calculated complexity of this exponential function is... When the length of the input sequence INseq As the number of self-attention mechanisms increases, the memory usage of these mechanisms grows exponentially.

[0123] exist Figure 4 The attention layer AL shown employs an incomplete Kolesky decomposition attention mechanism. ICF In the embodiments. For example... Figure 4 As shown, the intermediate matrix H T The dimension of V is The memory complexity used in the calculation of step S340 is In step S350, the decomposition matrix H (with dimension H) is... ) and intermediate matrix H T V (dimension is) The calculations are performed between ( ), so that the memory complexity used for the calculation in step S350 is also O(n). Therefore, the attention layer AL is calculated. ICF The memory space used is linearly related to the length of the input sequence INseq.

[0124] like Figure 2 and Figure 4 As shown, the output matrix M of an attention layer OUT It can be the input matrix of another attention layer. For example, the output matrix generated by attention layer 241 can be the input matrix of encoder-to-decoder attention layer 242.

[0125] In some embodiments, the output matrix M of the attention layer OUT It can be transmitted to, for example Figure 2 The fully connected layer 250 is shown. (As shown in the image.) Figure 2 As shown, the fully connected layer 250 is used to determine the output matrix M. OUT Generate the output sequence OUTseq.

[0126] By employing the incomplete Kolesky decomposition of the attention mechanism, the attention layer AL is used. ICF The converter model 200 is capable of handling input sequences with longer lengths. It is particularly useful when translating papers, magazines, or long articles, as it can obtain the attention matrix of the entire input sequence and generate the output accordingly without having to segment the long input sequence into several parts.

[0127] While certain embodiments of the present disclosure have been disclosed with respect to the above described embodiments, these embodiments are not intended to limit the present disclosure. Various alternatives and modifications can be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined in the appended claims. Therefore, the scope of the present disclosure is defined not by the detailed description of the disclosure, but by the appended claims.

Claims

1. A calculation method, characterized in that, This calculation method is applicable to a converter model and includes: An input matrix is ​​mapped to a query matrix based on multiple first learnable weights. The input matrix corresponds to an input sequence containing multiple feature vectors, and the query matrix contains multiple query vectors. The input matrix is ​​mapped to a numerical matrix based on multiple second learnable weights, and the numerical matrix contains multiple numerical vectors. A decomposition matrix is ​​generated by an incomplete Koleski decomposition based on the query matrix and a transpose of the query matrix, wherein the dimension of the decomposition matrix is ​​smaller than the dimension of the query matrix. Calculate an intermediate matrix based on the product of the transpose of the decomposition matrix and the numerical matrix; Calculate an output matrix based on the product of the decomposition matrix and the intermediate matrix; as well as An output sequence is generated using a fully connected layer based on the output matrix, wherein the converter model is used to translate the input sequence into the output sequence, or the converter model is used to extract, distinguish or generate the output sequence from the input sequence; The input sequence contains multiple characters or words in a first language, and the output sequence contains multiple characters or words in a second language; or The input sequence and the output sequence each contain multiple characters or words. The input sequence contains an article or a file. The output sequence contains a summary, a classification result, an answer to a question, or a title corresponding to the input sequence.

2. The calculation method as described in claim 1, wherein the dimension of the input matrix is... , Let be the length of a sequence of the input sequence, and The dimension values ​​of these feature vectors in the input sequence.

3. The computational method as described in claim 2, wherein the incomplete Koleski decomposition is used to approximate the product of the decomposition matrix and the transpose of the decomposition matrix as an exponential function of a shared query key attention matrix, the dimension of the decomposition matrix being... ,in, One parameter Corresponding to one iteration of the incomplete Koleski decomposition, the converter model uses the decomposition matrix and the transpose decomposition matrix to replace the shared query key attention matrix during the computation of the output matrix.

4. The calculation method as described in claim 3, wherein the parameter The parameters used in the approximation operation for this incomplete Koleski decomposition are given.

5. The calculation method as described in claim 4, wherein the parameter Set to be equal to or less than the dimension value of the feature vectors in the input sequence.

6. The calculation method as described in claim 4, wherein the parameter Set to be equal to or less than the rank of the shared query key attention matrix.

7. The computational method as described in claim 3, wherein the shared query key attention matrix is ​​defined based on the product of the query matrix and a transpose of the query matrix, and the dimension of the intermediate matrix is ​​smaller than the dimension of the shared query key attention matrix.

8. The calculation method as described in claim 7, wherein the dimension of the transpose query matrix is ​​. The dimensions of this shared query key attention matrix are: The dimensions of this intermediate matrix are .

9. The calculation method as described in claim 1, wherein the output matrix is ​​calculated as follows: ， in, This represents division between matrix elements. H represents a vector of all ones, and H represents the decomposition matrix. T H represents the transpose decomposition matrix. T V represents the intermediate matrix.

10. A computing system, characterized in that, This computing system includes: A memory for storing multiple computer-executable instructions; and A processor, coupled to the memory, is configured to execute computer-executable instructions to run a converter model, the converter model including an attention layer for: An input matrix is ​​mapped to a query matrix based on multiple first learnable weights. The input matrix corresponds to an input sequence containing multiple feature vectors, and the query matrix contains multiple query vectors. The input matrix is ​​mapped to a numerical matrix based on multiple second learnable weights, and the numerical matrix contains multiple numerical vectors. A decomposition matrix is ​​generated by an incomplete Koleski decomposition based on the query matrix and a transpose of the query matrix, wherein the dimension of the decomposition matrix is ​​smaller than the dimension of the query matrix. Calculate an intermediate matrix based on the product of the transpose of the decomposition matrix and the numerical matrix; Calculate an output matrix based on the product of the decomposition matrix and the intermediate matrix; as well as An output sequence is generated using a fully connected layer based on the output matrix, wherein the converter model is used to translate the input sequence into the output sequence, or the converter model is used to extract, distinguish or generate the output sequence from the input sequence; The input sequence contains multiple characters or words in a first language, and the output sequence contains multiple characters or words in a second language; or The input sequence and the output sequence each contain multiple characters or words. The input sequence contains an article or a file. The output sequence contains a summary, a classification result, an answer to a question, or a title corresponding to the input sequence.

11. The computing system of claim 10, wherein the dimension of the input matrix is... , Let be the length of a sequence of the input sequence, and The dimension values ​​of these feature vectors in the input sequence.

12. The computational system of claim 11, wherein the incomplete Koleski decomposition is used to approximate the product of the decomposition matrix and the transpose of the decomposition matrix as an exponential function of a shared query key attention matrix, the dimension of the decomposition matrix being... ,in, One parameter Corresponding to one iteration of the incomplete Koleski decomposition, the converter model uses the decomposition matrix and the transpose decomposition matrix to replace the shared query key attention matrix during the computation of the output matrix.

13. The computing system of claim 12, wherein the shared query key attention matrix is ​​defined based on the product of the query matrix and a transpose of the query matrix, and the dimension of the intermediate matrix is ​​smaller than the dimension of the shared query key attention matrix.

14. The computing system of claim 13, wherein the dimension of the transpose query matrix is ​​. The dimensions of this shared query key attention matrix are: The dimensions of this intermediate matrix are .

Images