Instruction generation method and reasoning method, apparatus, system, device and medium

Abstract

Embodiments of the present disclosure disclose an instruction generation method and inference method, device, system, equipment and medium, wherein an initial model corresponding to a to-be-compiled attention network is obtained, at least part of memory access processing in the initial model is removed, a target operation operation in the initial model that needs to be updated is determined based on the removed at least part of memory access operation and an updating manner of the target operation operation, then the target operation operation is updated based on the updating manner of the target operation processing, so that a first intermediate result obtained by processing an input object based on the to-be-compiled attention network is the same as a second intermediate result obtained by processing the input object based on an optimized attention network corresponding to an optimized model, the optimized model is obtained, and then a computer executable target instruction sequence corresponding to the optimized attention network is generated through compilation processing based on the optimized model, which helps to reduce the operation amount of memory access operation in the inference process of the attention mechanism.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to artificial intelligence technology, and in particular to an instruction generation method and reasoning method for attention mechanism, an apparatus, a system, a device and a medium. BACKGROUND

[0002] A deep learning model based on a Transformer architecture (hereinafter referred to as: Transformer model) can maintain good performance when processing long text, and can be calculated in parallel to improve processing speed. At present, it has been widely used in the field of natural language processing (NLP), such as machine translation, text classification, question and answer system, etc. As a basic component of the Transformer model, the attention mechanism (Attention) has become more and more important in NLP tasks. When using the Transformer model for NLP tasks, the Transformer model usually uses self-attention mechanism (Self-Attention) to calculate the attention distribution between each position and all positions in the input sequence, and obtains a weighted sum to represent the context information of the position. Therefore, optimizing the reasoning speed of the attention mechanism can effectively reduce the consumption of computing resources, reduce the reasoning time of the Transformer model, and improve the user experience.

[0003] In the related art, there are a large number of memory access operations in the reasoning process of the attention mechanism, which results in low reasoning speed and long reasoning time. SUMMARY

[0004] In order to solve the above technical problems, the embodiments of the present disclosure provide an instruction generation method and reasoning method for attention mechanism, an apparatus, a device and a medium to reduce the operation amount of memory access operations in the reasoning process of the attention mechanism.

[0005] In one aspect of the embodiments of the present disclosure, an instruction generation method for attention mechanism is provided, comprising:

[0006] obtaining an initial model corresponding to a to-be-compiled attention network;

[0007] removing at least part of memory access processing in the initial model;

[0008] determining a target operation operation in the initial model that needs to be updated and an update mode of the target operation operation based on the removed at least part of memory access operations;

[0009] updating the target operation operation based on the update mode of the target operation processing to obtain an optimized model;

[0010] generating a computer executable target instruction sequence based on the optimization model through compiling processing;

[0011] The first intermediate result obtained by processing an input object based on the attention network to be compiled is the same as the second intermediate result obtained by processing the input object based on the optimization attention network corresponding to the optimization model.

[0012] Another aspect of the embodiments of the present disclosure provides an inference method of an attention mechanism, comprising:

[0013] obtaining a computer executable target instruction sequence corresponding to the optimization attention network, the target instruction sequence being generated based on the instruction generation method of the attention mechanism according to any of the above embodiments of the present disclosure;

[0014] transmitting the target instruction sequence to a hardware platform;

[0015] processing an input parameter corresponding to a current sub-object in a received input object by executing the target instruction sequence by using the hardware platform, to obtain an inference result corresponding to the current sub-object, the input object comprising a plurality of sub-objects.

[0016] Still another aspect of the embodiments of the present disclosure provides an instruction generation device for an attention mechanism, comprising:

[0017] a first obtaining module configured to obtain an initial model corresponding to an attention network to be compiled;

[0018] a removing module configured to remove at least part of memory access operations in the initial model;

[0019] an updating determining module configured to determine a target operation operation in the initial model that needs to be updated and an updating manner of the target operation operation based on the at least part of memory access operations removed;

[0020] an updating module configured to update the target operation operation based on the updating manner of the target operation operation, to obtain an optimization model;

[0021] a compiling processing module configured to generate a computer executable target instruction sequence based on the optimization model through compiling processing;

[0022] The first intermediate result obtained by processing an input object based on the attention network to be compiled is the same as the second intermediate result obtained by processing the input object based on the optimization attention network corresponding to the optimization model.

[0023] Still another aspect of the embodiments of the present disclosure provides an inference system of an attention mechanism, comprising:

[0024] a second obtaining module, configured to obtain a computer executable target instruction sequence corresponding to the optimized attention network, the target instruction sequence being generated based on the instruction generation method for the attention mechanism according to any one of the embodiments of the present disclosure;

[0025] a transmission module, configured to transmit the target instruction sequence to a hardware platform;

[0026] the hardware platform, configured to execute the target instruction sequence, process an input parameter corresponding to a current sub-object in a received input object, and obtain an inference result corresponding to the current sub-object, the input object including a plurality of sub-objects.

[0027] In still another aspect of the embodiments of the present disclosure, a deep learning model is provided, including an encoder and a decoder.

[0028] The encoder includes an optimized attention network implemented based on an optimization model, the optimization model being based on the instruction generation method for the attention mechanism according to any one of the embodiments of the present disclosure, obtaining an initial model corresponding to a to-be-compiled attention network, optimizing at least part of memory access operations in the initial model, and updating target operation operations in the initial model.

[0029] In still another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, the storage medium storing a computer program, the computer program being used to execute the method according to any one of the embodiments of the present disclosure.

[0030] In still another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device including:

[0031] a processor;

[0032] a memory for storing executable instructions of the processor;

[0033] the processor, configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of the embodiments of the present disclosure.

[0034] In still another aspect of the embodiments of the present disclosure, a computer program product is provided, when an instruction processor in the computer program product executes, the method according to any one of the embodiments of the present disclosure is executed.

[0035] Based on the embodiments of the present disclosure, by obtaining an initial model corresponding to the attention network to be compiled, removing at least part of the memory access operations in the initial model, determining the target operation operation in the initial model that needs to be updated and the updating mode of the target operation operation based on the removed at least part of the memory access operations, then updating the target operation operation based on the updating mode of the target operation processing, obtaining an optimized model, so that the first intermediate result obtained by processing the input object based on the attention network to be compiled is the same as the second intermediate result obtained by processing the same input object based on the optimized attention network corresponding to the optimized model, and then generating a computer executable target instruction sequence by compiling the optimized model. Therefore, the operation amount (also referred to as the memory access amount) of the memory access operation can be reduced while ensuring that the intermediate results obtained by processing the same input object before (the attention network to be compiled) and after (the optimized attention network) optimization are unchanged. When the input object is processed by using the optimized attention network, the inference speed of the attention mechanism can be effectively improved, the inference time-consuming can be reduced, and the overall processing cost can be reduced.

[0036] The technical solutions of the present disclosure will be described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 An example of the time of memory access operation and operation in a memory-intensive algorithm.

[0038] Figure 2 An example of an application scenario of an embodiment of the present disclosure.

[0039] Figure 3 A flowchart of an instruction generation method for an attention mechanism provided by an example embodiment of the present disclosure.

[0040] Figure 4 A schematic diagram of processing in an optimized model in an example embodiment of the present disclosure.

[0041] Figure 5 A schematic diagram of processing in an optimized model in another example embodiment of the present disclosure.

[0042] Figure 6 A schematic diagram of processing in an optimized model in another example embodiment of the present disclosure.

[0043] Figure 7 A schematic diagram of processing in an optimized model in another example embodiment of the present disclosure.

[0044] Figure 8 A flowchart of an inference method of an attention mechanism provided by an example embodiment of the present disclosure.

[0045] Figure 9A flowchart of an inference method of an attention mechanism provided for another exemplary embodiment of the present disclosure.

[0046] Figure 10 A flowchart of an inference method of an attention mechanism provided for another exemplary embodiment of the present disclosure.

[0047] Figure 11 A flowchart of an inference method of an attention mechanism provided for another exemplary embodiment of the present disclosure.

[0048] Figure 12 A flowchart of an inference method of an attention mechanism provided for another exemplary embodiment of the present disclosure.

[0049] Figure 13 A structural diagram of an instruction generation device for an attention mechanism provided for an exemplary embodiment of the present disclosure.

[0050] Figure 14 A structural diagram of an inference system of an attention mechanism provided for an exemplary embodiment of the present disclosure.

[0051] Figure 15 A structural diagram of a deep learning model provided for an exemplary embodiment of the present disclosure.

[0052] Figure 16 A structural diagram of a deep learning model provided for an exemplary embodiment of the present disclosure.

[0053] Figure 17 A structural diagram of an electronic device provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] In order to explain the present disclosure, the example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, obviously, the described embodiments are only part of the embodiments of the present disclosure, not all the embodiments, it should be understood that the present disclosure is not limited by the example embodiments.

[0055] It should be noted that: unless otherwise specified, the relative arrangement, numerical expression and numerical value of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0056] It should also be understood that the description of the present disclosure for each embodiment emphasizes the differences between each embodiment, the same or similar parts can be referred to each other, for the sake of brevity, will not be repeated.

[0057] The technology, method and equipment known to those skilled in the related art can not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as part of the specification.

[0058] Summary of the application

[0059] In the process of implementing the present disclosure, the inventors have found through research that the attention mechanism is a memory-intensive algorithm. In the inference process of the related technology, there are a large number of memory operations, such as transpose (Transpose) operations and concatenation (Concat) operations. These operations do not have calculation operations themselves, and the core is data transfer, or data reading and rewriting of memory. The memory operation time of the memory-intensive algorithm is much longer than the calculation operation time. For example, as shown in FIG. 1, it is an example of memory operation time and calculation operation time in a memory-intensive algorithm, where the length of the rectangle represents the corresponding operation time. Because there are a large number of memory operations in the inference process of the attention mechanism, the inference speed of the attention mechanism is low, the inference time is long, and the overall overhead is large. Figure 1

[0060] Exemplary application scenarios

[0061] Figure 2 FIG. 1 is an exemplary application scenario of an embodiment of the present disclosure.

[0062] The instruction generation scheme for the attention mechanism according to the embodiment of the present disclosure can obtain an initial model corresponding to the attention network to be compiled, optimize at least part of the memory operations in the initial model, and update the target calculation operation in the initial model, so that the processing overhead of the obtained optimized model is less than that of the initial model. Then, based on the optimized model, the computer executable target instruction sequence corresponding to the optimized attention network is generated through compilation processing. The first intermediate result obtained by processing the input object through the initial instruction sequence is the same as the second intermediate result obtained by processing the input object through the target instruction sequence. Here, the initial instruction sequence is the computer executable instruction sequence corresponding to the attention network to be compiled.

[0063] The instruction generation scheme for the attention mechanism according to the embodiment of the present disclosure can be implemented by the compiler 102. The compiler 102 can include a front end, a middle end and a back end, which respectively include different modularized compiler components and tool chains. The front end can convert the attention network to be compiled from a programming language into an intermediate representation model. The middle end can optimize the intermediate representation model to obtain an optimized model. The back end converts the optimized model into a computer executable target instruction sequence.

[0064] ​The instruction generation scheme for the attention mechanism according to the embodiments of the present disclosure can be implemented at any stage of the front end, the middle end, and the back end of the compiling process, and the specific implementation is not limited. That is, the initial model can be a model described by a programming language at the front end, a model in an intermediate representation, or a model described by an initial instruction sequence generated at the back end, and the specific implementation can be set according to actual needs.

[0065] After obtaining the target instruction sequence by using the instruction generation scheme for the attention mechanism according to the embodiments of the present disclosure, the hardware platform 104 can execute the target instruction sequence to implement the inference of the attention mechanism and obtain an inference result.

[0066] The instruction generation scheme for the attention mechanism according to the embodiments of the present disclosure can be applied to task processing in any application scenario or field, for example, can be applied to task processing in the natural language processing (NLP) field, the automatic driving scenario, the medical image processing scenario, and the voice interaction scenario, such as machine translation, text classification, a question and answer system, image classification, and the like, and the embodiments of the present disclosure do not limit the same.

[0067] The instruction generation scheme for the attention mechanism according to the embodiments of the present disclosure is applicable to any hardware platform that can execute computer executable instructions, such as a graphic processing unit (GPU), a central processing unit (CPU), an accelerator, and the like, and the embodiments of the present disclosure do not limit the same.

[0068] Exemplary method

[0069] Figure 3 A flowchart of an instruction generation method for an attention mechanism according to an exemplary embodiment of the present disclosure is provided. The embodiment can be applied to any electronic device such as a terminal device, a server, and the like, as shown in FIG. 8, and the embodiment includes the following steps. Figure 3

[0070] 202, an initial model corresponding to a to-be-compiled attention network is obtained.

[0071] The to-be-compiled attention network can be an attention network in any type of pre-trained deep learning model, such as a Transformer model, a convolutional neural network (CNN), a recurrent neural network (RNN), and the like, and the embodiments of the present disclosure do not limit the applicable deep learning model.

[0072] ​In the embodiments of the present disclosure, the initial model can be a model described by a programming language corresponding to the attention network to be compiled, can be a model in intermediate representation corresponding to the attention network to be compiled, and can also be a model described by an initial instruction sequence generated corresponding to the attention network to be compiled. Taking the model in intermediate representation as an example, the attention network to be compiled can be converted from a programming language to intermediate representation through a model compiler to obtain an initial model in intermediate representation corresponding to the attention network to be compiled. The specific conversion from the programming language to the intermediate representation can be implemented in any manner according to actual requirements, and the present disclosure will not be repeated.

[0073] The intermediate representation can be an intermediate language description between a programming language and a machine-recognizable instruction sequence, and can also be referred to as a computation graph. The computation graph describes the computation logic of the model through operator nodes and the dependency relationship between the operator nodes. The operator nodes can include loading nodes, computation nodes, and storage nodes. The loading nodes are used to load the inputs required by the computation nodes for computation, the computation nodes are used to perform corresponding computation on the inputs, and the storage nodes are used to store the computation results of the computation nodes. The computation graph of the model is formed through the dependency topology relationship of the plurality of operator nodes. The types of the operator nodes correspond to the operation types in the model, such as an operator node corresponding to a matrix multiplication operation, an operator node corresponding to a normalization operation, and the like. According to different network layers in the model, the computation graph can also be divided into a plurality of computation subgraphs, which can be set according to actual requirements.

[0074] In step 202, the initial model corresponding to the attention network to be compiled can be directly obtained, or the initial model corresponding to the attention network to be compiled can be obtained from a deep learning model, and the embodiments of the present disclosure do not limit this.

[0075] 204, removing at least part of the memory access operations in the initial model.

[0076] Since there are various types of memory access operations in the initial model, and each type of memory access operation includes a plurality of times, in the embodiments of the present disclosure, optimizing at least part of the memory access operations in the initial model means optimizing the operation times of one or more types of memory access operations to a certain extent to reduce the memory access amount in the inference process of the attention mechanism, thereby improving the inference speed, reducing the inference time consumption, and reducing the overall processing overhead.

[0077] 206, determining a target operation in the initial model that needs to be updated and an updating manner of the target operation based on the removed at least part of the memory access operations.

[0078] Based on step 204, after removing at least part of the memory access operation in the initial model, the target operation operation in the initial model that needs to be updated and the updating manner of the target operation operation are determined to keep the intermediate result obtained by processing the input object by the initial model unchanged. Wherein, the operation operation type and quantity involved in the target operation operation and the corresponding updating manner are determined according to the type and quantity of the removed memory access operation in the initial model, and the present embodiment of the present disclosure does not limit this, as long as the intermediate result obtained by processing the same input object before (the to-be-compiled attention network) and after (the corresponding obtained optimized attention network) optimization is unchanged.

[0079] 208, based on the updating manner of the target operation processing, the target operation operation is updated to obtain an optimized model.

[0080] 210, based on the optimized model, a computer executable target instruction sequence is generated through compilation processing.

[0081] Wherein, the first intermediate result obtained by processing the input object based on the to-be-compiled attention network is the same as the second intermediate result obtained by processing the input object based on the optimized attention network corresponding to the optimized model. That is, the processing result of the same input object based on the optimized attention network obtained by the present embodiment and the to-be-compiled attention network is the same, and will not change the processing result of the input object. The optimized attention network here is the implementation form of the optimized model, that is, the attention network corresponding to the processing of the to-be-compiled attention network based on the present embodiment.

[0082] Wherein, the first intermediate result is the same as the second intermediate result. The first intermediate result here is the intermediate result obtained by processing the input object by executing the initial instruction sequence, and the initial instruction sequence is the computer executable instruction sequence corresponding to the to-be-compiled attention network. The second intermediate result is the intermediate result obtained by processing the input object by executing the target instruction sequence.

[0083] Based on the embodiment, by obtaining an initial model corresponding to the attention network to be compiled, removing at least part of the memory access operations in the initial model, and updating the target operation operation in the initial model that needs to be updated, the first intermediate result obtained by processing the input object based on the attention network to be compiled is the same as the second intermediate result obtained by processing the same input object based on the optimized attention network corresponding to the optimized model. Then, the computer executable target instruction sequence is generated by compiling the optimized model. Therefore, the operation amount (also referred to as the memory access amount) of the memory access operation can be reduced while ensuring that the intermediate results obtained by processing the same input object before (the attention network to be compiled) and after (the optimized attention network) optimization are unchanged. When the input object is processed by using the optimized attention network, the inference speed of the attention mechanism can be effectively improved, the inference time-consuming can be reduced, and the overall processing cost can be reduced.

[0084] Optionally, in some implementations, the initial model may, for example, include but is not limited to two types of processing operations, i.e., operation operations and memory access operations. The memory access operations may, for example, include but are not limited to two types of memory access operations, i.e., transpose operations and concatenation operations, and other types of memory access operations, such as simple memory data reading and writing.

[0085] The inventors have found that in the inference process of the attention mechanism, the memory access operations with large operation amounts mainly include transpose operations and concatenation operations. Based on the embodiment, the transpose operations and the concatenation operations with large memory access amounts in the inference process of the attention mechanism are optimized, which helps to effectively reduce the operation times of the memory access operations, greatly improve the inference speed, and reduce the long inference time-consuming.

[0086] For ease of understanding, the inference process of the attention mechanism and the input parameters involved in the embodiment of the present disclosure are described as follows.

[0087] The inference process of the attention mechanism in the embodiment of the present disclosure involves five input parameters, i.e., a query vector matrix (Query, Q), a key vector matrix (Key, K), a value vector matrix (Value, V), a key vector matrix cache (K CACHE), and a value vector matrix cache (V CACHE). The five inputs are all four-dimensional vector matrices, and their shapes are as follows: Q[batch, head, d_k, time1], K[batch, head, d_k, time1], V[batch, head, d_k, time1], K CACHE[batch, head, d_k, time2], and V CACHE[batch, head, d_k, time2].

[0088] wherein, batch represents the number of input objects processed at the same time, for example, the number of sentences processed at the same time when the input objects are sentences. head represents the number of heads of the attention mechanism in the attention network, the value of head is 1 when the attention network adopts a single-head attention mechanism, and the value of head is a specific number greater than 1 when the attention network adopts a multi-head attention mechanism (Multi-Head Attention). d_k represents the dimension of the sub-vector corresponding to each sub-object in the input object (the size of the hidden layer in the attention network to be compiled). head and d_k are parameters of the attention network, and their specific values are pre-set based on the corresponding attention network. time1 represents the unit number of the sub-object (i.e., the current sub-object) of the deep learning model in which the current input attention network structure is located, and time2 represents the unit number of the output result of the deep learning model for the input object. For example, when the input object is a sentence, time1 represents the number of words (e.g., 1) input into the deep learning model, and time2 represents the number of words (e.g., 9) generated by the deep learning model for the sentence.

[0089] In actual applications, the values of head and d_k are determined after the attention network used is determined. The values of batch, time1 and time2 are related to the specific input object. In general, the values of batch, head, d_k, time1 and time2 satisfy the following relationships: the value of head is less than the value of d_k (i.e., head < d_k), the value of time1 is much less than the value of time2 (time1 << time2), and the value of time1 is much less than the value of d_k (time1 << d_k).

[0090] Various possible ways can be used to obtain the above five input parameters corresponding to each sub-object, such as the ways described in the related art.

[0091] For example, in some implementations, the above input parameters of each sub-object can be obtained through a feature extraction layer (such as an Embedding layer) and a linear transformation layer.

[0092] For example, a feature extraction layer can be used to extract features of the input object to obtain an input vector matrix Y [y1, y2,..., y n], where n is the number of sub-objects included in the input object Y, and each yi is a d_k-dimensional vector, which is a sub-input vector corresponding to each sub-object.

[0093] Then, the sub-input vectors yi corresponding to each sub-object are linearly transformed by a linear transformation layer, for example, linearly transformed by K=yi*W_k, V=yi*W_v, Q=yi*W_q, to obtain the query vector matrix, the key vector matrix and the value vector matrix of each sub-object in the preset dimension. Wherein, W_k, W_v, W_q are parameter matrices learned by the deep learning model in which the attention network is located through the training stage.

[0094] By processing each sub-object in the input object through the deep learning model, after obtaining the key vector matrix and the value vector matrix corresponding to each sub-object, the key vector matrix of the sub-object can be spliced with the key vector matrix cache before the sub-object in the input object to obtain the key vector matrix cache of the sub-object; similarly, the value vector matrix of the sub-object can be spliced with the value vector matrix before the sub-object in the input object to obtain the value vector matrix cache of the sub-object, so as to take the query vector matrix, the key vector matrix, the value vector matrix, the key vector matrix cache and the value vector matrix cache of the sub-object as five input parameters of the sub-object for inference of the attention mechanism.

[0095] The specific implementation of obtaining the input parameters of each sub-object is not limited in the embodiments of the present disclosure.

[0096] In the inference process of the related art attention mechanism, for each sub-object in each input object, at least the following processing is included: splicing operation of K_CACHE and K, splicing operation of V_CACHE and V, transpose operation of Q, K and V respectively, transpose operation of the transposed K again, matrix multiplication (MatMul) operation of the transposed matrix K obtained by the transpose operation again and Q to obtain an attention score matrix (scores), normalization operation of scores, matrix multiplication operation of the attention score matrix (attn) obtained by normalization and V to obtain a matrix X, the shape of which is [b, h, t1, d_k], and transpose operation and rearrangement (view) operation of the matrix X to obtain an intermediate result, the shape of which is [b, h*d_k, t1]. Since the inference result of the attention mechanism is only an intermediate output in the deep learning model in which the attention network is located, not the final output of the deep learning model, the inference result of the attention mechanism is also called the intermediate result.

[0097] Optionally, in some of the implementations, at least part of the transpose operation in the initial model can be removed in step 204.

[0098] Accordingly, in step 208, the processing object of the first operation in the initial model and / or the input order of the processing object can be adjusted, and a transpose operation on the intermediate output can be added in the initial model. The target operation includes the first operation, where the first operation is an operation associated with the at least partially transposed operation in terms of processing object and processing result after the at least partially transposed operation is performed on the initial model. Specifically, to ensure that the intermediate result obtained by processing the same input object remains unchanged after the at least partially transposed operation is performed on the initial model, the operation and the updating method, such as the matrix multiplication operation, are updated accordingly. The specific type and number of the first operation and the processing object involved can be determined according to actual needs.

[0099] For example, in a specific implementation, the transpose operations performed on Q, K, and V in the inference process of the attention mechanism and the transpose operation performed on the matrix X in the related art can be removed. Accordingly, the processing object and the input order of the matrix multiplication operation performed on the transposed matrix K and Q obtained by performing the transpose operation again are adjusted accordingly, the transpose operation on the normalized attn is added, and the processing object and the input order of the matrix multiplication operation performed on the normalized attn and V are adjusted accordingly.

[0100] Based on the embodiment, the transpose operation with a relatively large memory access amount involved in the inference process of the attention mechanism is selected for optimization, the processing object of the first operation in the initial model or the input order of the processing object is adjusted, and the transpose operation on the intermediate output is added in the initial model, so that the operation number of the memory access operation is reduced, the inference speed is improved, and the inference time is reduced.

[0101] For the sake of quantitative understanding, the following describes the processing operation in the optimized model obtained based on steps 202-208 in the embodiments of the present disclosure, which refers to the operation processing on the five input parameters of each sub-object, and can include memory access operations and operation operations, by taking the shapes of the five input parameters of each sub-object as examples. For the sake of representation, b represents batch, h represents head, t1 represents time 1, and t2 represents time 2 in the following embodiments of the present disclosure.

[0102] Based on the processing operation in the optimized model obtained based on the above some implementation manners, as shown in FIG. 8, the processing operation can include: Figure 4

[0103] ​302, a concatenation operation on the key vector matrix cache (K_CACHE) and the first key vector matrix (K) in the first specified dimension, based on which a second key vector matrix can be obtained.

[0104] The first specified dimension here can be the dimension in which t2 in the key vector matrix cache is located, and the dimension in which t1 in the first key vector matrix is located.

[0105] Since the shape of K_CACHE is [b, h, d_k, t2], and the shape of K is [b, h, d_k, t1], for example, K can be concatenated with K_CACHE in the last dimension by K = cat ([K_CACHE, K], dim =-1), and the output is assigned to a new K (referred to as a second key vector matrix). The shape of the second key vector matrix is [b, h, d_k, t2+t1]. Wherein, dim =-1 represents the last dimension.

[0106] 304, a concatenation operation on the value vector matrix cache (V_CACHE) and the first value vector matrix (V) in the first specified dimension, based on which a second value vector matrix can be obtained.

[0107] The first specified dimension here can be the dimension in which t2 in the value vector matrix cache is located, and the dimension in which t1 in the first value vector matrix is located.

[0108] Since the shape of V_CACHE is [b, h, d_k, t2], and the shape of V is [b, h, d_k, t1], for example, V can be concatenated with V_CACHE in the last dimension by V = cat ([V_CACHE, V], dim =-1), and the output is assigned to a new V (referred to as a second value vector matrix). The shape of the second value vector matrix is [b, h, d_k, t2+t1].

[0109] 306, a transpose operation on the first query vector matrix (Q) in the first specified dimension pair, based on which a second query vector matrix can be obtained.

[0110] The first specified dimension pair here can be the dimension in which d_k and time1 in the first query vector matrix are located.

[0111] Since the shape of Q is [b, h, d_k, t1], Q can be transposed by Q = Q.transpose(2, 3), which is a transpose operation between the last two dimensions. The output is assigned to a new Q (referred to as a second query vector matrix) with a shape of [b, h, t1, d_k]. Since Q is a four-dimensional matrix, its four dimensions correspond to 0, 1, 2, and 3, and 2 and 3 in transpose(2, 3) represent the last two dimensions.

[0112] 308, the matrix multiplication operation between the second query vector matrix and the second key vector matrix can obtain a first attention score matrix (scores).

[0113] Since the shape of the second query vector matrix is [b, h, t1, d_k] and the shape of the second key vector matrix is [b, h, d_k, t2+t1], the matrix multiplication operation between Q and K can obtain a first attention score matrix with a shape of [b, h, t1, t2+t1] by the operation scores = matmul(Q, K).

[0114] The operation 308 adjusts the input order of the processing object of the first operation in the initial model. Specifically, the matrix multiplication operation in the operation 308 is the first operation in the embodiments of the present disclosure, and the operation 308 adjusts the input order of the processing object from the transposed matrix K and Q to the second query vector matrix and the second key vector matrix relative to the related art, which is a matrix multiplication (MatMul) operation between the transposed matrix K and Q in the inference process of the attention mechanism.

[0115] 310, a transpose operation on the first attention score matrix between the second specified dimension pair to obtain a second attention score matrix.

[0116] The second specified dimension pair here can be the dimensions where t1 and t2+t1 in the second attention score matrix are located.

[0117] The operation 302 belongs to the above-mentioned addition of a transpose operation on the intermediate output (the first attention score matrix) in the initial model.

[0118] Optionally, the processing operation in the optimized model can also selectively include a normalization operation on the first attention score matrix, based on which a normalized attention score matrix (attn) can be obtained.

[0119] For example, the last dimension (dim=-1) of the first attention score matrix can be normalized by attn=softmax(scores,dim=-1), and the normalized attention score matrix obtained by the normalization operation has the same shape, and the values of the elements in the normalized attention score matrix are normalized in the range of [0, 1].

[0120] Correspondingly, the operation 310 is a transpose operation on the normalized attention score matrix between the second specified dimensions. For example, the last two dimensions of the first attention score matrix can be transposed by attn=attn.transpose(2,3), and the shape of the second attention score matrix obtained after the transpose operation is [b, h, t2+t1, t1].

[0121] 312, the matrix multiplication operation of the second value vector matrix and the second attention score matrix can obtain the first intermediate result (X).

[0122] For example, the matrix multiplication operation of the second value vector matrix and the second attention score matrix can be performed by X=matmul(V,attn), and since the shape of the second value vector matrix is [b, h, d_k, t2+t1] and the shape of the second attention score matrix is [b, h, t2+t1, t1], the shape of the first intermediate result (X) obtained by the matrix multiplication operation is [b, h, d_k, t1].

[0123] The operation 312 adjusts the processing object of the first operation in the initial model and the input order of the processing object. Specifically, the matrix multiplication operation in the operation 312 is the first operation in the embodiment of the present disclosure, and the operation 312 adjusts the processing object from attn to the transpose matrix of attn and adjusts the input order of the processing object from attn and V to V and the transpose matrix of attn relative to the related art in the attention mechanism reasoning process. Therefore, the processing object of the first operation in the initial model or the input order of the processing object is adjusted.

[0124] 314, the rearrangement (view) operation of the first intermediate result can obtain the second intermediate result.

[0125] For example, the first intermediate result can be rearranged (view) by X=X.view(batch, d_k*head, 1, time1) to obtain the second intermediate result, which is the result that can be obtained by optimizing the attention network or by executing the target instruction sequence corresponding to the optimized attention network to process the input object.

[0126] Through the operation 314, the plurality of first intermediate results corresponding to the multi-head attention mechanism can be fused, and the first intermediate result is converted from the thinking matrix to the three-dimensional matrix to obtain the intermediate result of the entire optimized attention network. For the single-head attention mechanism, the value of the head number head is 1, and the operation 314 is also applicable.

[0127] Compared with the related art, the embodiment removes the transpose operation on the input parameters K and V, the transpose operation on the transposed K, and the transpose operation on the matrix X in the inference process of the attention mechanism, thereby eliminating the memory access amount and inference time of executing these transpose operations, and effectively improving the inference speed of the attention mechanism. Although the embodiment increases the transpose operation on the second attention score matrix as shown in operation 312 in the inference process of the attention mechanism compared with the related art, since the shape of the second attention score matrix is [b, h, t1, t2+t1] and the shape of K and V is [b, h, d_k, t2+t1], t1 << d_k, the reduced memory access amount is much larger than the increased memory access amount, so that in the inference scenario, the overall processing overhead of the optimized model is less than that of the initial model, and the inference acceleration effect is achieved overall. Moreover, by adjusting the input order of the matrix multiplication operation on the transposed matrix K and Q in the inference process of the attention mechanism of the related art to the matrix multiplication operation on the second query vector matrix and the second key vector matrix as shown in operation 308, the input order of the matrix multiplication operation on the normalized attention score matrix (attn) and V in the inference process of the attention mechanism of the related art is adjusted to the matrix multiplication operation on the second value vector matrix and the second attention score matrix as shown in operation 312, which is the same as the intermediate result obtained by processing the same input object in the inference process of the attention mechanism of the related art.

[0128] The above Figure 4 The embodiments shown in the above

[0129] Optionally, in some implementations, in operation 204, at least part of the concatenation operation in the initial model can be further removed based on the removal of at least part of the transposition operation in the initial model in some implementations described above.

[0130] Correspondingly, in this embodiment, in operation 208, the second operation in the initial model can be further converted into a plurality of sub-operations based on the adjustment of the processing object and / or the input order of the processing object of the first operation in the initial model and the addition of the transposition operation of the intermediate output in the initial model in some implementations described above. In this embodiment, the target operation described above specifically includes the first operation and the second operation. Here, the second operation is an operation that needs to be further updated, such as a matrix multiplication operation, to ensure that the intermediate result obtained by processing the same input object remains unchanged after at least part of the concatenation operation in the initial model is removed. The specific type and number of the second operation and the processing object involved can be determined according to actual needs.

[0131] For example, in a specific implementation, the concatenation operation in operation 302 and / or operation 304 can be selectively removed. Correspondingly, the matrix multiplication operation in operation 308 and / or operation 312 is converted into a plurality of sub-operations, respectively.

[0132] In some specific implementation examples, in operation 204, the operation shown in operation 302 in the initial model, that is, the concatenation operation on the corresponding key vector matrix cache and the first key vector matrix in the input object in the first specified dimension, can be removed.

[0133] Correspondingly, in this specific implementation example, converting the second operation in the initial model into a plurality of sub-operations can include:

[0134] The matrix multiplication operation on the second query vector matrix and the second key vector matrix shown in operation 308 is converted into a plurality of sub-operations: a matrix multiplication operation on the second query vector matrix and the key vector matrix cache, based on which a first sub-score matrix can be obtained; a matrix multiplication operation on the second query vector matrix and the first key vector matrix, based on which a second sub-score matrix can be obtained; and a concatenation operation on the first sub-score matrix and the second sub-score matrix in the first specified dimension.

[0135] Based on the embodiment, at least part of the concatenation operation involved in the inference process of the attention mechanism and having a relatively large memory access amount can be further optimized, and the second operation in the initial model is converted into a plurality of sub-operations, so as to reduce the operation times of the memory access operation, improve the inference speed, reduce the long inference time, and effectively improve the inference speed of the attention mechanism.

[0136] Based on some specific implementation examples, the processing operations included in the optimized model are as follows: Figure 5

[0137] 402, the concatenation operation of the value vector matrix cache (V_CACHE) and the first value vector matrix (V) in the first specified dimension, based on which the second value vector matrix can be obtained.

[0138] Here, the first specified dimension can be the dimension where t2 in the value vector matrix cache is located, and the dimension where t1 in the first value vector matrix is located.

[0139] Since the shape of V_CACHE is [b, h, d_k, t2], and the shape of V is [b, h, d_k, t1], for example, V can be obtained by cat([V_CACHE, V], dim=-1), which concatenates V_CACHE and V in the last dimension, and outputs the assignment to the new V (called the second value vector matrix). The shape of the second value vector matrix is [b, h, d_k, t2+t1].

[0140] 404, the transpose operation of the first query vector matrix (Q) in the first specified dimension pair, based on which the second query vector matrix can be obtained.

[0141] Here, the first specified dimension pair can be the dimensions where d_k and time1 in the first query vector matrix are located.

[0142] Since the shape of Q is [b, h, d_k, t1], for example, Q can be obtained by Q=Q.transpose(2,3), which transposes Q between the last two dimensions. The output is assigned to the new Q (called the second query vector matrix), and the shape of the second query vector matrix is [b, h, t1, d_k]. Since Q is a four-dimensional matrix, its four dimensions correspond to 0, 1, 2, and 3, and 2 and 3 in transpose(2,3) represent the last two dimensions.

[0143] ​406, the matrix multiplication operation on the second query vector matrix and the key vector matrix cache (K CACHE) based on which the first sub-score matrix can be obtained.

[0144] Since the shape of the second query vector matrix is [b, h, t1, d_k], and the shape of the key vector matrix cache is [b, h, d_k, t2], for example, the matrix multiplication operation on the second query vector matrix and the key vector matrix cache can be performed by scores_part1 = matmul(Q, K CACHE), and the shape of the first sub-score matrix scores_part1 obtained is [b, h, t1, t2].

[0145] 408, the matrix multiplication operation on the second query vector matrix and the first key vector matrix based on which the second sub-score matrix can be obtained.

[0146] Since the shape of the second query vector matrix is [b, h, t1, d_k], and the shape of the first key vector matrix is [b, h, d_k, t1], the matrix multiplication operation on the second query vector matrix and the first key vector matrix can be performed by scores_part2 = matmul(Q, K), and the shape of the second sub-score matrix obtained is [b, h, t1, t1].

[0147] 410, the splicing operation on the first sub-score matrix and the second sub-score matrix in the first specified dimension based on which the first attention score matrix (scores) with the shape of [b, h, t1, t2+t1] can be obtained.

[0148] Among them, operations 406-410 are converted multiple sub-operations of operation 308 on the second query vector matrix and the second key vector matrix. The matrix multiplication operation on the second query vector matrix and the second key vector matrix in operation 308 belongs to the above-mentioned second operation operation in the embodiment of the present disclosure.

[0149] 412, the transposition operation on the first attention score matrix between the second specified dimension pair based on which the second attention score matrix can be obtained.

[0150] Here, the second specified dimension pair can be the dimensions where t1 and t1+t2 in the first attention score matrix are located.

[0151] Optionally, the processing operation in the optimization model can also selectively include the processing operation: performing the normalization operation on the first attention score matrix to obtain the normalized attention score matrix (attn).

[0152] For example, the last dimension (dim=-1) of the first attention score matrix can be normalized by attn=softmax(scores,dim=-1), and the normalized attention score matrix obtained by the normalization operation has the same shape, and the values of the elements in the normalized attention score matrix are normalized in the range of [0, 1].

[0153] Correspondingly, in the operation 412, the normalized attention score matrix is transposed between the second specified dimensions to obtain the second attention score matrix. For example, the last two dimensions of the second attention score matrix can be transposed by attn=attn.transpose(2,3), and the shape of the first attention score matrix is [b, h, t1, t2+t1], and the shape of the second attention score matrix obtained after the transposition operation is [b, h, t2+t1, t1].

[0154] 414, the matrix multiplication operation of the second value vector matrix and the second attention score matrix can obtain the first intermediate result (X) based on the matrix multiplication operation.

[0155] For example, the matrix multiplication operation of the second value vector matrix and the second attention score matrix can be performed by X=matmul(V,attn), and since the shape of the second value vector matrix is [b, h, d_k, t2+t1] and the shape of the second attention score matrix is [b, h, t2+t1, t1], the shape of the first intermediate result (X) obtained by the matrix multiplication operation is [b, h, d_k, t1].

[0156] 416, the rearrangement operation of the first intermediate result can obtain the second intermediate result based on the rearrangement operation.

[0157] For example, the rearrangement (view) of the first intermediate result can be performed by X=X.view(batch,d_k*head,1,time1) to obtain the second intermediate result, which is the result obtained by processing the input object by optimizing the attention network or by executing the target instruction sequence corresponding to the optimized attention network.

[0158] Through the operation 416, the multiple first intermediate results corresponding to the multi-head attention mechanism can be fused, and the first intermediate result is converted from a thought matrix to a three-dimensional matrix to obtain the intermediate result of the entire optimized attention network. For a single-head attention mechanism, the value of the number of heads head is 1, and the operation 416 is also applicable.

[0159] In this embodiment, the operation 412 is performed on the first attention score matrix obtained by the operation 410. Figure 4On the basis of the embodiment shown, the splicing operation of the above operation 302 is further optimized, and the matrix multiplication operation in operation 308 is converted into a plurality of sub-operations shown in operations 406-410. Thereby, the memory access amount and inference time consumption of performing the splicing operation of operation 302 are eliminated, and the inference speed of the attention mechanism can be effectively improved; and the calculation amount of operation 308 and the converted operation 406-410 on matrix multiplication is the same, and the conversion does not increase the calculation amount and processing overhead. In addition, relative to the optimized splicing operation of the above operation 302, the embodiment adds the splicing operation of the first sub-score matrix and the second sub-score matrix as shown in operation 410. Since K_CACHE[b, h, d_k, t2] and K[b, h, d_k, t1] are spliced in operation 302, and scores_part1[b, h, t1, t2] and scores_part2[b, h, t1, t1] are spliced in operation 410, and t1 << d_k, therefore, the data amount involved in the splicing operation added by operation 410 is much smaller than the data amount involved in the splicing operation reduced in operation 302, so that the memory access amount can be greatly reduced, the inference time consumption is reduced, and the inference speed of the attention mechanism is effectively improved.

[0160] In some other specific implementation examples, the optimization of at least part of the splicing operation in the initial model can be that the operation shown in operation 304 in the initial model is removed, that is, the splicing operation of the corresponding value vector matrix cache and the first value vector matrix in the first specified dimension to obtain the second value vector matrix.

[0161] Correspondingly, in the specific implementation example, converting the second operation in the initial model into a plurality of sub-operations can include:

[0162] The matrix multiplication operation of the second value vector matrix and the second attention score matrix shown in operation 312 is converted into a plurality of sub-operations: a cutting operation of the second attention score matrix in the third specified dimension, based on which a third sub-score matrix and a fourth sub-score matrix can be obtained; a matrix multiplication operation of the value vector matrix cache and the third sub-score matrix, based on which a second intermediate sub-result can be obtained; a matrix multiplication operation of the first value vector matrix and the fourth sub-score matrix, based on which a third intermediate sub-result can be obtained; and an addition operation of the second intermediate sub-result and the third intermediate sub-result.

[0163] Based on the embodiment, at least part of the concatenation operation involved in the reasoning process of the attention mechanism and having a relatively large memory access amount can be further optimized, and the second operation in the initial model is converted into a plurality of sub-operations, so as to reduce the operation times of the memory access operation, improve the reasoning speed, and reduce the long reasoning time while ensuring that the intermediate results obtained by processing the same input object before and after model optimization are unchanged.

[0164] Based on the optimized model obtained based on the above other specific implementation examples, an example implementation of processing an input object by executing the target instruction sequence after generating the computer executable target instruction sequence through compilation processing is as follows. Figure 6 As shown in the following table, the example implementation includes the following.

[0165] 502, the concatenation operation of the key vector matrix cache (K CACHE) and the first key vector matrix (K) in the input object in the first specified dimension, based on which the second key vector matrix can be obtained.

[0166] The first specified dimension here can be the dimension where t2 in the key vector matrix cache is located, and the dimension where t1 in the first key vector matrix is located.

[0167] Since the shape of K CACHE is [b, h, d_k, t2], and the shape of K is [b, h, d_k, t1], for example, K CACHE and K can be concatenated in the last dimension by K = cat ([K CACHE, K], dim =-1), and the output is assigned to a new K (referred to as the second key vector matrix). The shape of the second key vector matrix is [b, h, d_k, t2+t1]. Wherein, dim =-1 represents the last dimension.

[0168] 504, the transpose operation of the corresponding first query vector matrix (Q) in the first specified dimension pair, based on which the second query vector matrix can be obtained.

[0169] The first specified dimension pair here can be the dimensions where d_k and time1 in the first query vector matrix are located.

[0170] Since the shape of Q is [b, h, d_k, t1], for example, Q can be transposed by Q = Q.transpose(2,3), and the transpose operation is performed between the last two dimensions. The output is assigned to a new Q (i.e. the second query vector matrix), and the shape of the second query vector matrix is [b, h, t1, d_k]. Since Q is a four-dimensional matrix, its four dimensions correspond to 0, 1, 2, and 3, and 2 and 3 in transpose(2,3) represent the last two dimensions.

[0171] 506, the matrix multiplication operation on the second query vector matrix and the second key vector matrix, based on which the first attention score matrix (scores) can be obtained.

[0172] Since the shape of the second query vector matrix is [b, h, t1, d_k] and the shape of the second key vector matrix is [b, h, d_k, t2+t1], the first attention score matrix can be obtained by performing the matrix multiplication operation on Q and K through scores=matmul(Q,K), and the shape of the first attention score matrix is [b, h, t1, t2+t1].

[0173] 508, the transposition operation on the first attention score matrix between the second specified dimension pair, based on which the second attention score matrix can be obtained.

[0174] The second specified dimension pair here can be the dimensions where t1 and t2+t1 in the second attention score matrix are located.

[0175] For example, the last two dimensions of the second attention score matrix can be transposed by attn=attn.transpose(2,3), and the shape of the second attention score matrix is [b, h, t1, t2+t1] before the transposition operation, and the shape of the second attention score matrix is [b, h, t2+t1, t1] after the transposition operation.

[0176] Optionally, the processing operation in the optimization model can also selectively include the normalization operation on the first attention score matrix, based on which the second attention score matrix (attn) can be obtained.

[0177] For example, the last dimension (dim=-1) of the first attention score matrix can be normalized by attn=softmax(scores,dim=-1), and the shape of the second attention score matrix obtained by the normalization operation remains unchanged, and the numerical values of the elements in the second attention score matrix are normalized in the range of [0, 1].

[0178] Correspondingly, in the operation 508, the transposition operation on the normalized attention score matrix between the second specified dimension pair is specifically performed.

[0179] 510, the cutting operation on the second attention score matrix in the third specified dimension, based on which the third sub-score matrix and the fourth sub-score matrix can be obtained.

[0180] The third specified dimension here can be the dimension where t2+t1 in the second attention score matrix is located.

[0181] For example, the second attention score matrix with shape [b, h, t2+t1, t1] can be split on the second to last dimension (dim=2) by attn_part1, attn_part2 = split(attn, [time2, time1], dim=2) to obtain two parts: a third sub-score matrix (attn_part1) with shape [b, h, t2, t1] and a fourth sub-score matrix (attn_part2) with shape [b, h, t1, t1].

[0182] 512, a matrix multiplication operation is performed on the value vector matrix cache (V_CACHE) and the third sub-score matrix, based on which a second intermediate sub-result (X_part1) can be obtained.

[0183] Since the shape of V_CACHE is [b, h, d_k, t2] and the shape of the third sub-score matrix is [b, h, t2, t1], a matrix multiplication operation can be performed on V_CACHE and attn_part1 by X_part1 = matmul(V_CACHE, attn_part1) to obtain the second intermediate sub-result X_part1 with shape [b, h, d_k, t1].

[0184] 514, a matrix multiplication operation is performed on the first value vector matrix (V) and the fourth sub-score matrix, based on which a third intermediate sub-result (X_part2) can be obtained.

[0185] Since the shape of V is [b, h, d_k, t1] and the shape of the fourth sub-score matrix is [b, h, t1, t1], a matrix multiplication operation can be performed on V and attn_part2 by X_part2 = matmul(V, attn_part2) to obtain the third intermediate sub-result X_part2 with shape [b, h, d_k, t1].

[0186] 516, an addition operation is performed on the second intermediate sub-result and the third intermediate sub-result, based on which a first intermediate sub-result (X) can be obtained.

[0187] Through the operation 516, the shape of the obtained first intermediate sub-result is [b, h, d_k, t1].

[0188] The operations 510-516 are converted sub-operations of the matrix multiplication operation of the operation 312 on the second value vector matrix and the second attention score matrix.

[0189] 518. The first intermediate sub-result is subjected to a rearrangement operation, and a second intermediate result can be obtained based on the rearrangement operation.

[0190] For example, the first intermediate sub-result can be rearranged (view) through X=X.view(batch, d_k*head, 1, time1), and the obtained second intermediate result is the result that can be obtained by optimizing the attention network or by processing the input object through the target instruction sequence corresponding to the optimized attention network.

[0191] Through the operation 518, the multiple first intermediate sub-results corresponding to the multi-head attention mechanism can be fused, and the first intermediate sub-result is converted from a thought matrix to a three-dimensional matrix to obtain an intermediate result of the entire optimized attention network. For a single-head attention mechanism, the value of the head number head is 1, and the operation 518 is also applicable.

[0192] Based on the embodiment shown in Figure 4 Based on the embodiment shown in The splicing operation of the operation 304 is further optimized, and the matrix multiplication operation in the operation 312 is converted into the multiple sub-operations shown in the operations 510-516. Thus, the memory access amount and inference time for executing the splicing operation of the operation 304 are eliminated, and the inference speed of the attention mechanism can be effectively improved. In addition, the calculation amount of the operations 312 and the converted operations 510-516 on the matrix multiplication is the same, and the conversion does not increase the calculation amount and processing overhead. In addition, compared with the optimized splicing operation of the operation 304, in the embodiment, the second attention score matrix is first split into two parts attn_part1[b, h, t2, t1] and attn_part2[b, h, t1, t1], and the calculation results are added after being calculated respectively. Since the shapes of attn_part1[b, h, t2, t1], attn_part2[b, h, t1, t1], and the input of the addition are all [b, h, d_k, t1], and the shapes of the inputs of the splicing operation in the operation 304 are [b, h, d_k, t2] and [b, h, d_k, t1], since t1 << d_k and t1 << t2, the data amount involved in the split (Split) operation and the data amount involved in the addition in the embodiment of the present disclosure are much smaller than the data amount involved in the reduced splicing operation, so that the memory access amount can be greatly reduced, the inference time can be reduced, and the inference speed of the attention mechanism can be effectively improved.

[0193] In some implementation examples, the optimization of the at least part of the splicing operations in the initial model can include: removing the operations in the initial model as shown in operation 302 and operation 304, i.e., the splicing operations of the corresponding key vector matrix cache and the first key vector matrix in the first specified dimension in the input object, based on which the second key vector matrix can be obtained; and the splicing operations of the value vector matrix cache and the first value vector matrix in the first specified dimension, based on which the second value vector matrix can be obtained.

[0194] Correspondingly, in some implementation examples, the conversion of the second operation in the initial model into a plurality of sub-operations can include:

[0195] The matrix multiplication operation of the second query vector matrix and the second key vector matrix as shown in operation 308 is converted into a plurality of sub-operations, including: the matrix multiplication operation of the second query vector matrix and the key vector matrix cache, based on which the first sub-score matrix can be obtained; the matrix multiplication operation of the second query vector matrix and the first key vector matrix, based on which the second sub-score matrix can be obtained; and the splicing operation of the first sub-score matrix and the second sub-score matrix in the first specified dimension.

[0196] The matrix multiplication operation of the second value vector matrix and the second attention score matrix as shown in operation 312 is converted into a plurality of sub-operations, including: the cutting operation of the second attention score matrix in the third specified dimension, based on which the third sub-score matrix and the fourth sub-score matrix can be obtained; the matrix multiplication operation of the value vector matrix cache and the third sub-score matrix, based on which the second intermediate sub-result can be obtained; the matrix multiplication operation of the first value vector matrix and the fourth sub-score matrix, based on which the third intermediate sub-result can be obtained; and the addition operation of the second intermediate sub-result and the third intermediate sub-result.

[0197] For details, please refer to the description of the related part in the above embodiments, which will not be repeated here.

[0198] Based on the processing operations in the optimized model obtained from the above-mentioned implementation examples, as shown in operation 604, the processing operations include: Figure 7

[0199] 602, the transposition operation of the first query vector matrix in the input object between the first specified dimensions, based on which the second query vector matrix can be obtained.

[0200] 604, the matrix multiplication operation of the second query vector matrix and the key vector matrix cache, based on which the first sub-score matrix can be obtained.

[0201] ​606, the matrix multiplication operation performed on the second query vector matrix and the first key vector matrix, based on which a second sub-score matrix can be obtained.

[0202] 608, the splicing operation performed on the first sub-score matrix and the second sub-score matrix in the first specified dimension, based on which a first attention score matrix can be obtained.

[0203] 610, the transposition operation performed on the first attention score matrix between the second specified dimension pairs, based on which a second attention score matrix can be obtained.

[0204] Optionally, the processing operation in the optimization model can also selectively include: a normalization operation performed on the first attention score matrix, based on which a normalized attention score matrix can be obtained. Correspondingly, in the operation 610, the transposition operation is performed on the normalized attention score matrix between the second specified dimension pairs.

[0205] 612, the cutting operation performed on the second attention score matrix in the third specified dimension, based on which a third sub-score matrix and a fourth sub-score matrix can be obtained.

[0206] 614, the matrix multiplication operation performed on the value vector matrix cache and the third sub-score matrix, based on which a second intermediate sub-result can be obtained.

[0207] 616, the matrix multiplication operation performed on the first value vector matrix and the fourth sub-score matrix, based on which a third intermediate sub-result can be obtained.

[0208] 618, the addition operation performed on the second intermediate sub-result and the third intermediate sub-result, based on which can be obtained.

[0209] 620, the rearrangement operation performed on the first intermediate sub-result to obtain a second intermediate result.

[0210] Figure 7 For specific implementation of each operation in the illustrated embodiment, please refer to the above Figures 4-6 For the description of the related part in the illustrated embodiment, details are not repeated here.

[0211] Based on the present embodiment, at least part of the transposition operation and at least part of the splicing operation with relatively large memory access amount in the inference process involving the attention mechanism can be further optimized simultaneously, and the second operation operation in the initial model is converted into multiple sub-operations, so as to reduce the operation times of the memory access operation, improve the inference speed, and reduce the long inference time while ensuring that the intermediate results obtained by processing the same input object before and after model optimization are unchanged.

[0212] Alternatively, in some implementations, in step 204, at least some transpose operations and at least some splicing operations in the initial model can be removed simultaneously.

[0213] Accordingly, in this embodiment, step 208 may include: adjusting the processing objects and input order of the processing objects in the initial model for the third operation, converting the third operation into multiple sub-operations, and adding a transpose operation to the intermediate output in the initial model. In this embodiment, the target operation specifically includes the third operation. Here, the third operation is an operation that updates the intermediate result obtained by processing the same input object after optimizing the initial model with at least some transpose and at least some concatenation operations, such as matrix multiplication. The specific type and number of the third operation, as well as the processing objects involved, can be determined according to actual needs.

[0214] For example, in one specific implementation, the transpose operations performed on Q, K, and V respectively, as well as the transpose operation on matrix X, during the inference process of the attention mechanism, can be removed, along with the concatenation operations of operations 302 and / or 304. Correspondingly, a transpose operation on the normalized attn is added, and the processing object and / or input order of the matrix multiplication operation between the transposed matrix K and Q (obtained by the second transpose operation) are adjusted accordingly and converted into multiple sub-operations; and / or the processing object and input order of the matrix multiplication operation between the normalized attn and V are adjusted accordingly and converted into multiple sub-operations. For details, please refer to the relevant sections of the above embodiments; further elaboration is not provided here.

[0215] The processing operations included in the optimized model based on these implementation methods can be referenced. Figure 7 This will not be elaborated upon here.

[0216] Based on this embodiment, optimization can be directly applied simultaneously to at least some transpose operations and at least some concatenation operations with relatively large memory accesses during the inference process of the attention mechanism in related technologies. The third operation in the initial model can also be updated accordingly. This ensures that the intermediate results obtained from processing the same input object remain unchanged before and after model optimization. Figure 5 , Figure 6 or Figure 7 The illustrated embodiment reduces the number of memory access operations, improves inference speed, and reduces the inference time.

[0217] The above disclosure Figures 5-7 The embodiments shown can be used for self-attention mechanisms, and this disclosure does not limit the scope of the embodiments.

[0218] Based on the optimization model obtained by the embodiment of the present disclosure, after generating the computer executable target instruction sequence through the compiling process, the processing of the input object can be realized by executing the target instruction sequence. The processing process can be referred to as the inference process or attention calculation process of the attention mechanism. For the convenience of quantitative understanding, the following still takes the above five input parameters and their shapes as an example to illustrate the processing of the input object by executing the target instruction sequence in the embodiment of the present disclosure.

[0219] Figure 8 The flowchart of the inference method of the attention mechanism provided by an exemplary embodiment of the present disclosure. The embodiment of the present disclosure can be applied to any electronic device such as a terminal device, a server, a hardware platform, etc. As shown in the figure, the embodiment includes the following steps: Figure 8

[0220] 702, obtaining the computer executable target instruction sequence corresponding to the optimized attention network.

[0221] The target instruction sequence here can be generated based on the instruction generation method of the attention mechanism of any embodiment of the present disclosure.

[0222] 704, transmitting the target instruction sequence to the hardware platform.

[0223] 706, using the hardware platform to execute the target instruction sequence to process the input parameters corresponding to the current sub-object in the received input object, and obtain the inference result corresponding to the current sub-object, that is, the attention distribution.

[0224] The input object includes a plurality of sub-objects, and the input parameters of each sub-object include a query vector matrix, a key vector matrix, a value vector matrix, a key vector matrix cache and a value vector matrix cache.

[0225] Optionally, before executing the embodiment, the input parameters of each sub-object can be obtained in advance.

[0226] For example, in some implementations, the input parameters of each sub-object can be obtained through a feature extraction layer (such as an Embedding layer) and a linear transformation layer.

[0227] For example, the feature extraction layer can be used to extract the features of the input object to obtain an input vector matrix Y [y1, y2,..., y n], where n is the number of sub-objects included in the input object Y, and each yi is a d_k-dimensional vector, which is a sub-input vector corresponding to each sub-object.

[0228] ​Then, the sub-input vectors yi corresponding to each sub-object are linearly transformed by a linear transformation layer, for example, linear transformation can be performed by K=yi*W_k, V=yi*W_v, Q=yi*W_q, to obtain the query vector matrix, the key vector matrix and the value vector matrix of each sub-object in the preset dimension. Wherein, W_k, W_v, W_q are parameter matrices learned by the deep learning model in the training stage.

[0229] By sequentially processing each sub-object in the input object by the deep learning model, after obtaining the key vector matrix and the value vector matrix corresponding to each sub-object, the key vector matrix of the sub-object can be spliced with the key vector matrix cache before the sub-object in the input object to obtain the key vector matrix cache of the sub-object; similarly, the value vector matrix of the sub-object can be spliced with the value vector matrix before the sub-object in the input object to obtain the value vector matrix cache of the sub-object, so as to take the query vector matrix, the key vector matrix, the value vector matrix, the key vector matrix cache and the value vector matrix cache of the sub-object as five input parameters of the sub-object for inference of the attention mechanism.

[0230] The specific implementation of obtaining the input parameters of each sub-object is not limited in the embodiments of the present disclosure.

[0231] Based on the present embodiment, by executing the target instruction sequence corresponding to the optimized attention network, the operation amount of the memory access operation is effectively reduced in the inference process of the input object by the attention mechanism, so as to effectively improve the inference speed of the attention mechanism, reduce the long inference time and reduce the overall processing overhead.

[0232] Optionally, in some implementations, in step 706, the input parameters corresponding to the current sub-object in the received input object are processed to obtain the inference result corresponding to the current sub-object, which can be implemented by executing the processing operation shown in the embodiment. Figure 4

[0233] Figure 9 The flowchart of the inference method of the attention mechanism provided by another exemplary embodiment of the present disclosure is shown. As shown in the embodiment, the inference method comprises: Figure 9

[0234] 802, the key vector matrix cache (K CACHE) and the first key vector matrix (K) corresponding to the current sub-object in the input object are spliced in the first specified dimension to obtain the second key vector matrix.

[0235] The input object comprises a plurality of sub-objects.

[0236] ​​804, perform a splicing operation on the value vector matrix corresponding to the current sub-object (V_CACHE) and the first value vector matrix (V) in the first specified dimension, to obtain a second value vector matrix.

[0237] 806, perform a transposition operation on the first query vector matrix corresponding to the current sub-object (Q) in the first specified dimension pair, to obtain a second query vector matrix.

[0238] Wherein, there is no execution order restriction between steps 802-806, and the three can be executed simultaneously in parallel, or in any time sequence, and the embodiments of the present disclosure do not limit this.

[0239] 808, perform a matrix multiplication operation on the second query vector matrix and the second key vector matrix, to obtain a first attention score matrix (scores).

[0240] 810, perform a transposition operation on the first attention score matrix in the second specified dimension pair, to obtain a second attention score matrix.

[0241] Optionally, before step 810, a normalization operation can also be selectively included on the first attention score matrix to obtain a normalized attention score matrix (attn).

[0242] Correspondingly, in this operation 810, a transposition operation is specifically performed on the normalized attention score matrix in the second specified dimension pair to obtain the second attention score matrix.

[0243] 812, perform a matrix multiplication operation on the second value vector matrix and the second attention score matrix, to obtain a first intermediate sub-result (X).

[0244] 814, perform a rearrangement operation on the first intermediate sub-result to obtain an inference result corresponding to the current sub-object.

[0245] Through this step 814, the multiple first intermediate sub-results corresponding to the multi-head attention mechanism can be fused, and the first intermediate sub-result is converted from a thinking matrix to a three-dimensional matrix to obtain an inference result corresponding to the current sub-object. For a single-head attention mechanism, the value of the number of heads head is 1, and this step 814 is also applicable.

[0246] This embodiment can be used for both self-attention mechanism (Self Attention) and cross-attention mechanism (Cross Attention), and the embodiments of the present disclosure do not limit this.

[0247] Corresponding to the embodiments shown in the above Figure 4 , please refer to the description of the above embodiments for specific implementation, and the related technical effects are also described in Figure 4The description of the embodiments shown will not be repeated here.

[0248] Optionally, in some other implementations, in step 706, the input parameter corresponding to the current sub-object in the received input object is processed to obtain the inference result corresponding to the current sub-object, which can be achieved by executing Figure 5 The processing operation of the embodiments shown is implemented.

[0249] Figure 10 The flowchart of the attention mechanism inference method provided for another exemplary embodiment of the present disclosure is shown. As shown, the inference method includes: Figure 10

[0250] 902, the value vector matrix cache (V_CACHE) and the first value vector matrix (V) corresponding to the current sub-object in the input object are spliced in the first specified dimension to obtain a second value vector matrix.

[0251] The input object includes a plurality of sub-objects.

[0252] 904, the first query vector matrix (Q) corresponding to the current sub-object is transposed between the first specified dimension pairs to obtain a second query vector matrix.

[0253] There is no execution order restriction between steps 902-904, and the three can be executed simultaneously in parallel or in any time sequence, and the present disclosure does not limit this.

[0254] 906, the second query vector matrix and the key vector matrix cache (K_CACHE) are subjected to matrix multiplication operation to obtain a first sub-score matrix.

[0255] 908, the second query vector matrix and the first key vector matrix are subjected to matrix multiplication operation to obtain a second sub-score matrix.

[0256] There is no execution order restriction between steps 906-908, and the two can be executed simultaneously in parallel or in any time sequence, and the present disclosure does not limit this.

[0257] 910, the first sub-score matrix and the second sub-score matrix are spliced in the first specified dimension to obtain a first attention score matrix (scores), which has a shape of [b, h, t1, t2+t1].

[0258] 912, the first attention score matrix is transposed between the second specified dimension pairs to obtain a second attention score matrix.

[0259] ​Optionally, before step 912, a normalization operation can also be selectively included, that is, normalization is performed on the first attention score matrix to obtain a normalized attention score matrix (attn).

[0260] Correspondingly, in the operation 912, the normalized attention score matrix is transposed between the second specified dimensions to obtain a second attention score matrix.

[0261] 914, the second value vector matrix and the second attention score matrix are multiplied to obtain a first intermediate result (X).

[0262] 916, the first intermediate result is rearranged to obtain the inference result corresponding to the current sub-object.

[0263] Through the step 916, the multiple first intermediate results corresponding to the multi-head attention mechanism can be fused, and the first intermediate result is converted from the thinking matrix to the three-dimensional matrix to obtain the inference result output by the entire optimization attention network for the current sub-object. For single-head attention mechanism, the value of head is 1, and the step 916 is also applicable.

[0264] The embodiment can be used for self-attention mechanism (Self Attention).

[0265] Corresponding to the embodiments shown in the above Figure 5 , please refer to the description of the above embodiments for specific implementation, and the related technical effects can also be seen in the description of the embodiments shown in Figure 5 , which will not be described here.

[0266] In addition, in some implementations, in step 706, the input parameters corresponding to the current sub-object in the received input object are processed to obtain the inference result corresponding to the current sub-object, which can be implemented by performing the processing operation of the embodiment shown in Figure 6 .

[0267] Figure 11 The flowchart of the inference method of the attention mechanism provided by another exemplary embodiment of the present disclosure is shown in Figure 11 , the inference method includes:

[0268] 1002, the key vector matrix cache (K CACHE) corresponding to the current sub-object in the input object and the first key vector matrix (K) are spliced in the first specified dimension to obtain a second key vector matrix.

[0269] 1004, the first query vector matrix (Q) corresponding to the current sub-object is transposed between the first specified dimensions to obtain a second query vector matrix.

[0270] There is no execution sequence restriction between step 1002 and step 1004, both can be executed simultaneously in parallel, or can be executed in any time sequence, and the embodiments of the present disclosure do not limit this.

[0271] 1006, the second query vector matrix and the second key vector matrix are subjected to matrix multiplication operation to obtain the first attention score matrix (scores).

[0272] 1008, the first attention score matrix is subjected to transposition operation between the second specified dimension pairs to obtain the second attention score matrix.

[0273] Optionally, before step 1008, a normalization operation on the first attention score matrix can also be selectively included to obtain a normalized attention score matrix (attn). Accordingly, in the operation 1008, the normalized attention score matrix is specifically subjected to transposition operation between the second specified dimension pairs to obtain the second attention score matrix.

[0274] 1010, the second attention score matrix is subjected to cutting on the third specified dimension to obtain the third sub-score matrix and the fourth sub-score matrix.

[0275] 1012, the value vector matrix cache (V_CACHE) and the third sub-score matrix are subjected to matrix multiplication operation to obtain the second intermediate sub-result (X_part1).

[0276] 1014, the first value vector matrix (V) and the fourth sub-score matrix are subjected to matrix multiplication operation to obtain the third intermediate sub-result (X_part2).

[0277] There is no execution sequence restriction between step 1012 and step 1014, both can be executed simultaneously in parallel, or can be executed in any time sequence, and the embodiments of the present disclosure do not limit this.

[0278] 1016, the second intermediate sub-result and the third intermediate sub-result are subjected to addition operation to obtain the first intermediate sub-result (X).

[0279] 1018, the first intermediate sub-result is subjected to rearrangement operation to obtain the inference result corresponding to the current sub-object.

[0280] Through the step 1018, the multiple first intermediate sub-results corresponding to the multi-head attention mechanism can be fused, and the first intermediate sub-result is converted from the thinking matrix to the three-dimensional matrix to obtain the inference result output by the entire optimized attention network for the current sub-object. For single-head attention mechanism, the value of the head number head is 1, and this step 1018 is also applicable.

[0281] The embodiment can be used for self-attention mechanism.

[0282] Corresponding to the embodiments shown in the above Figure 6 , please refer to the description of the above embodiments for specific implementation, which will not be repeated here.

[0283] Alternatively, in still other implementations, in step 706, the input parameters corresponding to the current sub-object in the received input object are processed to obtain the inference result corresponding to the current sub-object, which can be implemented by executing the processing operation of the embodiment shown in Figure 7 .

[0284] Figure 12 The flowchart of the inference method of the attention mechanism provided for another exemplary embodiment of the present disclosure is shown in Figure 12 . The inference method includes:

[0285] 1102, the first query vector matrix corresponding to the current sub-object in the input object is transposed between the first specified dimensions to obtain the second query vector matrix.

[0286] The input object includes a plurality of sub-objects.

[0287] 1104, the second query vector matrix and the key vector matrix cache are subjected to matrix multiplication operation to obtain the first sub-score matrix.

[0288] 1106, the second query vector matrix and the first key vector matrix are subjected to matrix multiplication operation to obtain the second sub-score matrix.

[0289] There is no execution order restriction between step 1104 and step 1106, both of which can be executed simultaneously in parallel, or in any time sequence, and the present embodiment does not limit this.

[0290] 1108, the first sub-score matrix and the second sub-score matrix are subjected to splicing operation in the first specified dimension to obtain the first attention score matrix.

[0291] 1110, the first attention score matrix is transposed between the second specified dimension pair to obtain the second attention score matrix.

[0292] Optionally, before step 1110, a normalization operation on the first attention score matrix can also be selectively included to obtain the normalized attention score matrix (attn).

[0293] Correspondingly, in the operation 1110, the normalized attention score matrix is transposed between the second specified dimension pair to obtain the second attention score matrix.

[0294] 1112, cut the second attention score matrix in the third specified dimension to obtain a third sub-score matrix and a fourth sub-score matrix.

[0295] 1114, perform a matrix multiplication operation on the value vector matrix buffer and the third sub-score matrix to obtain a second intermediate sub-result.

[0296] 1116, perform a matrix multiplication operation on the first value vector matrix and the fourth sub-score matrix to obtain a third intermediate sub-result.

[0297] Wherein, there is no execution order restriction between step 1114 and step 1116, both can be executed simultaneously in parallel, or can be executed in any time sequence, and the embodiments of the present disclosure do not limit this.

[0298] 1118, perform an addition operation on the second intermediate sub-result and the third intermediate sub-result to obtain a first intermediate sub-result (X).

[0299] 1120, perform a rearrangement operation on the first intermediate sub-result to obtain the inference result corresponding to the current sub-object.

[0300] The present embodiment can be used for self-attention mechanism (Self Attention).

[0301] Corresponding to the embodiments shown in the above Figure 7 , please refer to the description of the above embodiments for specific implementation, and the related technical effects also refer to the description of the embodiments shown in Figure 7 , which will not be repeated here.

[0302] Based on the inference result of the input object, it can be used for natural language processing (NLP) field, automatic driving scene, medical image processing scene, voice interaction scene task processing, such as machine translation, text classification, question and answer system, image classification, etc. Based on the inference result of the input object and the feature obtained by pre-extracting the feature of the input object, the corresponding task processing result can be obtained, and the embodiments of the present disclosure do not limit this.

[0303] Exemplary apparatus

[0304] Figure 13 The structure diagram of the instruction generation device for attention mechanism provided by an exemplary embodiment of the present disclosure is shown. The instruction generation device for attention mechanism of the present embodiment can be used to implement the instruction generation method for attention mechanism of any of the above embodiments of the present disclosure. As Figure 13 shown, the instruction generation device for attention mechanism of the present embodiment comprises:

[0305] The first obtaining module 1202 is configured to obtain an initial model corresponding to a to-be-compiled attention network.

[0306] The update determining module 1204 is configured to determine a target operation in the initial model that needs to be updated and an update manner of the target operation based on the removed at least part of the memory access operation.

[0307] The update module 1206 is configured to update the target operation based on the update manner of the target operation, to obtain an optimized model.

[0308] The compiling processing module 1208 is configured to generate a computer executable target instruction sequence through compiling processing based on the optimized model.

[0309] The first intermediate result obtained by processing an input object based on the to-be-compiled attention network is the same as a second intermediate result obtained by processing the input object based on an optimized attention network corresponding to the optimized model.

[0310] The first intermediate result obtained by processing an input object based on the to-be-compiled attention network is the same as a second intermediate result obtained by processing the input object based on an optimized attention network corresponding to the optimized model.

[0311] Optionally, in some implementations, the initial model can include, for example but not limited to, two types of processing, i.e., operation and memory access operation, wherein the memory access operation can include, for example but not limited to, two types of memory access operation, i.e., transpose operation and concatenation operation.

[0312] Optionally, in some implementations, the removing module 1204 is specifically configured to remove at least part of the transpose operation in the initial model. Correspondingly, the update module 1206 is specifically configured to adjust a processing object and / or input order of the processing object of a first operation in the initial model, and add a transpose operation of an intermediate output in the initial model, wherein the target operation includes the first operation.

[0313] The processing operation in the optimized model obtained based on the implementation can refer to the above Figure 4 , which will not be described herein again.

[0314] Optionally, in further implementations, the removing module 1204 is specifically further configured to remove at least part of the concatenation operation in the initial model. Correspondingly, the update module 1206 is specifically further configured to convert a second operation in the initial model into a plurality of sub-operations. In this embodiment, the target operation specifically includes the first operation and the second operation.

[0315] In some implementation examples, the removing module 1204 is specifically configured to remove the following operation in the initial model: a concatenation operation performed on the key vector matrix cache corresponding to the current sub-object in the input object and the first key vector matrix in the first specified dimension.

[0316] Correspondingly, the updating module 1206 is specifically configured to convert the matrix multiplication operation performed on the second query vector matrix and the second key vector matrix into the following multiple sub-operations: a matrix multiplication operation performed on the second query vector matrix and the key vector matrix cache, based on which a first sub-score matrix is obtained; a matrix multiplication operation performed on the second query vector matrix and the first key vector matrix, based on which a second sub-score matrix is obtained; a concatenation operation performed on the first sub-score matrix and the second sub-score matrix in the first specified dimension.

[0317] Based on the processing operation included in the optimized model obtained based on this implementation example, the processing operation can be referred to the above Figure 5 , which will not be described here again.

[0318] In some implementation examples, the removing module 1204 is specifically configured to remove the following operation in the initial model: a concatenation operation performed on the value vector matrix cache corresponding to the current sub-object and the first value vector matrix in the first specified dimension.

[0319] Correspondingly, the updating module 1206 is specifically configured to convert the matrix multiplication operation performed on the second value vector matrix and the third attention score matrix into the following multiple sub-operations: a cutting operation performed on the third attention score matrix in the third specified dimension, based on which a third sub-score matrix and a fourth sub-score matrix are obtained; a matrix multiplication operation performed on the value vector matrix cache and the third sub-score matrix, based on which a second intermediate sub-result is obtained; a matrix multiplication operation performed on the first value vector matrix and the fourth sub-score matrix, based on which a third intermediate sub-result is obtained; an addition operation performed on the second intermediate sub-result and the third intermediate sub-result.

[0320] Based on the processing operation included in the optimized model obtained based on this implementation example, the processing operation can be referred to the above Figure 6 , which will not be described here again.

[0321] In some implementation examples, the removing module 1204 is specifically configured to remove the following operation in the initial model: a concatenation operation performed on the key vector matrix cache and the first key vector matrix in the first specified dimension; a concatenation operation performed on the value vector matrix cache and the first value vector matrix in the first specified dimension.

[0322] Correspondingly, the updating module 1206 is specifically configured to:

[0323] The matrix multiplication operation of the second query vector matrix and the second key vector matrix is converted into the following multiple sub-operations: the matrix multiplication operation of the second query vector matrix and the key vector matrix cache, and the first sub-score matrix is obtained based on the matrix multiplication operation; the matrix multiplication operation of the second query vector matrix and the first key vector matrix, and the second sub-score matrix is obtained based on the matrix multiplication operation; the splicing operation of the first sub-score matrix and the second sub-score matrix in the first specified dimension;

[0324] The matrix multiplication operation of the second value vector matrix and the third attention score matrix is converted into the following multiple sub-operations: the cutting operation of the third attention score matrix in the third specified dimension, and the third sub-score matrix and the fourth sub-score matrix are obtained based on the cutting operation; the matrix multiplication operation of the value vector matrix cache and the third sub-score matrix, and the second intermediate sub-result is obtained based on the matrix multiplication operation; the matrix multiplication operation of the first value vector matrix and the fourth sub-score matrix, and the third intermediate sub-result is obtained based on the matrix multiplication operation; the addition operation of the second intermediate sub-result and the third intermediate sub-result.

[0325] Based on the processing operation included in the optimization model obtained based on the yet some implementation examples, the processing operation included in the optimization model obtained based on the yet some implementation examples can be referred to the above Figure 7 , which will not be repeated here.

[0326] Optionally, in some other implementation modes, the removing module 1204 is specifically configured to remove at least part of the transposition operation and at least part of the splicing operation in the initial model.

[0327] Correspondingly, the updating module 1206 is specifically configured to adjust the processing object and / or the input order of the processing object of the third operation in the initial model, and convert the third operation into multiple sub-operations, and add the transposition operation of the intermediate output in the initial model and adjust the processing object and the input order of the processing object of the third operation in the initial model, and convert the third operation into multiple sub-operations, and add the transposition operation of the intermediate output in the initial model.

[0328] Based on the processing operation included in the optimization model obtained based on the yet some implementation examples, the processing operation included in the optimization model obtained based on the yet some implementation examples can be referred to the above Figure 7 , which will not be repeated here.

[0329] The instruction generation apparatus for attention mechanisms in this disclosure corresponds to the instruction generation method for attention mechanisms described above, and the relevant content can be referred to each other, which will not be repeated here. The beneficial technical effects of the instruction generation apparatus for attention mechanisms in this disclosure can be found in the corresponding beneficial technical effects of the instruction generation method for attention mechanisms in the above-described embodiments, which will not be repeated here.

[0330] Figure 14 This is a schematic diagram of the structure of an attention mechanism inference system provided as an exemplary embodiment of this disclosure. The attention mechanism inference system of this disclosure can be used to implement the inference method of the attention mechanism in any of the above embodiments of this disclosure. Figure 14 As shown, the inference system for the attention mechanism in this embodiment includes:

[0331] The second acquisition module 1302 is used to acquire a computer-executable target instruction sequence corresponding to the optimized attention network, wherein the target instruction sequence is generated based on the instruction generation method for attention mechanism in any of the above embodiments of the present disclosure.

[0332] The transmission module 1304 is used to transmit the target instruction sequence to the hardware platform.

[0333] Hardware platform 1306 is used to execute target instruction sequence, process the input parameters corresponding to the current sub-object in the received input object, and obtain the inference result corresponding to the current sub-object. The input object includes multiple sub-objects.

[0334] The hardware platform of this disclosure may include, but is not limited to, GPUs, CPUs, accelerators, etc., and this disclosure does not limit it.

[0335] Figure 15 This is a schematic diagram illustrating the structure of a deep learning model provided in an exemplary embodiment of this disclosure. The deep learning model in this disclosure may include, but is not limited to, the Transformer model, convolutional neural network model, etc., and this disclosure does not limit the specific implementation type of the deep learning model. Figure 15 As shown, the deep learning model in this embodiment of the disclosure includes an encoder 1402 and a decoder 1404. The encoder 1402 includes an optimized attention network implemented based on an optimized model. This optimized model is obtained by obtaining an initial model corresponding to the attention network to be compiled based on the instruction generation method for attention mechanism in any of the above embodiments of the disclosure, optimizing at least some memory access operations in the initial model, and updating the target operation in the initial model.

[0336] In addition, such as Figure 16As shown, it is a structural diagram of a deep learning model provided by another exemplary embodiment of the present disclosure. In the deep learning model in the embodiment of the present disclosure, the encoder 1402 can further include a feature extraction layer (for example, an Embedding layer), a linear transformation layer and a first feed forward neural network.

[0337] In the feature extraction layer, the input object can be subjected to feature extraction to obtain an input vector matrix Y [y1, y2,..., y n], where n is the number of sub-objects included in the input object Y, and each yi is a d_k-dimensional vector, which is a sub-input vector corresponding to each sub-object.

[0338] Then, the linear transformation layer is used to perform linear transformation on the sub-input vector yi corresponding to each sub-object, for example, K=yi*W_k, V=yi*W_v, Q=yi*W_q can be used to perform linear transformation to obtain the query vector matrix, the key vector matrix and the value vector matrix corresponding to each sub-object. Where W_k, W_v, W_q are parameter matrices learned by the deep learning model through the training stage.

[0339] By sequentially processing each sub-object in the input object through the deep learning model, after obtaining the key vector matrix and the value vector matrix corresponding to each sub-object, the key vector matrix of the sub-object can be spliced with the key vector matrix cache of the sub-object before the sub-object in the input object to obtain the key vector matrix cache of the sub-object. Similarly, the value vector matrix of the sub-object can be spliced with the value vector matrix of the sub-object before the sub-object in the input object to obtain the value vector matrix cache of the sub-object. In order to input the query vector matrix (Q), the key vector matrix (K), the value vector matrix (V), the key vector matrix cache (K_CACHE) and the value vector matrix cache (V_CACHE) of the sub-object as five input parameters of the optimization attention network in the encoder, and output the inference result (i.e. attention distribution) of the sub-object through the optimization attention network. After using the key vector matrix cache (K_CACHE) and the value vector matrix cache (V_CACHE), when processing a new sub-object, the Key and Value calculated and stored before can be directly used, without the need to recalculate, thereby saving computing resources and improving processing speed.

[0340] After obtaining the attention distribution of the current sub-object through the optimization attention network in the encoder, the first feed forward neural network is used to perform nonlinear transformation on the attention distribution to obtain the encoding information matrix corresponding to each sub-object and input the decoder. The decoder can obtain the final processing result of each sub-object based on the encoding information matrix of each sub-object before the input object.

[0341] Optionally, the decoder can comprise an attention network and a second feedforward neural network. The attention network in the decoder can be an attention network before optimization according to an embodiment of the present disclosure, or can be an optimized attention network obtained by optimization according to an embodiment of the present disclosure. The attention network in the decoder can be used to further process the encoding information matrix of each sub-object output by the encoder to obtain a further attention distribution. The second feedforward neural network can obtain a final processing result of each sub-object based on the further attention distribution of each sub-object and its previous sub-object.

[0342] Each network or network layer comprised in the encoder and the decoder in an embodiment of the present disclosure can be one or more. Each network can comprise one or more network layers, and other network layers can also be set according to actual needs, which are not limited in the embodiments of the present disclosure.

[0343] The deep learning model according to an embodiment of the present disclosure can be used for image processing, machine translation, text classification, question and answer systems, and other tasks in various fields, and the final processing result is a corresponding task processing result.

[0344] For example, in a machine translation task, the deep learning model is used to convert a source language text into a target language text, and the final processing result is the target language text obtained by translation. The encoder and the decoder both use the attention mechanism, which can effectively capture the semantic information of the input text, thereby improving the translation quality.

[0345] For another example, in a text classification task, the deep learning model is mainly used to convert a text into a vector representation and classify using the vector representation, and the final processing result is the category to which the text belongs. When the deep learning model is a Transformer model, the Transformer model has an advantage in processing long texts, and therefore, when processing natural language processing tasks, good results are obtained.

[0346] For another example, in a question and answer system, the deep learning model is mainly used to match a question and an answer to provide an answer, and the final processing result is a matching score between the question and the answer.

[0347] Exemplary electronic device

[0348] In addition, an embodiment of the present disclosure further provides an electronic device, comprising:

[0349] a processor;

[0350] a memory for storing instructions executable by the processor;

[0351] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the instruction generation method for the attention mechanism or the inference method of the attention mechanism according to any one of the embodiments of the present disclosure.

[0352] Figure 17 An electronic device provided by an embodiment of the present disclosure includes at least one processor 11 and a memory 12.

[0353] The processor 11 can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device 10 to perform desired functions.

[0354] The memory 12 can include one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may, for example, include random access memory (RAM), cache, and / or the like. Non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 11 can execute the one or more computer program instructions to implement the instruction generation method for the attention mechanism or the inference method of the attention mechanism of various embodiments of the present disclosure and / or other desired functions.

[0355] In one example, the electronic device 10 can further include an input device 13 and an output device 14, which are interconnected through a bus system and / or other forms of connection mechanism (not shown).

[0356] The input device 13 can further include, for example, a keyboard, a mouse, and / or the like.

[0357] The output device 14 can output various information to the outside, which can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0358] Of course, in order to simplify, Figure 17 Only some of the components in the electronic device 10 related to the present disclosure are shown in the figure, and components such as buses, input / output interfaces, and / or the like are omitted. In addition, the electronic device 10 can further include any other appropriate components according to specific application cases.

[0359] Exemplary computer program product and computer readable storage medium

[0360] In addition to the above method and device, the embodiments of the present disclosure can also provide a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps of the instruction generation method for the attention mechanism or the inference method of the attention mechanism described in the above “Exemplary Method” section.

[0361] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0362] In addition, the embodiments of the present disclosure can also be a computer readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the instruction generation method for the attention mechanism or the inference method of the attention mechanism described in the above “Exemplary Method” section.

[0363] The computer readable storage medium can be any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium, for example, but not limited to, includes an electrical, a magnetic, an optical, an electromagnetic, an infrared, or a semiconductor system, device or apparatus, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0364] The above describes the basic principles of the present disclosure in combination with specific embodiments, but the advantages, benefits, effects and the like mentioned in the present disclosure are only examples and are not limiting, and it cannot be considered that each embodiment of the present disclosure must have them. In addition, the above specific details are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above details do not limit the present disclosure to the above specific details.

[0365] Those skilled in the art will appreciate that various adaptations and modifications of the application can be configured without departing from the scope and spirit of the application. Therefore, it is to be understood that, within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described herein.

Claims

1. An instruction generation method for attention mechanism, comprising: obtaining an initial model corresponding to an attention network to be compiled; removing at least part of memory access operations in the initial model; determining a target operation in the initial model that needs to be updated and an updating manner of the target operation based on the removed at least part of memory access operations; updating the target operation based on the updating manner of the target operation to obtain an optimized model; generating a computer executable target instruction sequence through compilation based on the optimized model; wherein a first intermediate result obtained by processing an input object based on the attention network to be compiled is the same as a second intermediate result obtained by processing the input object based on an optimized attention network corresponding to the optimized model.

2. The method of claim 1, wherein, The initial model comprises operation and memory access operation, and the memory access operation comprises transpose operation and concatenation operation.

3. The method of claim 2, wherein, Removing at least part of memory access operations in the initial model comprises: removing at least part of transpose operations in the initial model; updating the target operation based on the updating manner of the target operation comprises: adjusting a processing object and / or input order of the processing object of a first operation in the initial model, and adding a transpose operation of intermediate output in the initial model, wherein the target operation comprises the first operation.

4. The method of claim 3, wherein, The processing operation in the optimized model comprises: a concatenation operation of a key vector matrix buffer and a first key vector matrix in a first specified dimension, based on which a second key vector matrix is obtained; a concatenation operation of a value vector matrix buffer and a first value vector matrix in the first specified dimension, based on which a second value vector matrix is obtained; a transpose operation of a first query vector matrix between a first specified dimension pair, based on which a second query vector matrix is obtained; a matrix multiplication operation of the second query vector matrix and the second key vector matrix, based on which a first attention score matrix is obtained; a transpose operation of the first attention score matrix between a second specified dimension pair, based on which a second attention score matrix is obtained; a matrix multiplication operation of the second value vector matrix and the second attention score matrix, based on which a first intermediate sub-result is obtained; a rearrangement operation of the first intermediate sub-result, based on which the second intermediate result is obtained.

5. The method of claim 4, wherein, Removing at least part of memory access operations in the initial model further comprises: removing at least part of concatenation operations in the initial model; updating the target operation based on the updating manner of the target operation further comprises: converting a second operation in the initial model into a plurality of sub-operations.

6. The method of claim 5, wherein, Removing at least part of memory access operations in the initial model comprises: removing the following operation in the initial model: a concatenation operation of a key vector matrix buffer and a first key vector matrix in a first specified dimension; Converting a second operation operation in the initial model into a plurality of sub-operations, including: Converting the matrix multiplication operation of the second query vector matrix and the second key vector matrix into a plurality of sub-operations: a matrix multiplication operation of the second query vector matrix and the key vector matrix cache, based on which a first sub-score matrix is obtained; a matrix multiplication operation of the second query vector matrix and the first key vector matrix, based on which a second sub-score matrix is obtained; a splicing operation of the first sub-score matrix and the second sub-score matrix in the first specified dimension.

7. The method of claim 5, wherein, Removing at least part of the memory access operation in the initial model, including: Removing the following operation in the initial model: a splicing operation of the value vector matrix cache and the first value vector matrix in the first specified dimension; Converting a second operation operation in the initial model into a plurality of sub-operations, including: Converting the matrix multiplication operation of the second value vector matrix and the second attention score matrix into a plurality of sub-operations: a cutting operation of the second attention score matrix in the third specified dimension, based on which a third sub-score matrix and a fourth sub-score matrix are obtained; a matrix multiplication operation of the value vector matrix cache and the third sub-score matrix, based on which a second intermediate sub-result is obtained; a matrix multiplication operation of the first value vector matrix and the fourth sub-score matrix, based on which a third intermediate sub-result is obtained; an addition operation of the second intermediate sub-result and the third intermediate sub-result.

8. The method of claim 5, wherein, Removing at least part of the memory access operation in the initial model, including: Removing the following operation in the initial model: a splicing operation of the key vector matrix cache and the first key vector matrix in the first specified dimension; a splicing operation of the value vector matrix cache and the first value vector matrix in the first specified dimension; Converting a second operation operation in the initial model into a plurality of sub-operations, including: Converting the matrix multiplication operation of the second query vector matrix and the second key vector matrix into a plurality of sub-operations: a matrix multiplication operation of the second query vector matrix and the key vector matrix cache, based on which a first sub-score matrix is obtained; a matrix multiplication operation of the second query vector matrix and the first key vector matrix, based on which a second sub-score matrix is obtained; a splicing operation of the first sub-score matrix and the second sub-score matrix in the first specified dimension. The matrix multiplication operation of the second value vector matrix and the second attention score matrix is converted into a plurality of sub-operations: a cutting operation of the second attention score matrix in a third specified dimension, based on which a third sub-score matrix and a fourth sub-score matrix are obtained; a matrix multiplication operation of the value vector matrix buffer and the third sub-score matrix, based on which a second intermediate sub-result is obtained; a matrix multiplication operation of the first value vector matrix and the fourth sub-score matrix, based on which a third intermediate sub-result is obtained; and an addition operation of the second intermediate sub-result and the third intermediate sub-result.

9. The method of claim 2, wherein, The at least part of the memory access operations in the initial model is removed, including: The at least part of the transpose operations and the at least part of the splicing operations in the initial model are removed. The target operation is updated based on the updating manner of the target operation, including: The processing object of the third operation in the initial model and / or the input order of the processing object are adjusted, the third operation is converted into a plurality of sub-operations, and a transpose operation of an intermediate output is added in the initial model.

10. An attention mechanism reasoning method, comprising: obtaining a computer executable target instruction sequence corresponding to an optimized attention network, the target instruction sequence being generated based on the instruction generation method for the attention mechanism in any one of claims 1-9; transmitting the target instruction sequence to a hardware platform; processing an input parameter corresponding to a current sub-object in a received input object by executing the target instruction sequence using the hardware platform, and obtaining a reasoning result corresponding to the current sub-object, the input object including a plurality of sub-objects.

11. An instruction generation apparatus for an attention mechanism, comprising: a first obtaining module configured to obtain an initial model corresponding to a to-be-compiled attention network; a removing module configured to remove at least part of memory access operations in the initial model; an updating determining module configured to determine a target operation in the initial model that needs to be updated and an updating manner of the target operation based on the at least part of the memory access operations removed; an updating module configured to update the target operation based on the updating manner of the target operation, and obtain an optimized model; a compiling processing module configured to generate a computer executable target instruction sequence by compiling processing based on the optimized model; wherein a first intermediate result obtained by processing an input object based on the to-be-compiled attention network is the same as a second intermediate result obtained by processing the input object based on an optimized attention network corresponding to the optimized model.

12. An attention mechanism reasoning system, comprising: a second obtaining module configured to obtain a computer executable target instruction sequence corresponding to an optimized attention network, the target instruction sequence being generated based on the instruction generation method for the attention mechanism in any one of claims 1-9; a transmitting module configured to transmit the target instruction sequence to a hardware platform; and a processing module configured to process an input object based on the optimized attention network corresponding to the optimized model. The hardware platform is configured to execute the target instruction sequence, process an input parameter corresponding to a current sub-object in a received input object, and obtain an inference result corresponding to the current sub-object, wherein the input object includes a plurality of sub-objects.

13. A deep learning model comprising an encoder and a decoder. The encoder comprises an optimization attention network implemented based on an optimization model, the optimization model is based on the instruction generation method for attention mechanism in any one of claims 1-9, obtains an initial model corresponding to a to-be-compiled attention network, optimizes at least part of memory access operations in the initial model, and updates target operation operations in the initial model to obtain.

14. A computer-readable storage medium, the storage medium storing a computer program, the computer program being configured to execute the method in any one of claims 1-10.

15. An electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method in any one of claims 1-10.

Images