Method and apparatus for attention calculation in artificial intelligence model

Abstract

Embodiments of the present application relate to the technical field of artificial intelligence. Disclosed are a method and apparatus for attention calculation in an artificial intelligence model. The method comprises: acquiring a plurality of elements requiring attention calculation; generating a Q matrix, a K matrix, and a V matrix required for attention calculation and corresponding to the plurality of elements; determining a position of valid data in a relation matrix corresponding to the Q matrix and the K matrix; on the basis of the determined position of the valid data in the relation matrix, determining calculation data for matrix multiplication calculation in the Q matrix and the K matrix; performing matrix multiplication calculation on the determined calculation data in the Q matrix and the K matrix to obtain the relation matrix; and performing an operation on the relation matrix and a V matrix to implement attention calculation. By means of the present application, invalid data can be prevented from participating in attention calculation, thereby improving the attention calculation efficiency.

Description

Method and device for attention calculation in artificial intelligence models

[0001] This application claims priority to Chinese patent application No. 202410124904.9, filed on January 26, 2024, entitled “Method and device for attention calculation in artificial intelligence model”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for attention calculation in an artificial intelligence model. Background Art

[0003] The Transformer is a neural network architecture that forms the core structure of currently popular large-scale artificial intelligence (AI) models. The attention calculations involved in the Transformer can generate text features for the input text of the AI ​​model, enabling the AI ​​model to better "understand" the input text and process it more accurately.

[0004] In the attention calculation, we can first generate three matrices Q, K, and V for the input text, and then perform matrix operations on the three matrices Q, K, and V to obtain the text features of the input text.

[0005] However, in the current AI large model, a mask matrix is ​​still needed to participate in the matrix operations of the three matrices Q, K, and V to filter out invalid data in the three matrices Q, K, and V during the operation process, but this also reduces the efficiency of attention calculation. Summary of the Invention

[0006] The embodiments of the present application provide a method and apparatus for attention calculation in an artificial intelligence model, which can improve the efficiency of attention calculation. The corresponding technical solutions are as follows:

[0007] In a first aspect, a method for attention calculation in an artificial intelligence model is provided, the method comprising: obtaining multiple elements requiring attention calculation, wherein there is a sequential relationship between the multiple elements, and generating a Q matrix, a K matrix, and a V matrix required for attention calculation corresponding to the multiple elements. The position of valid data in the relationship matrix obtained after matrix multiplication of the Q matrix and the K matrix is ​​determined, wherein the valid data is used to identify the relationship between each element and the element at a specified position among the multiple elements. Based on the position of the determined valid data in the relationship matrix, the calculation data for matrix multiplication calculation in the Q matrix and the K matrix is ​​determined. Matrix multiplication calculation is performed on the determined calculation data of the Q matrix and the K matrix to obtain a relationship matrix. Operations are performed on the relationship matrix and the V matrix to implement attention calculation.

[0008] The multiple elements requiring attention calculation can be multiple characters in the input text, and the order of these elements represents the order of each character in the input text. The relationship matrix is ​​the similarity matrix used to calculate the corresponding Q and K matrices. Valid data are the elements in the relationship matrix that require attention calculation. The position of valid data in the relationship matrix can be configured by the technician.

[0009] In the solution shown in the present application, in the process of performing the attention calculation, first, when calculating the relationship matrix of the Q matrix and the K matrix, the calculation data for the Q matrix and the K matrix can be determined according to the position of the valid data in the relationship matrix, and then the valid data in the relationship matrix is ​​calculated based on the calculation data. In this way, in the process of calculating the relationship matrix, invalid data is avoided from participating in the calculation, the calculation efficiency of the relationship matrix is ​​improved, and the efficiency of the attention calculation is further improved. In addition, since the relationship matrix calculated in the present application mainly includes valid data, it is possible to further avoid invalid data from participating in subsequent attention calculations, which can further improve the efficiency of the attention calculation.

[0010] In one achievable method, determining the position of valid data in the relationship matrix obtained by performing a matrix multiplication operation on the Q matrix and the K matrix includes obtaining setting information for invalid data in the artificial intelligence model, where invalid data is the relationship between multiple elements that is not considered during attention calculation. Based on the setting information, determining the position of valid data in the relationship matrix obtained by performing a matrix multiplication operation on the Q matrix and the K matrix.

[0011] Among them, the setting information of invalid data in the artificial intelligence model may include the position of invalid data in the relationship matrix, so that the position in the relationship matrix other than the invalid data is the position corresponding to the valid data. In one example, the setting information can be a pre-set mask matrix with the same size as the relationship matrix, and each element value in the mask matrix is ​​used to indicate whether the element at the same position in the relationship matrix is ​​invalid data or valid data. In this way, by determining the position of the valid data in the relationship matrix, the position of the calculation data used to calculate the valid data in the Q matrix and the K matrix can be inferred through the matrix multiplication method. In this way, it is only necessary to perform matrix operations of the Q matrix and the K matrix based on the calculation data in the inferred Q matrix and the K matrix to calculate the valid data in the relationship matrix, thereby avoiding the operation of invalid data in the relationship matrix and improving the efficiency of the attention calculation.

[0012] In one feasible method, matrix multiplication is performed on the determined Q matrix and K matrix calculation data to obtain a relationship matrix, including: dividing the Q matrix into blocks to obtain a first matrix block, and dividing the K matrix into blocks to obtain a second matrix block; according to the first matrix block corresponding to the Q matrix and the second matrix block corresponding to the K matrix, matrix multiplication is performed on the determined Q matrix and K matrix calculation data to obtain a relationship matrix.

[0013] In the solution described in this application, when calculating the effective data in the relationship matrix corresponding to the Q matrix and the K matrix, the Q matrix and the K matrix can be divided into blocks to obtain a first matrix block corresponding to the Q matrix and a second matrix block corresponding to the K matrix. In this way, the effective data in the relationship matrix can be calculated by the first matrix block and the second matrix block, which can improve the efficiency of effective data calculation and further improve the efficiency of attention calculation.

[0014] In one feasible manner, the Q matrix is ​​divided into blocks to obtain first matrix blocks, and the K matrix is ​​divided into blocks to obtain second matrix blocks, including: dividing the Q matrix and the K matrix respectively according to specified sizes to obtain first matrix blocks corresponding to the Q matrix and second matrix blocks corresponding to the K matrix, wherein the specified size is determined by the size of the Q matrix or the K matrix.

[0015] In the solution shown in this application, each fifth matrix block in the relationship matrix calculated by the first matrix block and the second matrix block may include invalid data in some of the fifth matrix blocks. By determining the size of the blocks for the Q matrix and the K matrix based on the size of the Q matrix or the K matrix, the efficiency of calculating the third matrix blocks can be guaranteed, and the amount of invalid data included in some of the third matrix blocks can be reduced.

[0016] In one achievable method, before the Q matrix and the K matrix are divided into blocks according to the specified size, the method further includes: obtaining the Q matrix, the K matrix, and the V matrix generated by the multiple sample elements that need to perform attention calculation. Multiple candidate sizes are determined, and based on each candidate size, the Q matrix and the K matrix generated by the multiple sample elements are divided into blocks to obtain a third matrix block corresponding to the Q matrix generated by the multiple sample elements and a fourth matrix block corresponding to the K matrix generated by the multiple sample elements, where the candidate size is determined by the size of the Q matrix and the K matrix. Based on the third matrix block, the fourth matrix block, and the V matrix generated by the multiple sample elements corresponding to each candidate size, the attention calculation results corresponding to the multiple sample elements are determined in sequence, and the time consumption for obtaining the attention calculation result each time is determined. Based on the time consumption corresponding to each candidate size, a specified size is determined among the multiple candidate sizes.

[0017] In this way, different candidate sizes can be tested through sample elements to obtain the specified size with the highest computational efficiency. In practical applications, the Q matrix and the K matrix can be divided into blocks according to the determined specified size, thereby improving the efficiency of the computational relationship based on the first matrix block and the second matrix block, and thus improving the efficiency of the attention calculation.

[0018] In one achievable manner, the method further includes: determining the fifth matrix block located in the same row of the relationship matrix as the sixth matrix block of the relationship matrix, where the fifth matrix block in the relationship matrix is ​​calculated by the first matrix block in the same row of the Q matrix and the second matrix block in the same column of the K matrix. Obtaining a Mask matrix corresponding to each sixth matrix block, wherein each Mask matrix has the same size as the corresponding sixth matrix block. Based on the Mask matrix corresponding to each sixth matrix block, masking is performed on each sixth matrix block to obtain each masked sixth matrix block.

[0019] In the solution described in this application, each fifth matrix block in the relationship matrix calculated by the first matrix block and the second matrix block may contain invalid data. Therefore, in this application, the invalid data in the fifth matrix block can be masked on a row-by-row basis to prevent the invalid data from affecting the attention calculation results.

[0020] In one achievable method, performing operations on the relationship matrix and the V matrix to implement attention calculation includes: performing operations on the relationship matrix and the V matrix based on each masked sixth matrix block to obtain an attention calculation result corresponding to the attention calculation. Because the sixth matrix block primarily includes valid data in the relationship matrix, performing matrix multiplication operations on the similarity matrix and the V matrix based on the sixth matrix block can avoid a large amount of invalid data from participating in the matrix multiplication operation, thereby improving the efficiency of performing the matrix multiplication operation and, in turn, improving the efficiency of the attention calculation.

[0021] In a second aspect, a device for calculating attention in an artificial intelligence model is provided, the device comprising:

[0022] The acquisition module is used to obtain multiple elements that need to be calculated for attention. There is a sequential relationship between multiple elements;

[0023] The generation module is used to generate the Q matrix, K matrix, and V matrix required for attention calculation corresponding to multiple elements;

[0024] A determination module is used to determine the position of valid data in the relationship matrix obtained after the matrix multiplication operation of the Q matrix and the K matrix, wherein the valid data is used to identify the relationship between each element and the element at a specified position in the multiple elements;

[0025] A determination module, configured to determine the calculation data for performing matrix multiplication calculation in the Q matrix and the K matrix according to the position of the determined valid data in the relationship matrix;

[0026] The calculation module is used to perform matrix multiplication on the determined Q matrix and K matrix calculation data to obtain a relationship matrix, and perform operations on the relationship matrix and the V matrix to realize attention calculation.

[0027] In one feasible manner, the determination module is used to: obtain setting information of invalid data in the artificial intelligence model, where invalid data is the relationship between multiple elements that does not need to be considered when performing attention calculations; and determine, based on the setting information, the position of valid data in the relationship matrix obtained after matrix multiplication operations on the Q matrix and the K matrix.

[0028] In one implementable manner, the calculation module is used to: divide the Q matrix into blocks to obtain a first matrix block, and divide the K matrix into blocks to obtain a second matrix block; according to the first matrix block corresponding to the Q matrix and the second matrix block corresponding to the K matrix, perform matrix multiplication calculation on the determined Q matrix and K matrix calculation data to obtain a relationship matrix.

[0029] In one implementable manner, the calculation module is used to: divide the Q matrix and the K matrix into blocks according to specified sizes, respectively, to obtain a first matrix block corresponding to the Q matrix and a second matrix block corresponding to the K matrix, wherein the specified size is determined by the size of the Q matrix or the K matrix.

[0030] In one achievable manner, the device further includes a testing module, which is configured to: obtain a Q matrix, a K matrix, and a V matrix generated by multiple sample elements requiring attention calculation; determine multiple candidate sizes, and, based on each candidate size, block the Q matrix and the K matrix generated by the multiple sample elements to obtain a third matrix block corresponding to the Q matrix generated by the multiple sample elements and a fourth matrix block corresponding to the K matrix generated by the multiple sample elements, wherein the candidate size is determined by the size of the Q matrix and the K matrix; determine the attention calculation results corresponding to the multiple sample elements in sequence based on the third matrix block, the fourth matrix block, and the V matrix generated by the multiple sample elements corresponding to each candidate size, and determine the time consumed for obtaining the attention calculation result each time; and determine a specified size among the multiple candidate sizes based on the time consumed for each candidate size.

[0031] In one achievable manner, the device also includes a mask processing module, which is used to: determine the fifth matrix block located in the same row of the relationship matrix as the sixth matrix block of the relationship matrix, where the fifth matrix block in the relationship matrix is ​​calculated by the first matrix block in the same row of the Q matrix and the second matrix block in the same column of the K matrix; obtain the Mask matrix corresponding to each sixth matrix block, wherein each Mask matrix has the same size as the corresponding sixth matrix block; based on the Mask matrix corresponding to each sixth matrix block, perform mask processing on each sixth matrix block to obtain each sixth matrix block after mask processing.

[0032] In one implementable manner, the calculation module is used to: perform relation matrix and V matrix operations based on each sixth matrix block after mask processing to obtain an attention calculation result corresponding to the attention calculation.

[0033] In a third aspect, a computing device is provided, comprising a processor and a memory, wherein the processor is configured to execute instructions stored in the memory so that the computing device performs the method described in the first aspect and / or any implementable manner of the first aspect.

[0034] In a fourth aspect, a computer program product comprising instructions is provided, which, when executed by a computing device, causes the computing device to execute the method as described in the first aspect and / or any implementable manner of the first aspect.

[0035] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method described in the first aspect and / or any achievable method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] FIG1 is a schematic diagram of the structure of a Transformer provided in an embodiment of the present application;

[0037] Figure 2 is a flowchart of the attention calculation involved in Transformer;

[0038] FIG3 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0039] FIG4 is a flow chart of a method for attention calculation in an artificial intelligence model provided in an embodiment of the present application;

[0040] FIG5 is a schematic diagram of valid data in a relationship matrix provided in an embodiment of the present application;

[0041] FIG6 is a schematic diagram of valid data in another relationship matrix provided in an embodiment of the present application;

[0042] FIG7 is a schematic diagram of valid data in another relationship matrix provided in an embodiment of the present application;

[0043] FIG8 is a schematic diagram of a matrix multiplication operation provided in an embodiment of the present application;

[0044] FIG9 is a schematic diagram of invalid data provided by an embodiment of the present application;

[0045] FIG10 is a schematic diagram of a sixth matrix block and mask matrix provided in an embodiment of the present application;

[0046] FIG11 is a flow chart of a method for determining a specified size provided in an embodiment of the present application;

[0047] Figure 12 is a structural diagram of a device for attention calculation in an artificial intelligence model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0049] Attention calculation is a widely used method in artificial intelligence technology. It can be used to calculate the correlation between multiple elements, helping AI models "understand" multiple input elements. Generally, the multiple elements requiring attention calculation can be individual words in a sentence or individual pixels in an image.

[0050] The Transformer is a neural network architecture that forms the core structure of currently popular large AI models and involves attention calculations. The following uses the Transformer as an example to explain the attention calculation process. Figure 1 shows the Transformer architecture. In this figure, the Input Embedding represents the input vector generated from the input data of the large AI model. LayerNorm stands for Layer Normalization, which is used to normalize the input vector. Q, K, and V are the Q, K, and V matrices, respectively, generated based on the normalized input vector. Q represents the query, K represents the key, and V represents the value. Mask represents the mask matrix, with "+" representing matrix addition and "×" representing matrix multiplication. The Softmax layer normalizes the matrix.

[0051] Figure 2 is a flowchart of the attention calculation involved in the Transformer. Referring to Figure 2, the attention calculation process includes:

[0052] S1. After obtaining the Q matrix, K matrix, and V matrix, the transposed matrix of the Q matrix and the K matrix (KT matrix) can be multiplied to obtain the similarity matrix (Sim matrix) of the Q matrix and the K matrix, which can also be called a relationship matrix.

[0053] S2. After obtaining the similarity matrix Sim, the Sim matrix can be masked by the mask matrix, that is, the mask matrix and the Sim matrix are added to obtain the masked matrix (Sim-mask matrix).

[0054] In one example, the values ​​in the lower left triangle of the mask matrix are all 0, and the values ​​in the upper right triangle are all negative infinity. Therefore, after performing matrix addition on the mask matrix and the Sim matrix, the values ​​in the lower left triangle of the sim_masked matrix match the values ​​in the lower left triangle of Sim, while the values ​​in the upper right triangle become negative infinity. During the calculation, negative infinity can be set to a large negative number, such as -10000.

[0055] S3. After obtaining the Sim-mask matrix, the softmax calculation formula is used to calculate the probability matrix.

[0056] In step S3, the maximum value of each row of the Sim-mask matrix can be determined on a row-by-row basis, and then the maximum value of that row is subtracted from the value of each row element. After the subtraction, the element values ​​of each element in the lower left triangular area of ​​the Sim_masked matrix will be less than or equal to 0, while the upper right triangular area will still be negative infinity. Then, an exponential calculation is performed on each element. Due to the characteristics of exponential calculation, the element values ​​of the lower left triangular area of ​​the Sim_masked matrix will be between 0 and 1, and the values ​​of the upper right triangular area will be all 0. Then, the sum of the elements in each row of the Sim_masked matrix can be calculated, and the probability matrix Probs can be obtained by dividing each row element by the sum of the corresponding row.

[0057] Since the values ​​of the upper right triangular area of ​​Sim_masked are all 0 after the exponential operation, this part is meaningless when calculating the sum of each row, and the value after dividing this part by the sum of the corresponding row is still 0, that is, the values ​​of the upper right triangular area of ​​the probability matrix Probs are all 0.

[0058] S4. After obtaining the probability matrix Probs, a matrix multiplication operation can be performed on the probability matrix Probs and the V matrix to obtain the output matrix Out.

[0059] Since the values ​​in the upper right triangle of the probability matrix Probs are all 0, these elements have no effect on the result of the matrix multiplication operation. Therefore, it is meaningless to perform matrix multiplication on these elements.

[0060] From the above steps S1 to S4, it can be seen that there are a lot of meaningless calculations in the current attention calculation, so the efficiency of the current attention calculation needs to be further improved.

[0061] The present embodiment provides a method for attention calculation in an artificial intelligence model. This method is applicable to attention calculations in various scenarios, such as language processing and image processing. It can reduce the computation of invalid data in attention calculations and improve the efficiency of attention calculations. Figure 3 is a schematic diagram of the structure of a computing device for the method for attention calculation in an artificial intelligence model provided by the present embodiment. As shown in Figure 3, computing device 300 may include a bus 302, a processor 304, and a memory 306. Optionally, computing device 300 may also include a communication interface 308. The processor 304, memory 306, and communication interface 308 communicate with each other via bus 302. Computing device 300 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 300. The computing device 300 may be a device for running a model, a terminal, or a server. When the computing device 300 is a terminal, it includes, but is not limited to, a desktop computer, a mobile phone, a laptop, a tablet computer, etc. When the computing device 300 is a server, it can be a standalone server, i.e., a server that can independently perform AI model training, or it can be any device in a computing cluster that performs AI model training. If the computing device 300 is any device in the cluster, it is used to perform the attention calculation on a portion of the data in the AI ​​model. The computing device can run multiple virtual machines or containers, and each virtual machine or container can also independently perform the attention calculation.

[0062] Bus 302 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses can be categorized as address buses, value buses, control buses, and so on. For ease of illustration, FIG3 shows a single line, but this does not imply a single bus or a single type of bus. Bus 302 may include a path for transmitting information between various components of computing device 300 (e.g., memory 306, processor 304, and communication interface 308).

[0063] The processor 304 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP). Alternatively, the processor may be a system-on-chip (SOC) including one or more of the above-mentioned CPUs, GPUs, MPs, etc. The processor 304 may further include a matrix operation unit, which may be used to implement the matrix operations involved in the method for attention calculation in the artificial intelligence model provided in the embodiment of the present application, such as matrix multiplication or matrix addition.

[0064] The memory 306 may include volatile memory, such as random access memory (RAM). The memory 306 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0065] The memory 306 stores executable program code, and the processor 304 executes the executable program code to implement the method for attention calculation in the artificial intelligence model provided in the embodiment of the present application. For example, multiple elements requiring attention calculation are obtained, where there is a sequential relationship between the multiple elements, and the Q matrix, K matrix, and V matrix required for attention calculation corresponding to the multiple elements are generated. The position of valid data in the relationship matrix obtained after matrix multiplication of the Q matrix and the K matrix is ​​determined in the relationship matrix. The valid data is used to identify the relationship between each element and the element at a specified position in the multiple elements. Based on the position of the determined valid data in the relationship matrix, the calculation data for matrix multiplication calculation in the Q matrix and the K matrix is ​​determined. Matrix multiplication calculation is performed on the determined Q matrix and K matrix calculation data to obtain a relationship matrix. Operations are performed on the relationship matrix and the V matrix to implement attention calculation, etc.

[0066] The communication interface 308 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 300 and other devices or a communication network.

[0067] FIG4 is a flow chart of a method for calculating attention in an artificial intelligence model provided by an embodiment of the present application. The method can be executed by the computing device 300 described above, and further can be executed by the processor 304 included in the computing device 300. Referring to FIG4 , the method includes:

[0068] Step 401: The processor obtains multiple elements that require attention calculation.

[0069] There is a sequential relationship between multiple elements, and the multiple elements can be any elements suitable for attention calculation.

[0070] In one example, the attention calculation method provided in an embodiment of the present application is applied to an AI language macromodel. The multiple elements requiring attention calculation are the input text of the AI ​​language macromodel, such as conversational sentences input by a user into the AI ​​macromodel. Alternatively, the input text may refer to conversational text generated by the macromodel. Each character in the input text is an element, and the order of each character in the input text represents the ordering relationship between the multiple elements.

[0071] In another example, the attention calculation method provided in an embodiment of the present application is applied to an AI image model. The multiple elements requiring attention calculation are the input image of the AI ​​image model, such as an image to be semantically segmented. Each pixel in the input image is an element, and the positional relationship between the multiple pixels in the input image represents the order relationship between the multiple elements.

[0072] Step 402: Generate the Q matrix, K matrix, and V matrix required for attention calculation corresponding to multiple elements.

[0073] As shown in Figure 2, in the attention calculation, the Q matrix, K matrix, and V matrix can be generated for the input text. In one example, after the input text is processed by LayerNorm to obtain a normalized input vector, the normalized input vector can be compared with the weight matrix W. Q 、W K 、W V Perform matrix multiplication operations to obtain the Q matrix, K matrix, and V matrix.

[0074] Step 403: Determine the position of valid data in the relationship matrix obtained after matrix multiplication of the Q matrix and the K matrix. The valid data is used to identify the relationship between each element and the element at a specified position in the multiple elements.

[0075] Valid data refers to the data in the relationship matrix that needs to participate in the attention calculation, that is, the relationship between different elements that need to participate in the attention calculation. The designated position is the position of other elements with which each element's relationship is calculated. This designated position can be set by technical personnel based on the application scenario of the AI ​​model. Once the designated position is determined, the corresponding valid data's position in the relationship matrix is ​​also determined. Figures 5 to 7 respectively illustrate three designated positions and the corresponding valid data's position in the relationship matrix.

[0076] Figure 5 is a schematic diagram of valid data in a relationship matrix provided by an embodiment of the present application. The elements in the lower left triangle area of ​​the relationship matrix shown in Figure 5 are valid data in the relationship matrix. In one example, the attention calculation does not need to "pay attention" to the complete context, and in the generative model, it has the characteristic of "only looking forward, not looking backward" for the input text. Therefore, in the attention calculation, it is only necessary to pay attention to the relevance of each word in the input text with itself and the subsequent words. In other words, the specified position corresponding to each of the above elements is the various positions before the position of the element in the multiple elements and the position of the element itself. In the attention calculation, only the data in the lower left triangle area of ​​the relationship matrix need to participate in the calculation to obtain text features.

[0077] Figure 6 is a schematic diagram of valid data in another relationship matrix provided by an embodiment of the present application. In the relationship matrix shown in Figure 6, the elements located in the diagonal area are valid data in the relationship matrix. In one example, when the length of the input text is too long, for example, the input text exceeds the set length threshold, or the input text is an article. In this case, in the attention calculation, it is only necessary to pay attention to the correlation between each word and the words near it. For example, the designated position corresponding to each of the above elements is the n positions before the position of the element in the multiple elements and the position of the element itself. Wherein, n can be a preset value. When the number of positions before the position of the element is less than n, the designated position is each position before the element and the position of the element itself. In the attention calculation, it is only necessary to pay attention to the correlation between the first n words before each word.

[0078] Figure 7 is a schematic diagram of valid data in another relationship matrix provided by an embodiment of the present application. The elements in the lower left triangular area and the upper left matrix area of ​​the relationship matrix shown in Figure 7 represent valid data in the relationship matrix. In one example, the input text includes a user-entered conversational sentence (e.g., a question) and conversational text generated by a large AI model. In this case, during attention calculation, for the user-entered conversational sentence, attention can be paid to the correlation between each word in the conversational sentence. For the conversational text generated by the large AI model, attention can be paid to "looking forward only, not backward," meaning attention is only needed to focus on the correlation between each word in the conversational text and itself and the words that follow it. In other words, the designated position corresponding to each element can include all positions before the position of the element in the multiple elements, or n positions before the position of the element in the multiple elements, as well as the position of the element itself. During attention calculation, only the data in the lower left triangular area and the upper left matrix area of ​​the relationship matrix are involved in the calculation to obtain text features.

[0079] In one example, invalid data configuration information can be obtained from the AI ​​model. Based on this configuration information, the position of valid data in the relationship matrix obtained by performing matrix multiplication on the Q and K matrices is determined. Invalid data refers to data outside the valid data in the relationship matrix, specifically, relationships between multiple elements that are not considered during attention calculations.

[0080] The setting information of invalid data in the artificial intelligence model may include the position of the invalid data in the relationship matrix, so that the position in the relationship matrix other than the invalid data is the position corresponding to the valid data. The setting information can be set according to the above-mentioned specified position. The setting information can be a pre-set mask matrix with the same size as the relationship matrix, and each element value in the mask matrix is ​​used to indicate whether the element at the same position in the relationship matrix is ​​invalid data or valid data. For example, each element in the mask matrix consists of "0" and "-10000", where the element "0" is used to indicate that the element at the same position in the relationship matrix is ​​valid data, and the element "-10000" is used to indicate that the element at the same position in the relationship matrix is ​​invalid data. Therefore, the position of valid data and invalid data in the relationship matrix can be determined through the mask matrix.

[0081] Step 404: Determine the calculation data for matrix multiplication calculation in the Q matrix and the K matrix according to the position of the determined valid data in the relationship matrix.

[0082] After determining the position of the valid data in the relational matrix, the Q matrix and K matrix can be inferred by the matrix multiplication method. TThe position of the calculation data used to calculate the effective data in the matrix (the device matrix of the K matrix). For example, if the effective data is in the i-th row and j-th column of the relational matrix, the calculation data for calculating the effective data is the element in the i-th row of the Q matrix and the K T The element in the j-th column of the matrix.

[0083] In this way, it is only necessary to perform matrix operations on the Q matrix and the K matrix based on the calculated data in the inferred Q matrix and the K matrix to calculate the valid data in the relationship matrix, thereby avoiding the operation of invalid data in the relationship matrix and improving the efficiency of attention calculation.

[0084] Step 405: Perform matrix multiplication on the determined Q matrix and K matrix calculation data to obtain a relationship matrix.

[0085] In implementation, a vector operation unit included in the processor may be used to perform a matrix multiplication operation on the Q matrix and the K matrix calculation data to obtain each valid data included in the relational matrix.

[0086] In an achievable manner, in order to improve the efficiency of calculating the relationship matrix, after obtaining the Q matrix and K T After the matrix, we can first calculate the Q matrix and K T The matrix is ​​Tiling processed to obtain multiple Tiling blocks (which can be called the first matrix block) and K matrix. T The matrix includes multiple tiling blocks (which may be referred to as second matrix blocks). Then, based on the first matrix blocks and the second matrix blocks, valid data in the relationship matrix corresponding to the Q matrix and the K matrix may be calculated.

[0087] Accordingly, the processing of step 404 can be replaced by determining the Q matrix and K matrix according to the position of the determined valid data in the relationship matrix. T A first matrix block and a second matrix block in a matrix for performing matrix multiplication calculation.

[0088] FIG8 is a schematic diagram of a matrix multiplication operation provided by an embodiment of the present application. In FIG8, the elements in the lower left triangle area of ​​the relationship matrix are valid data. Therefore, the first matrix block, K T Each column of the second matrix block in the matrix calculates the third matrix block containing valid data in the relation matrix. For example, according to the first matrix block of the i-th row in the Q matrix, K T The second matrix block at the jth column of the matrix is ​​calculated, and the third matrix block at the ith row and jth column of the relation matrix is ​​calculated, where i is greater than or equal to j.

[0089] In implementation, the first matrix block of each row in the Q matrix and the transposed matrix of the K matrix (K T The second matrix blocks of each column in the matrix are calculated to obtain the fifth matrix blocks included in the relationship matrix in sequence.

[0090] In the embodiment of the present application, when performing the Q matrix and K T In the process of obtaining the corresponding relation matrix through matrix multiplication, there is no need to calculate all the elements in the relation matrix. Therefore, the amount of matrix multiplication in the attention calculation is reduced, which can improve the computational efficiency of the attention calculation.

[0091] Step 406: Perform operations on the relationship matrix and the V matrix to implement attention calculation.

[0092] Since the relationship matrix lacks invalid data, the matrix multiplication operation between the relationship matrix and the V matrix can be performed only through the invalid data of the relationship matrix. In this way, the calculation of invalid data can be omitted during the calculation process, thereby improving the efficiency of attention calculation.

[0093] The relationship matrix is ​​composed of the first matrix block in the Q matrix, K T When the second matrix block of the matrix is ​​calculated, there is a fifth matrix block in the relationship matrix that includes some invalid data. For example, each third matrix block located on the diagonal of the relationship matrix in Figure 8 includes invalid data. As shown in Figure 9, when the size of the third matrix block is 3×3, the three elements in the upper right corner of the third matrix block are invalid data. Therefore, in order to avoid the influence of this part of invalid data on the attention calculation, this embodiment of the application also provides a mask processing method, including:

[0094] The fifth matrix blocks located in the same row of the relationship matrix are determined as sixth matrix blocks. Based on the Mask matrix corresponding to each sixth matrix block, each sixth matrix block is masked to obtain each masked sixth matrix block.

[0095] In the implementation, the first matrix block in the Q matrix and K T After the second matrix block in the matrix is ​​calculated to obtain the fifth matrix block in the relationship matrix, each fifth matrix block corresponding to the same tiling row in the relationship matrix can be determined as the sixth matrix block in the relationship matrix. After each sixth matrix block is determined, a mask matrix corresponding to each sixth matrix block can be obtained according to the position of the sixth matrix block in the relationship matrix.

[0096] FIG10 is a schematic diagram of a sixth matrix block and a mask matrix provided in an embodiment of the present application. In one example, the size of each mask matrix is ​​the same as the size of the corresponding sixth matrix block, and each mask matrix includes elements with a value of 0 and elements with a value of negative infinity. The position of the element with a value of 0 in the mask matrix is ​​the same as the position of the valid data in the corresponding sixth matrix block, and the position of the element with a value of negative infinity in the mask matrix is ​​the same as the position of the invalid data in the corresponding sixth matrix block.

[0097] After obtaining the mask matrix corresponding to each sixth matrix block, a matrix addition operation can be performed on each sixth matrix block and the corresponding mask matrix, thereby masking each fourth matrix block to obtain a masked sixth matrix block. After obtaining the masked sixth matrix block, a relation matrix and V matrix operation can be performed based on each masked sixth matrix block and the V matrix to obtain the attention calculation result corresponding to the attention calculation.

[0098] In implementation, after obtaining the masked sixth matrix blocks, the elements in each row of each sixth matrix block can be normalized. Specifically, the maximum value of each row in the sixth matrix block is determined, and then the maximum value of each row is subtracted from the value of each row element. Then, an exponential calculation is performed on each element, and the elements in each row are summed. Each row element is then divided by the sum of the corresponding row to obtain valid data in the probability matrix.

[0099] Among them, after the number of elements that need to be calculated for attention is determined, the sizes of Q, K, and V can also be determined. T The specified size of the matrix for tiling can also be determined, that is, the size of the first matrix block, the second matrix block, and the fifth matrix block can be determined. In this way, the size of each sixth matrix block and the position of invalid data in each sixth matrix block can also be determined, and then the mask matrix corresponding to each sixth matrix block can also be determined. Therefore, according to the position of each sixth matrix block in the relationship matrix, the corresponding mask matrix is ​​pre-stored. Then, after the sixth matrix block is calculated, the sixth matrix block can be masked according to the pre-stored mask matrix.

[0100] After obtaining valid data in the probability matrix, the matrix multiplication operation of the probability matrix and the V matrix can be implemented based on the valid data included in the probability matrix to obtain the matrix multiplication operation result. The matrix multiplication operation result can then be normalized to obtain the attention calculation result. For example, the attention calculation result can be the text feature corresponding to the input text.

[0101] In the embodiment of the present application, since the sixth matrix block in the relationship matrix primarily includes valid data, when calculating the corresponding probability matrix based on the sixth matrix block, a large amount of invalid data can be avoided from participating in the calculation, thereby improving the efficiency of generating the probability matrix. Furthermore, since the probability matrix is ​​primarily calculated using the valid data in the relationship matrix, the calculated probability matrix only includes valid data. Therefore, when performing the matrix multiplication operation between the probability matrix and the V matrix using the valid data in the probability matrix, the participation of invalid data can be avoided, thereby improving the efficiency of executing the matrix multiplication operation.

[0102] As can be seen, in the embodiments of the present application, the efficiency of performing matrix multiplication operations on the Q and K matrices, the efficiency of mask processing corresponding to the relationship matrix, the efficiency of calculating the probability matrix, and the efficiency of performing matrix multiplication operations on the probability matrix and the V matrix in the attention calculation can be improved. Therefore, the embodiments of the present application can efficiently generate text features corresponding to the input text.

[0103] In one achievable embodiment, the specified sizes for tiling the Q, K, and V matrices can be determined by the size of the Q or K matrix, that is, by the length of the input text. In one example, a correspondence between the Q or K matrix and the specified sizes can be pre-stored. In implementation, the specified sizes for tiling the Q and K matrices can be determined based on this correspondence.

[0104] FIG11 is a flow chart of a method for determining the specified size for tiling the Q matrix and the K matrix provided by an embodiment of the present application. In one example, the method can be applied in the training phase of a large AI model. Referring to FIG11 , the method includes:

[0105] Step 1101: Obtain a Q matrix, a K matrix, and a V matrix generated by multiple sample elements that require attention calculation.

[0106] Among them, multiple sample elements can be sample texts for training the AI ​​large model. For generating the Q matrix, K matrix and V matrix corresponding to the sample text, you can refer to the processing of step 402 above, which will not be repeated here.

[0107] Step 1102: Determine multiple candidate sizes, and based on each candidate size, divide the Q matrix and the K matrix generated by the multiple sample elements into blocks to obtain a third matrix block corresponding to the Q matrix generated by the multiple sample elements and a fourth matrix block corresponding to the K matrix generated by the multiple sample elements. The candidate size is determined by the sizes of the Q matrix and the K matrix.

[0108] The candidate sizes can be pre-set by technicians according to the sizes of the Q matrix and the K matrix, for example, 2×2, 4×4, 8×8, etc. After determining the candidate sizes, the transposed matrices (K T Matrix) are processed in blocks to obtain the third matrix blocks, K T In one example, when the specified size is m×n, where m is not equal to n, the Q matrix can be divided into blocks of m×n size and the K matrix can be divided into blocks of n×m size. T The matrix is ​​divided into blocks.

[0109] Step 1103: Based on the V matrix generated by the third matrix block, the fourth matrix block, and the multiple sample elements corresponding to each candidate size, determine the attention calculation results corresponding to the multiple sample elements in sequence, and determine the time consumption for obtaining the attention calculation result each time.

[0110] Step 1104: Determine a designated size from multiple candidate sizes based on the time consumption corresponding to each candidate size.

[0111] In practice, during the training of the AI ​​model, the Q matrix and the K matrix can be divided into blocks according to each candidate size to obtain third and fourth matrix blocks, thereby generating text features corresponding to the test text. The processing of performing attention calculations based on the third and fourth matrix blocks can refer to the processing of performing attention calculations based on the first and second matrix blocks in steps 404-405 above, and will not be repeated here.

[0112] For each candidate size of the third matrix block and the fourth matrix block, the time consumed in generating text features from the third matrix block and the fourth matrix block of each candidate size can be recorded, and then the candidate size with the lowest time consumption can be determined as the designated size in the above embodiment. In this way, it is possible to avoid the situation where the designated size is too small, resulting in an excessive number of matrix blocks, which in turn reduces the efficiency of attention calculation, and it is also possible to avoid the situation where the designated size is too large, resulting in a large amount of invalid data in the matrix blocks, which in turn reduces the efficiency of attention calculation.

[0113] Based on the same inventive concept, an embodiment of the present application further provides an apparatus for calculating attention in an artificial intelligence model. This apparatus can be a computing device that performs the aforementioned method for calculating attention in an artificial intelligence model, or a program running on the computing device for performing the method for calculating attention in an artificial intelligence model. Figure 12 is a schematic diagram of the structure of an apparatus for calculating attention in an artificial intelligence model provided by an embodiment of the present application. Referring to Figure 12, the apparatus includes:

[0114] The acquisition module 1210 is used to obtain multiple elements that need to be calculated for attention. There is a sequential relationship between the multiple elements, which is specifically used to implement the acquisition function of the above step 401 and its hidden steps.

[0115] The generation module 1220 is used to generate the Q matrix, K matrix and V matrix required for attention calculation corresponding to multiple elements, and is specifically used to implement the generation function of the above step 402 and its hidden steps.

[0116] Determination module 1230 is used to determine the position of valid data in the relationship matrix obtained after matrix multiplication operation of Q matrix and K matrix in the relationship matrix, where the valid data is used to identify the relationship between each element and the element at a specified position among multiple elements; based on the position of the determined valid data in the relationship matrix, the calculation data for matrix multiplication calculation in Q matrix and K matrix is ​​determined, which is specifically used to implement the determination function of the above steps 403-404 and the hidden steps thereof.

[0117] The calculation module 1240 is used to perform matrix multiplication on the determined Q matrix and K matrix calculation data to obtain a relationship matrix, and to perform operations on the relationship matrix and the V matrix to implement attention calculation, specifically for implementing the calculation function of the above-mentioned step 405 and its hidden steps.

[0118] In one implementable manner, the determination module 1230 is used to: obtain setting information of invalid data in the artificial intelligence model, where invalid data is the relationship between multiple elements that does not need to be considered when performing attention calculations; and determine, based on the setting information, the position of valid data in the relationship matrix obtained after matrix multiplication operations of the Q matrix and the K matrix.

[0119] In one implementable manner, the calculation module 1240 is used to: divide the Q matrix into blocks to obtain a first matrix block, and divide the K matrix into blocks to obtain a second matrix block; according to the first matrix block corresponding to the Q matrix and the second matrix block corresponding to the K matrix, perform matrix multiplication calculation on the determined Q matrix and K matrix calculation data to obtain a relationship matrix.

[0120] In one achievable manner, the calculation module 1240 is used to: divide the Q matrix and the K matrix into blocks according to a specified size, respectively, to obtain a first matrix block corresponding to the Q matrix and a second matrix block corresponding to the K matrix, wherein the specified size is determined by the size of the Q matrix or the K matrix.

[0121] In one achievable manner, the device further includes a testing module, which is configured to: obtain a Q matrix, a K matrix, and a V matrix generated by multiple sample elements requiring attention calculation; determine multiple candidate sizes, and, based on each candidate size, block the Q matrix and the K matrix generated by the multiple sample elements to obtain a third matrix block corresponding to the Q matrix generated by the multiple sample elements and a fourth matrix block corresponding to the K matrix generated by the multiple sample elements, wherein the candidate size is determined by the size of the Q matrix and the K matrix; determine the attention calculation results corresponding to the multiple sample elements in sequence based on the third matrix block, the fourth matrix block, and the V matrix generated by the multiple sample elements corresponding to each candidate size, and determine the time consumed for obtaining the attention calculation result each time; and determine a specified size among the multiple candidate sizes based on the time consumed for each candidate size.

[0122] In one achievable manner, the device also includes a mask processing module, which is used to: determine the fifth matrix block located in the same row of the relationship matrix as the sixth matrix block of the relationship matrix, where the fifth matrix block in the relationship matrix is ​​calculated by the first matrix block in the same row of the Q matrix and the second matrix block in the same column of the K matrix; obtain the Mask matrix corresponding to each sixth matrix block, wherein each Mask matrix has the same size as the corresponding sixth matrix block; based on the Mask matrix corresponding to each sixth matrix block, perform mask processing on each sixth matrix block to obtain each sixth matrix block after mask processing.

[0123] In one implementable manner, the calculation module 1240 is configured to: perform a relation matrix and a V matrix operation based on each sixth matrix block after mask processing to obtain an attention calculation result corresponding to the attention calculation.

[0124] The division of modules in the embodiments of the present application is schematic and is only a logical function division. In actual implementation, there may be other division methods. In addition, the functional modules in the various embodiments of the present application can be integrated into one processor, or can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. In addition, the device for attention calculation in the artificial intelligence model provided in the above embodiment and the method embodiment for attention calculation in the artificial intelligence model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0125] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for enabling a terminal device (which can be a personal computer, mobile phone, or network device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program code.

[0126] The present application also provides a computer program product containing instructions. The computer program product may be software or a program product containing instructions that can be run on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the method for attention calculation in the artificial intelligence model provided in the present application.

[0127] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the method for attention calculation in the artificial intelligence model provided in the present application.

[0128] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items having substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is there a limit on quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various examples, the first matrix block can be referred to as the second matrix block, and similarly, the second matrix block can be referred to as the first matrix block. The first matrix block and the second matrix block can both be collectively referred to as matrix blocks, and in some cases, can be separate and different matrix blocks.

[0129] The term "at least one" in this application means one or more, and the term "plurality" in this application means two or more.

[0130] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for calculating attention in an artificial intelligence model, characterized in that The method includes: Obtaining a plurality of elements for which attention calculation is to be performed, where there is an order relationship among the plurality of elements; Generating a Q matrix, a K matrix, and a V matrix required for the attention calculation corresponding to the plurality of elements; Determining the positions of valid data in the relationship matrix obtained after performing matrix multiplication on the Q matrix and the K matrix, where the valid data is used to identify the relationship between each element and the element at a specified position among the plurality of elements; Determining the calculation data for matrix multiplication in the Q matrix and the K matrix according to the determined positions of the valid data in the relationship matrix; Performing matrix multiplication on the determined calculation data of the Q matrix and the K matrix to obtain the relationship matrix; Performing an operation on the relationship matrix and the V matrix to implement the attention calculation.

2. The method according to claim 1, wherein The determining the positions of valid data in the relationship matrix obtained after performing matrix multiplication on the Q matrix and the K matrix includes: Obtaining the setting information of invalid data in the artificial intelligence model, where the invalid data is the relationship between the plurality of elements that does not need to be considered when performing the attention calculation; Determining the positions of valid data in the relationship matrix obtained after performing matrix multiplication on the Q matrix and the K matrix according to the setting information, where the element at the specified position is determined by the setting information.

3. The method according to claim 1 or 2, characterized in that, The performing matrix multiplication on the determined calculation data of the Q matrix and the K matrix to obtain the relationship matrix includes: Partitioning the Q matrix to obtain a first matrix partition, and partitioning the K matrix to obtain a second matrix partition; Performing matrix multiplication on the determined calculation data of the Q matrix and the K matrix according to the first matrix partition corresponding to the Q matrix and the second matrix partition corresponding to the K matrix to obtain the relationship matrix.

4. The method according to claim 3, wherein The partitioning the Q matrix to obtain a first matrix partition and partitioning the K matrix to obtain a second matrix partition includes: Partitioning the Q matrix and the K matrix respectively according to a specified size to obtain the first matrix partition corresponding to the Q matrix and the second matrix partition corresponding to the K matrix, where the specified size is determined by the size of the Q matrix or the K matrix.

5. The method according to claim 4, wherein Before the partitioning the Q matrix and the K matrix respectively according to the specified size, it further includes: Obtaining a Q matrix, a K matrix, and a V matrix generated from a plurality of sample elements for which attention calculation is to be performed; Determining a plurality of candidate sizes, and respectively partitioning the Q matrix and the K matrix generated from the plurality of sample elements based on each candidate size to obtain a third matrix partition corresponding to the Q matrix generated from the plurality of sample elements and a fourth matrix partition corresponding to the K matrix generated from the plurality of sample elements, where the candidate sizes are determined by the sizes of the Q matrix and the K matrix; Generate a V matrix based on the third matrix block corresponding to each candidate size, the fourth matrix block, and the multiple sample elements, sequentially determine the attention calculation results corresponding to the multiple sample elements, and respectively determine the time taken for obtaining each attention calculation result. Based on the time taken corresponding to each candidate size, determine a specified size from the multiple candidate sizes.

6. The method according to any one of claims 3 to 5, characterized in that The method further includes: Determine the fifth matrix block in the same row of the relationship matrix as the sixth matrix block of the relationship matrix, where the fifth matrix block in the relationship matrix is calculated from the first matrix block in the same row of the Q matrix and the second matrix block in the same column of the K matrix. Obtain a Mask matrix corresponding to each sixth matrix block, where each Mask matrix has the same size as the corresponding sixth matrix block. Based on the Mask matrix corresponding to each sixth matrix block, perform masking processing on each sixth matrix block to obtain each masked sixth matrix block.

7. The method according to claim 6, wherein The performing an operation on the relationship matrix and the V matrix to implement the attention calculation includes: Based on each masked sixth matrix block, perform the operation on the relationship matrix and the V matrix to obtain the attention calculation result corresponding to the attention calculation.

8. An apparatus for calculating attention in an artificial intelligence model, characterized in that, The apparatus includes: An acquisition module, configured to acquire multiple elements that need to perform attention calculation, where there is an order relationship among the multiple elements. A generation module, configured to generate a Q matrix, a K matrix, and a V matrix required for the attention calculation corresponding to the multiple elements. A determination module, configured to determine the position of valid data in the relationship matrix obtained after performing matrix multiplication on the Q matrix and the K matrix, where the valid data is used to identify the relationship between each element and the element at a specified position among the multiple elements; according to the determined position of the valid data in the relationship matrix, determine the calculation data for performing matrix multiplication in the Q matrix and the K matrix. A calculation module, configured to perform matrix multiplication on the determined calculation data of the Q matrix and the K matrix to obtain the relationship matrix, and perform an operation on the relationship matrix and the V matrix to implement the attention calculation.

9. The device according to claim 8, characterized in that The determination module is configured to: Obtain the setting information of invalid data in the artificial intelligence model, where the invalid data is the relationship between the multiple elements that does not need to be considered when performing the attention calculation. According to the setting information, determine the position of the valid data in the relationship matrix obtained after performing matrix multiplication on the Q matrix and the K matrix, where the element at the specified position is determined by the setting information.

10. The device according to claim 8 or 9, characterized in that, The calculation module is configured to: Perform block division on the Q matrix to obtain a first matrix block, and perform block division on the K matrix to obtain a second matrix block. Based on the first matrix block corresponding to the Q matrix and the second matrix block corresponding to the K matrix, perform matrix multiplication on the calculated data of the determined Q matrix and K matrix to obtain the relationship matrix.

11. The device according to claim 10, wherein, The calculation module is used for: Block the Q matrix and the K matrix respectively according to the specified size to obtain the first matrix block corresponding to the Q matrix and the second matrix block corresponding to the K matrix, where the specified size is determined by the size of the Q matrix or the K matrix.

12. The device according to claim 11, wherein The device further includes a test module for: Obtain the Q matrix, K matrix, and V matrix generated by a plurality of sample elements that need to perform attention calculation; Determine a plurality of candidate sizes, and respectively based on each candidate size, block the Q matrix and K matrix generated by the plurality of sample elements to obtain the third matrix block corresponding to the Q matrix generated by the plurality of sample elements and the fourth matrix block corresponding to the K matrix generated by the plurality of sample elements, where the candidate size is determined by the sizes of the Q matrix and K matrix; Based on the third matrix block, the fourth matrix block corresponding to each candidate size, and the V matrix generated by the plurality of sample elements, sequentially determine the attention calculation results corresponding to the plurality of sample elements, and respectively determine the time consumed for obtaining the attention calculation results each time; Based on the time consumed corresponding to each candidate size, determine the specified size among the plurality of candidate sizes.

13. The device according to any one of claims 10 to 12, characterized in that, The device further includes a mask processing module for: Determine the fifth matrix block in the same row of the relationship matrix as the sixth matrix block of the relationship matrix, where the fifth matrix block in the relationship matrix is calculated from the first matrix block in the same row of the Q matrix and the second matrix block in the same column of the K matrix; Obtain the Mask matrix corresponding to each sixth matrix block, where each Mask matrix has the same size as the corresponding sixth matrix block; Based on the Mask matrix corresponding to each sixth matrix block, perform mask processing on each sixth matrix block to obtain each sixth matrix block after mask processing.

14. The device according to claim 13, characterized in that, The calculation module is used for: Based on each sixth matrix block after mask processing, perform the operation between the relationship matrix and the V matrix to obtain the attention calculation result corresponding to the attention calculation.

15. A computer program product comprising instructions, characterized in that, When the instruction is run by the computing device, the computing device is caused to execute the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, Including computer program instructions, when the computer program instructions are executed by the computing device, the computing device executes the method according to any one of claims 1 to 7.

Images