Media file processing method and apparatus

Abstract

The embodiments of the present application relate to the technical field of chips. Provided are a media file processing method and apparatus, which reduce computation overheads of a softmax link in an attention layer and memory access overheads of a computation process. The specific solution is: on the basis of an input matrix obtained by means of a media file, obtaining a query matrix, a key-value matrix and a value matrix, performing block-wise computation on the query matrix, the key-value matrix and the value matrix, and performing softmax on-the-fly computation on row blocks of the query matrix and column blocks of a transposed matrix of the key-value matrix. The embodiments of the present application are suitable for a computation process of an attention layer.

Description

Method and device for processing media files

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on February 22, 2024, with application number 202410199717.7 and application name “Method and Device for Processing Media Files”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of chip technology, and in particular to a method and device for processing media files. Background Art

[0003] With the continuous development of artificial intelligence (AI) technology, large language model (LLM) models and diffusion models are gaining more and more attention.

[0004] The large language model and diffusion model enable human-computer interaction through natural language. The system extracts key information from natural language sentences to identify their specific meaning. Furthermore, the large language model and diffusion model can also be used for image classification. By performing a series of processing on a target image, attention features are derived, and classification results are then derived based on these attention features.

[0005] The current large language model and diffusion model both include a large number of attention layer models. The formula of the attention layer can be described as: Q is the query matrix, K is the key matrix, and V is the value matrix. The operators in the attention layer model are computationally complex, involving a large number of matrix multiplications. Matrix multiplications involve memory accesses, which incurs significant memory overhead and increases system power consumption.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a method and device for processing media files, which reduce the computational overhead of the softmax link in the attention layer executed by the media file processing device and the memory access overhead of the calculation process.

[0008] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions.

[0009] In the first aspect, an embodiment of the present application provides a method for processing a media file, the method comprising: encoding the media file to obtain an input matrix, and obtaining a query matrix, a key-value matrix, and a value matrix based on the input matrix. Multiple row blocks of the query matrix are sequentially operated with multiple column blocks of the transposed matrix of the key-value matrix to obtain multiple first intermediate matrices. Whenever a first intermediate matrix is ​​obtained, the maximum value of the elements in the obtained first intermediate matrix and the previous maximum value stored in the first cache is determined, and the determined maximum value is stored in the first cache, wherein the j+1th maximum value covers the j-1th maximum value stored in the first cache, and j is an integer greater than 1. Multiple probability matrices are obtained based on the first intermediate matrix and the determined maximum value in turn, and the obtained probability matrices are stored in the second cache, wherein the j+1th probability matrix covers the j-1th probability matrix stored in the second cache. Whenever a probability matrix is ​​obtained, the current output matrix is ​​determined based on the obtained probability matrix, the row blocks of the value matrix, and the previously obtained output matrix to obtain a target output matrix, where the elements of the target output matrix are used to characterize the attention features of the media file, wherein the j+1th output matrix overwrites the j-1th output matrix stored in the second cache.

[0010] Therefore, in the method for processing media files provided by the embodiment of the present application, the query matrix, key-value matrix and value matrix can be obtained based on the input matrix obtained from the media file, and the query matrix, key-value matrix and value matrix are divided into blocks for calculation, and the row blocks of the query matrix and the column blocks of the transposed matrix of the key-value matrix are softmax-based calculation, which can reduce the computational overhead of the softmax link. In addition, compared with the method that requires complete access to the probability matrix in the calculation process of the attention layer, the method provided by the embodiment of the present application can cover the j-1 probability matrix with the j+1 probability matrix, and can cover the j-1 output matrix with the j+1 output matrix. In this method, less storage space can be used to store the probability matrix and the output matrix, that is, the probability matrix and the output matrix can be stored in a cache with a higher rate, without the need to store the probability matrix and the output matrix in an external memory, thereby reducing the memory access overhead in the calculation process of the attention layer and improving the computational efficiency.

[0011] In one possible design, a current output matrix is ​​determined based on the obtained probability matrix, the row blocks of the value matrix, and the previously obtained output matrix, including: performing matrix multiplication operations on the obtained probability matrix and the row blocks of the value matrix to obtain a second intermediate matrix, and obtaining the current output matrix based on the second intermediate matrix, the currently determined maximum value, the previously determined maximum value, and the previously obtained output matrix.

[0012] In one possible design, the method further includes: whenever a probability matrix is ​​obtained, obtaining a current accumulation matrix based on the obtained probability matrix and a previously obtained accumulation matrix, calibrating the current accumulation matrix to obtain a current calibrated accumulation matrix.

[0013] In one possible design, the method further includes: when obtaining the last output matrix, calibrating the last output matrix based on the last accumulated matrix to obtain a target output matrix.

[0014] In one possible design, multiple row blocks of the query matrix are sequentially operated on with multiple column blocks of the transposed matrix of the key-value matrix to obtain multiple first intermediate matrices, including: performing matrix multiplication operations on the row blocks of the query matrix and the column blocks of the transposed matrix of the key-value matrix to obtain a third intermediate matrix. The elements of the third intermediate matrix are numerically converted to obtain a first intermediate matrix, where the elements in the first intermediate matrix are fixed-point numbers. Therefore, by numerically converting the elements of the third intermediate matrix to obtain a first intermediate matrix whose elements are all fixed-point numbers, the calculation process of the attention layer can be converted from the floating-point domain to the fixed-point domain, further reducing circuit area and power consumption.

[0015] In one possible design, the number of row blocks of the query matrix, the number of column blocks of the transposed matrix of the key-value matrix, and the number of row blocks of the value matrix are the same.

[0016] In a second aspect, an embodiment of the present application provides a media file processing device, the device comprising: a matrix multiplier, a comparator, a first cache, an exponential operator, and a first calibration circuit. The matrix multiplier is used to encode the media file to obtain an input matrix, and to obtain a query matrix, a key-value matrix, and a value matrix based on the input matrix. The matrix multiplier is also used to sequentially operate on multiple row blocks of the query matrix with multiple column blocks of the transposed matrix of the key-value matrix to obtain multiple first intermediate matrices. The comparator is used to determine the maximum value between the elements in the obtained first intermediate matrix and the previous maximum value stored in the first cache each time a first intermediate matrix is ​​obtained, and store the determined maximum value in the first cache, wherein the j+1th maximum value overwrites the j-1th maximum value stored in the first cache, and j is an integer greater than 1. The exponential operator is used to sequentially obtain multiple probability matrices based on the first intermediate matrix and the determined maximum value, and store the obtained probability matrices in the second cache, wherein the j+1th probability matrix overwrites the j-1th probability matrix stored in the second cache. The first calibration circuit is used to determine the current output matrix based on the obtained probability matrix, the row blocks of the value matrix and the previously obtained output matrix whenever a probability matrix is ​​obtained, so as to obtain a target output matrix, wherein the elements of the target output matrix are used to characterize the attention features of the media file, wherein the j+1th output matrix overwrites the j-1th output matrix stored in the second cache.

[0017] In one possible design, the first calibration circuit is specifically configured to perform a matrix multiplication operation on the obtained probability matrix and the row blocks of the value matrix to obtain a second intermediate matrix, and obtain a current output matrix based on the second intermediate matrix, the currently determined maximum value, the previously determined maximum value, and the previously obtained output matrix.

[0018] In one possible design, the media file processing device also includes an adder, which is used to obtain a current cumulative matrix based on the obtained probability matrix and the previously obtained cumulative matrix whenever a probability matrix is ​​obtained, and calibrate the current cumulative matrix to obtain the current calibrated cumulative matrix.

[0019] In one possible design, the media file processing apparatus further includes a second calibration circuit, which is configured to calibrate the last output matrix based on the last accumulated matrix to obtain a target output matrix when the last output matrix is ​​obtained.

[0020] In one possible design, the matrix multiplier is specifically configured to perform a matrix multiplication operation on the row blocks of the query matrix and the column blocks of the transposed matrix of the key matrix to obtain a third intermediate matrix. Furthermore, the matrix performs a numerical conversion on the elements of the third intermediate matrix to obtain a first intermediate matrix, where the elements of the first intermediate matrix are fixed-point numbers.

[0021] In one possible design, the number of row blocks of the query matrix, the number of column blocks of the transposed matrix of the key-value matrix, and the number of row blocks of the value matrix are the same.

[0022] The beneficial effects of the second aspect can be found in the description of the first aspect.

[0023] In a third aspect, an embodiment of the present application provides a chip system comprising a second cache and a media file processing device according to the second aspect. The second cache is configured to store probability matrices and output matrices, wherein the j+1th probability matrix covers the j-1th probability matrix, and the j+1th output matrix covers the j-1th output matrix, where j is an integer greater than 1.

[0024] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, where the memory and the processor are coupled, the memory is used to store computer instructions, and the processor is used to execute the computer instructions to implement any one of the methods in the first aspect.

[0025] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method for processing media files in any of the above aspects and any possible implementation methods.

[0026] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer or processor, enables the computer or processor to execute a method for processing media files in any of the above aspects and any possible implementations.

[0027] It can be understood that any of the media file processing devices, chip systems, electronic devices, computer-readable storage media or computer program products provided above can be applied to the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods and will not be repeated here.

[0028] These and other aspects of the present application will become more readily apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is a calculation flow chart of an attention layer provided in an embodiment of the present application;

[0030] FIG2 is a calculation flow chart of another attention layer provided in an embodiment of the present application;

[0031] FIG3 is a flowchart of a method for processing a media file provided in an embodiment of the present application;

[0032] FIG4 is a schematic structural diagram of a media file processing device provided in an embodiment of the present application;

[0033] FIG5 is a schematic diagram of the circuit structure of a media file processing device provided in an embodiment of the present application;

[0034] FIG6 is a flowchart of another method for processing media files provided in an embodiment of the present application;

[0035] FIG7 is a calculation flow chart of S601 provided in an embodiment of the present application;

[0036] FIG8 is a calculation flow chart of S602 provided in an embodiment of the present application;

[0037] FIG9 is a calculation flow chart of S603 provided in an embodiment of the present application;

[0038] FIG10 is a calculation flow chart of S604 provided in an embodiment of the present application;

[0039] FIG11 is a calculation flow chart of S605 provided in an embodiment of the present application;

[0040] Figure 12 is a calculation flow chart of another attention layer provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] For ease of understanding, some examples of concepts related to the embodiments of this application are provided for reference as follows:

[0042] 1. Softmax function. The calculation formula of softmax function is Where "exp" represents the exponential operation with base e. Specifically, assuming the input variable x = [x0, x1, ..., xn], the calculation process of the softmax function may include: (1) Calculate the maximum value x in the input variable x max ; (2) Calculate each element x in the input variable x separately i With the maximum value x max The difference between the two, and do the exp operation, then sum the values ​​of each element in x after the operation, that is, ∑exp(x i -x max ); (3) Calculate the softmax value of each element in the input variable x, that is, calculate each element x in x separately i The difference between the maximum value and the maximum value is calculated by exp and divided by the sum obtained in step (2), that is, From this, we get the output variable s = [s0, s1, ..., sn], where s0 + s1 + ... + sn = 1. In other words, the softmax function maps the elements of the input variable to real numbers between 0 and 1, and normalizes them to ensure that the sum is 1. For example, in image classification, the elements of the output variable correspond to the probability of each class being chosen.

[0043] 2. Fixed-point numbers and floating-point numbers. The decimal point position of a fixed-point number is fixed, while the decimal point position of a floating-point number is not fixed and can float.

[0044] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0045] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0046] With the continuous development of artificial intelligence technology, large language models and diffusion models are gaining increasing attention. These models can enable human-machine interaction through natural language, image classification, object detection, image super-resolution reconstruction, and image restoration.

[0047] A typical model in a large language model is the transformer model, which introduces a self-attention mechanism. Specifically, the transformer model adopts an encoder-decoder architecture. The transformer model can include multiple encoding layers and multiple decoding layers. Taking the input variable as text data as an example, the text data can include multiple words. The process of the transformer model can include: (1) Obtaining the input vector X of each word in the input text data, the input vector X is obtained by adding the embedding of the word and the embedding of the word position, where the embedding is a feature extracted from the text data. (2) Inputting the input variable X into the encoding layer to obtain the encoding information matrix. (3) Inputting the encoding information matrix into the decoding layer, the decoding layer will translate the i+1th word based on the first i words translated. Each encoding layer can include a feedforward neural network and a self-attention layer, and each decoding layer can include an attention layer in addition to the feedforward neural network and the self-attention layer.

[0048] Specifically, the calculation process of the attention layer is shown in Figure 1, which is a calculation flow chart of the attention layer provided by an embodiment of the present application. The formula of the attention layer can be described as: Q is the query matrix, K is the key matrix, and V is the value matrix. Q, K, and V can be obtained by multiplying the input variables by three different weight matrices (W q 、W k and W v ) is obtained by transposing the matrix, Used to normalize the attention layer to maintain gradient stability.

[0049] Where, the number of tokens of the input variable x is n and the number of hidden state dimensions (hidden dim) is d. The calculation process of the attention layer may include: (1) The input variable x passes through three linear layers to obtain Q, K and V respectively. Specifically, x and W q T Multiplying them together gives Q, which has n rows and d columns. q ; x and W k T Multiplying them together gives us K, which has n rows and d columns. k ; x and W v T Multiplying them together gives us V, which has n rows and d columns. v(2) Perform a batched matrix multiplication (BMM) on the transposed matrix of the query matrix Q and the key matrix K to obtain the attention map matrix, namely QK T , QK T The number of rows and columns of the matrix are both n. (3) Perform softmax calculation on the attention map matrix to obtain the probability matrix A, where the number of rows and columns of the probability matrix A are both n. (4) Perform BMM on the probability matrix A and the value matrix V to obtain the output matrix Z, where the number of rows of the output matrix Z is n and the number of columns is d. v .

[0050] As you can understand, as the number of tokens in the input variable x increases, the size of the probability matrix A also increases. For example, when the number of tokens in x is 1024, the size of the probability matrix A is 1024×1024. When the number of tokens in x is 4096, the size of the probability matrix A is 4096×4096. If the elements in probability matrix A are stored as 16-bit floating-point numbers (float16), storing probability matrix A requires 32MB of memory. Due to the large storage space required for probability matrix A, it is generally not stored in the cache of the neural network processor unit (NPU) and must be written to external memory, which imposes significant storage pressure. In addition, when performing matrix multiplication on probability matrix A and value matrix V, probability matrix A must be read from external memory, resulting in significant memory access overhead and reducing chip performance and power consumption. In addition, as the size of probability matrix A increases, the computational complexity of the softmax function also increases.

[0051] To reduce the computational complexity of the attention layer, a simplified method for the softmax function is provided. This simplified method includes three approaches. Approach 1 replaces the base-e exponential calculation in the original softmax function calculation process with a base-2 exponential calculation. Approach 2 reduces one loop process based on Approach 1. Specifically, the steps of determining the maximum value in the input variable x and summing the values ​​of each element in the input variable x are combined, using only one loop process. Approach 3 extends Approach 2 by converting the input variables to numerical values, thereby implementing the base-2 exponential calculation using a table lookup and shifting.

[0052] However, all intermediate calculation results of the above three methods still need to be cached in memory and then read from memory for the next calculation, which still results in significant memory access overhead, additional performance overhead, and hardware cost. Furthermore, the quantization bit width of the input and output variables of the above three methods is relatively low, which cannot meet the computational accuracy of the neural network, and may require retraining of the neural network model.

[0053] In addition, in order to improve the calculation accuracy of the neural network, another calculation method of the softmax function is provided, as shown in Figure 2, which is a calculation flow chart of another attention layer provided in an embodiment of the present application. In which, matrix multiplication operation is performed on the transposed matrices of Q and K in the INT8 domain, and then softmax operation is performed in the INT32 domain to obtain the probability matrix, and finally matrix multiplication operation is performed on the probability matrix and V in the INT8 domain. In which, INT8 represents an 8-bit signed number, and INT32 represents a 32-bit signed number. Specifically, the calculation method includes: calculating the difference between the i-th element in the input variable x and the maximum value, and obtaining x norm_i , that is, x norm_i =x i -x max . For x norm_i Perform numerical conversion, that is, x norm_i =(-ln2)*u+v, where u is the integer part and v is the fractional part. The value range of v is (-ln2, 0). Calculate the value with e as the base and v as the exponent, where e v ≈0.3585*(v+1.353) 2 The subsequent steps are then calculated in the INT32 domain to obtain the softmax result.

[0054] However, the elements of the attention map matrix obtained during the calculation process need to be stored in the form of INT32, which causes great storage pressure and incurs additional performance and hardware overhead.

[0055] Therefore, the embodiment of the present application provides a method for processing media files, which performs block calculations on the query matrix, the key-value matrix, and the value matrix, and performs softmax path calculations on the row blocks of the query matrix and the column blocks of the transposed matrix of the key-value matrix, which can reduce the computational overhead of the softmax link. In addition, compared with the method that requires complete access to the probability matrix during the calculation process of the attention layer, the method provided by the embodiment of the present application can cover the j-1th probability matrix with the j+1th probability matrix, and can cover the j-1th output matrix with the j+1th output matrix. In this method, less storage space can be used to store the probability matrix and the output matrix, that is, the probability matrix and the output matrix can be stored in a cache with a higher rate, without the need to store the probability matrix and the output matrix in an external memory, thereby reducing the memory access overhead during the calculation process of the attention layer and improving the computational efficiency.

[0056] The media files in the embodiments of the present application may be media files such as images, sounds, and texts, or may be other types of media files, which are not limited in the present application.

[0057] Taking the media file as an image as an example, the media file processing method provided in the embodiment of the present application can be applied to a variety of scenarios such as image classification, target detection, image super-resolution reconstruction and image restoration. For example, in the image classification scenario, the device for processing media files in the present application uses the attention layer to classify images collected by a data acquisition device (such as a camera) to obtain image classification results, and can also perform corresponding subsequent operations based on the image classification results. The device for processing media files can be a terminal, such as a mobile phone, a tablet computer, a laptop computer, an augmented reality device (AR), a virtual reality device (VR), a vehicle-mounted terminal, etc., and can also be a server.

[0058] Taking media files as text data as an example, the media file processing method provided in the embodiments of the present application can be applied in text recognition scenarios. For example, a user uses a mobile phone to translate text data or convert text data into machine-readable binary data.

[0059] In some embodiments, the device for processing media files proposed in the embodiments of the present application may be a chip or a chip system, for example, a system on a chip (SoC). The SoC may include a processor, a memory, and an input / output (I / O) interface, among others, wherein the processor may be a single-core processor or a multi-core processor. The processor may load data and applications from the memory and then process the data, for example, performing the matrix multiplication operation described in the present application.

[0060] Applied to the above scenario, the process of the media file processing method provided by the embodiment of the present application is introduced below.

[0061] An embodiment of the present application provides a method for processing a media file, as shown in FIG3 . FIG3 is a flowchart of a method for processing a media file provided by an embodiment of the present application. The method may include S301 to S305 .

[0062] S301: Encode a media file to obtain an input matrix, and obtain a query matrix, a key-value matrix, and a value matrix based on the input matrix.

[0063] For example, taking the media file as text data as an example, the text data is encoded to obtain an input matrix. The token of the text data is n, and the feature dimension of each token is d. Then the number of rows of the input matrix is ​​n, and the number of columns is d, where each row of the input matrix can be understood as a word. Inputting the input matrix into three linear layers respectively can obtain a query matrix, a key-value matrix and a value matrix. Each linear layer can correspond to a weight matrix. Inputting the input matrix into three linear layers respectively for linear transformation can obtain a query matrix, a key-value matrix and a value matrix. Specifically, the input matrix can be multiplied by three weight matrices (W q 、W k and W v ) to obtain the query matrix, key value matrix and value matrix, W q 、W k and W v is a trainable parameter matrix. Each row in the query matrix, key-value matrix, and value matrix obtained at this time represents a word, and each column in the query matrix, key-value matrix, and value matrix represents the number of words.

[0064] S302 : sequentially perform operations on multiple row blocks of the query matrix and multiple column blocks of the transposed matrix of the key-value matrix to obtain multiple first intermediate matrices.

[0065] Exemplarily, the media file processing device can perform a matrix multiplication operation on the row blocks of the query matrix and the j+1th column block of the transposed matrix of the key value matrix to obtain the j+1th first intermediate matrix. In addition, the jth first intermediate matrix can also be obtained based on the row blocks of the query matrix and the jth column block of the transposed matrix of the key value matrix. That is, the media file processing device performs block calculations on the query matrix and the transposed matrix of the key value matrix, and the row block of the query matrix is ​​any row block in the query matrix, which is represented by "i" in subsequent embodiments, and i is an integer greater than or equal to 1.

[0066] S303. Whenever a first intermediate matrix is ​​obtained, determine the maximum value between an element in the obtained first intermediate matrix and a previous maximum value stored in the first cache, and store the determined maximum value in the first cache, wherein the j+1th maximum value overwrites the j-1th maximum value stored in the first cache, where j is an integer greater than 1.

[0067] Exemplarily, compared to the external memory mentioned above, the first cache can be a cache with a faster access rate but a smaller storage space. That is, reading data from the first cache is faster than reading data from the external memory, but the storage space of the first cache is smaller than the storage space of the external memory. The first cache can be used to store the maximum value in the element, wherein, in order to reduce the storage pressure of the first cache, the first cache can only store two maximum values ​​at the same time, for example, the first cache can store the jth maximum value and the j+1th maximum value at the same time, wherein, when storing the j+1th maximum value, the j+1th maximum value can be used to cover the j-1th maximum value.

[0068] Exemplarily, the jth maximum value can be stored in the first cache, and the jth maximum value can be recorded as max(j), where max(j) is a vector, and the number of elements in max(j) is the same as the number of rows of the jth first intermediate matrix. Specifically, taking the maximum value of the elements of the first row of the j+1th first intermediate matrix and the first element in max(j) as an example, the j+1th maximum value can be recorded as max(j+1). If the elements of the first row of the j+1th first intermediate matrix are all less than or equal to the first element in max(j), then the first element in max(j) is used as the first element in max(j+1). If there is at least one element in the first row of the j+1th first intermediate matrix that is greater than or equal to the first element in max(j), then the largest element among the elements in the first row of the j+1th first intermediate matrix is ​​used as the first element of max(j+1). Therefore, max(j+1) can be obtained by traversing the elements of each row in the j+1th first intermediate matrix, and max(j+1) is overwritten by max(j-1) stored in the first cache.

[0069] S304 . Obtain multiple probability matrices based on the first intermediate matrix and the determined maximum value in sequence, and store the obtained probability matrices in the second cache, wherein the j+1th probability matrix covers the j-1th probability matrix stored in the second cache.

[0070] For example, after obtaining the new max(j+1), the difference between the elements in the j+1th first intermediate matrix and max(j+1) is calculated, and operations with e as the base and the difference as the exponent are calculated to obtain the j+1th probability matrix, and the j+1th probability matrix is ​​overwritten with the j-1th probability matrix stored in the second cache. Specifically, assuming that the j+1th first intermediate matrix is ​​QK T (i, j+1) represents the j+1 probability matrix, and A(i, j+1) = exp(QK T (i, j+1)-max(j+1)). Wherein, i represents the row block of the query matrix and is an integer greater than or equal to 1.

[0071] Exemplary, compared to the external memory mentioned above, the second cache can be a cache with a faster access rate but smaller storage space. Specifically, the second cache can be a static random access memory (SRAM). In other words, reading data from the second cache is faster than reading data from the external memory, but the storage space of the second cache is smaller than the storage space of the external memory. Since the probability matrix in the embodiment of the present application is obtained by performing block calculations by the query matrix and the key value matrix, and the j+1th probability matrix will cover the j-1th probability matrix, the storage space required by the probability matrix is ​​small, so the probability matrix can be stored in the second cache, thereby improving memory access efficiency.

[0072] S305. Whenever a probability matrix is ​​obtained, the current output matrix is ​​determined based on the obtained probability matrix, the row blocks of the value matrix, and the previously obtained output matrix to obtain a target output matrix. The elements of the target output matrix are used to characterize the attention features of the media file, wherein the j+1th output matrix overwrites the j-1th output matrix stored in the second cache.

[0073] For example, the jth output matrix can be obtained by combining the jth probability matrix, the jth row block of the value matrix, and the j-1th output matrix. Each output matrix can be understood as an intermediate result, and each output matrix can be overwritten by the subsequent output matrix, which can reduce storage space. This can reduce the memory access overhead during the calculation of the attention layer and improve performance.

[0074] For ease of understanding, the following first introduces a media file processing device that applies a media file processing method provided in an embodiment of the present application.

[0075] In one possible implementation, as shown in FIG4 , FIG4 is a structural diagram of a media file processing device provided in an embodiment of the present application. The media file processing device may include a matrix multiplier and a processing engine, wherein the input of the matrix multiplier may be a left matrix, a right matrix and a bias term. In one example, the left matrix may be a query matrix, the right matrix may be a key-value matrix, and the matrix multiplier may perform matrix multiplication operations on the input left matrix and right matrix. The processing engine may include a cache and an attention engine. The attention engine may include an accumulator, an exponent operator, a comparator and a divider. In one example, the divider may also be replaced by a multiplier.

[0076] In some embodiments, based on the media file processing device shown in FIG4 , FIG5 shows a circuit diagram of a media file processing device. As shown in FIG5 , the media file processing device 50 may include a matrix multiplier 51, a comparator 52, a first buffer 53, an exponential operator 54, and a first calibration circuit 55. Furthermore, the media file processing device 50 may also include a subtractor 56, an adder 57, and a second calibration circuit 58.

[0077] Specifically, FIG5 shows a first cache_A and a first cache_B. In one possible implementation, the first cache_A and the first cache_B may be two storage spaces within the same cache, or may be two caches. The first cache_A may be used to store the maximum value determined in the softmax on-path calculation, and the first cache_A may also be referred to as a "max buffer." The first cache_B may be used to store the summation matrix determined in the softmax on-path calculation, and the first cache_B may also be referred to as a "sum_buffer."

[0078] For example, FIG5 shows a matrix multiplier 51_A, a matrix multiplier 51_B, and a matrix multiplier 51_C. Matrix multiplier 51_A, matrix multiplier 51_B, and matrix multiplier 51_C can be the same matrix multiplier, and time-division multiplexing can be used to enable the matrix multiplier to perform matrix multiplication operations based on different inputs. Specifically, the inputs to matrix multiplier 51_A can be the row blocks of the query matrix, the column blocks of the key-value matrix, and the bias term; the inputs to matrix multiplier 51_B can be the probability matrix, the row blocks of the value matrix, and the bias term; and the inputs to matrix multiplier 51_C can be a zero matrix, a zero matrix, and an output matrix.

[0079] For example, the comparator 52 can obtain the maximum value of the previous step from the first cache_A, represented by max(j), compare the elements in the first intermediate matrix with max(j), determine the maximum value of the current step, represented by max(j+1), and overwrite max(j-1) stored in the first cache_A with max(j+1), where max(j-1) is the maximum value of the step before the previous step. In other words, the first cache stores the maximum values ​​determined in both steps simultaneously, so the first cache can use the cache with the higher rate.

[0080] For example, the subtractor 56 can calculate the difference between the elements in the first intermediate matrix and the maximum value max(j+1) of the current step. The exponential operator 54 can calculate an operation with e as the base and the difference as the exponent. The adder 57 can obtain the cumulative matrix of the previous step from the first cache_B, represented by d(j), calibrate d(j), and add it to the output of the exponential operator 54 to obtain the cumulative matrix of the current step, represented by d(j+1), and overwrite d(j+1) with d(j) stored in the first cache_B. The first calibration circuit 55 can calibrate the sum of the j+1th second intermediate matrix and the jth output matrix based on the jth maximum value and the j+1th maximum value to obtain the j+1th output matrix. The second calibration circuit 58 can obtain the cumulative matrix of the last step from the second cache_B to calibrate the output matrix to obtain a calibrated output matrix.

[0081] It is understandable that the specific implementation of each circuit of the media file processing device can be found in the following description.

[0082] In conjunction with the circuit structure of the media file processing device shown in FIG5 , the media file processing method provided in the embodiment of the present application is further introduced below.

[0083] Optionally, S302 may include: performing matrix multiplication operations on row blocks of the probability matrix and the value matrix to obtain a second intermediate matrix, and obtaining a current output matrix based on the second intermediate matrix, the currently determined maximum value, the previously determined maximum value, and the previously obtained output matrix.

[0084] Exemplarily, the media file processing apparatus may perform a matrix multiplication operation on the j+1th probability matrix and the j+1th row block of the value matrix through a matrix multiplier to obtain the j+1th second intermediate matrix.

[0085] Exemplarily, the jth maximum value and the j+1th maximum value are stored in the first cache, the jth maximum value can be recorded as max(j), and the j+1th maximum value can be recorded as max(j+1). The media file processing device can obtain max(j+1) and max(j) from the first cache through a first calibration circuit, and calculate an exponential operation with base e and max(j)-max(j-1) as the exponent, that is, calculate exp(max(j)-max(j+1)). Assuming that the j+1th second intermediate matrix is ​​represented by O(i, j+1) and the j+1th output matrix is ​​represented by Z(i, j+1), then Z(i, j+1)=(O(i, j+1)+Z(i, j)) / exp(max(j)-max(j+1)).

[0086] Optionally, the media file processing method further includes: each time a probability matrix is ​​obtained, obtaining a current accumulation matrix based on the obtained probability matrix and a previously obtained accumulation matrix, and calibrating the current accumulation matrix to obtain a current calibrated accumulation matrix.

[0087] Exemplarily, the j+1th accumulation matrix is ​​the value obtained by summing the j+1th probability matrix and then summing it with the calibrated j-th accumulation matrix. Before accumulating the j-th accumulation matrix, the j-th accumulation matrix needs to be calibrated. Assuming that the j+1th accumulation matrix is ​​represented by d(j+1) and the j-th accumulation matrix is ​​represented by d(j), the calibrated j-th accumulation matrix can be expressed as d(j)*exp(max(j)-max(j+1)), then d(j+1)=sum(A(i, j+1))+d(j)*exp(max(j)-max(j+1)). Wherein, sum represents the sum of the elements in each row of A(i, j+1).

[0088] Optionally, the media file processing method further includes: when obtaining the last output matrix, calibrating the last output matrix based on the last accumulated matrix to obtain a target output matrix.

[0089] For example, assuming that the transposed matrix of the key-value matrix includes n column blocks, the last accumulation matrix can be expressed as the nth accumulation matrix, and the last output matrix can be expressed as the nth output matrix. The nth accumulation matrix can be expressed as d(n), and the nth output matrix can be expressed as O(i, n). O(i, n) can be calibrated based on d(n), and the calibrated Z(i, n) = O(i, n) / d(n).

[0090] Exemplarily, Z(i, n) is the element of the i-th row of the target output matrix. The next row block of the query matrix and each column block of the transposed matrix of the key-value matrix and each row block of the value matrix can be traversed to obtain the elements of other rows of the target output matrix, thereby completing the calculation process of the attention layer.

[0091] In a possible example, the calculation process of the attention layer is introduced by taking i=1, j=1 as an example, where the query matrix is ​​represented by "Q", the key-value matrix is ​​represented by "K", the value matrix is ​​represented by "V", the probability matrix is ​​represented by "A", and the first intermediate matrix is ​​represented by "QK T ", the second intermediate matrix is ​​represented by "O", and the output matrix is ​​represented by "Z". The processing flow of the media file is shown in Figure 6, which is a flowchart of another media file processing method provided by an embodiment of the present application. The method includes S601 to S607.

[0092] S601. Calculate Q(1)*K(1), and perform softmax calculation along the path to obtain A(1, 1).

[0093] For example, as shown in FIG7 , the query matrix may include m row blocks, and the transposed matrix of the key matrix may include n column blocks. A matrix multiplication operation is performed on the first row block Q(1) of the query matrix and the first column block K(1) of the key matrix to obtain a first intermediate matrix QK T (1, 1).

[0094] Exemplarily, the softmax path calculation step may include: at this time j=1, the elements in max(j-1) in the first cache are all negative infinity, that is, the first intermediate matrix QK T The elements of each row in (1,1) are greater than the corresponding elements in max(j-1), which can be obtained from the first intermediate matrix QK T (1,1) determines the maximum value max(1), then A(1,1)=exp(QK T (1,1)-max(1)), store A(1,1) in the second cache.

[0095] Exemplarily, when j=1, d(j-1)=0, then the accumulation matrix d(1)=sum(A(1,1)), and d(1) is stored in the first cache.

[0096] S602: Calculate Q(1)*K(2), and perform softmax calculation along the path to obtain A(1, 2).

[0097] For example, as shown in FIG8 , a matrix multiplication operation is performed on the first row block Q(1) of the query matrix and the first column block K(2) of the key matrix to obtain a first intermediate matrix QK T (1, 2).

[0098] Exemplarily, the step of calculating the softmax path may include: at this time j=2, max(1) is stored in the first cache. Specifically, the first intermediate matrix QK T The maximum value is determined from the elements of the first row of (1, 2). If the maximum value is greater than or equal to the first element in max(1), the maximum value is used as the first element in max(2). If the maximum value is less than the first element in max(1), the first element in max(1) is used as the first element in max(2). In this way, the first intermediate matrix QK can be traversed. T The elements of each row of (1, 2) get all the elements in max(2), and store max(2) in the first cache. At this time, the first cache stores max(1) and max(2). Among them, A(1, 2) = exp(QK T (1, 2)-max(2)), store A(1, 2) in the second cache.

[0099] Exemplarily, when j=2, the matrix d(2)=sum(A(1,2))+d(1)*exp(max(1)-max(2)) is accumulated, and d(2) is used to overwrite d(1) stored in the first cache.

[0100] S603 : Calculate A(1, 1)*V(1), and perform calibration based on the maximum value to obtain Z(1, 1).

[0101] Exemplarily, as shown in FIG9 , a matrix multiplication operation is performed on the probability matrix A(1, 1) and the first row block V(1) of the value matrix to obtain a second intermediate matrix O(1, 1).

[0102] Exemplarily, the calibration process for the second intermediate matrix O(1, 1) can be expressed as: Z(1, 1) = O(1, 1) / exp(max(2) - max(1)). At this time, Z(1, 1) is stored in the second cache.

[0103] S604: Calculate Q(1)*K(3), and perform softmax calculation along the path to obtain A(1, 3).

[0104] For example, as shown in FIG10 , a matrix multiplication operation is performed on the first row block Q(1) of the query matrix and the first column block K(3) of the key matrix to obtain a first intermediate matrix QK T (1, 3).

[0105] Exemplarily, the step of calculating the softmax path may include: at this time j=3, max(2) is stored in the first cache. Specifically, the first intermediate matrix QK T The maximum value is determined from the elements of the first row of (1, 3). If the maximum value is greater than or equal to the first element in max(2), the maximum value is used as the first element in max(3). If the maximum value is less than the first element in max(2), the first element in max(2) is used as the first element in max(3). In this way, the first intermediate matrix QK can be traversed. T The elements of each row of (1, 3) get all the elements in max(3), and max(3) overwrites max(1) stored in the first cache. Where A(1, 3) = exp(QK T (1, 3)-max(3)), A(1, 3) overwrites A(1, 1) stored in the second cache.

[0106] Exemplarily, when j=3, the cumulative matrix d(3)=sum(A(1,3))+d(2)*exp(max(2)-max(3)), and d(3) overwrites d(2) stored in the first cache.

[0107] S605 : Calculate A(1, 2)*V(2), and perform calibration based on the maximum value to obtain Z(1, 2).

[0108] Exemplarily, as shown in FIG11 , a matrix multiplication operation is performed on the probability matrix A(1, 2) and the first row block V(2) of the value matrix to obtain a second intermediate matrix O(1, 2).

[0109] Exemplarily, the calibration process for the second intermediate matrix O(1, 2) can be expressed as: Z(1, 2) = (O(1, 2) + Z(1, 1)) / exp(max(3) - max(2)). At this time, Z(1, 2) is stored in the second cache.

[0110] Repeat S604 and S605 to obtain the final O(1, n) and d(n).

[0111] S606. Repeat S604 and S605 to obtain the final O(1, n) and d(n).

[0112] S607. Calculate O(1, n) / d(n) to obtain Z(1, n).

[0113] It can be seen that the staggered arrangement of the matrix multiplication operations of QK and AV in the embodiment of the present application can adapt to the calculation process of the attention layer and ensure the accuracy of the calculation.

[0114] As shown in Figure 12, Figure 12 is a calculation flow chart of another attention layer provided by an embodiment of the present application. The process of this flow chart corresponds to the process in Figure 7. For example, (1) corresponds to S601, that is, (1) is the step of obtaining the probability matrix A(1, 1). At this time, max(1) and d(1) are stored in the first cache. (2) corresponds to S602, that is, (2) is the step of obtaining the probability matrix A(1, 2). At this time, max(1), max(2) and d(2) are stored in the first cache. (3) corresponds to S603, that is, (3) is the step of obtaining the output matrix Z(1, 1). At this time, Z(1, 1) is stored in the second cache. (4) corresponds to S604, that is, (4) is the step of obtaining the probability matrix A(1, 3). At this time, max(2), max(3) and d(3) are stored in the first cache. (5) corresponds to S605, that is, (5) is the step of obtaining the output matrix Z(1, 2). At this time, Z(1, 2) is stored in the second cache. In addition, (6) is the step of obtaining the probability matrix A(1, 4). At this time, max(3), max(4), and d(4) are stored in the first cache. (7) is the step of obtaining the output matrix Z(1, 3). At this time, Z(1, 3) is stored in the second cache. (8) is the step of obtaining the output matrix O(1, 4). At this time, O(1, 4) is stored in the second cache. (9) is the process of calibrating the output matrix O(1, 4), obtaining the calibrated output matrix Z(1, 4).

[0115] It is understandable that if the amount of data stored in the first cache and the second cache is small, a memory with a faster access rate, such as SRAM, can be used to reduce memory access overhead.

[0116] Optionally, S301 may also include: performing matrix multiplication operations on the row blocks of the query matrix and the column blocks of the transposed matrix of the key-value matrix to obtain a third intermediate matrix, performing numerical conversion on the elements of the third intermediate matrix to obtain a first intermediate matrix, where the elements in the first intermediate matrix are fixed-point numbers.

[0117] For example, by performing numerical conversion on the elements of the third intermediate matrix to obtain the first intermediate matrix, the calculation process of the attention layer can be converted from the floating-point domain to the fixed-point domain, which can further reduce circuit area and power consumption.

[0118] The following only introduces the content of the numerical conversion part, and the rest is not repeated.

[0119] For example, due to e x =2 x / ln2 If true, then when performing exponential operations, you can use 2 x / ln2 Replace e x Perform the calculation, where 1 / ln2≈1.4427.

[0120] Assume that y=x / ln2 is expressed in the form of a and b, where a is the integer digit of y and b is the decimal digit of y, then 2 x / ln2 =2 y =2 a *2 b , since a is an integer, 2 a It can be represented by shifting. In addition, b can be discarded when quantizing softmax, then e x It can be approximately expressed as 2(Round(1.4427x)), where Round means rounding 1.4427x to the specified number of decimal places. This converts the floating-point number to a fixed-point number.

[0121] It can be understood that the operation of calculating exp(max(j)-max(j+1)) mentioned above can be replaced by the operation of left shifting max(j)-max(j+1). In addition, since max(j+1) is the maximum value of the elements of the current first intermediate matrix and max(j) when determining max(j+1), max(j+1) always holds true, and the operation of exp(max(j)-max(j+1)) is a right shift operation.

[0122] It can be understood that due to the use of block calculation and softmax on-path calculation, memory access is reduced. The input matrix, output matrix and intermediate matrix in the embodiment of the present application can all use 16 bits (bit), which increases the quantization bit width, that is, improves the quantization accuracy of the model.

[0123] An embodiment of the present application provides a chip system including a second cache and a media file processing device according to the second aspect. The second cache is configured to store a probability matrix and an output matrix, wherein the j+1th probability matrix covers the j-1th probability matrix, and the j+1th output matrix covers the j-1th output matrix, where j is an integer greater than 1.

[0124] An embodiment of the present application further provides an electronic device comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors and are configured to store computer program code, the computer program code comprising computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the above-described related method steps to implement the media file processing method of the above-described embodiment.

[0125] An embodiment of the present application further provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the media file processing method in the above-mentioned embodiment.

[0126] An embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the media file processing method executed by the electronic device in the above-mentioned embodiment.

[0127] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the media file processing method executed by the electronic device in the above-mentioned method embodiments.

[0128] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0129] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0131] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0133] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0134] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for processing a media file, characterized in that: The method comprises: Encoding the media file to obtain an input matrix, and obtaining a query matrix, a key-value matrix, and a value matrix based on the input matrix; Sequentially performing operations on the multiple row blocks of the query matrix and the multiple column blocks of the transposed matrix of the key-value matrix to obtain multiple first intermediate matrices; Whenever a first intermediate matrix is ​​obtained, determining a maximum value between an element in the obtained first intermediate matrix and a previous maximum value stored in the first cache, and storing the determined maximum value in the first cache, wherein the j+1th maximum value overwrites the j-1th maximum value stored in the first cache, where j is an integer greater than 1; Sequentially obtaining a plurality of probability matrices based on the first intermediate matrix and the determined maximum value, and storing the obtained probability matrices in a second cache, wherein the j+1th probability matrix overwrites the j-1th probability matrix stored in the second cache; Whenever a probability matrix is ​​obtained, a current output matrix is ​​determined based on the obtained probability matrix, the row blocks of the value matrix, and the previously obtained output matrix to obtain a target output matrix, wherein the elements of the target output matrix are used to characterize the attention features of the media file, wherein the j+1th output matrix overwrites the j-1th output matrix stored in the second cache.

2. The method according to claim 1, characterized in that The determining of the current output matrix based on the obtained probability matrix, the row blocks of the value matrix and the previously obtained output matrix includes: Performing a matrix multiplication operation on the obtained probability matrix and the obtained row blocks of the value matrix to obtain a second intermediate matrix; A current output matrix is ​​obtained based on the second intermediate matrix, the currently determined maximum value, the previously determined maximum value, and the previously obtained output matrix.

3. The method according to claim 2, characterized in that The method further comprises: Whenever a probability matrix is ​​obtained, a current accumulation matrix is ​​obtained based on the obtained probability matrix and the previously obtained accumulation matrix; The current accumulation matrix is ​​calibrated to obtain the current calibrated accumulation matrix.

4. The method according to claim 3, characterized in that The method further comprises: When the last output matrix is ​​obtained, the last output matrix is ​​calibrated based on the last accumulation matrix to obtain the target output matrix.

5. The method according to claim 1, characterized in that The step of sequentially operating the plurality of row blocks of the query matrix with the plurality of column blocks of the transposed matrix of the key-value matrix to obtain a plurality of first intermediate matrices includes: Performing a matrix multiplication operation on the row blocks of the query matrix and the column blocks of the transposed matrix of the key-value matrix to obtain a third intermediate matrix; Numerical conversion is performed on elements of the third intermediate matrix to obtain the first intermediate matrix, where elements in the first intermediate matrix are fixed-point numbers.

6. The method according to any one of claims 1 to 5, characterized in that The number of row blocks of the query matrix, the number of column blocks of the transposed matrix of the key-value matrix, and the number of row blocks of the value matrix are the same.

7. A media file processing device, characterized in that: include: A matrix multiplier, a comparator, a first cache, an exponential operator and a first calibration circuit; The matrix multiplier is configured to: encode the media file to obtain an input matrix, and obtain a query matrix, a key-value matrix, and a value matrix based on the input matrix; The matrix multiplier is further configured to: sequentially operate the multiple row blocks of the query matrix with the multiple column blocks of the transposed matrix of the key-value matrix to obtain multiple first intermediate matrices; The comparator is configured to: whenever a first intermediate matrix is ​​obtained, determine a maximum value between an element in the obtained first intermediate matrix and a previous maximum value stored in the first cache, and store the determined maximum value in the first cache, wherein the j+1th maximum value overwrites the j-1th maximum value stored in the first cache, where j is an integer greater than 1; The exponential operator is configured to sequentially obtain a plurality of probability matrices based on the first intermediate matrix and the determined maximum value, and store the obtained probability matrices in a second cache, wherein the j+1th probability matrix overwrites the j-1th probability matrix stored in the second cache; The first calibration circuit is configured to: whenever a probability matrix is ​​obtained, determine a current output matrix based on the obtained probability matrix, the row blocks of the value matrix and the previously obtained output matrix to obtain a target output matrix, wherein the elements of the target output matrix are used Characterize the attention features of the media file, wherein the j+1th output matrix covers the j-1th output matrix stored in the second cache.

8. The device according to claim 7, characterized in that The first calibration circuit is specifically used for: Performing a matrix multiplication operation on the obtained probability matrix and the obtained row blocks of the value matrix to obtain a second intermediate matrix; A current output matrix is ​​obtained based on the second intermediate matrix, the currently determined maximum value, the previously determined maximum value, and the previously obtained output matrix.

9. The device according to claim 8, characterized in that The device also includes an adder; The adder is configured to: whenever a probability matrix is ​​obtained, obtain a current accumulation matrix based on the obtained probability matrix and a previously obtained accumulation matrix; The current accumulation matrix is ​​calibrated to obtain the current calibrated accumulation matrix.

10. The device according to claim 9, characterized in that The apparatus further includes a second calibration circuit; The second calibration circuit is configured to: when obtaining the last output matrix, calibrate the last output matrix based on the last accumulation matrix to obtain the target output matrix.

11. The device according to claim 7, characterized in that The matrix multiplier is specifically used for: Performing a matrix multiplication operation on the row blocks of the query matrix and the column blocks of the transposed matrix of the key-value matrix to obtain a third intermediate matrix; Numerical conversion is performed on elements of the third intermediate matrix to obtain the first intermediate matrix, where elements in the first intermediate matrix are fixed-point numbers.

12. The device according to any one of claims 7 to 11, characterized in that: The number of row blocks of the query matrix, the number of column blocks of the transposed matrix of the key-value matrix, and the number of row blocks of the value matrix are the same.

13. A chip system, characterized in that: A device for processing media files comprising a second cache and any one of claims 7 to 12; The second cache is used to store probability matrices and output matrices, wherein the j+1th probability matrix covers the j-1th probability matrix, and the j+1th output matrix covers the j-1th output matrix, where j is an integer greater than 1.

14. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is coupled to the processor, the memory is used to store computer instructions, and the processor is used to execute the computer instructions to implement the method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 6.

Images