Matrix calculation method and related equipment

Abstract

The invention provides a matrix calculation method and related equipment. The method is applied to the computing device, an external memory and an internal memory are carried in the computing device, a first matrix to be computed is stored in the external memory, and a second matrix to be computed is stored in the internal memory. Wherein the data volume contained in the first matrix is greater than the storage capacity allocated to the first matrix in the memory. The method comprises the following steps: dividing the first matrix stored in the external memory into a plurality of sub-matrixes; wherein the data volume contained in each sub-matrix in the plurality of sub-matrixes is not greater than the storage capacity allocated to the first matrix in the memory; and loading the plurality of sub-matrixes from the external memory to the internal memory in batches, and carrying out matrix multiplication calculation on the sub-matrixes and the second matrix.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer technology, and in particular, to a matrix calculation method and related devices. Background Art

[0002] In the calculation process of various application scenarios such as face recognition, privacy computing and large model dialogue, matrix calculations, especially matrix multiplication calculations, are often involved. In addition, with the development of science and technology and the growth of user needs, the scale of matrix calculations is also constantly increasing. However, in the matrix multiplication calculation of ultra-large-scale matrices, the memory of existing computing devices may not even be able to store a single matrix involved in the calculation, let alone store the intermediate results in the matrix multiplication calculation process, so that the matrix multiplication calculation cannot be performed smoothly. For example, taking a computing device with a 16c32g configuration as an example, its memory storage capacity is 32G, but the dimension of the ultra-large-scale matrix involved in the calculation may be 4 million * 4096. If it is stored in a single-precision floating-point data format, each single-precision floating-point number occupies 4 bytes, then the storage space occupied by the large-scale matrix is ​​4 million * 4096 * 4 bytes ≈ 61G, which far exceeds the memory capacity, so the calculation of the ultra-large-scale matrix cannot be directly implemented in the memory.

[0003] In order to solve the calculation problem of ultra-large-scale matrices, existing technologies often split the ultra-large-scale matrices into the memory of multiple computing devices for distributed computing, which seriously increases the equipment cost and network communication cost. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a matrix calculation method and related devices.

[0005] In a first aspect, the present specification provides a matrix calculation method, which is applied to a computing device, wherein the computing device is equipped with an external memory and a internal memory, wherein the external memory stores a first matrix to be calculated, and the internal memory stores a second matrix to be calculated; wherein the amount of data contained in the first matrix is ​​greater than the storage capacity allocated to the first matrix in the internal memory; the method comprises:

[0006] Divide the first matrix stored in the external memory into a plurality of sub-matrices; wherein the amount of data contained in each of the plurality of sub-matrices is not greater than the storage capacity allocated to the first matrix in the internal memory;

[0007] The plurality of sub-matrices are loaded from the external memory into the internal memory in batches, and matrix multiplication calculations are performed with the second matrix.

[0008] In a second aspect, the present specification provides a matrix calculation device, which is applied to a computing device, wherein the computing device is equipped with an external memory and a internal memory, wherein the external memory stores a first matrix to be calculated, and the internal memory stores a second matrix to be calculated; wherein the amount of data contained in the first matrix is ​​greater than the storage capacity allocated to the first matrix in the internal memory; the device comprises:

[0009] A matrix division unit, used for dividing the first matrix stored in the external memory into a plurality of sub-matrices; wherein the amount of data contained in each of the plurality of sub-matrices is not greater than the storage capacity allocated to the first matrix in the internal memory;

[0010] A matrix calculation unit is used to load the multiple sub-matrices from the external memory into the internal memory in batches, and perform matrix multiplication calculation with the second matrix.

[0011] Accordingly, this specification also provides a computing device, comprising: a memory and a processor; the memory stores a computer program / instruction that can be executed by the processor; when the processor executes the computer program / instruction, it executes the matrix calculation method described in the first aspect above.

[0012] Accordingly, the present specification also provides a computer-readable storage medium having a computer program / instruction stored thereon, and when the computer program / instruction is executed by a processor, the matrix calculation method as described in the first aspect above is executed.

[0013] Accordingly, the present specification also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the matrix calculation method described in the first aspect above is performed.

[0014] In summary, the present application can be applied to a computing device, which is equipped with an external memory and a memory, wherein a first matrix to be calculated is stored in the external memory, and a second matrix to be calculated is stored in the memory. Wherein, the amount of data contained in the first matrix is ​​greater than the storage capacity allocated to the first matrix in the memory. In an embodiment shown, the computing device can divide the first matrix stored in the external memory into multiple sub-matrices; wherein, the amount of data contained in each of the multiple sub-matrices is not greater than the storage capacity allocated to the first matrix in the memory. Further, the computing device can load multiple sub-matrices from the external memory to the memory in batches, and perform matrix multiplication calculations with the second matrix. In this way, in the case where the memory of the computing device cannot directly store a large-scale matrix involved in the matrix multiplication calculation, the present application first stores the large-scale matrix in the external memory of the computing device, and divides it into multiple smaller sub-matrices, and then loads the multiple sub-matrices into the memory in batches, and performs matrix multiplication calculations with another small matrix in the memory, so that large-scale matrix calculations can be realized only through the local external memory of the device, without increasing additional equipment costs and network communication costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic diagram of a system architecture provided by an exemplary embodiment;

[0016] Figure 2 is a flowchart of a matrix calculation method provided by an exemplary embodiment;

[0017] Figure 3 is a schematic diagram of a matrix multiplication calculation provided by an exemplary embodiment;

[0018] Figure 4 is a structural schematic diagram of a matrix computing device provided by an exemplary embodiment;

[0019] Figure 5 It is a structural diagram of a computing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0020] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0021] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0022] It should be noted that the “plurality” mentioned in this application refers to two or more than two.

[0023] In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0024] This specification provides a technical solution that divides a large-scale matrix in an external memory into multiple smaller sub-matrices, and then loads the multiple sub-matrices into the internal memory in batches to perform matrix multiplication calculations with another small matrix. In this way, large-scale matrix calculations can be achieved only through the local external memory of the device, without increasing additional equipment costs and network communication costs.

[0025] In implementation, the present application can be applied to a computing device equipped with an external memory and a memory, wherein the external memory stores a first matrix to be calculated, and the memory stores a second matrix to be calculated. The amount of data contained in the first matrix is ​​greater than the storage capacity allocated to the first matrix in the memory. In an illustrated embodiment, the computing device can divide the first matrix stored in the external memory into a plurality of sub-matrices; wherein the amount of data contained in each of the plurality of sub-matrices is not greater than the storage capacity allocated to the first matrix in the memory. Furthermore, the computing device can load the plurality of sub-matrices from the external memory to the memory in batches, and perform matrix multiplication calculations with the second matrix.

[0026] In the above technical scheme, when the memory of the computing device cannot directly store the large-scale matrix involved in the matrix multiplication calculation, the present application first stores the large-scale matrix in the external memory of the computing device and divides it into multiple smaller sub-matrices. The multiple sub-matrices are then loaded into the memory in batches and matrix multiplication calculations are performed with another small matrix in the memory. In this way, large-scale matrix calculations can be achieved only through the local external memory of the device, which is efficient and convenient without increasing additional equipment costs and network communication costs.

[0027] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a system architecture provided by an exemplary embodiment. Figure 1 As shown, one or more embodiments provided in this specification may be Figure 1 The present invention is specifically implemented in the system architecture shown or a similar system architecture.

[0028] like Figure 1 As shown, the computing device is equipped with an external memory and an internal memory, wherein the external memory stores a first matrix to be calculated, and the internal memory stores a second matrix to be calculated. In an embodiment shown, the amount of data contained in the first matrix is ​​greater than the storage capacity allocated to the first matrix in the internal memory, and may even be greater than the storage capacity of the entire internal memory, but less than the storage capacity of the external memory (i.e., the external memory capacity).

[0029] It should be noted that this specification does not specifically limit the specific implementation manner in which the computing device obtains the first matrix and the second matrix and stores them.

[0030] In an illustrated embodiment, the computing device may obtain the first matrix and the second matrix from other remote devices through a network. Further, in response to the amount of data contained in the first matrix being greater than the storage capacity that can be allocated to the first matrix in the memory of the computing device, or even the amount of data contained in the first matrix being greater than the entire memory capacity, the computing device may store the first matrix in the external memory of the computing device; and, in response to the amount of data contained in the second matrix being less than the storage capacity that can be allocated to the second matrix in the memory of the computing device, the second matrix is ​​stored in the memory of the computing device.

[0031] In an illustrated embodiment, the computing device may first obtain original input data from other remote devices through a network, and perform a series of processing on the input data locally to obtain the first matrix and the second matrix. Accordingly, the first matrix may be originally stored in the external memory of the computing device, the second matrix may be originally stored in the internal memory of the computing device, and so on, which is not specifically limited in this specification.

[0032] Exemplarily, the dimension of the first matrix can be 4 million * 4096, that is, 4 million rows and 4096 columns. If stored in single-precision floating-point data format, each single-precision floating-point number occupies 4 bytes, then the amount of data contained in the first matrix (that is, the occupied storage space) is 4 million * 4096 * 4 bytes ≈ 61G; if stored in double-precision floating-point data format, each double-precision floating-point number occupies 8 bytes, then the amount of data contained in the first matrix is ​​4 million * 4096 * 8 bytes ≈ 122G.

[0033] Exemplarily, the dimension of the second matrix can be 4096*4096, that is, 4096 rows and 4096 columns. If stored in single-precision floating-point data format, the amount of data contained in the second matrix is ​​4096*4096*4 bytes=64M; if stored in double-precision floating-point data format, each double-precision floating-point number occupies 8 bytes, then the amount of data contained in the second matrix is ​​4096*4096*8 bytes=128M.

[0034] Exemplarily, the configuration of the computing device may be 16c32g, that is, including 16 CPU cores and 32G of memory. Obviously, 32G of memory cannot store the first matrix of about 61G or 122G, but can store the second matrix of 64M or 128M.

[0035] In an illustrated embodiment, after storing the first matrix and the second matrix, the computing device may perform calculations on the first matrix and the second matrix, specifically, matrix multiplication calculations. As described above, after the second matrix is ​​stored in the memory, since the amount of data contained in the first matrix has exceeded the storage capacity that can be allocated in the memory, the first matrix cannot be directly loaded into the memory to perform calculations with the second matrix. Therefore, the computing device may first divide the first matrix into a plurality of sub-matrices, wherein the amount of data contained in each sub-matrix is ​​not greater than the storage capacity allocated to the first matrix in the memory. Furthermore, the computing device may load the divided multiple sub-matrices from the external memory into the memory in batches, and perform matrix multiplication calculations with the second matrix to obtain corresponding calculation results. Finally, the computing device may merge the multiple calculation results of the multiple sub-matrices with the second matrix, so as to obtain the matrix multiplication calculation results of the first matrix and the second matrix, and so on. For details, please refer to the following Figure 2 The description of the corresponding embodiments will not be elaborated in detail here.

[0036] It should be noted that this specification does not specifically limit the specific type of memory installed in the computing device. In an illustrated embodiment, the memory installed in the computing device may be a random access memory (RAM), etc., which is not specifically limited in this specification.

[0037] It should be noted that this specification does not specifically limit the specific type of external memory carried by the computing device. In an embodiment shown, the external memory carried by the computing device can be any of the following: a disk, an optical disk, a USB flash drive, and a mobile hard disk, etc., and this specification does not specifically limit this. In some possible embodiments, the external memory can also be a remote database, a network disk, and a cloud disk connected to the computing device, etc., and this specification does not specifically limit this.

[0038] In one illustrated embodiment, the computing device may be a smart wearable device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a server, or a server cluster consisting of multiple servers, etc., which has the above functions, and this specification does not make specific limitations on this.

[0039] See also Figure 2 , Figure 2 is a flowchart of a matrix calculation method provided by an exemplary embodiment. The method can be applied to Figure 1 The computing device in the system architecture shown in the figure has internal memory and external memory. Figure 2 As shown, the method may specifically include the following steps S201 to S204.

[0040] Step S201, divide a first matrix stored in an external memory of a computing device into a plurality of sub-matrices; wherein the amount of data contained in the first matrix is ​​greater than the storage capacity allocated to the first matrix in the memory of the computing device, and the amount of data contained in each of the plurality of sub-matrices is not greater than the storage capacity allocated to the first matrix in the memory.

[0041] In one illustrated embodiment, based on the current matrix multiplication calculation requirements, the computing device may first obtain the first matrix and the second matrix to be calculated, and determine whether the amount of data contained in the first matrix and the second matrix is ​​greater than the storage capacity that can be allocated in the memory mounted in the computing device.

[0042] In an illustrated embodiment, in response to the amount of data contained in the first matrix being greater than the storage capacity that can be allocated to the first matrix in the memory of the computing device, and may even be greater than the entire memory capacity, the computing device may store the first matrix in the external memory of the computing device; and, in response to the amount of data contained in the second matrix being less than the storage capacity that can be allocated to the second matrix in the memory of the computing device, the computing device may store the second matrix in the memory of the computing device.

[0043] It should be noted that this specification does not specifically limit the application scenarios of matrix multiplication calculations. In some possible implementations, matrix multiplication calculations may be involved in various application scenarios such as face recognition, privacy computing, and large-scale language model (LLM) dialogue. Accordingly, the first matrix and the second matrix can be calculation data related to various application scenarios such as face recognition, privacy computing, and LLM model dialogue, and this specification does not specifically limit this.

[0044] Exemplarily, taking the LLM model dialogue scenario as an example, the computing device is equipped with an LLM model, which can be a pre-trained LLM basic model, or a LLM service model obtained by fine-tuning the pre-trained LLM basic model based on a data set related to the target application scenario, and so on. This specification does not make specific limitations on this.

[0045] In an illustrated embodiment, the user can conduct multiple rounds of dialogue with the LLM model through an input device (such as a keyboard or a touch screen, etc.) provided by the computing device to consult the LLM model about the target application scenario and obtain corresponding answers, etc. In an illustrated embodiment, the above-mentioned target application scenario can be a medical consultation scenario, a life assistant scenario, or a sentiment analysis scenario, etc., which is not specifically limited in this specification.

[0046] It should be understood that most of the existing LLM models are built based on the Transformer architecture. The core of the Transformer architecture is the self-attention mechanism, which allows the model to focus on different parts of the sequence when processing the input sequence, and assign different attention weights according to the correlation between them. Specifically, the self-attention mechanism determines how to weight the input by calculating the similarity between the query (Query, Q), key (Key, K), and value (Value, V). The calculation of the multi-head self-attention layer included in the Transformer architecture involves matrix multiplication calculations related to QKV, and the longer the input sequence of the model, the larger the matrix.

[0047] Below, the matrix multiplication calculation involved in the multi-head self-attention layer contained in the Transformer architecture is taken as an example to explain the first matrix and the second matrix to be calculated in this application.

[0048] In an illustrated embodiment, the computing device may obtain target data input by a user and convert the text content contained in the target data into a target ID sequence. For example, the target data may be data in various formats such as an image, text, a word document, or a PDF. For example, the computing device may convert each word in the text content into a corresponding ID based on a predefined dictionary, thereby obtaining a target ID sequence.

[0049] Furthermore, the computing device may input the target ID sequence into an LLM model based on the Transformer architecture. Accordingly, the LLM model may perform a series of preprocessing on the target ID sequence and then generate a first matrix as an intermediate representation (ie, hidden state (hidden_states)).

[0050] It should be noted that this specification does not particularly limit the specific content of the pre-processing.

[0051] In an illustrated embodiment, a series of preprocessing is performed on the target ID sequence, which may specifically include: embedding the target ID sequence into words through the embedding layer in the Transformer architecture to convert each ID into a corresponding embedding vector; further, in order to retain the position information of the input target ID sequence, positional embeddings may be added to the word embeddings; in addition, the vector may be normalized, etc., which is not specifically limited in this specification.

[0052] In an illustrated embodiment, the first matrix generated after preprocessing the input target ID sequence can be used to represent semantic information and context information contained in the sequence, etc., which is not specifically limited in this specification.

[0053] Furthermore, after the first matrix is ​​generated, the first matrix can be input into the multi-head self-attention layer included in the Transformer architecture for linear transformation, thereby obtaining a query matrix, a key matrix, and a value matrix for attention calculation.

[0054] In one illustrated embodiment, the second matrix in the memory can be a query weight matrix used in the multi-head self-attention layer to perform a linear transformation on the first matrix; correspondingly, the calculation result (for example, the third matrix) obtained after matrix multiplication of the first matrix and the second matrix can be the query matrix used for attention calculation in the multi-head self-attention layer, that is, the Q matrix.

[0055] In one illustrated embodiment, the second matrix in the memory can be a key weight matrix used in the multi-head self-attention layer to perform a linear transformation on the first matrix; correspondingly, the calculation result (for example, the third matrix) obtained after matrix multiplication of the first matrix and the second matrix can be the key matrix used for attention calculation in the multi-head self-attention layer, that is, the K matrix.

[0056] In one illustrated embodiment, the second matrix in the memory can be a value weight matrix used in the multi-head self-attention layer to perform a linear transformation on the first matrix; correspondingly, the calculation result (for example, the third matrix) obtained after matrix multiplication of the first matrix and the second matrix can be the value matrix used for attention calculation in the multi-head self-attention layer, that is, the V matrix.

[0057] In an illustrated embodiment, for example, in a long text conversation scenario, the target data input by the above user may be very large, for example, it may be an insurance policy document of dozens or even hundreds of pages, containing 4 million words, and the user requires the LLM model to analyze the cancer insurance services in the insurance policy. Based on this, the target ID sequence input into the LLM model will also be as long as 4 million IDs. Assuming that the dimension of the query vector / key vector / value vector is 4096, the dimension of the first matrix generated after preprocessing the target sequence ID is 4 million * 4096. If stored in a single-precision floating-point data format, the storage space required for the first matrix is ​​about 61G. Exemplarily, the dimension of the second matrix as a query weight matrix / key weight matrix / value weight matrix can be 4096*4096. If stored in a single-precision floating-point data format, the storage space required for the second matrix is ​​64M.

[0058] Exemplarily, the configuration of the computing device may be 16c32g, i.e., it includes 16 CPU cores and 32G of memory. Obviously, 32G of memory cannot store the first matrix of about 61G, but can store the second matrix of 64M. Based on this, the computing device may store the generated first matrix in the external memory and store the second matrix in the memory.

[0059] Exemplarily, the configuration of the computing device may be 8c16g, i.e., it includes 8 CPU cores and 16G of memory. Obviously, 16G of memory cannot store the first matrix of about 61G, but can store the second matrix of 64M. Based on this, the computing device may store the generated first matrix in external memory and store the second matrix in memory.

[0060] Furthermore, after storing the first matrix and the second matrix, the computing device may perform matrix multiplication calculation on the first matrix and the second matrix.

[0061] In an illustrated embodiment, before performing matrix multiplication calculation, the computing device may first divide the first matrix stored in the external memory into a plurality of sub-matrices, wherein the amount of data contained in each of the plurality of sub-matrices is not greater than the storage capacity that can be allocated to the first matrix in the internal memory. It should be understood that, considering that the second matrix and the intermediate results in the matrix multiplication calculation process also need to be stored in the internal memory, the amount of data contained in the divided sub-matrices should be less than the memory capacity while leaving a margin.

[0062] It should be noted that this specification does not specifically limit the specific implementation method of dividing the sub-matrices.

[0063] In an illustrated embodiment, considering the calculation principle of matrix multiplication calculation, the first matrix stored in the external memory can be divided into a plurality of sub-matrices along the row direction of the first matrix; wherein each sub-matrix contains part of the rows in the first matrix. Specifically, the number of sub-matrices to be divided and the number of rows contained in each sub-matrix can be determined based on the amount of data contained in the first matrix and the storage capacity that can be allocated to the first matrix in the memory. Further, the first matrix stored in the external memory can be divided into a plurality of sub-matrices of the number determined based on the number of rows contained in each sub-matrix along the row direction of the first matrix.

[0064] For example, taking the configuration of the computing device as 16c32g, the storage capacity that can be allocated to the first matrix in the memory is 24G, the dimension of the first matrix is ​​4 million * 4096, and the first matrix is ​​stored in the single-precision floating-point data format, the computing device can divide the first matrix into 10 sub-matrices along the row direction of the first matrix. The dimension of each sub-matrix is ​​400,000 * 4096, and the storage space occupied by each sub-matrix stored in the single-precision floating-point data format is approximately 6.1G, which is less than the 24G storage capacity allocated to the first matrix in the memory, and of course less than the 32G memory capacity of the computing device.

[0065] For example, taking the configuration of the computing device as 4c8g, the storage capacity that can be allocated to the first matrix in the memory is 6G, the dimension of the first matrix is ​​4 million*4096, and the first matrix is ​​stored in the single-precision floating-point data format, the computing device can divide the first matrix into 20 sub-matrices along the row direction of the first matrix. The dimension of each sub-matrix is ​​200,000*4096, and the storage space occupied by each sub-matrix stored in the single-precision floating-point data format is approximately 3G, which is less than the 6G storage capacity allocated to the first matrix in the memory, and of course less than the 8G memory capacity of the computing device.

[0066] For example, taking the configuration of the computing device as 8c16g, the storage capacity that can be allocated to the first matrix in the memory is 8G, the dimension of the first matrix is ​​1 million * 4096, and the first matrix is ​​stored in the single-precision floating-point data format, the computing device can divide the first matrix into 4 sub-matrices along the row direction of the first matrix. The dimension of each sub-matrix is ​​250,000 * 4096, and the storage space occupied by each sub-matrix stored in the single-precision floating-point data format is approximately 4G, which is less than the 8G storage capacity allocated to the first matrix in the memory, and of course less than the 16G memory capacity of the computing device.

[0067] In an illustrated embodiment, in addition to the above-mentioned even division method, an uneven division method may be used to divide the first matrix, that is, the data amounts contained in the divided sub-matrices may also be unequal.

[0068] Exemplarily, taking the configuration of the computing device as 16c32g, the storage capacity that can be allocated to the first matrix in the memory as 24G, and the dimension of the first matrix as 4 million * 4096, the computing device can divide the first matrix into 8 sub-matrices along the row direction of the first matrix, of which 4 sub-matrices can have a dimension of 400,000 * 4096, and each sub-matrix occupies approximately 6.1G of storage space when stored in single-precision floating-point data format; the remaining 4 sub-matrices can have a dimension of 600,000 * 4096, and each sub-matrix occupies approximately 9.2G of storage space when stored in single-precision floating-point data format, which are all less than the 24G storage capacity allocated to the first matrix in the memory, and of course less than the 32G memory capacity of the computing device.

[0069] Exemplarily, taking the configuration of the computing device as 8c16g, the storage capacity that can be allocated to the first matrix in the memory as 12G, and the dimension of the first matrix as 1 million * 4096, the computing device can divide the first matrix into 6 sub-matrices along the row direction of the first matrix, of which 4 sub-matrices can have a dimension of 200,000 * 4096, and each sub-matrix occupies approximately 3G of storage space when stored in single-precision floating-point data format; the remaining 2 sub-matrices can have a dimension of 100,000 * 4096, and each sub-matrix occupies approximately 1.5G of storage space when stored in single-precision floating-point data format, both of which are smaller than the 12G storage capacity allocated to the first matrix in the memory, and of course smaller than the 16G memory capacity of the computing device.

[0070] Step S202: Load the multiple sub-matrices from the external memory into the internal memory in batches, and perform matrix multiplication calculation with the second matrix stored in the internal memory.

[0071] Furthermore, after dividing the first matrix in the external memory into multiple sub-matrices, the computing device can load the multiple sub-matrices from the external memory to the internal memory in batches, and perform matrix multiplication calculations with the second matrix stored in the internal memory, thereby obtaining calculation results of each sub-matrix and the second matrix.

[0072] It should be noted that this specification does not specifically limit the number of sub-matrices included in each batch, and the appropriate number can be determined based on actual conditions and requirements.

[0073] In one illustrated embodiment, if the difference between the amount of data contained in each submatrix and the storage capacity allocated to the first matrix in the memory is small, the computing device can load each of the multiple submatrices from the external memory to the memory in turn, and perform matrix multiplication calculations on each submatrix with the second matrix.

[0074] For example, if the amount of data contained in each submatrix is ​​approximately 6.1G, and the storage capacity allocated to the first matrix in the memory of the computing device is only 8G, each of the multiple submatrices can be loaded from the external memory to the memory in turn, and matrix multiplication calculations can be performed with the second matrix.

[0075] For example, if the amount of data contained in each submatrix is ​​approximately 3G, and the storage capacity allocated to the first matrix in the memory of the computing device is only 4G, each of the multiple submatrices can be loaded from the external memory to the internal memory in turn, and matrix multiplication calculations can be performed with the second matrix.

[0076] In an illustrated embodiment, if there is a large difference between the amount of data contained in the submatrix and the storage capacity allocated to the first matrix in the memory, in order to ensure the computing performance as much as possible, the computing device may load at least two of the multiple submatrices from the external memory into the memory in parallel, and perform matrix multiplication calculations on the at least two submatrices in parallel with the second matrix, respectively. The total amount of data contained in the at least two submatrices is not greater than the storage capacity allocated to the first matrix in the memory.

[0077] Exemplarily, if the amount of data contained in each submatrix is ​​about 3G, and the storage capacity allocated to the first matrix in the memory of the computing device is 32G, then 4 matrices (with a total data amount of 12G) in the multiple submatrices can be loaded from the external memory to the internal memory in parallel, and the 4 submatrices are respectively subjected to matrix multiplication calculations with the second matrix in parallel. Alternatively, 6 matrices (with a total data amount of 18G) in the multiple submatrices can be loaded from the external memory to the internal memory in parallel, and the 6 submatrices are respectively subjected to matrix multiplication calculations with the second matrix in parallel, so as to further improve the computing performance, and this specification does not make specific restrictions on this.

[0078] It should be noted that this specification does not specifically limit the specific implementation method of matrix multiplication calculation.

[0079] In an illustrated embodiment, the computing device may call a predefined linear algebra library, and the linear algebra library may perform matrix multiplication on the submatrix in each batch and the second matrix. Exemplarily, the linear algebra library may be OpenBLAS, or may be NumPy of Python and Eigen of C++, etc., which is not specifically limited in this specification.

[0080] Further, after completing the matrix multiplication calculation of each submatrix and the second matrix, the computing device can store the calculation results of each submatrix and the second matrix in the external memory. Further, the computing device can merge the multiple calculation results in the external memory to obtain a third matrix. It should be understood that the third matrix is ​​the matrix multiplication calculation result of the first matrix and the second matrix.

[0081] Specifically, since the calculation result of each sub-matrix and the second matrix is ​​also a small matrix, the computing device can merge multiple small matrices along the row direction based on the calculation principle of matrix multiplication calculation, so as to obtain the third matrix.

[0082] For example, see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a matrix multiplication calculation provided by an exemplary embodiment. Figure 3 As shown, the dimension of the first matrix is ​​4 million * 4096, the dimension of the second matrix is ​​4096 * 4096, and the dimension of the third matrix finally calculated is 4 million * 4096.

[0083] like Figure 3 As shown, the computing device divides the first matrix into N sub-matrices along the row direction of the first matrix, which are sub-matrix 1, sub-matrix 2, ..., sub-matrix N, where N is an integer greater than 1. Figure 3 As shown, the calculation result obtained after matrix multiplication of each submatrix in the N submatrices with the second matrix is ​​a submatrix in the third matrix; for example, after matrix multiplication of submatrix 1 in the first matrix with the second matrix, calculation result 1 can be obtained, and this calculation result 1 is the first submatrix in the third matrix. For another example, after matrix multiplication of submatrix 2 in the first matrix with the second matrix, calculation result 2 can be obtained, and this calculation result 2 is the first submatrix in the third matrix, and so on. It will not be repeated here. Figure 3 As shown, after the N calculation results obtained by performing matrix multiplication calculations on the N sub-matrices and the second matrix are combined, the final complete calculation result, that is, the third matrix, is obtained.

[0084] As described above, in the present application, when the memory of the computing device cannot directly store the large-scale matrix involved in the matrix multiplication calculation, the large-scale matrix is ​​first stored in the external memory of the computing device and divided into multiple smaller sub-matrices. The multiple sub-matrices are then loaded into the memory in batches and matrix multiplication calculations are performed with another small matrix in the memory. In this way, large-scale matrix calculations can be achieved only through the local external memory of the device, which is simple and efficient without increasing additional equipment costs and network communication costs.

[0085] It should be noted that in addition to the application scenarios of the above-mentioned LLM model based on the Transformer architecture, the matrix calculation method provided in this application can also be applied to various scenarios. For example, in the face recognition scenario, the above-mentioned first matrix can be a large-scale matrix corresponding to the face image information, and the matrix calculation method provided in this application can also be used to implement its matrix multiplication calculation with a smaller second matrix. This specification does not make specific limitations on this.

[0086] In addition, in some possible implementations, the second matrix may also be any sub-matrix of a plurality of sub-matrices obtained by dividing a large-scale matrix, that is, through the matrix calculation method provided in the present application, the matrix multiplication calculation between a large-scale matrix and another large-scale matrix can be implemented based on the external memory of the computing device, and this specification does not make any specific limitations on this.

[0087] Corresponding to the implementation of the above method flow, the embodiment of this specification also provides a matrix calculation device. The device can be applied to Figure 1 The computing device in the system architecture shown in the figure is equipped with an external memory and a memory, the external memory stores a first matrix to be calculated, and the memory stores a second matrix to be calculated; wherein the amount of data contained in the first matrix is ​​greater than the storage capacity allocated to the first matrix in the memory. Figure 4 , Figure 4 FIG. 1 is a schematic diagram of a matrix computing device provided by an exemplary embodiment. Figure 4 As shown, the device 40 includes:

[0088] A matrix division unit 401 is used to divide the first matrix stored in the external memory into a plurality of sub-matrices; wherein the amount of data contained in each of the plurality of sub-matrices is not greater than the storage capacity allocated to the first matrix in the internal memory;

[0089] The matrix calculation unit 402 is used to load the multiple sub-matrices from the external memory into the internal memory in batches, and perform matrix multiplication calculation with the second matrix.

[0090] In an illustrated embodiment, the matrix division unit 401 is specifically configured to:

[0091] The first matrix stored in the external memory is divided into a plurality of sub-matrices along the row direction of the first matrix; wherein each sub-matrix contains a portion of the rows in the first matrix.

[0092] In an illustrated embodiment, the matrix division unit 401 is specifically configured to:

[0093] Determine the number of sub-matrices to be divided and the number of rows contained in each sub-matrix based on the amount of data contained in the first matrix and the storage capacity allocated to the first matrix in the memory;

[0094] Along the row direction of the first matrix, based on the determined number of rows contained in each sub-matrix, the first matrix stored in the external memory is divided into the plurality of sub-matrices.

[0095] In an illustrated embodiment, the matrix calculation unit 402 is specifically configured to:

[0096] Each submatrix of the multiple submatrices is loaded from the external memory to the internal memory in sequence, and matrix multiplication calculation is performed on each submatrix and the second matrix.

[0097] In an illustrated embodiment, the matrix calculation unit 402 is specifically configured to:

[0098] At least two sub-matrices among the multiple sub-matrices are loaded from the external memory to the internal memory in parallel, and matrix multiplication calculations are performed on the at least two sub-matrices in parallel with the second matrix respectively; wherein the total amount of data contained in the at least two sub-matrices is not greater than the storage capacity allocated to the first matrix in the internal memory.

[0099] In an illustrated embodiment, the device 40 further includes a merging unit 403, which is used to:

[0100] The plurality of calculation results of the plurality of sub-matrices and the second matrix are stored in the external memory, and the plurality of calculation results are combined to obtain a third matrix; wherein the third matrix is ​​a matrix multiplication calculation result of the first matrix and the second matrix.

[0101] In an illustrated embodiment, the computing device carries an LLM model based on a Transformer architecture; the apparatus 40 further includes a matrix generation unit 404, which is used to:

[0102] Obtain target data input by a user, and convert text content contained in the target data into a target ID sequence;

[0103] The target ID sequence is input into the LLM model, and the LLM model preprocesses the target ID sequence to generate the first matrix as an intermediate representation.

[0104] In one illustrated embodiment, the second matrix is ​​a query weight matrix used for linearly transforming the intermediate representation in the multi-head self-attention layer included in the Transformer architecture; the third matrix is ​​a query matrix used for attention calculation in the multi-head self-attention layer.

[0105] In one illustrated embodiment, the second matrix is ​​a key weight matrix used for linearly transforming the intermediate representation in the multi-head self-attention layer included in the Transformer architecture; the third matrix is ​​a key matrix used for attention calculation in the multi-head self-attention layer.

[0106] In one illustrated embodiment, the second matrix is ​​a value weight matrix used for linear transformation of the intermediate representation in the multi-head self-attention layer included in the Transformer architecture; the third matrix is ​​a value matrix used for attention calculation in the multi-head self-attention layer.

[0107] In an illustrated embodiment, the matrix calculation unit 402 is specifically configured to:

[0108] A predefined linear algebra calculation library is called, and the linear algebra calculation library performs matrix multiplication calculation on the sub-matrix of each batch and the second matrix.

[0109] The implementation process of the functions and effects of each unit in the above-mentioned device 40 is specifically described in the above-mentioned embodiment, and will not be repeated here. It should be understood that the above-mentioned device 40 can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor (CPU) of the device where it is located reading the corresponding computer program instructions into the memory and running them. From the hardware level, in addition to the CPU and memory, the device where the above-mentioned device is located usually also includes other hardware such as chips for wireless signal transmission and reception, and / or other hardware such as boards and cards for realizing network communication functions.

[0110] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Those of ordinary skill in the art may understand and implement it without creative work.

[0111] The devices, units, and modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, a vehicle-mounted computer, or a combination of any of these devices.

[0112] Corresponding to the above method embodiment, the embodiment of this specification also provides a computing device. Figure 5 , Figure 5 FIG. 1 is a schematic diagram of a computing device provided by an exemplary embodiment. Figure 5 As shown, the computing device includes a processor 1001 and a memory 1002, and may further include an input device 1004 (such as a keyboard, etc.) and an output device 1005 (such as a display, etc.). The processor 1001, the memory 1002, the input device 1004 and the output device 1005 may be connected via a bus or other means. The memory 1002 may include an internal memory and an external memory, wherein the external memory stores a first matrix to be calculated, and the internal memory stores a second matrix to be calculated, and the amount of data contained in the first matrix is ​​greater than the storage capacity allocated to the first matrix in the internal memory. Figure 5 As shown, the memory 1002 includes a computer-readable storage medium 1003, which stores a computer program that can be run by the processor 1001. The processor 1001 can be a CPU, a microprocessor, or an integrated circuit for controlling the execution of the above method embodiment. When the processor 1001 runs the stored computer program, it can execute the various steps of the matrix calculation method in the embodiment of this specification, including: dividing the first matrix stored in the external memory into multiple sub-matrices; wherein the amount of data contained in each of the multiple sub-matrices is not greater than the storage capacity allocated to the first matrix in the memory; loading the multiple sub-matrices from the external memory to the memory in batches, and performing matrix multiplication calculations with the second matrix, and so on.

[0113] For a detailed description of each step of the above matrix calculation method, please refer to the previous content and will not be repeated here.

[0114] Corresponding to the above method embodiment, the embodiment of this specification also provides a computer-readable storage medium, on which a computer program is stored, and when these computer programs are executed by a processor, the various steps of the matrix calculation method in the embodiment of this specification are executed. Please refer to the description of the above embodiment for details, which will not be repeated here.

[0115] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

[0116] In a typical configuration, a computing device includes one or more CPUs, input / output interfaces, network interfaces, and memory.

[0117] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0118] Computer readable media include permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer readable instructions, data structures, program modules or other data.

[0119] Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0120] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0121] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

Claims

1. A matrix calculation method, characterized in that: Applied to a computing device, the computing device is equipped with an external memory and a internal memory, the external memory stores a first matrix to be calculated, and the internal memory stores a second matrix to be calculated; wherein the amount of data contained in the first matrix is ​​greater than the storage capacity allocated to the first matrix in the internal memory; the method comprises: Divide the first matrix stored in the external memory into a plurality of sub-matrices; wherein the amount of data contained in each of the plurality of sub-matrices is not greater than the storage capacity allocated to the first matrix in the internal memory; The plurality of sub-matrices are loaded from the external memory into the internal memory in batches, and matrix multiplication calculations are performed with the second matrix.

2. The method according to claim 1, characterized in that The step of dividing the first matrix stored in the external memory into a plurality of sub-matrices comprises: The first matrix stored in the external memory is divided into a plurality of sub-matrices along the row direction of the first matrix; wherein each sub-matrix contains a portion of the rows in the first matrix.

3. The method according to claim 2, characterized in that The first matrix stored in the external memory is divided into a plurality of sub-matrices along the row direction of the first matrix, including: Determine the number of sub-matrices to be divided and the number of rows contained in each sub-matrix based on the amount of data contained in the first matrix and the storage capacity allocated to the first matrix in the memory; Along the row direction of the first matrix, based on the determined number of rows contained in each sub-matrix, the first matrix stored in the external memory is divided into the plurality of sub-matrices.

4. The method according to claim 1, characterized in that: The step of loading the plurality of sub-matrices from the external memory to the internal memory in batches and performing matrix multiplication calculation with the second matrix includes: At least two sub-matrices among the multiple sub-matrices are loaded from the external memory to the internal memory in parallel, and matrix multiplication calculations are performed on the at least two sub-matrices in parallel with the second matrix respectively; wherein the total amount of data contained in the at least two sub-matrices is not greater than the storage capacity allocated to the first matrix in the internal memory.

5. The method according to claim 1, characterized in that The method further comprises: The plurality of calculation results of the plurality of sub-matrices and the second matrix are stored in the external memory, and the plurality of calculation results are combined to obtain a third matrix; wherein the third matrix is ​​a matrix multiplication calculation result of the first matrix and the second matrix.

6. The method according to claim 1, characterized in that The computing device is equipped with an LLM model based on the Transformer architecture; the method further includes: Obtain target data input by a user, and convert text content contained in the target data into a target ID sequence; The target ID sequence is input into the LLM model, and the LLM model preprocesses the target ID sequence to generate the first matrix as an intermediate representation.

7. The method according to claim 6, characterized in that The second matrix is ​​a query weight matrix used for linear transformation of the intermediate representation in the multi-head self-attention layer included in the Transformer architecture; the third matrix is ​​a query matrix used for attention calculation in the multi-head self-attention layer.

8. The method according to claim 6, characterized in that The second matrix is ​​a key weight matrix used for linear transformation of the intermediate representation in the multi-head self-attention layer included in the Transformer architecture; the third matrix is ​​a key matrix used for attention calculation in the multi-head self-attention layer.

9. The method according to claim 6, characterized in that The second matrix is ​​a value weight matrix used for linear transformation of the intermediate representation in the multi-head self-attention layer included in the Transformer architecture; the third matrix is ​​a value matrix used for attention calculation in the multi-head self-attention layer.

10. The method according to any one of claims 1 to 9, characterized in that: The performing matrix multiplication calculation with the second matrix comprises: A predefined linear algebra calculation library is called, and the linear algebra calculation library performs matrix multiplication calculation on the sub-matrix of each batch and the second matrix.

11. A matrix computing device, characterized in that: Applied to a computing device, the computing device is equipped with an external memory and a internal memory, the external memory stores a first matrix to be calculated, and the internal memory stores a second matrix to be calculated; wherein the amount of data contained in the first matrix is ​​greater than the storage capacity allocated to the first matrix in the internal memory; the device comprises: A matrix division unit, used for dividing the first matrix stored in the external memory into a plurality of sub-matrices; wherein the amount of data contained in each of the plurality of sub-matrices is not greater than the storage capacity allocated to the first matrix in the internal memory; A matrix calculation unit is used to load the multiple sub-matrices from the external memory into the internal memory in batches, and perform matrix multiplication calculation with the second matrix.

12. A computing device, characterized in that: include: Memory and processor; The memory has stored thereon computer programs / instructions executable by the processor; When the processor runs the computer program / instructions, it performs the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images