Model calculation method and related device

Abstract

The embodiment of the present invention provides a model calculation method and related devices, belonging to the field of artificial intelligence technology. The method includes: dividing the model parameter matrix of the target machine learning model into multiple sub-matrices according to the video memory capacity of the physical device; determining the target sub-matrix to be calculated according to the importance levels corresponding to the multiple sub-matrices; determining the target calculation unit from the multiple calculation units according to the real-time load information corresponding to the multiple calculation units contained in the physical device; loading the target sub-matrix into the video memory of the physical device, so that the target calculation unit performs the matrix calculation task corresponding to the target sub-matrix based on the target sub-matrix loaded in the video memory. The method divides the model parameter matrix into sub-matrices, and performs sub-matrix calculations in sequence according to the importance levels of the sub-matrices. The amount of data of the sub-matrices loaded in sequence in the video memory is smaller than that of the model parameter matrix, which can effectively solve the problems of high video memory occupancy and insufficient computing resources.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model calculation method and related devices. Background Art

[0002] In machine learning, for various machine learning models, it is usually necessary to save model parameters and use the saved model parameters to perform matrix calculations. As the scale of machine learning models continues to increase, the number of model parameters for a single machine learning model is also increasing. Therefore, in the process of saving model parameters through a single physical device and using the saved model parameters to perform matrix calculations, the physical device has high video memory usage and insufficient computing resources, resulting in a slow model inference speed. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to provide a model calculation method and related devices, aiming to solve the problems of high video memory occupancy of the physical device, insufficient computing resources provided, and slow model reasoning speed in the process of saving model parameters through a single physical device and using the saved model parameters for matrix calculation.

[0004] In a first aspect, an embodiment of the present invention provides a model calculation method, including:

[0005] Divide the model parameter matrix of the target machine learning model into multiple sub-matrices according to the video memory capacity of the physical device;

[0006] Determine the importance levels respectively corresponding to the multiple sub-matrices;

[0007] Determine a target submatrix to be calculated according to the importance levels respectively corresponding to the multiple submatrices, wherein the target submatrix is ​​a submatrix that has not been subjected to matrix calculation among the multiple submatrices, and the importance level of the target submatrix is ​​higher than the importance levels of other submatrices that have not been subjected to matrix calculation;

[0008] Acquire real-time load information corresponding to each of the plurality of computing units included in the physical device;

[0009] Determine a target computing unit from the multiple computing units according to the real-time load information respectively corresponding to the multiple computing units, wherein the current load of the target computing unit is lower than the current loads of other computing units;

[0010] The target sub-matrix is ​​loaded into the video memory of the physical device, so that the target calculation unit performs a matrix calculation task corresponding to the target sub-matrix based on the target sub-matrix loaded into the video memory.

[0011] In a second aspect, an embodiment of the present invention provides a model calculation device, including:

[0012] A matrix partitioning module, used to partition the model parameter matrix of the target machine learning model into multiple sub-matrices according to the video memory capacity of the physical device;

[0013] A determination module is used to determine the importance levels respectively corresponding to the multiple sub-matrices; determine the target sub-matrix to be calculated currently according to the importance levels respectively corresponding to the multiple sub-matrices, wherein the target sub-matrix is ​​a sub-matrix in the multiple sub-matrices that has not been subjected to matrix calculation, and the importance level of the target sub-matrix is ​​higher than the importance levels of other sub-matrices that have not been subjected to matrix calculation; obtain the real-time load information respectively corresponding to the multiple computing units included in the physical device; determine the target computing unit from the multiple computing units according to the real-time load information respectively corresponding to the multiple computing units, and the current load of the target computing unit is lower than the current load of other computing units;

[0014] The task calculation module is used to load the target sub-matrix into the video memory of the physical device, so that the target calculation unit performs the matrix calculation task corresponding to the target sub-matrix based on the target sub-matrix loaded into the video memory.

[0015] In a third aspect, an embodiment of the present invention further provides a physical device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of any model calculation method provided in the specification of the present invention are implemented.

[0016] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any model calculation method provided in the specification of the present invention.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor coupled to a transceiver, and is used to execute the model calculation method provided in the first aspect of the embodiment of the present application. In a possible design, the chip can also be a dedicated hardware structure for implementing the model calculation method provided in the first aspect above, for example, the processing involving the neural network model can be implemented by a dedicated neural network processor or a graphics processor.

[0018] In a sixth aspect, an embodiment of the present application provides a chip system, which includes a processor for implementing the functions involved in the above-mentioned first aspect, for example, generating or processing the information involved in the model calculation method provided in the above-mentioned first aspect.

[0019] In a possible design, the chip system also includes a memory, which is connected to the processor through a circuit structure. The memory is used to store program instructions and data required by the terminal. The chip system can be composed of chips, or it can include chips and other discrete devices. Further optionally, the chip also includes a communication interface, and the processor is connected to the communication interface. The communication interface is used to receive data and / or information to be processed, the processor obtains the data and / or information from the communication interface, processes the data and / or information, and outputs the processing results through the communication interface. The communication interface can be an input and output interface.

[0020] The embodiment of the present invention provides a model calculation method and related devices, the method comprising: dividing the model parameter matrix of the target machine learning model into multiple sub-matrices according to the video memory capacity of the physical device; determining the importance levels corresponding to the multiple sub-matrices; determining the target sub-matrix to be calculated currently according to the importance levels corresponding to the multiple sub-matrices, the target sub-matrix is ​​a sub-matrix that has not been subjected to matrix calculation among the multiple sub-matrices, and the importance level of the target sub-matrix is ​​higher than the importance level of other sub-matrices that have not been subjected to matrix calculation; obtaining the real-time load information corresponding to the multiple computing units contained in the physical device; determining the target computing unit from the multiple computing units according to the real-time load information corresponding to the multiple computing units, the current load of the target computing unit is lower than the current load of other computing units; loading the target sub-matrix into the video memory of the physical device, so that the target computing unit performs the matrix calculation task corresponding to the target sub-matrix based on the target sub-matrix loaded in the video memory. The method divides the model parameter matrix into sub-matrices, so that the matrix calculation task corresponding to the model parameter matrix is ​​divided into multiple different sub-tasks, that is, the matrix calculation tasks corresponding to the multiple sub-matrices. Thus, when performing matrix calculations, only the submatrix that needs to be calculated is loaded into the video memory, rather than the full model parameter matrix, which can effectively solve the problem of high video memory usage; because the calculation unit performs the matrix calculation task corresponding to the submatrix, rather than the full calculation task corresponding to the model parameter matrix, it can not only ensure the smooth execution of the matrix calculation task, but also avoid the problem of insufficient computing resources. The model calculation method provided by the embodiment of the present invention utilizes the limited video memory space and computing resources in the physical device, and can run a machine learning model with more parameters without modifying the model framework and without sacrificing model performance, solving the problems of high video memory usage of the physical device, insufficient computing resources provided, and slow model reasoning speed in the process of saving model parameters through a single physical device and using the saved model parameters for matrix calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A schematic diagram of a flow chart of a model calculation method provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a partitioning model parameter matrix provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the module structure of a model calculation device provided by an embodiment of the present invention;

[0025] Figure 4 A schematic block diagram of the structure of a physical device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0028] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0029] The embodiment of the present invention provides a model calculation method and related devices. The model calculation method can be applied to a physical device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The physical device can be a server or a server cluster.

[0030] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0031] In machine learning, for various machine learning models (hereinafter referred to as models), such as Convolutional Neural Network (CNN) models, Recurrent Neural Network (RNN) models, Large Language Models (LLMs), etc., it is usually necessary to save model parameters and perform matrix calculations using the saved model parameters. The matrix calculations using the saved model parameters can occur in the model training stage or in the model use stage.

[0032] As the scale of machine learning models continues to grow, the number of model parameters for a single machine learning model is also increasing. Therefore, in the process of saving model parameters through a single physical device and using the saved model parameters for matrix calculation, the physical device's video memory usage is high and the computing resources provided are insufficient, resulting in a slow model inference speed.

[0033] In response to at least one of the above technical problems, in the related technology, the preservation of model parameters and the matrix calculation process based on the model parameters are optimized through various methods such as model parallelism, data parallelism, and hybrid parallelism, so as to solve the problems of high video memory occupancy of physical devices and insufficient computing resources provided.

[0034] Model parallelism refers to allocating different layers or parts of a model to multiple different physical devices for parallel computing, that is, deploying the model in a distributed manner. In model parallelism, using the storage resources and computing resources of multiple different physical devices can solve the problem of insufficient storage resources and computing resources of a single physical device. However, model parallelism usually requires adjustments to the model architecture, which is highly complex to implement, and multiple devices need to communicate frequently, resulting in high communication overhead, which is not conducive to improving the efficiency of model computing.

[0035] Data parallelism refers to copying the same model to multiple physical devices, and then performing matrix calculations on different data through different physical devices. Copying the same model to multiple physical devices includes copying the model parameters of the same model to multiple physical devices. The data parallel approach is applicable to most models and does not require adjustments to the model architecture. However, it cannot solve the problem of too many model parameters. When there are too many model parameters, the problem of excessive physical device memory usage will still occur. When the physical device's video memory capacity cannot meet the needs of storing model parameters, the video memory will frequently exchange data with the internal memory to read the model parameters from the internal memory, thereby affecting the model's computational efficiency.

[0036] Hybrid parallelism refers to the combination of model parallelism and data parallelism, which not only distributes different layers or parts of the model to multiple different physical devices, but also performs matrix calculations on different data through different physical devices. However, hybrid parallelism, like model parallelism, has the problems of high complexity in adjusting the model architecture and high communication overhead.

[0037] In addition, the above three parallel strategies also have the problem of poor flexibility and lack of dynamic adjustment of the use of computing resources. Specifically, the objects processed by the same model in actual use may be different. For example, in a certain model reasoning task, the processing object is text, while in another model reasoning task, the processing object is a picture. It can be understood that the processing objects are different, and the model's demand for computing resources is also different. However, in the above three parallel strategies, since the parallel strategy has been pre-selected, the computing resources in the corresponding physical device are only used to process the matrix calculation tasks assigned to it. Therefore, when the computing resources required to execute the matrix calculation tasks corresponding to the model change dynamically with the processing objects, the computing resources in each physical device will inevitably have the problem of computing resource redundancy, which will cause a waste of computing resources.

[0038] In order to solve at least one of the above technical problems, an embodiment of the present invention provides a model calculation method, in which the model parameter matrix of the model is split and the scheduling of the computing resources corresponding to the multiple computing units contained in the physical device is optimized, so as to reduce the memory usage when storing the model parameters in a single physical device and make full use of the computing resources when performing matrix calculations without sacrificing the model performance. In short, under the condition of ensuring the same model performance, the same physical device can run models with more model parameters using the model calculation method provided by the embodiment of the present invention.

[0039] The model calculation method provided by the embodiment of the present invention is described below.

[0040] Please refer to Figure 1 , Figure 1A schematic flow chart of a model calculation method provided in an embodiment of the present invention.

[0041] like Figure 1 As shown, the model calculation method includes steps S101 to S106.

[0042] Step S101: divide the model parameter matrix of the target machine learning model into multiple sub-matrices according to the video memory capacity of the physical device.

[0043] In actual applications, after the sample data is input into the target machine learning model, the target machine learning model running in the physical device performs corresponding matrix operations based on the model parameter matrix to generate model output. The sample data can be training sample data in the model training process or actual sample data in the model use process.

[0044] Since the amount of data in the model parameter matrix is ​​usually large, when the model parameter matrix is ​​loaded into the video memory or matrix calculation is performed as a whole, the problem of high video memory occupancy and insufficient computing resources is likely to occur. For this reason, in an embodiment of the present invention, before performing the matrix operation corresponding to the model parameter matrix, the model parameter matrix is ​​firstly processed in blocks, that is, the model parameter matrix of the target machine learning model is divided into multiple sub-matrices. By sequentially loading the video memory and performing matrix calculation on each sub-matrix, the problem of high video memory occupancy and insufficient computing resources can be effectively solved.

[0045] To facilitate understanding, we first briefly explain the data flow during the model operation process. If you want to run a machine learning model on a physical device, generally, you need to first load the model matrix parameters of the model into the memory of the physical device; when performing matrix calculations, first load the model parameter matrix in the memory into the video memory of the physical device; then, the computing unit in the physical device loads the model parameter matrix in the video memory into the register of the computing unit to perform matrix calculations in the register; finally, store the calculation results of the matrix calculation in the cache of the physical device.

[0046] In an optional embodiment, in the process of dividing the model parameter matrix into a plurality of sub-matrices, the data volume of each sub-matrix in the plurality of sub-matrices is less than or equal to the video memory capacity of the physical device.

[0047] It is understandable that when the data volume of a single sub-matrix is ​​larger than the video memory capacity of the physical device, since the video memory capacity is insufficient to accommodate the data volume of the entire sub-matrix at one time, the video memory needs to perform multiple data transfers with the internal memory to load the entire single sub-matrix into the video memory. This will inevitably lead to the problem of high video memory occupancy, and frequent data transfers between the video memory and the internal memory will also reduce the efficiency of matrix calculations.

[0048] When the data volume of a single submatrix is ​​less than or equal to the memory capacity of the physical device, the memory capacity is sufficient to accommodate the data volume of the entire submatrix at one time. The memory and the internal memory perform a data transfer once to load the entire submatrix into the memory. It can be seen that dividing the model parameter matrix into multiple submatrices with data volumes less than or equal to the memory capacity of the physical device can help solve the problem of high memory usage and improve the efficiency of matrix calculation.

[0049] To make it easier to understand, let’s take an example. Figure 2 A schematic diagram of a partitioning model parameter matrix provided by an embodiment of the present invention. Figure 2 As shown, the model parameter matrix W is a matrix with m rows and n columns. Based on the memory capacity of the physical device, it can be divided into p×q sub-matrices, where any sub-matrix Wij is a matrix with m / q rows and n / q columns. The amount of data corresponding to the sub-matrix Wij is less than or equal to the memory capacity of the physical device, i is the row index of the sub-matrix, and j is the column index of the sub-matrix.

[0050] Optionally, in the process of dividing the model parameter matrix, the model parameter matrix may be divided according to a row priority order, or may be divided according to a column priority order, which is not limited in this embodiment.

[0051] In another optional embodiment, in order to fully utilize the video memory capacity of the physical device, the model parameter matrix of the target machine learning model is divided into multiple sub-matrices according to the video memory capacity of the physical device, including:

[0052] According to multiple partitioning strategies, the model parameter matrix of the target machine learning model is partitioned to obtain multiple groups of sub-matrices corresponding to the multiple partitioning strategies, wherein each group of sub-matrices contains multiple sub-matrices, and the data volume of sub-matrices in different groups is different, and the data volume of each sub-matrix is ​​less than or equal to the video memory capacity of the physical device;

[0053] For any group of sub-matrices among the multiple groups of sub-matrices, determine the total amount of data of the sub-matrices that can be accommodated by the video memory capacity of the physical device;

[0054] A target group submatrix is ​​determined from multiple groups of submatrices, so that the multiple submatrices contained in the target group submatrix are used as the submatrix partitioning result of the model parameter matrix of the target machine learning model, and the total amount of data corresponding to the target group submatrix is ​​greater than the total amount of data corresponding to other groups of submatrices.

[0055] To elaborate, according to a variety of blocking strategies, the model parameter matrix of the target machine learning model is subjected to matrix partitioning processing respectively. This can be understood as performing matrix partitioning processing on the model parameter matrix of the target machine learning model according to different sub-matrix data amounts or different sub-matrices.

[0056] For example, multiple block strategies may include: Block strategy 1: Figure 2 The model parameter matrix W in is divided into sub-matrices with data size of s×h; Block strategy 2: Divide the model parameter matrix W into sub-matrices with data size of s'×h', etc. Among them, s and s' are less than m, and h and h' are less than n.

[0057] For another example, multiple block strategies may include: Block strategy 3: Figure 2 The model parameter matrix W in is evenly divided into x sub-matrices; Block strategy 4: The model parameter matrix W is evenly divided into y sub-matrices, etc., where x is not equal to y.

[0058] It is understandable that for any of the above-mentioned multiple block strategies, the model parameter matrix W can be divided into multiple sub-matrices. For the convenience of distinction, in this embodiment, multiple sub-matrices obtained by division based on a block strategy are grouped together, so that multiple groups of sub-matrices can be obtained by dividing the model parameter matrix W through multiple block strategies. Since different groups of sub-matrices correspond to different block strategies, the data amounts of the sub-matrices of different groups are different.

[0059] Since the data amounts of different groups of sub-matrices are different, the number of sub-matrices that can be accommodated at one time by a fixed-capacity video memory is different for different groups of sub-matrices, that is, the total amount of sub-matrix data that can be accommodated at one time by the video memory capacity of the physical device is different.

[0060] In this embodiment, for any group of sub-matrices, the sum of the data amounts corresponding to the sub-matrices that can be accommodated by the video memory capacity of the physical device at one time is determined respectively, so as to obtain multiple sums of data amounts corresponding to multiple groups of sub-matrices. Then, the largest sum of data amounts is selected from the multiple sums of data amounts, and the target group sub-matrix corresponding to the largest sum of data amounts is used as the sub-matrix partition result of the model parameter matrix.

[0061] For example, for a group of sub-matrices corresponding to block strategy 1, assuming that the video memory can accommodate 3 sub-matrices at a time, the total amount of data of the sub-matrices that the video memory capacity can accommodate at one time is 3*(s×h); for a group of sub-matrices corresponding to block strategy 2, assuming that the video memory can accommodate 2 sub-matrices at a time, the total amount of data of the sub-matrices that the video memory capacity can accommodate at one time is 2*(s'×h'). If the total amount of data 3*(s×h) is greater than the total amount of data 2*(s'×h'), the group of sub-matrices corresponding to block strategy 1 is determined to be the target sub-matrix, and the multiple sub-matrices contained in the target group of sub-matrices are used as the sub-matrix partitioning result of the model parameter matrix of the target machine learning model, that is, the multiple sub-matrices contained in the group of sub-matrices corresponding to block strategy 1 are used as the sub-matrix partitioning result of the model parameter matrix.

[0062] Step S102: Determine the importance levels corresponding to the multiple sub-matrices.

[0063] It is understandable that if multiple sub-matrices are processed for matrix calculation without distinguishing between the primary and the secondary, it may cause the video memory, computing resources or cache to be occupied by unimportant sub-matrices first, while important sub-matrices are frequently read and written, which will inevitably reduce the computing efficiency.

[0064] Therefore, after determining the multiple sub-matrices obtained by dividing the model parameter matrix, in order to ensure that the multiple sub-matrices are calculated efficiently and orderly, that is, to realize pipeline parallel calculation of the multiple sub-matrices, in this embodiment, by determining the importance levels corresponding to the multiple sub-matrices, the multiple sub-matrices are calculated in sequence according to the importance levels.

[0065] In an optional embodiment, determining the importance levels corresponding to the plurality of sub-matrices respectively includes:

[0066] For any submatrix among the multiple submatrices, calculate the gradient amplitude of each matrix element in the any submatrix relative to the loss function of the target machine learning model to determine the target gradient amplitude corresponding to the any submatrix;

[0067] According to the target gradient amplitudes respectively corresponding to the multiple sub-matrices, the importance levels respectively corresponding to the multiple sub-matrices are determined.

[0068] The calculation process of the gradient amplitude may refer to the relevant technology, which will not be described in detail in this implementation.

[0069] Optionally, for any submatrix among the multiple submatrices, the gradient amplitude of each matrix element relative to the loss function of the target machine learning model during multiple rounds of training can be obtained; then, the average of the multiple gradient amplitudes corresponding to the same matrix element during multiple rounds of training is taken as the gradient amplitude of the element, thereby determining the target gradient amplitude corresponding to any submatrix.

[0070] In machine learning, the larger the gradient magnitude (or absolute value of the gradient) of a matrix element, the greater the influence of the matrix element on the loss function. Therefore, the larger the target gradient magnitude corresponding to the submatrix, the higher the importance level of the submatrix.

[0071] Step S103: Determine the target submatrix to be calculated according to the importance levels corresponding to the multiple submatrices, the target submatrix being a submatrix that has not been calculated among the multiple submatrices, and the importance level of the target submatrix is ​​higher than the importance levels of other submatrices that have not been calculated.

[0072] When the video memory capacity and computing resources are limited, the limited resources are allocated to the sub-matrices with high importance first, which can ensure that the key matrix calculations are completed as early as possible, reduce the waiting time for subsequent calculations, and improve the overall matrix calculation efficiency.

[0073] Step S104: Acquire real-time load information corresponding to each of the plurality of computing units included in the physical device.

[0074] Among them, the physical device includes multiple computing units, each computing unit is configured with corresponding computing resources, which can be used for matrix operations of sub-matrices. The physical device can monitor the real-time load information of each computing unit, that is, the actual usage of the computing resources of each computing unit.

[0075] Step S105 : determining a target computing unit from the multiple computing units according to the real-time load information corresponding to the multiple computing units respectively, wherein the current load of the target computing unit is lower than the current loads of other computing units.

[0076] Step S106: Load the target sub-matrix into the video memory of the physical device, so that the target computing unit performs the matrix computing task corresponding to the target sub-matrix based on the target sub-matrix loaded into the video memory.

[0077] In this embodiment, in order to ensure that the sub-matrices with higher importance levels are calculated first, a computing unit with lower current load is selected to perform the matrix calculation task corresponding to the sub-matrix with higher importance level.

[0078] In an optional embodiment, a submatrix queue and a computing unit queue may be pre-established. In the submatrix queue, multiple submatrices are arranged in descending order according to the importance level of the submatrix; in the computing unit queue, multiple computing units are arranged in descending order according to the real-time load of the computing unit. The submatrix queue stores submatrices that have not been subjected to matrix calculation.

[0079] During matrix calculation of the submatrix, the first submatrix in the submatrix queue is determined as the target submatrix to be calculated currently, and the first computing unit in the computing unit queue is determined as the target computing unit for executing the matrix calculation task corresponding to the target submatrix.

[0080] It should be noted that after the target sub-matrix is ​​loaded into the video memory of the physical device, the target sub-matrix is ​​removed from the sub-matrix queue to update the sub-matrix queue; in the process of the target computing unit executing the matrix calculation task corresponding to the target sub-matrix, the order of multiple computing units in the computing unit queue is updated based on the real-time load of the target computing unit.

[0081] Based on the above example process, sub-matrices of different importance levels can be assigned to corresponding computing units in sequence, and matrix computing tasks corresponding to multiple sub-matrices can be executed in sequence by multiple computing units. Among them, multiple computing units can simultaneously execute matrix computing tasks corresponding to different sub-matrices, and the same computing unit can also simultaneously execute matrix computing tasks corresponding to multiple different sub-matrices.

[0082] After executing the matrix calculation tasks corresponding to multiple sub-matrices through multiple computing units, the calculation results of the matrix calculation tasks corresponding to the multiple sub-matrices can be merged to determine the output result of the target machine learning model.

[0083] During the specific implementation process, after completing the matrix calculation task of the corresponding sub-matrix, each computing unit can store the corresponding calculation results in the cache and wait for merging; after the calculation results corresponding to multiple sub-matrices are obtained, the calculation results corresponding to the multiple sub-matrices in the cache are spliced ​​into a complete matrix according to the block order of the model parameter matrix, as the output result of the target machine learning model.

[0084] In the model calculation method provided by the embodiment of the present invention, by dividing the model parameter matrix into sub-matrices, the matrix calculation task corresponding to the model parameter matrix is ​​divided into multiple different sub-tasks, that is, the matrix calculation tasks corresponding to the multiple sub-matrices. Therefore, when performing matrix calculation, only the sub-matrix that needs to be calculated is loaded into the video memory, rather than the full model parameter matrix, which can effectively solve the problem of high video memory occupancy; because the calculation unit executes the matrix calculation task corresponding to the sub-matrix, rather than all the calculation tasks corresponding to the model parameter matrix, it can not only ensure the smooth execution of the matrix calculation task, but also avoid the problem of insufficient computing resources. In addition, when the video memory capacity and computing resources are limited, the limited resources are preferentially allocated to the sub-matrix with high importance, which can ensure that the key matrix calculation is completed as soon as possible, reduce the waiting time of subsequent calculations, and improve the overall matrix calculation efficiency. The model calculation method provided in the embodiment of the present invention utilizes the limited video memory space and computing resources in the physical device, and can run a machine learning model with more parameters without modifying the model framework and without sacrificing model performance. This solves the problem of high video memory occupancy of the physical device, insufficient computing resources provided, and slow model inference speed in the process of saving model parameters through a single physical device and performing matrix calculations using the saved model parameters.

[0085] It is understandable that the matrix calculation of the model parameter matrix requires, in addition to the model parameter matrix itself, the input data of the model and various weight parameters, etc. In this embodiment, the input data and weight parameters, etc. are collectively referred to as data blocks for matrix calculation.

[0086] In an optional embodiment, in the above step S106, the target sub-matrix is ​​loaded into the video memory of the physical device, so that the target computing unit performs the matrix computing task corresponding to the target sub-matrix based on the target sub-matrix loaded into the video memory, specifically including:

[0087] The target submatrix in the video memory and the data block for matrix calculation of the target submatrix in the cache of the physical device are loaded into the register of the target calculation unit, so that the target calculation unit performs the matrix calculation task corresponding to the target submatrix based on the target submatrix and the data block loaded in the register.

[0088] The data block in the cache of the physical device for matrix calculation of the target sub-matrix is ​​read from the memory of the physical device.

[0089] In practical applications, during the matrix calculation process of multiple sub-matrices, the calculation result of one sub-matrix may be used for the matrix calculation of other sub-matrices, and the same data block may be used for the matrix calculation of multiple sub-matrices. If these calculation results or data blocks can be reused, data transmission can be effectively reduced and the efficiency of matrix calculation can be improved.

[0090] As an optional reuse method, the calculation results of the matrix calculation tasks corresponding to the target submatrix can be stored in the cache of the physical device; if the matrix calculation tasks corresponding to other submatrices in the multiple submatrices need to use the calculation results, the calculation units used to execute the matrix calculation tasks corresponding to the other submatrices read the calculation results from the cache.

[0091] As another optional reuse method, a target data block shared by matrix calculation tasks corresponding to multiple sub-matrices can be determined; if there is a corresponding matrix calculation task in the sub-matrix that does not perform matrix calculation and needs to use a sub-matrix of the target data block, the target data block is retained in the cache of the physical device; if there is no corresponding matrix calculation task in the sub-matrix that does not perform matrix calculation and needs to use a sub-matrix of the target data block, the target data block is deleted from the cache to release storage space in the cache. Through this solution, data transmission between the cache and the memory can be effectively reduced, and there is no need to repeatedly load the target data block in the memory into the cache.

[0092] In practical applications, during the matrix calculation process of a submatrix, the matrix elements in the submatrix may be used multiple times. For example, when performing matrix multiplication, a column of matrix elements in the submatrix may need to be multiplied with multiple rows of matrix elements, so the column of matrix elements needs to be accessed multiple times. If the matrix elements in the submatrix can be reused, the data transmission between the register and the video memory can be reduced, that is, the same matrix elements do not need to be loaded from the video memory to the register multiple times, thereby improving the efficiency of matrix calculation.

[0093] As an optional reuse method, the target matrix elements in the target submatrix that participate in the matrix calculation tasks corresponding to the target submatrix multiple times can be determined; after the target matrix elements are loaded into the register, if some of the matrix calculation tasks corresponding to the target submatrix need to use the target matrix elements, the target matrix elements are retained in the register; if the matrix calculation tasks corresponding to the target submatrix do not need to use the target matrix elements, the target matrix elements are deleted from the register to release storage space in the register.

[0094] The above describes how to improve the matrix calculation efficiency in the model calculation method provided by the embodiment of the present invention from the dimension of data reuse. The following describes how to improve the matrix calculation efficiency from the dimension of preloading sub-matrices.

[0095] After the computing unit completes the matrix calculation task corresponding to the currently assigned sub-matrix, the computing resources will be released so that the matrix calculation tasks corresponding to other sub-matrices can continue to be calculated. It can be understood that if the computing unit loads data for the next sub-matrix to be executed after completing the matrix calculation task corresponding to the currently assigned sub-matrix (including: loading the sub-matrix from the memory to the video memory, and then loading it from the video memory to the register; and loading the data block used for the sub-matrix calculation from the memory to the cache, etc.), there will be a period of data loading time between the completed matrix calculation task and the matrix calculation task that is about to start. During this period of data loading time, the computing resources of the computing unit are not utilized, which will cause a waste of computing resources on the one hand, and reduce the efficiency of matrix calculation due to waiting for data loading on the other hand.

[0096] In order to avoid wasting computing resources and improve the efficiency of matrix calculation, the model calculation method provided in the embodiment of the present invention takes the target computing unit as an example to illustrate how to preload the sub-matrix to be calculated corresponding to the computing unit when the computing unit executes the matrix calculation task corresponding to the sub-matrix.

[0097] In an optional embodiment, preloading the sub-matrix to be calculated includes: determining the sub-matrix to be calculated for which the target computing unit performs matrix calculation after the target sub-matrix; loading the sub-matrix to be calculated into the video memory of the physical device during the process of the target computing unit executing the matrix calculation task corresponding to the target sub-matrix; loading the sub-matrix to be calculated from the video memory into the register of the target computing unit, so that after the target computing unit completes the matrix calculation task corresponding to the target sub-matrix, the target computing unit executes the matrix calculation task corresponding to the sub-matrix to be calculated based on the sub-matrix to be calculated loaded in the register.

[0098] In another optional embodiment, preloading the sub-matrix to be calculated includes: determining the sub-matrix to be calculated for which the target computing unit performs matrix calculation after the target sub-matrix; and loading the data blocks used for calculating the sub-matrix to be calculated from the memory to the cache during the process of the target computing unit executing the matrix calculation task corresponding to the target sub-matrix, so that after the target computing unit completes the matrix calculation task corresponding to the target sub-matrix, the target computing unit executes the matrix calculation task corresponding to the sub-matrix to be calculated based on the data blocks used for calculating the sub-matrix to be calculated in the cache.

[0099] Optionally, the submatrix to be calculated can be determined from the submatrix that has not undergone matrix calculation based on the estimated computing resources that can be released by the target computing unit after executing the matrix calculation task corresponding to the target submatrix, and the importance level corresponding to the submatrix that has not undergone matrix calculation.

[0100] In this solution, by preloading the sub-matrix to be calculated and the calculation data block for the sub-matrix to be calculated, the degradation of calculation performance caused by data loading delay in the matrix calculation process can be reduced.

[0101] In the process of performing matrix calculation on sub-matrices through multiple computing units of a physical device, the computing units may be overloaded. To ensure the smooth implementation of the matrix calculation task, the model calculation method of this embodiment provides a dynamic adjustment strategy for the computing tasks to balance the load of multiple computing units.

[0102] Taking the target computing unit as an example, the computing task dynamic adjustment strategy may include:

[0103] Obtain the real-time computing resource usage indicator of the target computing unit during the process of executing the matrix computing task corresponding to the target sub-matrix.

[0104] The real-time computing resource usage indicators include: the number of matrix computing tasks currently executed by the target computing unit, the time consumption of the currently executed matrix computing tasks, and the speed of completing the matrix computing tasks.

[0105] If the real-time computing resource usage indicator indicates that the load of the target computing unit is higher than the first set load threshold, the matrix computing tasks of other sub-matrices whose importance levels are lower than the target sub-matrix currently executed by the target computing unit are transferred to other computing units among the multiple computing units, and / or the importance levels corresponding to the sub-matrices in the target computing unit that have not executed the matrix computing tasks are reduced, so as to allocate the sub-matrices in the target computing unit that have not executed the corresponding matrix computing tasks to other computing units among the multiple computing units.

[0106] If the real-time computing resource usage indicator indicates that the load of the target computing unit is lower than the second set load threshold, the importance level corresponding to the sub-matrix that has not executed the matrix calculation task in the target computing unit is increased so that the target computing unit executes the matrix calculation tasks corresponding to two or more sub-matrices in parallel.

[0107] Among them, the first set load threshold and the second set load threshold can be customized according to actual conditions.

[0108] In this solution, through the dynamic adjustment strategy of computing tasks, when some computing units are under high load due to heavy matrix computing tasks, the matrix computing tasks assigned to them are adjusted in time, and the matrix computing tasks corresponding to some sub-matrices are assigned to other computing units with lighter loads, which can ensure the load balancing of multiple computing units of the entire physical device and avoid slowing down the overall computing progress due to insufficient computing resources for a single computing unit. When some computing units are under low load due to fewer matrix computing tasks, executing the matrix computing tasks corresponding to multiple sub-matrices in parallel can also make full use of computing resources and avoid resource waste.

[0109] It can be understood that in step S101, the model parameter matrix of the target machine learning model is divided into multiple sub-matrices according to the video memory capacity of the physical device. Although this can achieve the improvement of computing efficiency by reducing the video memory capacity occupancy, in actual applications, it is also possible to find a more appropriate blocking strategy by continuously optimizing the blocking strategy through feedback, thereby further reducing the video memory capacity occupancy and improving computing efficiency.

[0110] In the model calculation method provided by the embodiment of the present invention, after the matrix calculation tasks corresponding to the multiple sub-matrices are executed by the multiple calculation units, the following steps may be performed:

[0111] Obtaining at least one evaluation index for evaluating the execution of matrix calculation tasks corresponding to the plurality of sub-matrices;

[0112] According to at least one evaluation metric, a model parameter matrix of a target machine learning model is re-divided into a plurality of sub-matrices.

[0113] After the model parameter matrix is ​​re-divided, the above steps 102 to 106 are executed again to obtain the corresponding video memory occupancy and computing efficiency after the sub-matrix is ​​re-divided, so as to obtain the model parameter matrix block partitioning strategy that is most conducive to matrix calculation, such as: the model parameter matrix block partitioning strategy with the highest computing efficiency.

[0114] In an optional embodiment, at least one evaluation index includes: the load of each of the multiple computing units; based on the at least one evaluation index, the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, including: if the load of each of the multiple computing units indicates that the loads between the multiple computing units are unbalanced, then when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data volume of a single sub-matrix is ​​reduced. This solution can reduce the data volume of the sub-matrix and refine the division of the sub-matrix, so that the matrix calculation tasks corresponding to the sub-matrix can be more flexibly scheduled among multiple computing units.

[0115] In an optional embodiment, at least one evaluation index includes: the frequency of data transmission between the memory storing the model parameter matrix and the video memory; based on the at least one evaluation index, the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, including: if the data transmission frequency is greater than the first threshold, then when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data volume of a single sub-matrix is ​​increased. This solution reduces the number of sub-matrices by increasing the data volume of a single sub-matrix, thereby optimizing the scheduling efficiency between the memory and the video memory.

[0116] In an optional embodiment, at least one evaluation index includes: the frequency of data transmission between the memory storing the model parameter matrix and the video memory, and the video memory occupancy rate; based on at least one evaluation index, the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, including: if the data transmission frequency is greater than a first threshold and the video memory occupancy rate is less than a second threshold, then when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data volume of a single sub-matrix is ​​increased. This solution reduces the number of sub-matrices by increasing the data volume of a single sub-matrix, optimizes the scheduling efficiency between the memory and the video memory, and improves the video memory occupancy rate.

[0117] In an optional embodiment, at least one evaluation index includes: data transmission frequency between the memory storing the model parameter matrix and the video memory, video memory occupancy, and multiple computing unit loads; based on at least one evaluation index, the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, including: if the data transmission frequency is less than a third threshold, the video memory occupancy is greater than a fourth threshold, and the load of multiple computing units indicates that there is a computing unit load greater than a fifth threshold, then when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data volume of a single sub-matrix is ​​reduced. This solution reduces the amount of data in a single sub-matrix, reduces the amount of computation of the matrix computation task corresponding to a single sub-matrix, and reduces the demand for video memory.

[0118] See also Figure 3 , Figure 3A model calculation device provided in an embodiment of the present application includes a matrix division module 11, a determination module 12, and a task calculation module 13. Among them, the matrix division module 11 is used to divide the model parameter matrix of the target machine learning model into multiple sub-matrices according to the video memory capacity of the physical device; the determination module 12 is used to determine the importance levels corresponding to the multiple sub-matrices; according to the importance levels corresponding to the multiple sub-matrices, determine the target sub-matrix to be calculated currently, the target sub-matrix is ​​a sub-matrix in the multiple sub-matrices that has not been subjected to matrix calculation, and the importance level of the target sub-matrix is ​​higher than the importance level of other sub-matrices that have not been subjected to matrix calculation; obtain the real-time load information corresponding to the multiple computing units contained in the physical device; according to the real-time load information corresponding to the multiple computing units, determine the target computing unit from the multiple computing units, and the current load of the target computing unit is lower than the current load of other computing units; the task calculation module 13 is used to load the target sub-matrix into the video memory of the physical device, so that the target computing unit performs the matrix calculation task corresponding to the target sub-matrix based on the target sub-matrix loaded in the video memory.

[0119] In some implementations, the amount of data in each of the multiple sub-matrices is less than or equal to the video memory capacity, and the matrix division module 11 performs the following steps in the process of dividing the model parameter matrix of the target machine learning model into multiple sub-matrices according to the video memory capacity of the physical device:

[0120] According to a plurality of block partitioning strategies, matrix partitioning processing is performed on the model parameter matrix of the target machine learning model respectively to obtain a plurality of groups of sub-matrices corresponding to the plurality of block partitioning strategies, each group of sub-matrices contains a plurality of sub-matrices, and the data amounts of sub-matrices in different groups are different;

[0121] For any group of sub-matrices among the multiple groups of sub-matrices, determining the total amount of data of the sub-matrices that can be accommodated by the video memory capacity of the physical device;

[0122] A target group submatrix is ​​determined from the multiple groups of submatrices, so that the multiple submatrices contained in the target group submatrix are used as the submatrix partitioning result of the model parameter matrix of the target machine learning model, and the total amount of data corresponding to the target group submatrix is ​​greater than the total amount of data corresponding to other groups of submatrices.

[0123] In some implementations, the determination module 12 performs, during the process of determining the importance levels respectively corresponding to the plurality of sub-matrices:

[0124] For any one of the multiple sub-matrices, calculate the gradient amplitude of each matrix element in the any one of the sub-matrices relative to the loss function of the target machine learning model to determine the target gradient amplitude corresponding to the any one of the sub-matrices;

[0125] The importance levels respectively corresponding to the multiple sub-matrices are determined according to the target gradient amplitudes respectively corresponding to the multiple sub-matrices.

[0126] In some implementations, the task computing module 13 is further configured to execute:

[0127] The target submatrix in the video memory and the data block for matrix calculation of the target submatrix in the cache of the physical device are loaded into the register of the target calculation unit, so that the target calculation unit performs the matrix calculation task corresponding to the target submatrix based on the target submatrix loaded in the register and the data block.

[0128] In some implementations, the task computing module 13 is further configured to execute:

[0129] Storing the calculation result of the matrix calculation task corresponding to the target submatrix into the cache;

[0130] If matrix calculation tasks corresponding to other sub-matrices among the multiple sub-matrices need to use the calculation result, the calculation unit used to execute the matrix calculation tasks corresponding to the other sub-matrices reads the calculation result from the cache.

[0131] In some implementations, the task computing module 13 is further configured to execute:

[0132] Determine a target data block shared by matrix calculation tasks corresponding to the plurality of sub-matrices respectively;

[0133] If there is a corresponding matrix calculation task in the sub-matrix that has not been subjected to matrix calculation and needs to use the sub-matrix of the target data block, retaining the target data block in the cache;

[0134] If there is no corresponding matrix calculation task in the sub-matrix that has not been subjected to matrix calculation and needs to use the sub-matrix of the target data block, the target data block is deleted from the cache.

[0135] In some implementations, the task computing module 13 is further configured to execute:

[0136] Determine a target matrix element in the target submatrix that participates multiple times in the matrix calculation task corresponding to the target submatrix;

[0137] After the target matrix element is loaded into the register, if some matrix calculation tasks corresponding to the target submatrix need to use the target matrix element, the target matrix element is retained in the register;

[0138] If the matrix calculation task corresponding to the target submatrix does not need to use the target matrix element, the target matrix element is deleted from the register.

[0139] In some embodiments, the determination module 12 is further used to execute: determining the sub-matrix to be calculated for the target computing unit to perform matrix calculation after the target sub-matrix; the task calculation module 13 is further used to execute: in the process of executing the matrix calculation task corresponding to the target sub-matrix, loading the sub-matrix to be calculated into the video memory of the physical device; loading the sub-matrix to be calculated from the video memory to the register of the target computing unit, so that the target computing unit executes the matrix calculation task corresponding to the sub-matrix to be calculated based on the sub-matrix to be calculated loaded in the register after completing the matrix calculation task corresponding to the target sub-matrix.

[0140] In some implementations, the task computing module 13 is further configured to execute:

[0141] Acquire a real-time computing resource usage indicator of the target computing unit in the process of executing the matrix computing task corresponding to the target sub-matrix;

[0142] If the real-time computing resource usage indicator indicates that the load of the target computing unit is higher than a first set load threshold, the matrix computing tasks of other sub-matrices whose importance levels are currently executed by the target computing unit are lower than those of the target sub-matrix, and / or the importance levels corresponding to the sub-matrices in the target computing unit that do not execute the matrix computing tasks are reduced, so as to allocate the sub-matrices in the target computing unit that do not execute the corresponding matrix computing tasks to other computing units in the multiple computing units;

[0143] If the real-time computing resource usage indicator indicates that the load of the target computing unit is lower than the second set load threshold, the importance level corresponding to the sub-matrix that has not executed the matrix calculation task in the target computing unit is increased, so that the target computing unit executes the matrix calculation tasks corresponding to two or more sub-matrices in parallel.

[0144] In some implementations, the real-time computing resource usage indicator includes: the number of matrix computing tasks currently executed by the target computing unit, the time consumption of the currently executed matrix computing tasks, and the speed of completing the matrix computing tasks.

[0145] In some implementations, after the plurality of computing units have completed the matrix computing tasks respectively corresponding to the plurality of sub-matrices, the task computing module 13 executes:

[0146] Acquire at least one evaluation index for evaluating the execution of the matrix calculation tasks respectively corresponding to the plurality of sub-matrices;

[0147] According to the at least one evaluation indicator, the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices.

[0148] In some implementations, the at least one evaluation indicator includes: the load of each of the multiple computing units, and the task computing module 13 performs, in the process of re-dividing the model parameter matrix of the target machine learning model into multiple sub-matrices according to the at least one evaluation indicator:

[0149] If the loads of the multiple computing units indicate that the loads between the multiple computing units are unbalanced, then when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the amount of data in a single sub-matrix is ​​reduced.

[0150] In some implementations, the at least one evaluation indicator includes: a data transmission frequency between a memory storing the model parameter matrix and the video memory, and the task calculation module 13 performs, in the process of re-dividing the model parameter matrix of the target machine learning model into a plurality of sub-matrices according to the at least one evaluation indicator:

[0151] If the data transmission frequency is greater than the first threshold, when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data amount of a single sub-matrix is ​​increased.

[0152] In some implementations, the at least one evaluation indicator includes: a data transmission frequency between a memory storing the model parameter matrix and the video memory, and a video memory occupancy rate. The task calculation module 13 performs the following steps in the process of re-dividing the model parameter matrix of the target machine learning model into a plurality of sub-matrices according to the at least one evaluation indicator:

[0153] If the data transmission frequency is greater than the first threshold and the video memory occupancy rate is less than the second threshold, when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data amount of a single sub-matrix is ​​increased.

[0154] In some implementations, the at least one evaluation indicator includes: a data transmission frequency between a memory storing the model parameter matrix and the video memory, a video memory occupancy rate, and a load of the plurality of computing units. The task computing module 13 performs, in the process of re-dividing the model parameter matrix of the target machine learning model into a plurality of sub-matrices according to the at least one evaluation indicator:

[0155] If the data transmission frequency is less than the third threshold, the video memory occupancy rate is greater than the fourth threshold, and the load indication of the multiple computing units indicates that the computing unit load is greater than the fifth threshold, then when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data amount of a single sub-matrix is ​​reduced.

[0156] In some embodiments, the task calculation module 13 is also used to perform: merging the calculation results of the matrix calculation tasks corresponding to the multiple sub-matrices respectively to determine the output result of the target machine learning model.

[0157] In some implementations, the model computing device may be applied to a physical device.

[0158] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the model calculation device described above can refer to the corresponding process in the aforementioned model calculation method embodiment, and will not be repeated here.

[0159] See also Figure 4 , FIG. is a schematic block diagram of the structure of a physical device provided in an embodiment of the present invention.

[0160] like Figure 4 As shown, the physical device includes a processor 21 and a memory 22 , and the processor 21 and the memory 22 are connected via a bus 23 , such as an I2C (Inter-integrated Circuit) bus.

[0161] Specifically, the processor 21 is used to provide computing and control capabilities to support the operation of the entire physical device. The processor 21 can be a central processing unit (CPU), and the processor 21 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0162] Specifically, the memory 22 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0163] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the embodiment of the present invention, and does not constitute a limitation on the physical device to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0164] The processor 21 is used to run the computer program stored in the memory, and implement any one of the model calculation methods provided by the embodiments of the present invention when executing the computer program.

[0165] In one embodiment, the processor 21 is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:

[0166] Divide the model parameter matrix of the target machine learning model into multiple sub-matrices according to the video memory capacity of the physical device;

[0167] Determine the importance levels respectively corresponding to the multiple sub-matrices;

[0168] Determine a target submatrix to be calculated according to the importance levels respectively corresponding to the multiple submatrices, wherein the target submatrix is ​​a submatrix that has not been subjected to matrix calculation among the multiple submatrices, and the importance level of the target submatrix is ​​higher than the importance levels of other submatrices that have not been subjected to matrix calculation;

[0169] Acquire real-time load information corresponding to each of the plurality of computing units included in the physical device;

[0170] Determine a target computing unit from the multiple computing units according to the real-time load information respectively corresponding to the multiple computing units, wherein the current load of the target computing unit is lower than the current loads of other computing units;

[0171] The target sub-matrix is ​​loaded into the video memory of the physical device, so that the target calculation unit performs a matrix calculation task corresponding to the target sub-matrix based on the target sub-matrix loaded into the video memory.

[0172] In some implementations, the amount of data of each submatrix in the multiple submatrices is less than or equal to the video memory capacity, and the processor 21 performs, in the process of dividing the model parameter matrix of the target machine learning model into multiple submatrices according to the video memory capacity of the physical device:

[0173] According to a plurality of block partitioning strategies, matrix partitioning processing is performed on the model parameter matrix of the target machine learning model respectively to obtain a plurality of groups of sub-matrices corresponding to the plurality of block partitioning strategies, each group of sub-matrices contains a plurality of sub-matrices, and the data amounts of sub-matrices in different groups are different;

[0174] For any group of sub-matrices among the multiple groups of sub-matrices, determining the total amount of data of the sub-matrices that can be accommodated by the video memory capacity of the physical device;

[0175] A target group submatrix is ​​determined from the multiple groups of submatrices, so that the multiple submatrices contained in the target group submatrix are used as the submatrix partitioning result of the model parameter matrix of the target machine learning model, and the total amount of data corresponding to the target group submatrix is ​​greater than the total amount of data corresponding to other groups of submatrices.

[0176] In some implementations, the processor 21 performs, during the process of determining the importance levels respectively corresponding to the plurality of sub-matrices:

[0177] For any one of the multiple sub-matrices, calculate the gradient amplitude of each matrix element in the any one of the sub-matrices relative to the loss function of the target machine learning model to determine the target gradient amplitude corresponding to the any one of the sub-matrices;

[0178] The importance levels respectively corresponding to the multiple sub-matrices are determined according to the target gradient amplitudes respectively corresponding to the multiple sub-matrices.

[0179] In some implementations, the processor 21 is further configured to execute:

[0180] The target submatrix in the video memory and the data block for matrix calculation of the target submatrix in the cache of the physical device are loaded into the register of the target calculation unit, so that the target calculation unit performs the matrix calculation task corresponding to the target submatrix based on the target submatrix loaded in the register and the data block.

[0181] In some implementations, the processor 21 is further configured to execute:

[0182] Storing the calculation result of the matrix calculation task corresponding to the target submatrix into the cache;

[0183] If matrix calculation tasks corresponding to other sub-matrices among the multiple sub-matrices need to use the calculation result, the calculation unit used to execute the matrix calculation tasks corresponding to the other sub-matrices reads the calculation result from the cache.

[0184] In some implementations, the processor 21 is further configured to execute:

[0185] Determine a target data block shared by matrix calculation tasks corresponding to the plurality of sub-matrices respectively;

[0186] If there is a corresponding matrix calculation task in the sub-matrix that has not been subjected to matrix calculation and needs to use the sub-matrix of the target data block, retaining the target data block in the cache;

[0187] If there is no corresponding matrix calculation task in the sub-matrix that has not been subjected to matrix calculation and needs to use the sub-matrix of the target data block, the target data block is deleted from the cache.

[0188] In some implementations, the processor 21 is further configured to execute:

[0189] Determine a target matrix element in the target submatrix that participates multiple times in the matrix calculation task corresponding to the target submatrix;

[0190] After the target matrix element is loaded into the register, if some matrix calculation tasks corresponding to the target submatrix need to use the target matrix element, the target matrix element is retained in the register;

[0191] If the matrix calculation task corresponding to the target submatrix does not need to use the target matrix element, the target matrix element is deleted from the register.

[0192] In some implementations, the processor 21 is further configured to execute:

[0193] Determine a submatrix to be calculated for matrix calculation by the target computing unit after the target submatrix;

[0194] In the process of executing the matrix calculation task corresponding to the target sub-matrix, loading the sub-matrix to be calculated into the video memory of the physical device;

[0195] The sub-matrix to be calculated is loaded from the video memory to the register of the target computing unit, so that after the target computing unit completes the matrix calculation task corresponding to the target sub-matrix, it performs the matrix calculation task corresponding to the sub-matrix to be calculated based on the sub-matrix to be calculated loaded in the register.

[0196] In some implementations, the processor 21 is further configured to execute:

[0197] Acquire a real-time computing resource usage indicator of the target computing unit in the process of executing the matrix computing task corresponding to the target sub-matrix;

[0198] If the real-time computing resource usage indicator indicates that the load of the target computing unit is higher than a first set load threshold, the matrix computing tasks of other sub-matrices whose importance levels are currently executed by the target computing unit are lower than those of the target sub-matrix, and / or the importance levels corresponding to the sub-matrices in the target computing unit that do not execute the matrix computing tasks are reduced, so as to allocate the sub-matrices in the target computing unit that do not execute the corresponding matrix computing tasks to other computing units in the multiple computing units;

[0199] If the real-time computing resource usage indicator indicates that the load of the target computing unit is lower than the second set load threshold, the importance level corresponding to the sub-matrix that has not executed the matrix calculation task in the target computing unit is increased, so that the target computing unit executes the matrix calculation tasks corresponding to two or more sub-matrices in parallel.

[0200] In some implementations, the real-time computing resource usage indicator includes: the number of matrix computing tasks currently executed by the target computing unit, the time consumption of the currently executed matrix computing tasks, and the speed of completing the matrix computing tasks.

[0201] In some implementations, after the processor 21 completes the matrix calculation tasks respectively corresponding to the multiple sub-matrices through the multiple calculation units, the processor 21 executes:

[0202] Acquire at least one evaluation index for evaluating the execution of the matrix calculation tasks respectively corresponding to the plurality of sub-matrices;

[0203] According to the at least one evaluation indicator, the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices.

[0204] In some implementations, the at least one evaluation indicator includes: the load of each of the plurality of computing units, and the processor 21 performs, in the process of re-dividing the model parameter matrix of the target machine learning model into a plurality of sub-matrices according to the at least one evaluation indicator:

[0205] If the loads of the multiple computing units indicate that the loads between the multiple computing units are unbalanced, then when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the amount of data in a single sub-matrix is ​​reduced.

[0206] In some implementations, the at least one evaluation indicator includes: a data transmission frequency between a memory storing the model parameter matrix and the video memory, and the task calculation module 13 performs, in the process of re-dividing the model parameter matrix of the target machine learning model into a plurality of sub-matrices according to the at least one evaluation indicator:

[0207] If the data transmission frequency is greater than the first threshold, when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data amount of a single sub-matrix is ​​increased.

[0208] In some implementations, the at least one evaluation indicator includes: a data transmission frequency between a memory storing the model parameter matrix and the video memory, and a video memory occupancy rate. In the process of re-dividing the model parameter matrix of the target machine learning model into a plurality of sub-matrices according to the at least one evaluation indicator, the processor 21 executes:

[0209] If the data transmission frequency is greater than the first threshold and the video memory occupancy rate is less than the second threshold, when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data amount of a single sub-matrix is ​​increased.

[0210] In some implementations, the at least one evaluation indicator includes: a data transmission frequency between a memory storing the model parameter matrix and the video memory, a video memory occupancy rate, and a load of the plurality of computing units. The processor 21 performs, in the process of re-dividing the model parameter matrix of the target machine learning model into a plurality of sub-matrices according to the at least one evaluation indicator:

[0211] If the data transmission frequency is less than the third threshold, the video memory occupancy rate is greater than the fourth threshold, and the load indication of the multiple computing units indicates that the computing unit load is greater than the fifth threshold, then when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data amount of a single sub-matrix is ​​reduced.

[0212] In some embodiments, the processor 21 is further used to execute: merging the calculation results of the matrix calculation tasks corresponding to the multiple sub-matrices respectively to determine the output result of the target machine learning model.

[0213] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the physical device described above can refer to the corresponding process in the aforementioned model calculation method embodiment, and will not be repeated here.

[0214] An embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any model calculation method provided in the description of the embodiment of the present invention.

[0215] The storage medium may be an internal storage unit of the physical device of the aforementioned embodiment, such as a hard disk or memory of the physical device. The storage medium may also be an external storage device of the physical device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the physical device.

[0216] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0217] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0218] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A model calculation method, characterized in that: The method comprises: Divide the model parameter matrix of the target machine learning model into multiple sub-matrices according to the video memory capacity of the physical device; Determine the importance levels respectively corresponding to the multiple sub-matrices; Determine a target submatrix to be calculated according to the importance levels respectively corresponding to the multiple submatrices, wherein the target submatrix is ​​a submatrix that has not been subjected to matrix calculation among the multiple submatrices, and the importance level of the target submatrix is ​​higher than the importance levels of other submatrices that have not been subjected to matrix calculation; Acquire real-time load information corresponding to each of the plurality of computing units included in the physical device; Determine a target computing unit from the multiple computing units according to the real-time load information respectively corresponding to the multiple computing units, wherein the current load of the target computing unit is lower than the current loads of other computing units; Loading the target sub-matrix into the video memory of the physical device, so that the target computing unit performs a matrix computing task corresponding to the target sub-matrix based on the target sub-matrix loaded into the video memory; Wherein, the data volume of each submatrix in the plurality of submatrices is less than or equal to the video memory capacity; According to the memory capacity of the physical device, the model parameter matrix of the target machine learning model is divided into a plurality of sub-matrices, including: According to a plurality of block partitioning strategies, matrix partitioning processing is performed on the model parameter matrix of the target machine learning model respectively to obtain a plurality of groups of sub-matrices corresponding to the plurality of block partitioning strategies, each group of sub-matrices contains a plurality of sub-matrices, and the data amounts of sub-matrices in different groups are different; For any group of sub-matrices among the multiple groups of sub-matrices, determining the total amount of data of the sub-matrices that can be accommodated by the video memory capacity of the physical device; A target group submatrix is ​​determined from the multiple groups of submatrices, so that the multiple submatrices contained in the target group submatrix are used as the submatrix partitioning result of the model parameter matrix of the target machine learning model, and the total amount of data corresponding to the target group submatrix is ​​greater than the total amount of data corresponding to other groups of submatrices.

2. The method according to claim 1, characterized in that The determining the importance levels respectively corresponding to the plurality of sub-matrices comprises: For any one of the multiple sub-matrices, calculate the gradient amplitude of each matrix element in the any one of the sub-matrices relative to the loss function of the target machine learning model to determine the target gradient amplitude corresponding to the any one of the sub-matrices; The importance levels respectively corresponding to the multiple sub-matrices are determined according to the target gradient amplitudes respectively corresponding to the multiple sub-matrices.

3. The method according to claim 1, characterized in that The method further comprises: The target submatrix in the video memory and the data block for matrix calculation of the target submatrix in the cache of the physical device are loaded into the register of the target calculation unit, so that the target calculation unit performs the matrix calculation task corresponding to the target submatrix based on the target submatrix loaded in the register and the data block.

4. The method according to claim 3, characterized in that The method further comprises: Storing the calculation result of the matrix calculation task corresponding to the target submatrix into the cache; If matrix calculation tasks corresponding to other sub-matrices among the multiple sub-matrices need to use the calculation result, the calculation unit used to execute the matrix calculation tasks corresponding to the other sub-matrices reads the calculation result from the cache.

5. The method according to claim 3, characterized in that: The method further comprises: Determine a target data block shared by matrix calculation tasks corresponding to the plurality of sub-matrices respectively; If there is a corresponding matrix calculation task in the sub-matrix that has not been subjected to matrix calculation and needs to use the sub-matrix of the target data block, retaining the target data block in the cache; If there is no corresponding matrix calculation task in the sub-matrix that has not been subjected to matrix calculation and needs to use the sub-matrix of the target data block, the target data block is deleted from the cache.

6. The method according to claim 3, characterized in that The method further comprises: Determine a target matrix element in the target submatrix that participates multiple times in the matrix calculation task corresponding to the target submatrix; After the target matrix element is loaded into the register, if some matrix calculation tasks corresponding to the target submatrix need to use the target matrix element, the target matrix element is retained in the register; If the matrix calculation task corresponding to the target submatrix does not need to use the target matrix element, the target matrix element is deleted from the register.

7. The method according to claim 1, characterized in that The method further comprises: Determine a submatrix to be calculated for matrix calculation by the target computing unit after the target submatrix; In the process of executing the matrix calculation task corresponding to the target sub-matrix, loading the sub-matrix to be calculated into the video memory of the physical device; The sub-matrix to be calculated is loaded from the video memory to the register of the target computing unit, so that after the target computing unit completes the matrix calculation task corresponding to the target sub-matrix, it performs the matrix calculation task corresponding to the sub-matrix to be calculated based on the sub-matrix to be calculated loaded in the register.

8. The method according to claim 1, characterized in that The method further comprises: Acquire a real-time computing resource usage indicator of the target computing unit in the process of executing the matrix computing task corresponding to the target sub-matrix; If the real-time computing resource usage indicator indicates that the load of the target computing unit is higher than a first set load threshold, the matrix computing tasks of other sub-matrices whose importance levels are currently executed by the target computing unit are lower than those of the target sub-matrix, and / or the importance levels corresponding to the sub-matrices in the target computing unit that do not execute the matrix computing tasks are reduced, so as to allocate the sub-matrices in the target computing unit that do not execute the corresponding matrix computing tasks to other computing units in the multiple computing units; If the real-time computing resource usage indicator indicates that the load of the target computing unit is lower than the second set load threshold, the importance level corresponding to the sub-matrix that has not executed the matrix calculation task in the target computing unit is increased, so that the target computing unit executes the matrix calculation tasks corresponding to two or more sub-matrices in parallel.

9. The method according to claim 8, characterized in that The real-time computing resource usage index includes: the number of matrix computing tasks currently executed by the target computing unit, the time consumption of the currently executed matrix computing tasks, and the speed of completing the matrix computing tasks.

10. The method according to claim 1, characterized in that After the plurality of computing units have completed the matrix computing tasks respectively corresponding to the plurality of sub-matrices, the method further includes: Acquire at least one evaluation index for evaluating the execution of the matrix calculation tasks respectively corresponding to the plurality of sub-matrices; According to the at least one evaluation indicator, the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices.

11. The method according to claim 10, characterized in that The at least one evaluation indicator includes: the load of each of the multiple computing units; and the re-dividing the model parameter matrix of the target machine learning model into multiple sub-matrices according to the at least one evaluation indicator includes: If the loads of the multiple computing units indicate that the loads between the multiple computing units are unbalanced, then when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the amount of data in a single sub-matrix is ​​reduced.

12. The method according to claim 10, characterized in that The at least one evaluation indicator includes: a data transmission frequency between a memory storing the model parameter matrix and the video memory; and re-dividing the model parameter matrix of the target machine learning model into a plurality of sub-matrices according to the at least one evaluation indicator includes: If the data transmission frequency is greater than the first threshold, when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data amount of a single sub-matrix is ​​increased.

13. The method according to claim 10, characterized in that The at least one evaluation index includes: a data transmission frequency between a memory storing the model parameter matrix and the video memory, and a video memory occupancy rate; and re-dividing the model parameter matrix of the target machine learning model into a plurality of sub-matrices according to the at least one evaluation index includes: If the data transmission frequency is greater than the first threshold and the video memory occupancy rate is less than the second threshold, when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data amount of a single sub-matrix is ​​increased.

14. The method according to claim 10, characterized in that The at least one evaluation index includes: the data transmission frequency between the memory storing the model parameter matrix and the video memory, the video memory occupancy rate, and the load of the multiple computing units; the re-dividing the model parameter matrix of the target machine learning model into multiple sub-matrices according to the at least one evaluation index includes: If the data transmission frequency is less than the third threshold, the video memory occupancy rate is greater than the fourth threshold, and the load indication of the multiple computing units indicates that the computing unit load is greater than the fifth threshold, then when the model parameter matrix of the target machine learning model is re-divided into multiple sub-matrices, the data amount of a single sub-matrix is ​​reduced.

15. The method according to any one of claims 1 to 14, characterized in that The method further comprises: The calculation results of the matrix calculation tasks corresponding to the multiple sub-matrices are combined to determine the output result of the target machine learning model.

16. A model calculation device, characterized in that: include: A matrix partitioning module, used to partition the model parameter matrix of the target machine learning model into multiple sub-matrices according to the video memory capacity of the physical device; A determination module, used to determine the importance levels respectively corresponding to the multiple sub-matrices; Determine a target submatrix to be calculated according to the importance levels respectively corresponding to the multiple submatrices, wherein the target submatrix is ​​a submatrix that has not been subjected to matrix calculation among the multiple submatrices, and the importance level of the target submatrix is ​​higher than the importance levels of other submatrices that have not been subjected to matrix calculation; Acquire real-time load information corresponding to each of the plurality of computing units included in the physical device; determine a target computing unit from the plurality of computing units according to the real-time load information corresponding to the plurality of computing units, wherein a current load of the target computing unit is lower than a current load of other computing units; A task calculation module, used for loading the target sub-matrix into the video memory of the physical device, so that the target calculation unit performs the matrix calculation task corresponding to the target sub-matrix based on the target sub-matrix loaded into the video memory; Wherein, the data volume of each submatrix in the plurality of submatrices is less than or equal to the video memory capacity; The matrix partitioning module divides the model parameter matrix of the target machine learning model into a plurality of sub-matrices according to the video memory capacity of the physical device, for: According to a plurality of block partitioning strategies, matrix partitioning processing is performed on the model parameter matrix of the target machine learning model respectively to obtain a plurality of groups of sub-matrices corresponding to the plurality of block partitioning strategies, each group of sub-matrices contains a plurality of sub-matrices, and the data amounts of sub-matrices in different groups are different; For any group of sub-matrices among the multiple groups of sub-matrices, determining the total amount of data of the sub-matrices that can be accommodated by the video memory capacity of the physical device; A target group submatrix is ​​determined from the multiple groups of submatrices, so that the multiple submatrices contained in the target group submatrix are used as the submatrix partitioning result of the model parameter matrix of the target machine learning model, and the total amount of data corresponding to the target group submatrix is ​​greater than the total amount of data corresponding to other groups of submatrices.

17. A physical device, characterized in that The physical device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the model calculation method according to any one of claims 1 to 15 when executing the computer program.

18. A computer storage medium for computer storage, characterized in that: The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the model calculation method according to any one of claims 1 to 15.

19. A chip, characterized in that: The chip includes a processor coupled to a transceiver, and is used to execute the model calculation method as described in any one of claims 1-15.

20. A chip system, characterized in that: The chip system includes: A communication interface for inputting and / or outputting information; A processor is used to execute a computer executable program so that a device equipped with the chip system executes the model calculation method as described in any one of claims 1 to 15.

Images