A method, apparatus and readable storage medium for determining sample batch size

Abstract

Embodiments of the present application disclose a method, device and readable storage medium for determining a sample batch size. Embodiments of the present application determine N average calculation cycle numbers of a deep learning model based on N sample batch sizes, wherein N is a positive integer greater than or equal to 1, each average calculation cycle number corresponds to a sample batch size; determine a first candidate sample batch size according to the N average calculation cycle numbers; determine N calculation intensities of the deep learning model based on the N sample batch sizes, wherein each calculation intensity corresponds to a sample batch size; determine a second candidate sample batch size according to the N calculation intensities; and determine the maximum value of the first candidate sample batch size and the second candidate sample batch size as the target sample batch size. Through the above method, an optimal sample batch size can be accurately determined, so that the deep learning model achieves an optimal performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a method and device for determining sample batch size and readable storage medium. BACKGROUND

[0002] Deep learning (DL) model is an important part of machine learning, and deployment of the deep learning model on hardware can be generally divided into training scene and inference scene. In both scenes, sample batch size (bs) has a certain degree of influence on the optimization degree and speed of training, and the performance of inference.

[0003] For the inference scene, although the sample batch size does not affect the accuracy of the inference result, reasonable sample batch selection will improve the inference efficiency of the deep learning model. In the inference scene, the chip uses DDR SDRAM (Double Data Rate Synchronous Dynamic Random Access Memory) and other storage media with relatively limited bandwidth, so the bandwidth is an important factor limiting the improvement of inference performance. If the sample batch size (bs) is too small, the deep learning model has small calculation amount, and cannot fully use the computing power of the hardware. If the sample batch size (bs) is too large, it will bring a large demand for storage of intermediate calculation results, and will cause multiple data transfers between cache and low cache in the inference process. The limitation of hardware bandwidth will cause the decline of inference performance. Moreover, the compiler will also schedule the calculation of the deep learning model according to the bandwidth limitation of the hardware, that is, adjust the calculation order of different samples, so that the deep learning model can save as much intermediate calculation result as possible on the hardware cache during the calculation process, and reduce the data transfer amount between caches, thereby improving the inference performance of the deep learning model. When the sample batch size (bs) increases, the complexity of the scheduling algorithm of the compiler will also increase accordingly. When the sample batch size is too large, the scheduling algorithm complexity may be too high to schedule the optimal calculation scheme.

[0004] In the prior art, the method of grid search is used to traverse the possible batch sample number. For different sample batch sizes, the deep learning model is actually inferred, and the performance of inferring the deep learning model on the hardware is recorded, so as to select the optimal sample batch size. Since the search space of the grid search method is large, and each candidate sample batch size needs to be actually run, and then the optimal selection is made, it will bring huge time overhead, and the selection efficiency is too low.

[0005] In summary, how to accurately determine the optimal sample batch size is a problem to be solved at present. SUMMARY

[0006] Therefore, the embodiments of the present application provide a method, device and readable storage medium for allocating computing resources based on a tensor calculation graph, which can accurately determine an optimal sample batch size and make a deep learning model achieve an optimal performance.

[0007] In a first aspect, the embodiments of the present application provide a method for determining a sample batch size, which comprises:

[0008] determining N average calculation cycle numbers of a deep learning model based on N sample batch sizes, wherein N is a positive integer greater than or equal to 1, and each average calculation cycle number corresponds to a sample batch size;

[0009] determining a first candidate sample batch size according to the N average calculation cycle numbers;

[0010] determining N calculation intensities of the deep learning model based on the N sample batch sizes, wherein each calculation intensity corresponds to a sample batch size;

[0011] determining a second candidate sample batch size according to the N calculation intensities;

[0012] determining the target sample batch size as the maximum value of the first candidate sample batch size and the second candidate sample batch size.

[0013] Optionally, the step of determining N average calculation cycle numbers of a deep learning model based on N sample batch sizes comprises:

[0014] determining the calculation cycle numbers of a plurality of operators in the deep learning model for each sample batch size in the N sample batch sizes; and determining the sum of the calculation cycle numbers of the plurality of operators as the total calculation cycle number corresponding to the sample batch size;

[0015] determining the average calculation cycle number as the ratio of each total calculation cycle to the sample batch size corresponding thereto.

[0016] Optionally, the step of determining a first candidate sample batch size according to the N average calculation cycle numbers comprises:

[0017] determining a first inflection point according to the order from small to large of the N sample batch sizes, wherein the first inflection point is an inflection point of the N average calculation cycle numbers;

[0018] determining the sample batch size corresponding to the first inflection point as the first candidate sample batch size.

[0019] Optionally, the determining the first inflection point according to the order of the N sample batch sizes from small to large specifically comprises:

[0020] Taking the N average calculation periods as N logarithms;

[0021] Determining N first slope values according to the N logarithms;

[0022] Sorting the N first slope values according to the corresponding sample batch sizes from small to large;

[0023] Determining the average calculation period corresponding to the first time when the N first slope values in the sorting process are greater than a first set threshold as the first inflection point.

[0024] Optionally, the determining the N calculation intensities of the deep learning model based on the N sample batch sizes specifically comprises:

[0025] For each of the N sample batch sizes, determining the ratio of the total calculation amount of the deep learning model to the total memory amount of the deep learning model as the calculation intensity corresponding to the sample batch size.

[0026] Optionally, the determining the N calculation intensities of the deep learning model based on the N sample batch sizes specifically comprises:

[0027] For each of the N sample batch sizes, determining the calculation amount of a plurality of operators in the deep learning model;

[0028] Determining the sum of the calculation amounts of the plurality of operators in the deep learning model as the total calculation amount of the deep learning model;

[0029] For each of the N sample batch sizes, determining the data memory amount of input and output data of the plurality of operators in the deep learning model, the parameter memory amount of the plurality of operators in the deep learning model, and the memory amount of intermediate calculation results of the plurality of operators in the deep learning model;

[0030] Determining the sum of the data memory amount of input and output data of the plurality of operators in the deep learning model, the parameter memory amount of the plurality of operators in the deep learning model, and the memory amount of intermediate calculation results of the plurality of operators in the deep learning model as the total memory amount of the deep learning model.

[0031] Optionally, the determining the second candidate sample batch size according to the N calculation intensities specifically comprises:

[0032] determine a second inflection point according to an order of the N sample batch sizes from small to large, the second inflection point being an inflection point of the N calculation intensities;

[0033] determine the sample batch size corresponding to the second inflection point as a second candidate sample batch size.

[0034] Optionally, the determining the second inflection point according to the order of the N sample batch sizes from small to large specifically includes:

[0035] take N logarithms of the N calculation intensities;

[0036] determine N second slope values according to the N logarithms;

[0037] sort the N second slope values according to an order corresponding to the sample batch sizes from small to large;

[0038] determine the calculation intensity corresponding to a first time when the N second slope values are greater than a second set threshold as the second inflection point.

[0039] In a second aspect, an embodiment of the present application provides a device for determining a sample batch size, the device comprising:

[0040] a first determining unit configured to determine N average calculation cycle numbers of a deep learning model based on N sample batch sizes, wherein N is a positive integer greater than or equal to 1, and each of the average calculation cycle numbers corresponds to a sample batch size;

[0041] a second determining unit configured to determine a first candidate sample batch size according to the N average calculation cycle numbers;

[0042] a third determining unit configured to determine N calculation intensities of the deep learning model based on the N sample batch sizes, wherein each of the calculation intensities corresponds to a sample batch size;

[0043] a fourth determining unit configured to determine a second candidate sample batch size according to the N calculation intensities;

[0044] a fifth determining unit configured to determine a maximum value of the first candidate sample batch size and the second candidate sample batch size as the target sample batch size.

[0045] Optionally, the first determining unit is specifically configured to:

[0046] determine calculation cycle numbers of a plurality of operators in the deep learning model for each of the N sample batch sizes;

[0047] Summing up the number of calculation periods of the plurality of operators to determine the total number of calculation periods corresponding to the sample batch size;

[0048] Determining the ratio of each of the total number of calculation periods to the sample batch size corresponding thereto as the average number of calculation periods.

[0049] Optionally, the second determining unit is specifically configured to:

[0050] Determining a first inflection point according to the order of the N sample batch sizes from small to large, the first inflection point being an inflection point of the N average numbers of calculation periods;

[0051] Determining the sample batch size corresponding to the first inflection point as a first candidate sample batch size.

[0052] Optionally, the second determining unit is specifically configured to:

[0053] Taking the logarithm of the N average numbers of calculation periods;

[0054] Determining N first slope values according to the N logarithms;

[0055] Sorting the N first slope values according to the order of the corresponding sample batch sizes from small to large;

[0056] Determining the average number of calculation periods corresponding to the first time when the N first slope values exceed a first set threshold value in the sorting process as the first inflection point.

[0057] Preferably, the third determining unit is specifically configured to: for each of the N sample batch sizes, determining the ratio of the total calculation amount of the deep learning model to the total memory amount of the deep learning model as the calculation intensity corresponding to the sample batch size.

[0058] Preferably, the determining N calculation intensities of the deep learning model based on the N sample batch sizes specifically includes:

[0059] For each of the N sample batch sizes, determining the calculation amount of the plurality of operators in the deep learning model;

[0060] Determining the sum of the calculation amount of the plurality of operators in the deep learning model as the total calculation amount of the deep learning model;

[0061] For each of the N sample batch sizes, determining the data memory amount of the input and output data of the plurality of operators in the deep learning model, the parameter memory amount of the plurality of operators in the deep learning model, and the memory amount of the intermediate calculation results of the plurality of operators in the deep learning model;

[0062] The sum of the data access amount of the input and output data of the plurality of operators in the deep learning model, the parameter access amount of the plurality of operators in the deep learning model, and the access amount of the intermediate calculation results of the plurality of operators in the deep learning model is determined as the total access amount of the deep learning model.

[0063] Optionally, the fourth determining unit is specifically configured to:

[0064] According to the order from small to large of the N sample batch sizes, a second inflection point is determined, the second inflection point being an inflection point of the N calculation intensities.

[0065] The sample batch size corresponding to the second inflection point is determined as a second candidate sample batch size.

[0066] Optionally, the fourth determining unit is specifically configured to:

[0067] The N calculation intensities are taken N logarithms.

[0068] N second slope values are determined according to the N logarithms.

[0069] The N second slope values are sorted according to the order from small to large of the corresponding sample batch sizes.

[0070] The calculation intensity corresponding to the first time when the N second slope values are greater than a second set threshold value is determined as the second inflection point.

[0071] In a third aspect, an embodiment of the present application provides a computer program instruction, which, when executed by a processor, implements the method in the first aspect or any one of the possible implementations of the first aspect.

[0072] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program instruction, and the computer program instruction, when executed by a processor, implements the method in the first aspect or any one of the possible implementations of the first aspect.

[0073] In a fifth aspect, an embodiment of the present application provides a chip, including a memory and a processing core, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processing core to implement the method in the first aspect or any one of the possible implementations of the first aspect.

[0074] In a sixth aspect, an embodiment of the present application provides a board card, the board card including the chip in the fifth aspect.

[0075] In a seventh aspect, an embodiment of the present application provides a server, the server including the board card in the sixth aspect.

[0076] The embodiment of the present application determines N average calculation cycle numbers of a deep learning model based on N sample batch sizes, wherein N is a positive integer greater than or equal to 1, each average calculation cycle number corresponds to a sample batch size; determines a first candidate sample batch size according to the N average calculation cycle numbers; determines N calculation intensities of the deep learning model based on the N sample batch sizes, wherein each calculation intensity corresponds to a sample batch size; determines a second candidate sample batch size according to the N calculation intensities; and determines the maximum value of the first candidate sample batch size and the second candidate sample batch size as the target sample batch size. Through the above method, an optimal sample batch size can be accurately determined, so that the deep learning model achieves an optimal performance. BRIEF DESCRIPTION OF DRAWINGS

[0077] The above and other objects, features and advantages of the present application will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0078] Figure 1 is a method flowchart for determining a sample batch size according to an embodiment of the present application;

[0079] Figure 2 is a relationship diagram between an average calculation cycle number and a sample batch size according to an embodiment of the present application;

[0080] Figure 3 is a method flowchart for determining a first candidate sample batch size according to an embodiment of the present application;

[0081] Figure 4 is a relationship diagram between a calculation intensity and a sample batch size according to an embodiment of the present application;

[0082] Figure 5 is a method flowchart for determining a second candidate sample batch size according to an embodiment of the present application;

[0083] Figure 6 is a relationship diagram between a calculation intensity and a sample batch size according to an embodiment of the present application;

[0084] Figure 7 is a relationship diagram between an average calculation cycle and a sample batch size according to an embodiment of the present application;

[0085] Figure 8 is a relationship diagram between a calculation intensity and a sample batch size according to an embodiment of the present application;

[0086] Figure 9 is a device diagram for determining a sample batch size according to an embodiment of the present application. DETAILED DESCRIPTION

[0087] The present disclosure is described in connection with the embodiments below; however, the present disclosure is not limited to only these embodiments. In the following detailed description of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, the present disclosure can be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present disclosure.

[0088] In addition, it should be understood that the drawings provided herein are for illustrative purposes and are not necessarily to scale.

[0089] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this present disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an overly literal or overly formal sense unless expressly so defined herein.

[0090] In the description of the present disclosure, it should be understood that the terms "first", "second", etc. are used only for descriptive purposes and are not to be construed as indicating or implying relative importance. In addition, in the description of the present disclosure, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0091] In the prior art, the deployment of a deep learning model on hardware can be generally divided into training scenarios and inference scenarios. In both scenarios, the batch size (bs) of samples processed each time has a certain degree of influence on the optimization degree and speed of training, and the performance of inference. A reasonable batch size bs can find the best balance between hardware bandwidth, memory capacity, hardware computing power and model structure, thereby exerting the hardware computing power and achieving the optimal performance of model calculation and deployment.

[0092] In the training scene, the sample batch size bs affects the training efficiency and accuracy. If bs is too small, the training data is difficult to converge, which may cause underfitting; if bs is too large, the required memory capacity increases accordingly, and the number of training set epochs needs to be increased to achieve the optimal result, which reduces the training efficiency. Therefore, selecting a reasonable bs can not only improve the training efficiency, but also reduce the amplitude of training shock and improve the training accuracy; in the inference scene, although the sample batch size does not affect the accuracy of the inference result, reasonable sample batch selection can improve the inference efficiency of the deep learning model; in the inference scene, the chip uses DDR SDRAM (Double Data Rate Synchronous Dynamic Random Access Memory) and other storage media with relatively limited bandwidth, so bandwidth is an important factor limiting the improvement of inference performance. If the sample batch size (bs) is too small, the deep learning model has small calculation amount, and cannot fully use the hardware computing power; if the sample batch size (bs) is too large, it will bring a large demand for storage of intermediate calculation results, which will cause multiple data transfers between cache and low cache during the inference process, and the limitation of hardware bandwidth will cause the decline of inference performance. Moreover, the compiler will also schedule the deep learning model calculation according to the bandwidth limitation of the hardware, that is, adjust the calculation order of different samples, so that the deep learning model can save as much intermediate calculation result as possible in the hardware cache during the calculation process, reducing the data transfer amount between caches, thereby improving the inference performance of the deep learning model. When the sample batch size (bs) increases, the complexity of the compiler scheduling algorithm will increase accordingly; when the sample batch size is too large, the scheduling algorithm complexity may be too high to schedule the optimal calculation scheme.

[0093] In the prior art, the grid search method is used to traverse the possible batch sample number, the deep learning model is actually inferred for different sample batch sizes, the performance of inferring the deep learning model on the hardware is recorded, and the sample batch size is selected by optimization. Since the search space of the grid search method is large, and each candidate sample batch size needs to be actually run, then the optimal selection will bring huge time overhead, and the selection efficiency is too low. Therefore, how to accurately determine the optimal sample batch size is a problem to be solved at present.

[0094] In the embodiment of the present application, in the inference scene of deploying the deep learning model on the neural model processor (Neural-network Processing Unit, NPU) hardware, in order to determine the suitable sample batch size of the deep learning model, a method for determining the sample batch size is proposed, as shown in Figure 1 Figure 1 ​is a method flowchart for determining a sample batch size according to an embodiment of the present application, and specifically includes the following steps:

[0095] In step S100, based on N sample batch sizes, N average calculation period numbers of the deep learning model are respectively determined, wherein N is a positive integer greater than or equal to 1, and each average calculation period number corresponds to a sample batch size.

[0096] Specifically, the step of determining N average calculation period numbers of the deep learning model according to N sample batch sizes specifically includes: for each sample batch size in the N sample batch sizes, determining the calculation period numbers of a plurality of operators in the deep learning model; and determining the sum of the calculation period numbers of the plurality of operators as the total calculation period number corresponding to the sample batch size. The plurality of operators in the deep learning model can be selected according to requirements, and are generally core operators in the deep learning model; and the number of the plurality of operators in the deep learning model is the number of the selected operators. The ratio of each total calculation period to the sample batch size corresponding thereto is determined as an average calculation period number, wherein N total calculation period numbers correspond to N average calculation period numbers.

[0097] In a possible implementation, based on the structure of the NPU calculation unit, the total calculation period number is estimated when the deep learning model selects different sample batch sizes, and specifically, the sum of the calculation period numbers of the plurality of operators is the total calculation period number, that is:

[0098]

[0099] wherein C i represents the calculation period number of the operator OP i on the NPU, and C B represents the total calculation period number, and n is the total number of operators. The n operators can be all operators in the deep learning model, or can be part of the operators in the deep learning model, such as computation-intensive operators.

[0100] In a possible implementation, the ratio of the total calculation period number C B to the sample batch size bs is the average calculation period number c b when a single sample is inferred.

[0101]

[0102] Specifically, the average calculation period number c b can reflect the utilization rate of the computing resources of the deep learning model during calculation, and the smaller the value of the average calculation period number c b , the higher the utilization rate of the computing resources.

[0103] In one possible implementation, the NPU hardware has multiple MAC (multiplier and accumulation) computation units forming a MAC array. When the shape of the MAC array matches the shape of the operator input data of the deep neural network model, all computational resources can be utilized, resulting in high utilization. If the shape of the MAC array does not match the shape of the operator input data of the deep neural network model, some MAC computation units will idle, reducing the utilization rate of computational resources. Since the utilization rate of computational resources is reflected by the average number of computation cycles of the deep learning model, the sample batch size affects the shape of the operator input data, which in turn affects the utilization rate of the NPU hardware's computational units by the deep learning model.

[0104] For example, taking the Click Through Rate (CTR) model in deep learning as an example, the sample batch size *bs* is directly proportional to the number of rows *m* in the left matrix of the matrix multiplication (Matmul), where *m* = *k*bs*, and *k* is a coefficient. Whether the value of *m* is divisible by the number of rows in the MAC matrix will affect the utilization of hardware computing resources. If the value of *m* *bs* is not divisible by the number of rows *x* in the MAC computing array, it will result in a waste of computing resources for rows *x* - *(k*bs)%* in the MAC. A reasonable value of *bs* will maximize the utilization of NPU computing resources during deep learning model inference. Furthermore, for a single operator, as the value of *bs* increases, the number of computation cycles for the operator initially increases synchronously with the value of *bs*, and then the average number of computation cycles decreases. It will remain stable, as detailed below. Figure 2 As shown, when the sample batch size (bs) is too small, the MAC calculation unit cannot be fully utilized by the input data, resulting in computational waste and an average theoretical number of calculation cycles for the operator. The value of bs is relatively large. As the bs value increases, the utilization rate of the operator for computational resources gradually reaches its maximum, and the average theoretical number of cycles tends to stabilize. For the overall deep learning model, the deep learning model consists of multiple operators, such as... Figure 2 As shown, since there is a similar correlation between the average number of computation cycles for each operator and the bs value, the overall deep learning model can be regarded as the sum of the individual operators, thereby obtaining a reasonable bs value for the model to maximize the utilization of computing resources. Figure 2 In this study, the utilization rate of computing units by the deep learning model tends to plateau after bs1, suggesting that the utilization rate reaches its maximum when the number of samples is greater than or equal to bs1. Figure 2 In the diagram, the dashed line represents the relationship between the number of computation cycles for each operator and the sample batch size, while the solid line represents the relationship between the average number of computation cycles and the sample batch size in the deep learning model.

[0105] In the embodiments of the present application, the first candidate sample batch size is determined according to the N average calculation periods and the N sample batch sizes. Figure 2 The value of bs1 in the above formula is determined according to the method described in step S101.

[0106] Step S101: determining a first candidate sample batch size according to the N average calculation periods.

[0107] Specifically, the first candidate sample batch size is determined according to the N average calculation periods and the N sample batch sizes. Figure 3 The method comprises the following steps:

[0108] Step S300: calculating the N logarithms of the N average calculation periods.

[0109] In the embodiments of the present application, the average calculation periods are large, and in order to simplify the calculation, the logarithms of the average calculation periods and the sample batch sizes corresponding to the average calculation periods are calculated.

[0110] Step S301: determining N first slope values according to the N logarithms.

[0111] Specifically, the N points can be determined according to the logarithms of the average calculation periods and the sample batch sizes corresponding to the average calculation periods, and then the slopes between two adjacent points on the horizontal axis are determined.

[0112] Step S302: sorting the N first slope values in ascending order according to the corresponding sample batch sizes.

[0113] Step S303: determining the average calculation period corresponding to the first time when the N first slope values in the sorting process are greater than the first set threshold value as the first inflection point.

[0114] Step S304: determining the sample batch size corresponding to the first inflection point as the first candidate sample batch size.

[0115] Step S102: determining N calculation intensities of the deep learning model based on the N sample batch sizes, wherein each calculation intensity corresponds to a sample batch size.

[0116] Specifically, the N computing intensities of the deep learning model are determined by determining a ratio of the total computation amount of the deep learning model to the total memory amount of the deep learning model as the computing intensity, wherein the N computing intensities correspond to N sample batch sizes. The total computation amount of the deep learning model is the sum of operator computation amounts of each operator in the deep learning model, or the sum of operator computation amounts of part of the operators in the deep learning model, and the part of the operators can be computation-intensive operators. The operator computation amount refers to the number of floating-point operations of the operator when computing an input single sample, and can also be referred to as the time complexity of the deep learning model.

[0117] In a possible implementation, the total computation amount of the deep learning model is the sum of the computation amounts of a plurality of operators in the deep learning model; and the total memory amount of the deep learning model is the sum of the data memory amount of input and output data of the deep learning model, the parameter memory amount of the deep learning model, and the memory amount of intermediate calculation results of the deep learning model.

[0118] For example, the computing intensity I (FLOPs / Bytes) of the deep learning model is equal to the ratio of the total computation amount R (FLOPs) of the deep learning model to the total memory amount L (Bytes) of the deep learning model, that is:

[0119]

[0120] The total computation amount R of the deep learning model is equal to the sum of the computation amounts of each operator in the deep learning model; the operator computation amount refers to the number of floating-point operations of the operator when computing an input sample, and can also be referred to as the time complexity of the deep learning model.

[0121] Specifically, R i represents the number of floating-point operations of the i-th operator OP i in the single-sample deep learning model, and the operator computation amount of the operator OP i is bs*R i when the inference sample batch size is bs. The total computation amount of the deep learning model is as follows:

[0122]

[0123] In the embodiment of the application, the computation amounts of different operators are also different, and the computation amount formulas of part of the computation-intensive operators are listed in Table 1 as follows:

[0124] Table 1

[0125]

[0126]

[0127] In this embodiment of the invention, the total memory access L (Bytes) of the deep learning model represents the size of the memory units accessed during the deep learning model's computation, and also reflects the deep learning model's bandwidth requirements for memory units.

[0128] In one possible implementation, within the NPU, due to the data movement between caches during the inference process of the deep learning model, the memory access volume of the deep learning model is represented by the amount of data movement. The total memory access volume of the deep learning model consists of three parts: the memory access volume P of the input and output data, the memory access volume W of the parameters of the deep learning model, and the memory access volume S of the intermediate computation results of the deep learning model. These three parts are described in detail below.

[0129] Specifically, regarding the data access volume P for input and output data, during inference in a deep learning model, input data must be sequentially moved from the external cache to the NPU's on-chip cache before computation begins; after the entire deep learning model computation is completed, output data is sequentially moved from the NPU's on-chip cache to the external cache. Therefore, the amount of input and output data of a deep learning model is directly proportional to the sample batch size bs required for inference. Assuming f in and f out Let f represent the single-sample data access volume of the input and output of the deep learning model, respectively. Then, the data access volume of the input and output of the deep learning model, P = bs * (f in +f out ).

[0130] Regarding the parameter memory access W of a deep learning model, the parameter memory access volume represents the memory access volume of parameters participating in the deep learning model during inference, such as the convolution kernel in convolution calculation. Different parameters in the deep learning model may be moved between the cache and external buffer one or more times during inference. Here, this is represented by the variable α. i This indicates the operator OP i The number of transports of the parameters used in the calculation; where p i Operator OP i The parameters used in the calculation are memory access count; α i Values ​​and Operators OP i The memory requirements during computation are related to the total amount of NPU memory. Specifically, the number of operator samples (b) that the hardware can store. i The memory requirement S of the operator i The cache resources M of the NPU are calculated for each operator OP. i The maximum number of samples that an NPU can compute and store is bi, specifically:

[0131]

[0132] In a possible implementation, when the sample batch size bs of inference is less than or equal to b i , for the operator OP i , through the optimization scheduling of the compiler, after reasonable scheduling, the intermediate result and the calculation result can be all resident on the NPU cache, without the need of data transmission between the cache and the low-speed cache; when the sample batch size bs is greater than n i , the operator OP i needs to be divided into multiple operations, and the operation amount of each operation is b i (or less than b i ) samples, and the parameters of OP i are carried on the NPU for a i times, specifically:

[0133]

[0134] wherein, the above ceil is the upward rounding.

[0135] In the embodiment of the application, the parameter memory access amount W of the deep learning model is the sum of the parameter memory access amounts of each operator in the deep learning model:

[0136]

[0137] For the memory access amount S of the intermediate calculation result of the deep learning model, the memory access amount S of the intermediate calculation result indicates that, when the NPU cache size cannot save all the intermediate calculation results in the deep learning model inference process, part of the intermediate calculation results need to be overflowed from the NPU cache to the low-speed cache (i.e., the external cache in the above), and when the intermediate calculation result is needed to be continuously used in subsequent calculation, the intermediate calculation result is loaded back to the cache from the low-speed cache; since the bandwidth is the performance bottleneck in the deep learning model inference, the compiler usually reduces the data carrying between different cache levels of the calculation intermediate calculation result through scheduling. Therefore, although the actual memory access amount of the deep learning model calculation intermediate calculation result cannot be determined before the compiler scheduling, the intermediate calculation result memory access amount S can be estimated, and the S is:

[0138]

[0139] wherein, the S is proportional to bs, and according to the above formula, the minimum value of the S is 0, indicating that all the intermediate results of the deep learning model inference process are resident on the cache, and no data needs to be carried between the cache and the low-speed cache; and the maximum value of the S is This represents the maximum amount of data that can be moved between the high-speed cache and the low-speed cache for intermediate computation results. It represents the batch size *bs* of samples processed in the topological order of the deep neural network model without scheduling optimization. When the NPU high-speed cache cannot hold the intermediate computation results, the results overflow into the low-speed cache. Where *β*... i Operator OP i The number of times data is moved between the cache and the cache, s i Operator OP i The calculation determines the size of the output data. The main reason for multiplying by 2 in the above formula is that if intermediate calculation results are moved out of the cache, subsequent calculations that depend on these intermediate results will need to load them back into the cache. Therefore, the memory access volume of intermediate calculation results should be multiplied by two. Simultaneously, S is positively correlated with the sample batch size bs. As the sample batch size increases, the memory requirements of intermediate calculation results for deep learning model inference gradually increase. Once this exceeds the actual cache capacity of the hardware, the data movement between caches will also increase accordingly.

[0140] Combining the above three parts, the total memory access of the deep learning model can be expressed as:

[0141]

[0142] The calculated strength can be expressed as:

[0143]

[0144] Substituting the range of values ​​for S into the above equation, we can obtain:

[0145]

[0146] In one possible implementation, due to compiler scheduling optimizations, the memory access volume of intermediate computation results in the actual computation of the deep learning model can be considered minimal during the actual computation of the deep learning model. and Therefore, As an estimate of the calculated intensity, let I ≈ I est .

[0147] For I est A curve can be plotted to show the relationship between the calculated intensity and the sample batch size (bs), as shown below. Figure 4 As shown, by Figure 4 It can be seen that when the sample batch size bs is small, the computational intensity is small. As the sample batch size increases, the computational intensity also increases and reaches an inflection point. After reaching the inflection point bs2, the computational intensity tends to stabilize. Therefore, it is believed that the computational intensity of the deep learning model is the greatest when the sample batch size reaches bs2.

[0148] In the embodiment of the present application, the corresponding multiple sample batch sizes in the above description need to be determined in multiple sample batches Figure 4 The value of bs2 in the above description is calculated according to the method described in step S103.

[0149] Step S103, determining a second candidate sample batch size according to the N calculation intensities.

[0150] Specifically, the second candidate sample batch size is determined according to the N calculation intensities, and the specific method is as follows Figure 5 The method comprises the following steps:

[0151] Step S500, taking the N calculation intensities as N logarithms.

[0152] In the embodiment of the present application, because the calculation intensity is large, in order to simplify the calculation, the logarithm of the calculation intensity and the sample batch size corresponding to the calculation intensity is taken.

[0153] Step S501, determining N second slope values according to the N logarithms.

[0154] Specifically, because the N points can be determined according to the calculation intensity after taking the logarithm and the sample batch size corresponding to the calculation intensity, and then the slope between the adjacent two points on the horizontal axis is determined.

[0155] Step S502, sorting the N second slope values in the order corresponding to the sample batch size from small to large.

[0156] Step S503, determining the calculation intensity corresponding to the first time when the N second slope values are greater than the second set threshold value as the second inflection point.

[0157] Step S504, determining the sample batch size corresponding to the second inflection point as the second candidate sample batch size.

[0158] Step S104, determining the maximum value of the first candidate sample batch size and the second candidate sample batch size as the target sample batch size.

[0159] For example, assuming that the first candidate sample batch size is bs1, and the second candidate sample batch size is bs2, max(bs1, bs2) is taken as the target sample batch size.

[0160] In a possible implementation, the bs corresponding to the inflection point in Figure 2 and Figure 4 is selected, and the reason for not setting a large enough bs is as follows:

[0161] First, in estimating the computing intensity of the deep learning model, the memory access amount S of the intermediate calculation result is zero in the estimation, but in the actual calculation process, the memory access amount S of the intermediate calculation result increases with the increase of the sample batch size, when the sample batch size bs is small, the estimation of the computing intensity of the deep learning model is more accurate, and when the sample batch size bs increases, since the memory access amount S of the intermediate calculation result is positively correlated with the sample batch size, the larger the bs is, the greater the difference between the estimated total memory access amount L est of the deep learning model and the actual total memory access amount L em of the deep learning model is, that is, L em -L est , and the difference between L em and L est increases with the increase of bs. Therefore, as shown in the following formula, the actual computing intensity I Figure 6 of the deep learning model under a large sample batch size tends to be stable or slightly smaller than I em , that is, I est .

[0162] Secondly, when the compiler schedules and optimizes the deep learning model, the optimization algorithm searches for the optimal deep learning model inference order within a given sample batch size range to reduce the memory access amount of the intermediate calculation result of the deep learning model. The larger the sample batch size is, the higher the complexity of the optimization algorithm is; when different optimization scheduling algorithms are selected, the increase of the sample batch size will cause the complexity of the optimization algorithm to increase exponentially, which is unacceptable in actual operation.

[0163] The following will describe a method for determining a sample batch size in detail through a specific embodiment.

[0164] Suppose that the optimal sample batch size for the resnet50 model on the NPU 1 hardware is inferred, the NPU 1 has 8 MAC computing units that can perform efficient matrix operations, the MAC shape is 32*64 (float16) and 32*128 (int8), each computing unit has an exclusive 1280KB first-level cache, and the 8 computing units share a large enough second-level cache; according to the hardware structure of the eight computing units and the MAC shape of the NPU 1, therefore, only the sample batch size of 2 power needs to be considered as the candidate sample batch size.

[0165] In the estimation of the calculation intensity of the resnet50 model, the first-level cache on each calculation unit is taken as the main part, the storage capacity of the cache of the NPU 1 is 8*1280KB=10240KB, and the total memory access amount of the resnet50 model refers to the data exchange between the first-level cache and the second-level cache.

[0166] Under the above premise, the calculation utilization rate of the resnet50 model on the MAC array of the NPU 1 is estimated, that is, the average calculation cycle number of the resnet50 model is estimated. Table 2 below lists the calculation cycle numbers of various calculation types of operators in the NPU 1, as follows:

[0167] Table 2

[0168]

[0169] In the embodiment of the application, assuming that the candidate sample batch size is 1, 2, 4, 8, 16, 32, 64 and 128, the average calculation cycle number of the Resnet50 model under different sample batch sizes is calculated, and the following data can be obtained, as shown in Table 3:

[0170] Table 3

[0171]

[0172]

[0173] In a possible implementation, in order to calculate the inflection point bs1 of the Resnet50 model according to the average calculation cycle number, the average calculation cycle number (avg cycle num) and the sample batch size are logarithmized to obtain Table 4, as follows:

[0174] Table 4

[0175]

[0176] According to the logarithmic value corresponding to each sample batch size in Table 4 above, a curve Figure 7 is drawn, and the slope value (slope) of the connecting line of the points on the curve corresponding to the adjacent two points in Figure 7 is determined, as shown in Table 5:

[0177] Table 5

[0178]

[0179] Assuming the first set threshold is -0.02, the sample batch size at which the slope value first exceeds the first set threshold is determined as the first candidate sample batch size. That is, -0.01 is the value that first exceeds the first set threshold. Then, the bs value 8 corresponding to -0.01 is determined as the first candidate sample batch size, that is, the value of the inflection point bs1 is 8.

[0180] In one possible implementation, the computational intensity of the ResNet50 model calculated using the formulas in the above method for different sample batch sizes is shown in Table 6 below:

[0181] Table 6

[0182]

[0183]

[0184] Following the same algorithm as Tables 4 and 5 above, first calculate the logarithm, then calculate the slope, and finally plot the logarithm based on the logarithm value corresponding to each sample batch size. Figure 8 ,Sure Figure 8 The slope value of the line connecting the points on the curve corresponding to two adjacent points is used to determine the sample batch size when the slope value is greater than the second set threshold for the first time. The corresponding bs value 32 is determined as the second candidate sample batch size, that is, the value of the inflection point bs2 is 8.

[0185] In the above specific embodiment, bs1 = 8, bs2 = 32, bs = max(bs1, bs2) = 32.

[0186] In one possible implementation, the above method can be applied to computing hardware outside the NPU, such as a GPU. The specific architecture of the GPU, such as the number of SPs (streaming processes), SMs (streaming multiprocessors), and warp threads, can be used as the basis for calculating the average number of computation cycles. The storage capacity of the GPU's shared memory and the amount of data transferred between shared memory and global memory can be used to estimate the computational intensity of the model. Based on the above indicators, the optimal batch size for inference samples on the GPU can be evaluated.

[0187] Figure 9 This is a schematic diagram of a device for determining sample batch size according to an embodiment of the present invention. Figure 9 As shown, the device in this embodiment includes a first determining unit 901, a second determining unit 902, a third determining unit 903, a fourth determining unit 904, and a fifth determining unit 905.

[0188] The first determining unit 901 is configured to determine N average calculation period numbers of the deep learning model based on the N sample batch sizes, where N is a positive integer greater than or equal to 1, and each average calculation period number corresponds to a sample batch size; the second determining unit 902 is configured to determine a first candidate sample batch size according to the N average calculation period numbers; the third determining unit 903 is configured to determine N calculation intensities of the deep learning model based on the N sample batch sizes, where each calculation intensity corresponds to a sample batch size; the fourth determining unit 904 is configured to determine a second candidate sample batch size according to the N calculation intensities; and the fifth determining unit 905 is configured to determine the maximum value of the first candidate sample batch size and the second candidate sample batch size as the target sample batch size.

[0189] Optionally, the first determining unit is specifically configured to:

[0190] determine, for each of the N sample batch sizes, a calculation period number of a plurality of operators in the deep learning model;

[0191] determine a total calculation period number corresponding to the sample batch size as the sum of the calculation period numbers of the plurality of operators;

[0192] determine an average calculation period number as the ratio of each total calculation period to the sample batch size corresponding thereto.

[0193] Optionally, the second determining unit is specifically configured to:

[0194] determine a first inflection point according to the order of the N sample batch sizes from small to large, the first inflection point being an inflection point of the N average calculation period numbers;

[0195] determine a first candidate sample batch size as the sample batch size corresponding to the first inflection point.

[0196] Optionally, the second determining unit is specifically configured to:

[0197] take N logarithms of the N average calculation period numbers;

[0198] determine N first slope values according to the N logarithms;

[0199] sort the N first slope values according to the order of the corresponding sample batch sizes from small to large;

[0200] determine the first inflection point as the average calculation period number corresponding to the first time when the N first slope values exceed a first preset threshold value in the sorting process.

[0201] Preferably, the third determining unit is specifically configured to: for each of the N sample batch sizes, determine the ratio of the total computation amount of the deep learning model to the total memory amount of the deep learning model as the calculation intensity corresponding to the sample batch size.

[0202] Preferably, the determining of the N calculation intensities of the deep learning model based on the N sample batch sizes specifically includes:

[0203] for each of the N sample batch sizes, determining the computation amount of the plurality of operators in the deep learning model;

[0204] determining the sum of the computation amount of the plurality of operators in the deep learning model as the total computation amount of the deep learning model;

[0205] for each of the N sample batch sizes, determining the data memory amount of the input and output data of the plurality of operators in the deep learning model, the parameter memory amount of the plurality of operators in the deep learning model, and the memory amount of the intermediate calculation results of the plurality of operators in the deep learning model;

[0206] determining the sum of the data memory amount of the input and output data of the plurality of operators in the deep learning model, the parameter memory amount of the plurality of operators in the deep learning model, and the memory amount of the intermediate calculation results of the plurality of operators in the deep learning model as the total memory amount of the deep learning model.

[0207] Optionally, the fourth determining unit is specifically configured to:

[0208] determine a second inflection point according to the order of the N sample batch sizes from small to large, the second inflection point being an inflection point of the N calculation intensities;

[0209] determine the sample batch size corresponding to the second inflection point as a second candidate sample batch size.

[0210] Optionally, the fourth determining unit is specifically configured to:

[0211] take the N calculation intensities as N logarithms;

[0212] determine N second slope values according to the N logarithms;

[0213] sort the N second slope values according to the order of the sample batch sizes from small to large;

[0214] determine the calculation intensity corresponding to the first time when the N second slope values are greater than a second set threshold value as the second inflection point.

[0215] The embodiment of the present application also provides a computer program instruction, which realizes the method in any one of the above embodiments when executed by a processor.

[0216] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program instruction, and the computer program instruction realizes the method in any one of the above embodiments when executed by a processor.

[0217] The embodiment of the present application provides a chip, comprising a memory and a processing core, the memory is used for storing one or more computer program instructions, wherein the one or more computer program instructions are executed by the processing core to realize the method in any one of the above embodiments.

[0218] The embodiment of the present application provides a board card, and the chip is included on the board card.

[0219] The embodiment of the present application provides a server, and the board card is included on the server.

[0220] As those skilled in the art will realize, the various aspects of the embodiments of the present application can be implemented as a system, a method or a computer program product. Therefore, the various aspects of the embodiments of the present application can take the form of a complete hardware embodiment, a complete software embodiment (including firmware, persistent software, microcode, etc.) or an embodiment combining software and hardware aspects that can all be generally referred to herein as a "circuit", "module" or "system". Furthermore, the various aspects of the embodiments of the present application can take the form of a computer program product implemented in one or more computer readable media having computer readable program code embodied thereon.

[0221] Any combination of one or more computer readable medium can be utilized. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the present embodiments, a computer readable storage medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0222] A computer readable signal medium can include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport program code items structured for execution by, or in connection with, the instruction execution system, apparatus, or device.

[0223] The program code embodied on the computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0224] Computer program code for carrying out operations for aspects of the present embodiments can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0225] The flow diagrams depicted herein and / or described above are merely illustrative examples, and are used merely to provide an understanding of various aspects of the present embodiments. One will readily appreciate that each block of the flow diagrams and / or combinations of blocks in the flow diagrams can be implemented by computer program instructions. Such program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flow diagrams and / or combinations of blocks in the flow diagrams.

[0226] These computer program instructions can also be stored in a computer readable medium that can direct a computer, a programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flow diagrams and / or combinations of blocks in the flow diagrams.

[0227] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0228] The preferred embodiments of the application are described above with the specific embodiments. The application can be modified and changed by various manners and can have various changes for those skilled in the art. Any modification, equivalent replacement, improvement and the like within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A method for determining sample batch size, characterized in that, The method comprises: determining N average calculation cycle numbers of the deep learning model based on N sample batch sizes, wherein N is a positive integer greater than or equal to 1, each average calculation cycle number corresponds to a sample batch size; determining a first candidate sample batch size according to the N average calculation cycle numbers; determining N calculation intensities of the deep learning model based on the N sample batch sizes, wherein each calculation intensity corresponds to a sample batch size; determining a second candidate sample batch size according to the N calculation intensities; determining a target sample batch size as the maximum value of the first candidate sample batch size and the second candidate sample batch size; wherein the determination of the first candidate sample batch size according to the N average calculation cycle numbers comprises: determining a first inflection point according to the order of the N sample batch sizes from small to large, the first inflection point being an inflection point of the N average calculation cycle numbers, the first inflection point being the first point at which the average calculation cycle number tends to be stable; determining the first candidate sample batch size as the sample batch size corresponding to the first inflection point; the determination of the second candidate sample batch size according to the N calculation intensities comprises: determining a second inflection point according to the order of the N sample batch sizes from small to large, the second inflection point being an inflection point of the N calculation intensities, the second inflection point being the first point at which the calculation intensity tends to be stable; determining the second candidate sample batch size as the sample batch size corresponding to the second inflection point; the determination of the N calculation intensities of the deep learning model based on the N sample batch sizes comprises: for each sample batch size in the N sample batch sizes, determining the calculation intensity corresponding to the sample batch size as the ratio of the total calculation amount of the deep learning model to the total memory amount of the deep learning model.

2. The method of claim 1, wherein, the determination of the N average calculation cycle numbers of the deep learning model based on the N sample batch sizes comprises: for each sample batch size in the N sample batch sizes, determining the calculation cycle numbers of a plurality of operators in the deep learning model; determining the total calculation cycle number corresponding to the sample batch size as the sum of the calculation cycle numbers of the plurality of operators; determining the average calculation cycle number as the ratio of each total calculation cycle to the sample batch size corresponding thereto.

3. The method of claim 1, wherein, the determination of the first inflection point according to the order of the N sample batch sizes from small to large comprises: taking N logarithms of the N average calculation cycle numbers; determining N first slope values according to the N logarithms; sorting the N first slope values according to the order of the corresponding sample batch sizes from small to large; determining the average calculation cycle number corresponding to the first time when the N first slope values exceed a first set threshold value in the sorting process as the first inflection point.

4. The method of claim 1, wherein, the determination of the N calculation intensities of the deep learning model based on the N sample batch sizes comprises: for each sample batch size in the N sample batch sizes, determining the calculation amounts of a plurality of operators in the deep learning model; Sum the total amount of computation of the plurality of operators in the deep learning model as the total computation amount of the deep learning model; For each of the N sample batch sizes, determine the data memory amount of the input and output data of the plurality of operators in the deep learning model, the parameter memory amount of the plurality of operators in the deep learning model, and the memory amount of the intermediate calculation results of the plurality of operators in the deep learning model; Sum the data memory amount of the input and output data of the plurality of operators in the deep learning model, the parameter memory amount of the plurality of operators in the deep learning model, and the memory amount of the intermediate calculation results of the plurality of operators in the deep learning model as the total memory amount of the deep learning model.

5. The method of claim 1, wherein, The second inflection point is determined according to the order of the N sample batch sizes from small to large, and specifically includes: Taking N logarithms of the N calculation intensities; Determine N second slope values according to the N logarithms; Sort the N second slope values according to the order corresponding to the sample batch sizes from small to large; The calculation intensity corresponding to the first time when the N second slope values are greater than a second set threshold value is determined as the second inflection point.

6. An apparatus for determining a sample batch size, the apparatus comprising: The device includes: A first determination unit configured to determine N average calculation cycle numbers of a deep learning model based on N sample batch sizes, wherein N is a positive integer greater than or equal to 1, and each average calculation cycle number corresponds to a sample batch size; A second determination unit configured to determine a first candidate sample batch size according to the N average calculation cycle numbers; A third determination unit configured to determine N calculation intensities of the deep learning model based on the N sample batch sizes, wherein each calculation intensity corresponds to a sample batch size; A fourth determination unit configured to determine a second candidate sample batch size according to the N calculation intensities; A fifth determination unit configured to determine a target sample batch size as the maximum value of the first candidate sample batch size and the second candidate sample batch size; The first candidate sample batch size is determined according to the N average calculation cycle numbers, and specifically includes: Determine a first inflection point according to the order of the N sample batch sizes from small to large, wherein the first inflection point is an inflection point of the N average calculation cycle numbers, and the first inflection point is the first point at which the average calculation cycle number tends to be stable; Determine a first candidate sample batch size corresponding to the first inflection point; The second candidate sample batch size is determined according to the N calculation intensities, and specifically includes: Determine a second inflection point according to the order of the N sample batch sizes from small to large, wherein the second inflection point is an inflection point of the N calculation intensities, and the second inflection point is the first point at which the calculation intensity tends to be stable; Determine a second candidate sample batch size corresponding to the second inflection point; The N calculation intensities of the deep learning model are determined based on the N sample batch sizes, and specifically include: For each of the N sample batch sizes, a ratio of the total computation amount of the deep learning model to the total memory amount of the deep learning model is determined as a computation intensity corresponding to the sample batch size.

7. Computer program instructions, characterized in that, The computer program instructions are stored on a computer readable storage medium, and when executed by a processor, implement the method of any one of claims 1-5.

Images