Optimization method of artificial intelligence model based on processor storage space
By dividing the input data of the artificial intelligence model into multiple target batches and optimizing the computing process, the performance differences and storage bandwidth bottlenecks between computing devices and hardware architectures are solved, and the computing efficiency and performance of the artificial intelligence model are improved.
Patent Information
- Application Number
- CN202410865437.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-01
AI Technical Summary
The existing technology is difficult to effectively optimize the computing performance of large-scale artificial intelligence models on different computing devices and hardware architectures, resulting in high computing costs and low algorithm development efficiency, especially the bottleneck of data transmission and storage bandwidth between multiple operators. .
By dividing the input data of the artificial intelligence model into multiple target batches, the calculation process is optimized based on the size of each target batch and the input and output data tensor size of the operator, and the intermediate calculation results are stored in the target buffer. In order to meet the buffer space requirements of each operator, reduce the bandwidth requirements for external memory.
It improves the effective utilization rate of the target buffer, reduces the bandwidth requirement for external memory, and improves the overall performance and computing efficiency of the artificial intelligence model.
Smart Images

Figure CN118426989B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to an optimization method for an artificial intelligence model based on processor storage space, an electronic device, and a storage medium. Background Art
[0002] In the field of artificial intelligence (AI), as the application scenarios of deep learning continue to generalize, AI computing tasks need to be deployed on different computing devices and hardware architectures. For example, the optimization of computing performance of a large-scale artificial intelligence model can be directly converted into savings in computing costs; at the same time, for AI computing tasks, the optimization of artificial intelligence model performance can provide more space for algorithms, allowing algorithm developers to try larger or more complex models in a reasonable time to handle more complex computing tasks. Summary of the invention
[0003] At least one embodiment of the present disclosure provides an optimization method for an artificial intelligence model based on a processor storage space, wherein the artificial intelligence model includes multiple operators, and the input data of the artificial intelligence model includes input batches, each batch in the input batches corresponds to a data tensor of the input data, and the optimization method of the artificial intelligence model includes: in response to the size of the storage space occupied by the input data and output data of the multiple operators being larger than the size of a target buffer, dividing the input batch of the input data of the artificial intelligence model into multiple target batches, wherein the size of each target batch in the multiple target batches meets the alignment requirement of each operator in the multiple operators; based on the size of each target batch and the size of the data tensor of the input data and output data of each operator, using the multiple operators to calculate the input data of the artificial intelligence model to obtain intermediate calculation results of the multiple operators; and storing the intermediate calculation results of the multiple operators in the target buffer.
[0004] For example, in the optimization method of an artificial intelligence model provided by at least one embodiment of the present disclosure, the input data of the artificial intelligence model is stored in a first memory, and the optimization method of the artificial intelligence model also includes: based on the size of each target batch and the size of the data tensor of the input data and output data of each operator, using the multiple operators to calculate the input data of the artificial intelligence model to obtain the output data of the artificial intelligence model; and storing the output data of the artificial intelligence model in the first memory.
[0005] For example, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure also includes: in response to the size of the storage space occupied by the input data and output data of the multiple operators being less than or equal to the size of the target buffer, using the multiple operators to calculate the input data of the artificial intelligence model to obtain intermediate calculation results of the multiple operators; and storing the intermediate calculation results of the multiple operators in the target buffer.
[0006] For example, in the optimization method of an artificial intelligence model provided by at least one embodiment of the present disclosure, the input data of the artificial intelligence model is calculated using the multiple operators based on the size of each target batch and the size of the data tensor of the input data and output data of each operator to obtain intermediate calculation results of the multiple operators, including: calculating the buffer space size requirement of each operator based on the size of each target batch and the size of the data tensor of the input data and output data of each operator; in response to the buffer space size requirement of each operator being less than or equal to the size of the target buffer, using the multiple operators to calculate the data corresponding to each target batch, to obtain the intermediate calculation results of the multiple operators corresponding to each target batch.
[0007] For example, in the optimization method of the artificial intelligence model provided in at least one embodiment of the present disclosure, the multiple operators include the 1st operator to the Nth operator, N is an integer greater than 1, i=1, 2, ..., N, and the buffer space size requirement of each operator is calculated based on the size of each target batch and the size of the data tensor of the input data and output data of each operator, including: based on the size Nb of each target batch and the size Ta of the data tensor of the input data of the i-th operator among the multiple operators, the buffer space size requirement R (Op i-input ), where R(Op i-input )=Nb×Ta×α, α is the byte size of the data type; based on the size Nb of each target batch and the size To of the data tensor of the output data of the i-th operator, calculate the buffer space size requirement R(Op i-output ), where R(Op i-output )=Nb×To×α;Based on the buffer space size requirement R(Op i-input ) and the buffer space requirement for output data R(Op i-output ), calculate the buffer space requirement R(Op i ), where R(Op i )=R(Op i-input )+R(Op i-output ).
[0008] For example, in the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure, the multiple target batches include the 1st target batch to the Mth target batch, M is an integer greater than 1, and in response to the buffer space size requirement of each operator being less than or equal to the size of the target buffer, the multiple operators are used to calculate the data corresponding to each target batch, including: using the multiple operators to sequentially calculate the data corresponding to the 1st target batch to the data corresponding to the Mth target batch.
[0009] For example, in the optimization method of an artificial intelligence model provided by at least one embodiment of the present disclosure, the multiple operators include the 1st operator to the Nth operator in a first calculation order, where N is an integer greater than 1, and in response to the buffer space size requirement of each operator being less than or equal to the size of the target buffer, the intermediate calculation results of the multiple operators include the output data of the 1st operator to the output data of the N-1th operator, wherein, among the multiple operators, the output data of the kth operator serves as the input data of the k+1th operator, k=1, 2,…, N-1.
[0010] For example, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure also includes: in response to the buffer space size requirement of at least one operator among the multiple operators being greater than the size of the target buffer, dividing the artificial intelligence model into multiple first-level sub-models according to preset segmentation rules.
[0011] For example, in the optimization method of an artificial intelligence model provided by at least one embodiment of the present disclosure, the multiple first-level sub-models include the 1st sub-model to the Qth sub-model in a second calculation order, where Q is an integer greater than 1, wherein, among the multiple first-level sub-models, the output data of the jth sub-model serves as the input data of the j+1th sub-model, j=1, 2,…, Q-1, wherein the input data and output data of each sub-model in the multiple first-level sub-models are stored in a first memory.
[0012] For example, in the optimization method of an artificial intelligence model provided by at least one embodiment of the present disclosure, the multiple first-level sub-models include a multi-operator sub-model, and the multi-operator sub-model includes at least two operators. The optimization method of the artificial intelligence model also includes: in response to the minimum batch value that satisfies the alignment requirement of each operator in the multi-operator sub-model being smaller than the size of the target batch, dividing the multiple target batches input into the multi-operator sub-model into multiple first sub-batches, wherein the size of each first sub-batch in the multiple first sub-batches is equal to the minimum batch value that satisfies the alignment requirement of each operator in the multi-operator sub-model.
[0013] For example, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure also includes: calculating the buffer space size requirement of each operator in the multi-operator sub-model based on the size of each first sub-batch and the size of the data tensor of the input data and output data of each operator in the multi-operator sub-model; in response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, using the multi-operator sub-model to calculate the data corresponding to each first sub-batch, and obtaining the intermediate calculation results of the multiple operators in the multi-operator sub-model corresponding to each first sub-batch; and storing the intermediate calculation results of the multiple operators in the multi-operator sub-model corresponding to each first sub-batch in the target buffer.
[0014] For example, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure also includes: in response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, using the multi-operator sub-model to calculate the data corresponding to each first sub-batch, and obtaining the output data of the multi-operator sub-model corresponding to each first sub-batch; storing the output data of the multi-operator sub-model corresponding to each first sub-batch in a first memory.
[0015] For example, in the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure, the multiple first sub-batches include the 1st first sub-batch to the Pth first sub-batch, P is an integer greater than 1, and in response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, the multi-operator sub-model is used to calculate the data corresponding to each first sub-batch, including: in response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, the multi-operator sub-model is used to sequentially calculate the data corresponding to the 1st first sub-batch to the data corresponding to the Pth first sub-batch.
[0016] For example, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure also includes: in response to the buffer space size requirement of at least one operator in the multi-operator sub-model being greater than the size of the target buffer, dividing the multi-operator sub-model into multiple second-level sub-models according to the preset segmentation rule.
[0017] For example, in the optimization method of an artificial intelligence model provided by at least one embodiment of the present disclosure, the multiple operators of the artificial intelligence model include the 1st operator to the Nth operator in a first calculation order, where N is an integer greater than 1, and in response to the buffer space size requirement of at least one operator among the multiple operators being greater than the size of the target buffer, the artificial intelligence model is divided into multiple first-level sub-models according to a preset segmentation rule, including: in response to the buffer space size requirement of at least one operator among the multiple operators being greater than the size of the target buffer, a segmentation operator is selected from the at least one operator, and the segmentation operator is used as a single operator sub-model; in accordance with the first calculation order, the operators from the 1st operator to the operator before the segmentation operator are used as the first sub-model, and the operators after the segmentation operator to the Nth operator are used as the second sub-model, wherein the single operator sub-model, the first sub-model and the second sub-model are sub-models of the first level.
[0018] For example, in the optimization method of the artificial intelligence model provided in at least one embodiment of the present disclosure, the segmentation operator is the operator with the largest buffer space size requirement among the at least one operator.
[0019] For example, in the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure, in response to the at least one operator including at least two operators with the largest buffer space size requirements, the splitting operator is the operator that comes first in the first calculation order among the at least two operators with the largest buffer space size requirements.
[0020] For example, in the optimization method of the artificial intelligence model provided in at least one embodiment of the present disclosure, at least one of the first sub-model and the second sub-model is a multi-operator sub-model.
[0021] At least one embodiment of the present disclosure further provides an electronic device. The electronic device includes: a processor; a memory including one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the optimization method of the artificial intelligence model based on the processor storage space provided in any embodiment of the present disclosure.
[0022] At least one embodiment of the present disclosure further provides a storage medium storing non-temporary computer-readable instructions, which, when executed by a computer, implements the optimization method of the artificial intelligence model based on the processor storage space provided by any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.
[0024] Figure 1A A schematic diagram of an example of data reading and calculation in an artificial intelligence model;
[0025] Figure 1B A schematic diagram of another example of data reading and calculation in an artificial intelligence model;
[0026] Figure 2 An exemplary flow chart of an optimization method of an artificial intelligence model provided for at least one embodiment of the present disclosure;
[0027] Figure 3 for Figure 2 An exemplary flowchart of an example of step S120 in FIG.
[0028] Figure 4 A schematic diagram of an example of an optimization method for an artificial intelligence model provided by at least one embodiment of the present disclosure;
[0029] Figure 5 An exemplary flow chart of another example of a method for optimizing an artificial intelligence model provided by at least one embodiment of the present disclosure;
[0030] Fig. 6A A schematic diagram of an example of an artificial intelligence model provided for at least one embodiment of the present disclosure;
[0031] Figure 6B A schematic diagram of an example of a plurality of first-level sub-models provided for at least one embodiment of the present disclosure;
[0032] Figure 6C A schematic diagram of another example of an optimization method for an artificial intelligence model provided by at least one embodiment of the present disclosure;
[0033] Figure 7 A schematic block diagram of an optimization device for an artificial intelligence model provided in at least one embodiment of the present disclosure;
[0034] Figure 8 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure;
[0035] Fig. 9 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure; and
[0036] Fig.10 A schematic diagram of a storage medium provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0038] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0039] The present disclosure is described below through several specific embodiments. In order to keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components may be omitted. When any component of the embodiments of the present disclosure appears in more than one figure, the component is represented by the same or similar reference numeral in each figure.
[0040] The artificial intelligence model consists of multiple operators (Operators, Ops) to handle different computing tasks. In the artificial intelligence model, the operator corresponds to the computing logic of the layer or node in the model. For example, the convolution calculation task in the convolution layer (Convolution Layer) can be an operator; the weight summation calculation task in the fully-connected layer (FC layer) can also be an operator. When implementing the artificial intelligence model, the operator can be implemented by software (for example, through a computer program) or by hardware (for example, through a circuit).
[0041] During the training or reasoning process of the artificial intelligence model, it is necessary to call multiple operators in the artificial intelligence model to perform predetermined computing tasks. When executing the computing task, the execution unit reads the input data and weight data of the operator from the storage device, and the read input data and weight data are calculated in the operator and the calculation result is used as the output data of the operator. For example, the weight data is only accessed at a specific location in the model, while the input data and output data will always propagate forward (also known as data transfer between operators). For example, the output data of the previous operator is the input data of the next operator, so the input data and output data need to be read or stored repeatedly; while the weight data is only the input of the operator, not the output of the operator, nor the input of the next operator, and does not need to be read or stored repeatedly.
[0042] For example, for data transfer between operators, except for a small number of operators with fixed fusion modes, the calculation process of most operators is completed in blocks, and the input data and output data generated by these operators during the data transfer process need to be stored in the storage device. For example, on the one hand, since the storage space of the register resources of the execution unit is much smaller than the storage space of the storage device, the execution unit can only store a small part of the input data and output data of the operator; on the other hand, the positions of the two upstream and downstream operators that the execution unit needs to read may also be different, for example, data at certain positions need to be shared. Therefore, after the previous operator is calculated, the output data obtained needs to be placed in a storage hierarchy visible to the execution unit (for example, a storage device that meets the above requirements) as the input data of the next operator; the next operator then reads the input data at the corresponding position of the storage device as required.
[0043] Figure 1A A schematic diagram of an example of data reading and calculation in an artificial intelligence model.
[0044] For example, Figure 1A As shown, when executing a computing task, each operator needs to read input data and weight data from an external memory to complete the calculation of the current operator and obtain output data. For example, operator 0 reads input data 0 and weight data 0 from an external memory, and obtains output data 0 after performing calculations (not shown in the figure); output data 0 is stored in the external memory as input data 1 of the next operator 1. For example, further, operator 1 reads input data 1 and weight data 1 from an external memory and obtains output data 1 after performing calculations (not shown in the figure); output data 1 is stored in the external memory as input data 2 of the next operator 2... and so on, until the calculations of all operators in the artificial intelligence model are completed.
[0045] For example, in Figure 1AIn the example, data transfer between multiple operators is performed through external memory. For example, during the calculation process of each operator, the input data and weight data of the original size need to be read from the external memory at the same time. After the calculation is completed in the current operator, the output data obtained is stored in the external memory before entering the calculation of the next operator. In the case of limited bandwidth of the external memory, the bandwidth of the external memory may not be sufficient to support the loading and storage of multiple large-size data. The serious bandwidth bottleneck leads to a large waste of computing power of the artificial intelligence model, thereby reducing the overall performance of the model.
[0046] Figure 1B A schematic diagram of another example of data reading and calculation in an artificial intelligence model.
[0047] For example, Figure 1B As shown in the figure, for some small artificial intelligence models, if the sum of the input data and output data sizes of all operators in the model does not exceed the size of a certain intermediate buffer, the input data and output data of each operator can be stored in the intermediate buffer instead of the external memory, that is, the data transfer between operators can be carried out through the intermediate buffer. For example, during the operator calculation process, only the weight data needs to be read from the external memory, and the input data and output data can be read from the high-bandwidth intermediate buffer, reducing the bandwidth demand for the external memory.
[0048] For example, Figure 1B As shown, operator 0 reads input data 0 from the intermediate buffer and weight data 0 from the external memory, and obtains output data 0 after performing calculation (not shown in the figure); output data 0 is stored in the intermediate buffer as input data 1 of the next operator 1. For example, further, operator 1 reads input data 1 from the intermediate buffer and weight data 1 from the external memory, and obtains output data 1 after performing calculation (not shown in the figure); output data 1 is stored in the intermediate buffer as input data 2 of the next operator 2... and so on, until the calculation of all operators in the artificial intelligence model is completed.
[0049] For example, for Figure 1B For example, it is required that the sum of the input data and output data sizes of all operators in the model does not exceed the size of the intermediate buffer; if this size requirement is not met, it can only be implemented as follows Figure 1AThe scheme shown. For example, the size of the input data and output data of an operator is the product of the data tensor size of a single sample and the batch size, and the batch size of the input data and output data represents the number of samples contained in the data. For example, for computing tasks such as deep learning, larger batches of input data are required to meet the computing requirements, and as the network model becomes larger and larger, the number of input data and output data of multiple operators also increases significantly, resulting in most artificial intelligence models being unable to meet the size requirements of the intermediate buffer. Therefore, Figure 1B The solution has a small scope of application, resulting in the intermediate buffer not being effectively utilized.
[0050] At least one embodiment of the present disclosure provides an optimization method for an artificial intelligence model based on processor storage space. The artificial intelligence model includes multiple operators, and the input data of the artificial intelligence model includes input batches, and each batch in the input batches corresponds to a data tensor of the input data. The optimization method of the artificial intelligence model includes: in response to the size of the storage space occupied by the input data and output data of multiple operators being larger than the size of the target buffer, the input batch of the input data of the artificial intelligence model is divided into multiple target batches, wherein the size of each target batch in the multiple target batches meets the alignment requirements of each operator in the multiple operators; based on the size of each target batch and the size of the data tensor of the input data and output data of each operator, multiple operators are used to calculate the input data of the artificial intelligence model to obtain the intermediate calculation results of multiple operators; the intermediate calculation results of multiple operators are stored in the target buffer.
[0051] At least one embodiment of the present disclosure also provides an optimization device, electronic device and storage medium for an artificial intelligence model, which are used to implement the optimization method of the artificial intelligence model of the above-mentioned embodiment.
[0052] The method, apparatus, electronic device, and storage medium provided by at least one embodiment of the present disclosure can divide the input batch of the input data of the artificial intelligence model into multiple target batches, and can store the intermediate calculation results of multiple operators calculated based on the target batches in a target buffer, so that the target buffer can also be used when the size of the input batch of the input data is large, thereby improving the effective utilization rate of the target buffer, reducing the bandwidth requirement for the external memory, and improving the performance of the artificial intelligence model.
[0053] Hereinafter, at least one embodiment of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the same reference numerals in different drawings will be used to refer to the same elements described.
[0054] The artificial intelligence model provided by at least one embodiment of the present disclosure includes multiple operators, and the input data of the artificial intelligence model includes an input batch (Batch), and each batch in the input batch corresponds to a data tensor (Tensor) of the input data. For example, the size of the input data of each operator is the product of the data tensor size of a single sample and the input batch size, and the batch size of the input data represents the number of samples contained in the input data. For example, taking four dimensions [N, C, H, W] as an example, the input batch size of the input data with a layout of [N, C, H, W] is N, indicating that the number of samples contained in the input data is N, and the data tensor size of a single sample is the product of the number of channels C, the height H and the width W; the shape of the input data can also be two-dimensional, three-dimensional, four-dimensional, etc., and input batches and data tensors of different dimensions can be selected according to actual needs, and the embodiments of the present disclosure are not limited to this.
[0055] Figure 2 An exemplary flow chart of a method for optimizing an artificial intelligence model provided for at least one embodiment of the present disclosure.
[0056] For example, Figure 2 As shown, at least one embodiment of the present disclosure provides an optimization method for an artificial intelligence model, which can be executed on a hardware device such as a processor and a memory in an artificial intelligence chip. For example, multiple operators of the artificial intelligence model can be calculated in a processor. For example, the artificial intelligence chip also includes a target buffer, which can be an intermediate buffer (buffer) with a high bandwidth and access speed, and developers can control the type of data stored therein. For example, the optimization method of the artificial intelligence model can include the following steps S110 to S130.
[0057] Step S110: In response to the size of the storage space occupied by the input data and output data of multiple operators being larger than the size of the target buffer, an input batch of the input data of the artificial intelligence model is divided into multiple target batches.
[0058] For example, in step S110, when the size of the storage space occupied by the input data and output data of all operators in the artificial intelligence model is larger than the size of the target buffer (i.e., it is not possible to use, for example, Figure 1B In the scheme shown in FIG. 1 , the processor can split the input batch of input data into multiple target batches so that the subsequent calculation process meets the size requirements of the target buffer. For example, since the calculation process of most deep learning operators is independent of the input batch, after the input batch is split, the total number of samples in the calculation process will not decrease, and the calculation result of the data tensor of each sample will not be affected.
[0059] For example, in step S110, the size of each target batch obtained by segmentation meets the alignment requirement of each operator. For example, for the alignment requirement of a specific operator, the batch N of the input data of the operator needs to be an integer multiple of a specific value Nop, and the specific value Nop is determined by the inherent properties of the operator or the specific implementation of the operator.
[0060] Step S120: Based on the size of each target batch and the size of the data tensor of the input data and output data of each operator, multiple operators are used to calculate the input data of the artificial intelligence model to obtain the intermediate calculation results of multiple operators.
[0061] For example, in step S120, based on the size of each target batch and the size of the data tensor of the input data and output data of each operator, the processor can determine whether the storage space size required by each operator in the model meets the size requirement of the target buffer; if there are operators that do not meet the above size requirements, the processor can further split the target batch. For example, the intermediate calculation results of multiple operators calculated by the processor all meet the size requirement of the target buffer.
[0062] For example, in step S120, the current operator can only complete the calculation of the data corresponding to the split batch and then enter the calculation of the next operator, without having to complete the calculation of the data corresponding to the entire input batch at one time, thereby reducing the scale of data transmission between operators and reducing the size requirement of the target buffer.
[0063] Step S130: storing the intermediate calculation results of the multiple operators in the target buffer.
[0064] For example, in step S130, since the intermediate calculation results of the multiple operators calculated in step S120 all meet the size requirements of the target buffer, the processor can store the intermediate calculation results of the multiple operators in the target buffer, thereby reducing the bandwidth requirements for the external memory.
[0065] It should be noted that each step in the optimization method of the artificial intelligence model provided in at least one embodiment of the present disclosure can be executed by a hardware device such as a processor, and the calculated results are also stored in a corresponding memory or buffer. The specific selection can be made according to actual needs, and the embodiments of the present disclosure do not limit this.
[0066] In some examples, the input data of the artificial intelligence model is stored in the first memory. For example, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure may also include steps S140 to S150.
[0067] Step S140: Based on the size of each target batch and the size of the data tensor of the input data and output data of each operator, use multiple operators to calculate the input data of the artificial intelligence model to obtain the output data of the artificial intelligence model;
[0068] Step S150: Store the output data of the artificial intelligence model in the first memory.
[0069] For example, in step S140, after the input data of the artificial intelligence model is calculated by multiple operators, in addition to the intermediate calculation results of the multiple operators obtained in step S120, the final output data of the artificial intelligence model is also obtained. For example, further, in step S150, the obtained output data of the artificial intelligence model is stored in the first memory.
[0070] For example, since the initial input data of the artificial intelligence model only needs to be read once, and the final output data only needs to be stored once, the input data and output data of the artificial intelligence model can be stored in the first memory; since the intermediate calculation result of the previous operator is the input of the next operator, and the reading and writing speed of the target buffer is several to dozens of times faster than that of the first memory, using the target buffer to store the intermediate calculation results of multiple operators can improve the data transmission efficiency between operators.
[0071] It should be noted that in at least one embodiment of the present disclosure, the target buffer may be an intermediate cache device such as a tensor buffer (GemmMain Buffer, GMB), the first memory may be a storage device such as a high bandwidth memory (HighBandwidth Memory, HBM), and the target buffer and the first memory may also select other types of storage devices according to actual needs, and the embodiments of the present disclosure are not limited to this; in addition, corresponding to different types of artificial intelligence models or different algorithm execution processes of artificial intelligence models, operators and their input data and output data may be of different types, and the input batch size, data tensor size and target batch size obtained by segmentation may also be selected according to actual conditions, and the embodiments of the present disclosure are not limited to this.
[0072] In some examples, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure may also include steps S160 to S170.
[0073] Step S160: In response to the size of the storage space occupied by the input data and output data of the multiple operators being less than or equal to the size of the target buffer, using the multiple operators to calculate the input data of the artificial intelligence model to obtain intermediate calculation results of the multiple operators;
[0074] Step S170: storing the intermediate calculation results of the multiple operators in the target buffer.
[0075] For example, in step S160, when the size of the storage space occupied by the input data and output data of all operators in the artificial intelligence model is less than or equal to the size of the target buffer, multiple operators can be used to calculate the input data of the artificial intelligence model to obtain the intermediate calculation results of multiple operators.
[0076] For example, further in step S170, the intermediate calculation result of each operator is stored in the target buffer to improve the data access efficiency in the calculation process of multiple operators.
[0077] Figure 3 for Figure 2 An exemplary flowchart of an example of step S120 in FIG.
[0078] For example, Figure 3 As shown, Figure 2 The step S120 may further include steps S121 to S122.
[0079] Step S121: Calculate the buffer space size requirement of each operator based on the size of each target batch and the size of the data tensor of the input data and output data of each operator.
[0080] For example, in step S121, the buffer space size requirement of each operator is used to determine whether the storage space size required for the input data and output data of the operator meets the size requirement of the target buffer. The buffer space size requirement of each operator is related to the size of the target batch and the size of the data tensor of the input data and output data of the operator.
[0081] For example, specifically, the multiple operators include the 1st operator to the Nth operator, where N is an integer greater than 1, i=1, 2, ..., N. For example, step S121 may further include steps S1211 to S1213:
[0082] Step S1211: Based on the size Nb of each target batch and the size Ta of the data tensor of the input data of the ith operator among the multiple operators, calculate the buffer space size requirement R (Op i-input ), where R(Op i-input ) = Nb×Ta×α, where α is the byte size of the data type;
[0083] Step S1212: Based on the size Nb of each target batch and the size To of the data tensor of the output data of the ith operator, calculate the buffer space size requirement R (Op i-output ), where R(Op i-output ) = Nb × To × α;
[0084] Step S1213: Based on the buffer space size requirement R(Op i-input ) and the buffer space requirement for output data R(Op i-output ), calculate the buffer space requirement R(Op i ), where R(Op i )=R(Op i-input )+R(Op i-output ).
[0085] For example, through S1211 to S1213, based on the size Nb of each target batch, the size Ta of the data tensor of the input data of the ith operator, and the size To of the data tensor of the output data, the buffer space size requirement R (Op i )=Nb×(Ta+To)×α. In some examples, α may represent the byte size of the data type, and its specific value may be selected according to the actual data type, which is not limited in the embodiments of the present disclosure.
[0086] Step S122: In response to the buffer space size requirement of each operator being less than or equal to the size of the target buffer, multiple operators are used to calculate the data corresponding to each target batch to obtain intermediate calculation results of the multiple operators corresponding to each target batch.
[0087] For example, in step S122, the buffer space requirement R (Op i ) is compared with the size of the target buffer; if the buffer space size requirement R(Op i ) are both smaller than or equal to the size of the target buffer, the target batch can be used as the batch for performing calculations without further splitting the target batch.
[0088] For example, in step S122, by using multiple operators of the artificial intelligence model to calculate the data corresponding to each target batch, the intermediate calculation results of the multiple operators corresponding to each target batch can be obtained. For example, specifically, the multiple target batches include the 1st target batch to the Mth target batch, where M is an integer greater than 1; step S122 may further include: using multiple operators to sequentially calculate the data corresponding to the 1st target batch to the Mth target batch.
[0089] For example, specifically, all operators may be used to calculate the current target batch first, and then all operators may be used to calculate the next target batch, thereby obtaining intermediate calculation results of multiple operators corresponding to each target batch.
[0090] For example, in step S122, the plurality of operators include the 1st operator to the Nth operator in the first calculation order, where N is an integer greater than 1. In some examples, in response to the buffer space size requirement R(Op i ) is less than or equal to the size of the target buffer, and the intermediate calculation results of multiple operators may include the output data of the 1st operator to the output data of the N-1th operator. For example, among multiple operators, the output data of the kth operator is used as the input data of the k+1th operator, where k = 1, 2, ..., N-1.
[0091] For example, for the current target batch, the first calculation order may include: inputting the input data corresponding to the current target batch into the first operator for calculation, using the k+1th operator to calculate the output data of the kth operator, until the Nth operator is calculated. For example, the output data of the Nth operator is the output data of the artificial intelligence model corresponding to the current target batch. For example, the intermediate calculation results of the artificial intelligence model corresponding to each target batch can be stored in the target buffer, and the input data and output data of the artificial intelligence model corresponding to each target batch can be stored in the first memory.
[0092] Figure 4 A schematic diagram of an example of an optimization method for an artificial intelligence model provided by at least one embodiment of the present disclosure. For example, Figure 4 for Figure 2 Steps S110 to S130 and Figure 3 A specific example of steps S121 to S122 in .
[0093] For example, Figure 4 As shown, assuming that the artificial intelligence model includes two operators (operator 0 and operator 1), if the size of the storage space occupied by the input data and output data of operator 1 and operator 2 is larger than the size of the target buffer, then execute Figure 2 In step S110, the input batch of the input data of the artificial intelligence model is divided into a plurality of target batches (for example, Figure 4 For example, the sizes of target batches B0 and B1 both meet the alignment requirements of operator 0 and operator 1, that is, target batches B0 and B1 are integer multiples of the specific value Nop0 of operator 0 and the specific value Nop1 of operator 1, respectively.
[0094] For example, Figure 4 As shown, in Figure 2 In step S120, operator 0 and operator 1 are used to calculate the input data of the artificial intelligence model to obtain the intermediate calculation results of operator 0 and operator 1 (for example, Figure 4 Output B0-operator 0 and output B1-operator 1) shown in FIG. 1 ; further, in Figure 2In step S130, the intermediate calculation results of operator 0 and operator 1 are stored in the target buffer.
[0095] For example, specifically, step S121 is executed, assuming that the sizes of the target batches B0 and B1 are both Nb, based on the size Nb of each target batch, the size of the data tensor of the input data of operator 0 and operator 1, and the size of the data tensor of the output data, the buffer space size requirements R(Op0) and R(Op1) of operator 0 and operator 1 can be calculated. For example, if R(Op0) and R(Op1) are both less than or equal to the size of the target buffer, step S122 is executed, and operator 0 and operator 1 are used to calculate the data corresponding to the target batches B0 and B1 respectively, without further splitting the target batches B0 or B1.
[0096] For example, Figure 4 As shown, the target batch B0 can be calculated using operator 0 and operator 1 first, and then the target batch B1 can be calculated using operator 0 and operator 1. For example, when calculating the target batch B0, according to the first calculation order, the input data (input B0) corresponding to the target batch B0 is read from the first memory to operator 0, and after calculation by operator 0, the intermediate calculation result (output B0-operator 0) of operator 0 corresponding to the target batch B0 is obtained, and the output B0-operator 0 is used as the input data of operator 1 (input B0-operator 1) and is stored in the target buffer; further, the input B0-operator 1 is read from the target buffer to operator 1, and after calculation by operator 1, the output data (output B0) obtained is the output data of the artificial intelligence model corresponding to the target batch B0, and the output B0 is stored in the first memory.
[0097] For example, Figure 4 As shown, when calculating the target batch B1, according to the first calculation order, the input data (input B1) corresponding to the target batch B1 is read from the first memory to operator 0, and after calculation by operator 0, the intermediate calculation result of operator 0 corresponding to the target batch B1 (output B1-operator 0) is obtained, and the output B1-operator 0 is used as the input data of operator 1 (input B1-operator 1) and is stored in the target buffer; further, the input B1-operator 1 is read from the target buffer to operator 1, and after calculation by operator 1, the output data (output B1) obtained is the output data of the artificial intelligence model corresponding to the target batch B1, and the output B1 is stored in the first memory.
[0098] It should be noted that Figure 4 The example only shows the process of transferring input data and output data of different target batches between operators. The weight data is in Figure 4 The specific transmission process of weight data can refer to the description in the art and will not be repeated here.
[0099] It should be noted that Figure 4 The optimization method of the artificial intelligence model shown is only exemplary, and the architecture of the artificial intelligence model, the types and number of operators included, the number and size of the target batches obtained by segmentation, the specific implementation method of the target buffer and the first memory, etc. can all be selected according to actual needs, and the embodiments of the present disclosure do not limit this.
[0100] At least one embodiment of the present disclosure provides an optimization method for an artificial intelligence model. By dividing an input batch of input data of the artificial intelligence model into multiple target batches, intermediate calculation results of multiple operators calculated based on the target batches can be stored in a target buffer. Therefore, the target buffer can also be used when the size of the input batch of the input data is large, thereby improving the effective utilization rate of the target buffer, reducing the bandwidth requirement for the external memory, and improving the performance of the artificial intelligence model.
[0101] In some examples, such as Figure 3 As shown, Figure 2 The step S120 may further include a step S123.
[0102] Step S123: In response to the buffer space size requirement of at least one operator among the multiple operators being greater than the size of the target buffer, the artificial intelligence model is divided into multiple first-level sub-models according to a preset segmentation rule.
[0103] For example, in step S123, the buffer space requirement R (Op i ) is compared with the size of the target buffer; if there is one or more operators whose buffer space size requirement is greater than the size of the target buffer, the artificial intelligence model can be further segmented according to the preset segmentation rule to obtain a sub-model that meets the size requirement of the target buffer. For example, the preset segmentation rule can be selected according to actual conditions, and the embodiments of the present disclosure are not limited to this.
[0104] In some examples, the plurality of first-level sub-models include the 1st sub-model to the Qth sub-model in the second calculation order, where Q is an integer greater than 1. For example, among the plurality of first-level sub-models, the output data of the jth sub-model is used as the input data of the j+1th sub-model, where j=1, 2, ..., Q-1.
[0105] For example, the second calculation order may include: inputting the input data into the 1st sub-model for calculation, using the j+1th sub-model to calculate the output data of the jth sub-model, until the Qth sub-model is calculated. For example, the intermediate calculation results of each sub-model may be stored in the target buffer, and the initial input data and final output data of each sub-model may be stored in the first memory.
[0106] In some examples, the plurality of first-level sub-models include a multi-operator sub-model, and the multi-operator sub-model includes at least two operators. For example, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure may also include step S1231.
[0107] Step S1231: In response to the minimum batch value satisfying the alignment requirement of each operator in the multi-operator sub-model being smaller than the size of the target batch, multiple target batches input into the multi-operator sub-model are divided into multiple first sub-batches.
[0108] For example, in step S1231, for the first-level sub-model obtained by segmentation in step S123, if there is a multi-operator sub-model, the minimum batch value that meets the alignment requirements of each operator in the sub-model can be determined, and the minimum batch value can be compared with the target batch size.
[0109] For example, in step S1231, for the N operators in the sub-model, the alignment requirement Ni=n×Nopi of the ith operator is determined (i.e., Ni satisfies an integer multiple of Nopi), where n is a positive integer, i=1,2,…,N, and Nopi is a specific value related to the properties of the ith operator itself. For example, the minimum value is selected from Ni (i=1,2,…,N) of the N operators as the minimum batch value. If the minimum batch value is smaller than the size of the target batch, the target batch can be further divided into multiple first sub-batches. -
[0110] For example, in step S1231, the size of each first sub-batch obtained by segmentation is equal to the minimum batch value that satisfies the alignment requirement Ni of each operator in the multi-operator sub-model.
[0111] In some examples, after obtaining the first sub-batch of the multi-operator sub-model, the multi-operator sub-model may be used to calculate the data corresponding to each first sub-batch. For example, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure may also include steps S1232 to S1234.
[0112] Step S1232: Based on the size of each first sub-batch and the size of the data tensors of the input data and output data of each operator in the multi-operator sub-model, calculate the buffer space size requirement of each operator in the multi-operator sub-model.
[0113] For example, in step S1232, in order to determine whether the storage space size required by the multi-operator sub-model based on the input data and output data of the first sub-batch meets the size requirement of the target buffer, based on the size Nb' of each first sub-batch and the size Ta' of the data tensor of the input data and the size To' of the data tensor of the output data of each operator in the multi-operator sub-model, the buffer space size requirement R' (Op i )=Nb'×(Ta'+To')×α.
[0114] Step S1233: In response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, the multi-operator sub-model is used to calculate the data corresponding to each first sub-batch, and the intermediate calculation results of the multiple operators in the multi-operator sub-model corresponding to each first sub-batch are obtained.
[0115] For example, in step S1233, the buffer space requirement R'(Op i ) is compared with the size of the target buffer; if the buffer space size requirement of all operators is R'(Op i ) are both smaller than or equal to the size of the target buffer, the first sub-batch can be used as the batch for performing calculations without further splitting the first sub-batch.
[0116] For example, specifically, the multiple first sub-batches include the 1st first sub-batch to the Pth first sub-batch, where P is an integer greater than 1. For example, step S1233 may further include: in response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, using the multi-operator sub-model to sequentially calculate data corresponding to the 1st first sub-batch to data corresponding to the Pth first sub-batch.
[0117] For example, specifically, all operators of the multi-operator sub-model can be used to calculate the current first sub-batch, and then all operators can be used to calculate the next first sub-batch, so as to obtain the intermediate calculation results of multiple operators in the multi-operator sub-model corresponding to each first sub-batch.
[0118] Step S1234: Store the intermediate calculation results of the multiple operators in the multi-operator sub-model corresponding to each first sub-batch in the target buffer.
[0119] For example, in step S1234, since the intermediate calculation results of the multi-operator sub-model calculated in step S1233 all meet the size requirements of the target buffer, the intermediate calculation results of the multi-operator sub-model can be stored in the target buffer, thereby reducing the bandwidth requirements for the external memory.
[0120] In some examples, after calculation, in addition to obtaining the intermediate calculation results of the multi-operator sub-model, the final output data of the multi-operator sub-model is also obtained. For example, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure may also include steps S1235 to S1236.
[0121] Step S1235: In response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, the multi-operator sub-model is used to calculate the data corresponding to each first sub-batch, and the output data of the multi-operator sub-model corresponding to each first sub-batch is obtained;
[0122] Step S1236: Store the output data of the multi-operator sub-model corresponding to each first sub-batch in the first memory.
[0123] For example, in step S1235, if the buffer space size requirement R'(Op i ) are less than or equal to the size of the target buffer, the multi-operator sub-model is used to calculate the data corresponding to each first sub-batch. In addition to the intermediate calculation results obtained in step S1233, the output data of the multi-operator sub-model corresponding to each first sub-batch will also be obtained.
[0124] For example, further in step S1236, the output data of the obtained multi-operator sub-model corresponding to each first sub-batch is stored in the first memory.
[0125] In some examples, if the storage space size required by the multi-operator sub-model based on the input data and output data of the first sub-batch does not meet the size requirement of the target buffer, the multi-operator sub-model needs to be further segmented. For example, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure may also include step S1237.
[0126] Step S1237: In response to the buffer space size requirement of at least one operator in the multi-operator sub-model being greater than the size of the target buffer, the multi-operator sub-model is divided into a plurality of second-level sub-models according to a preset division rule.
[0127] For example, in step S1237, if the multi-operator sub-model has one or more operators with a buffer space size requirement R'(Op i ) is larger than the size of the target buffer, the multi-operator sub-model is divided into a plurality of second-level sub-models according to a preset segmentation rule to obtain a sub-model that meets the size requirement of the target buffer. For example, the preset segmentation rule may be the same as step S123, which will not be described in detail herein.
[0128] Figure 5An exemplary flowchart of another example of an optimization method for an artificial intelligence model provided by at least one embodiment of the present disclosure. For example, Figure 5 for Figure 3 A specific example of steps S121 to S123 in FIG.
[0129] For example, Figure 5 As shown, after the data corresponding to each target batch is input into the artificial intelligence model, in step S121, the buffer space size requirement R (Op i For example, further, in step S122, the buffer space size requirement R (Op i ) meets the size requirement of the target buffer; if so, use the artificial intelligence model to calculate the data corresponding to each target batch to obtain the intermediate calculation results and output results of the artificial intelligence model; if not, execute step S123 to divide the artificial intelligence model into multiple first-level sub-models according to the preset segmentation rules.
[0130] For example, Figure 5 As shown, for the sub-models obtained by segmentation, it is determined whether there are unprocessed sub-models. For example, if there are no unprocessed sub-models, it means that all sub-models have been calculated, so that the obtained intermediate calculation results can be stored in the target buffer, and the output results can be stored in the first memory. For example, if there are unprocessed sub-models, step S1231 is executed to select one of the unprocessed sub-models, and calculate the minimum sub-batch that meets the alignment requirements of each operator in the sub-model to determine the sub-batch to be further segmented (for the multi-operator sub-model of the first level, the sub-batch to be further segmented is, for example, the first sub-batch).
[0131] For example, Figure 5 As shown, further, in step S1232, the buffer space size requirement R'(Op i ), and determine the buffer space requirement R'(Op i ) satisfies the size requirement of the target buffer; if so, execute step S1233, use the sub-model to calculate the data corresponding to the sub-batch, and mark the sub-model as processed; if not, execute step S1237, and further split the sub-model into sub-models of the next level according to the preset splitting rules (for the multi-operator sub-model of the first level, the sub-model of the next level obtained by splitting is, for example, the sub-model of the second level).
[0132] For example, Figure 5As shown, for the sub-model of the next level obtained by further segmentation, the above-described method is used again to process it to determine whether it needs to be further segmented; the specific calculation process and judgment method can be referred to the above description and will not be repeated here.
[0133] The optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure can split the artificial intelligence model once or multiple times to obtain sub-models that meet the size requirements of the target buffer, so that the target buffer can also be used when the artificial intelligence model contains operators that do not meet the size requirements of the target buffer, thereby expanding the applicable scenarios of the optimization method, further improving the effective utilization rate of the target buffer, and enhancing the performance of the artificial intelligence model.
[0134] In some examples, for the preset segmentation rule of step S123, in one segmentation rule, operators whose buffer space size requirements do not meet the size of the target buffer can be segmented separately to further segment the artificial intelligence model. For example, the multiple operators of the artificial intelligence model include the 1st operator to the Nth operator in the first calculation order, where N is an integer greater than 1. For example, step S123 can further include steps S12301 to S12302.
[0135] Step S12301: In response to a buffer space size requirement of at least one operator among the multiple operators being greater than a size of a target buffer, a split operator is selected from the at least one operator, and the split operator is used as a single operator sub-model;
[0136] Step S12302: According to the first calculation order, the operators from the 1st operator to the operator before the splitting operator are taken as the first sub-model, and the operators after the splitting operator to the Nth operator are taken as the second sub-model.
[0137] For example, in step S12301, if there are one or more operators whose buffer space size requirement is greater than the size of the target buffer, a splitting operator is selected from one or more operators that do not meet the constraint condition, and the splitting operator is used as a single operator sub-model.
[0138] For example, specifically, the splitting operator is selected as the operator with the largest buffer space requirement among the one or more operators. For example, if the one or more operators include at least two operators with the largest buffer space requirement, the splitting operator is selected as the operator that is preceding in the first calculation order among the at least two operators with the largest buffer space requirement.
[0139] For example, in step S12302, according to the first calculation order, one or more operators before the splitting operator in the artificial intelligence model are used as the first sub-model, and one or more operators after the splitting operator are used as the second sub-model. For example, the single operator sub-model, the first sub-model, and the second sub-model obtained by splitting are all sub-models of the first level.
[0140] For example, at least one of the first sub-model and the second sub-model may be a multi-operator sub-model. The specific method of using the multi-operator sub-model for calculation may refer to the above steps S1231 to S1237 and Figure 5 The description will not be repeated here.
[0141] Fig. 6A A schematic diagram of an example of an artificial intelligence model provided in at least one embodiment of the present disclosure, Figure 6B A schematic diagram of an example of multiple first-level sub-models provided for at least one embodiment of the present disclosure. For example, Fig. 6A and Figure 6B for Figure 2 Steps S110 to S130 and Figure 3 A specific example of steps S121 to S123 in FIG.
[0142] For example, Fig. 6A As shown, the example artificial intelligence model includes 6 operators, which are operators Op0 to Op5 in the first calculation order; the size of the input data and output data of each operator is represented by (x, y), where x represents the batch size and y represents the size of the data tensor corresponding to each batch. For example, operators Op0 to Op5 can be Gemm type operators, ReLU type operators, Gemm type operators, BatchNorm type operators, Gemm type operators and Softmax type operators, respectively, and can also be selected as other types of operators according to actual needs, and the embodiments of the present disclosure are not limited to this.
[0143] For example, assuming that Figure 2The target batch Nb=1024 obtained by the step S110 is split, and the size of the target batch Nb meets the alignment requirements of operators Op0~Op5. For example, for the alignment requirements of operators Op0, Op2 and Op4, the batch size of the input data of operators Op0, Op2 and Op4 is required to be an integer multiple of 128 (i.e., Nop=128); for the alignment requirements of operators Op1 and Op5, the batch size of the input data of operators Op1 and Op5 is required to be an integer multiple of 32 (i.e., Nop=32); for the alignment requirements of operator Op3, the batch size of the input data of operator Op3 is required to be an integer multiple of 1024. For example, when Op3 is an operator of the BatchNorm type, in some cases, Op3 requires that the input batch cannot be split. At this time, Op3 needs to be processed separately based on the input batch.
[0144] For example, Fig. 6A As shown in the figure, the data tensor of the input data corresponding to the target batch Nb (Nb=1024) is 1024, that is, the size of the input data (input 0) of the operator Op0 is (1024, 1024). For example, the target batch Nb is used as the batch for calculation of the artificial intelligence model, that is, each operator uses the target batch Nb for calculation; after calculation by different operators, the size of the data tensor corresponding to each target batch will change.
[0145] For example, correspondingly, Fig. 6A As shown, the output data of operator Op0 is used as the input data (input 1) of operator Op1, and the corresponding size of input 1 is (1024, 2048); the output data of operator Op1 is used as the input data (input 2) of operator Op2, and the corresponding size of input 2 is (1024, 2048); the output data of operator Op2 is used as the input data (input 3) of operator Op3, and the corresponding size of input 3 is (1024, 2048); the output data of operator Op3 is used as the input data (input 4) of operator Op4, and the corresponding size of input 4 is (1024, 2048); the output data of operator Op4 is used as the input data (input 5) of operator Op5, and the corresponding size of input 5 is (1024, 512); the output data of operator Op5 (output 5) is used as the final output data of the artificial intelligence model, and the corresponding size of output 5 is (1024, 512).
[0146] For example, in Figure 3 In step S121, based on the size of the target batch Nb and the size of the data tensors of the input data and output data of the operators Op0 to Op5, the buffer space size requirements R(Op0) to R(Op5) of the operators Op0 to Op5 are calculated respectively, and it is determined whether the buffer space size requirement of each operator is less than or equal to the size of the target buffer (for example, 4 MB):
[0147] R(Op0)=Nb×(1024+2048)×α<=4MB;
[0148] R(Op1)=Nb×(2048+2048)×α<=4MB;
[0149] R(Op2)=Nb×(2048+2048)×α<=4MB;
[0150] R(Op3)=Nb×(2048+2048)×α<=4MB;
[0151] R(Op4)=Nb×(2048+512)×α<=4MB;
[0152] R(Op5)=Nb×(512+512)×α<=4MB.
[0153] For example, the target batch Nb = 1024, the byte size of the data type α = 4B; from the calculation results, it can be seen that the buffer space size requirement R(Op3) of the operator Op3 among the operators Op0 to Op5 is greater than the size of the target buffer (for example, 4MB), resulting in the constraint relationship between the buffer space size requirement R(Op3) of the operator Op3 and the size of the target buffer in the above inequality not being established. For example, further, executing Figure 3 In step S123, the artificial intelligence model is divided into multiple first-level sub-models according to a preset segmentation rule.
[0154] For example, Figure 6B An example of multiple first-level sub-models obtained by segmentation is shown. For example, specifically, step S12301 is executed. Since the buffer space size requirement R(Op3) of operator Op3 does not meet the size requirement of the target buffer (for example, 4MB), operator Op3 is selected as the segmentation operator, and the segmentation operator Op3 is used as a single operator sub-model. For example, further, step S12302 is executed. According to the first calculation order, operators Op0 to Op2 before the segmentation operator Op3 are used as the first sub-model, and operators Op4 to Op5 after the segmentation operator Op3 are used as the second sub-model. For example, the first sub-model and the second sub-model obtained by segmentation are both multi-operator sub-models.
[0155] For example, Figure 6B As shown, the batch used for calculation by the single operator sub-model is the target batch Nb=1024, and the data corresponding to the target batch Nb (ie, input 3') is calculated using the single operator sub-model. The calculated output data (ie, output 3') can be stored in the first memory.
[0156] For example, Figure 6BAs shown, for the first sub-model and the second sub-model, the minimum batch value that satisfies the alignment requirements of each operator in the first sub-model and the second sub-model is first determined. For example, for the first sub-model, the batch size of the input data of operators Op0 and Op2 is required to be an integer multiple of 128 (ie, Nop=128), and the batch N size of the input data of operator Op1 is required to be an integer multiple of 32 (ie, Nop=32). Therefore, the minimum batch value that meets the above alignment requirements is 128. For example, for the second sub-model, the batch N of the input data of operator Op4 is required to be an integer multiple of 128 (ie, Nop=128), and the batch N of the input data of operator Op5 is required to be an integer multiple of 32 (ie, Nop=32). Therefore, the minimum batch value that meets the above alignment requirements is also 128.
[0157] For example, further, step S1231 is performed to compare the minimum batch values of the first sub-model and the second sub-model with the size of the target batch Nb respectively; since the minimum batch values Nb1 and Nb2 are both smaller than the target batch Nb, the target batch Nb can be divided into multiple first sub-batches. For example, the first sub-batch Nb1 of the first sub-model is equal to the minimum batch value that meets the above alignment requirements (i.e., Nb1=128), and the first sub-batch Nb2 of the second sub-model is equal to the minimum batch value that meets the above alignment requirements (i.e., Nb2=128).
[0158] For example, further, step S1232 is executed. For the first sub-model, based on the size of each first sub-batch Nb1 and the size of the data tensor of the input data and the output data of each operator, the buffer space size requirement R'(Op0) to R'(Op2) of each operator in the first sub-model is calculated, and it is determined whether the buffer space size requirement of each operator is less than or equal to the size of the target buffer (for example, 4 MB):
[0159] R'(Op0)=Nb1×(1024+2048)×α<=4MB;
[0160] R'(Op1)=Nb1×(2048+2048)×α<=4MB;
[0161] R'(Op2)=Nb1×(2048+2048)×α<=4MB.
[0162] For example, for the second sub-model, based on the size of each first sub-batch Nb2 and the size of the data tensor of the input data and output data of each operator, the buffer space size requirement R'(Op4)~R'(Op5) of each operator is calculated, and it is determined whether the buffer space size requirement of each operator is less than or equal to the size of the target buffer (for example, 4MB):
[0163] R'(Op4)=Nb2×(2048+512)×α<=4MB;
[0164] R'(Op5)=Nb2×(512+512)×α<=4MB.
[0165] For example, the first sub-batch Nb1 of the first sub-model is 128, the first sub-batch Nb2 of the second sub-model is 128, and the byte size of the data type α is 4B. From the calculation results, it can be seen that the buffer space size requirement R'(Op i ) are all smaller than the target buffer size of 4MB, that is, the constraint relationship between the buffer space size requirements of operators Op0-Op2 and Op4-Op5 and the target buffer size in the above inequality holds. For example, further, Figure 6B As shown, execute steps S1233 and S1235, use the first sub-model to calculate the data corresponding to the first sub-batch Nb1, and obtain the intermediate calculation results of operators Op0~Op2 corresponding to the first sub-batch Nb1 (i.e., input 1' and input 2'), and the output data of the first sub-model corresponding to the first sub-batch Nb1 (i.e., output 2'); use the second sub-model to calculate the data corresponding to the first sub-batch Nb2, and obtain the intermediate calculation results of operators Op4~Op5 corresponding to the first sub-batch Nb2 (i.e., input 5'), and the output data of the second sub-model corresponding to the first sub-batch Nb2 (i.e., output 5').
[0166] For example, further, steps S1234 and S1236 are executed to store the intermediate calculation results of the first sub-model and the second sub-model (i.e., input 1', input 2', and input 5') in the target buffer, and the output data of the first sub-model and the second sub-model (i.e., output 2' and output 5') are stored in the first memory.
[0167] Based on Fig. 6A and Figure 6B As can be seen from the examples shown, the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure, by taking the calculations in the artificial intelligence model that do not meet the size requirements of the target buffer as splitting points, can find the possibility of using the target buffer for optimization in the split sub-models, thereby expanding the applicable scenarios of the artificial intelligence model optimization method, further improving the effective utilization rate of the target buffer, and enhancing the performance of the artificial intelligence model.
[0168] Figure 6C A schematic diagram of another example of an optimization method for an artificial intelligence model provided by at least one embodiment of the present disclosure. For example, Figure 6C for Figure 6B A specific example of the calculation process of the three sub-models in .
[0169] For example, Figure 6C As shown, the artificial intelligence model is divided into three first-level sub-models, including the first sub-model, the single-operator sub-model and the second sub-model in the second calculation order. For example, the output data of the first sub-model is the input data of the single-operator sub-model, and the output data of the single-operator sub-model is the input data of the second sub-model. For example, the second calculation order is the calculation order between sub-models, which may specifically include: inputting the input data corresponding to the target batch Nb into the first sub-model, inputting the output data calculated by the first sub-model into the single-operator sub-model, and inputting the output data calculated by the single-operator sub-model into the second sub-model, and the output data of the second sub-model is the output data of the artificial intelligence model corresponding to the target batch Nb.
[0170] For example, Figure 6C As shown, for the first sub-model and the second sub-model, within each sub-model, all operators are first used to calculate the current first sub-batch, and then all operators are used to calculate the next first sub-batch, thereby obtaining intermediate calculation results of multiple operators corresponding to each first sub-batch.
[0171] For example, specifically, Figure 6C As shown, for the first sub-model, the target sub-batch Nb is divided into three first sub-batches Nb1-0, Nb1-1 and Nb1-2. For example, within the first sub-model, the input data corresponding to the first sub-batch Nb1-0 is input into Op0, and is calculated by Op0, Op1 and Op2 according to the first calculation order. The data output by Op0 and Op1 are respectively used as the intermediate calculation results of Op0 and Op1 corresponding to the first sub-batch Nb1-0, and the data output by Op2 is used as the output data of the first sub-model corresponding to the first sub-batch Nb1-0; further, the input data corresponding to the first sub-batch Nb1-1 is input into Op0, and is calculated by Op0, Op1 and Op2 according to the first calculation order. The data output by Op0 and Op1 are respectively used as the intermediate calculation results of Op0 and Op1 corresponding to the first sub-batch Nb1-0, and the data output by Op2 is used as the output data of the first sub-model corresponding to the first sub-batch Nb1-0; further, the input data corresponding to the first sub-batch Nb1-1 is input into Op0, and is calculated by Op0, Op1 and Op2 according to the first calculation order. The data output by Op0 and Op1 are used as the intermediate calculation results of Op0 and Op1 corresponding to the first sub-batch Nb1-1, and the data output by Op2 is used as the output data of the first sub-model corresponding to the first sub-batch Nb1-1; further, the input data corresponding to the first sub-batch Nb1-2 is input into Op0, and calculated by Op0, Op1 and Op2 in the first calculation order, and the data output by Op0 and Op1 are used as the intermediate calculation results of Op0 and Op1 corresponding to the first sub-batch Nb1-2, and the data output by Op2 is used as the output data of the first sub-model corresponding to the first sub-batch Nb1-2.
[0172] For example, further, multiple intermediate calculation results of the first sub-model are stored in the target buffer, and the output data of the first sub-model corresponding to the first sub-batch Nb1-0, Nb1-1 and Nb1-2 are respectively stored in the first memory.
[0173] For example, Figure 6C As shown, the single operator sub-model reads the output data of the first sub-model from the first memory as the input data of the single operator sub-model; since the single operator sub-model does not divide the target sub-batch Nb, the input data corresponding to the target sub-batch Nb is input into Op3, and calculated by Op3, and the data output by Op3 is used as the output data of the single operator sub-model corresponding to the target sub-batch Nb. For example, further, the output data of the single operator model corresponding to the target sub-batch Nb is stored in the first memory.
[0174] For example, Figure 6C As shown, for the second sub-model, the target sub-batch Nb is divided into three first sub-batches Nb2-0, Nb2-1 and Nb2-2. For example, the second sub-model reads the output data of the single operator sub-model from the first memory as the input data of the second sub-model. For example, within the second sub-model, the input data corresponding to the first sub-batch Nb2-0 is input into Op4, and is calculated by Op4 and Op5 according to the first calculation order. The data output by Op4 is used as the intermediate calculation result of Op4 corresponding to the first sub-batch Nb2-0, and the data output by Op5 is used as the output data of the second sub-model corresponding to the first sub-batch Nb2-0; further, the input data corresponding to the first sub-batch Nb2-1 is input into Op4, and is calculated by Op4 and Op5 according to the first calculation order. The data output by Op4 is used as the intermediate calculation result of Op4 corresponding to the first sub-batch Nb2-1, and the data output by Op5 is used as the output data of the second sub-model corresponding to the first sub-batch Nb2-1; further, the input data corresponding to the first sub-batch Nb2-2 is input into Op4, and is calculated by Op4 and Op5 according to the first calculation order. The data output by Op4 is used as the intermediate calculation result of Op4 corresponding to the first sub-batch Nb2-2, and the data output by Op5 is used as the output data of the second sub-model corresponding to the first sub-batch Nb2-2.
[0175] For example, further, multiple intermediate calculation results of the second sub-model are stored in the target buffer, and the output data of the second sub-model corresponding to the first sub-batches Nb2-0, Nb2-1 and Nb2-2 are respectively stored in the first memory.
[0176] For example, Figure 6C As shown, the overall calculation order of the first sub-model is as follows:
[0177] 1) Input Nb1-0 (first memory) -> Op0 -> intermediate calculation result 1-1 (target buffer) -> Op1 -> intermediate calculation result 1-2 (target buffer) -> Op2 -> output Nb1-0 (first memory);
[0178] 2) Input Nb1-1 (first memory) -> Op0 -> intermediate calculation result 2-1 (target buffer) -> Op1 -> intermediate calculation result 2-2 (target buffer) -> Op2 -> output Nb1-1 (first memory);
[0179] 3) Input Nb1-2 (first memory) -> Op0 -> intermediate calculation result 3-1 (target buffer) -> Op1 -> intermediate calculation result 3-2 (target buffer) -> Op2 -> output Nb1-2 (first memory).
[0180] For example, Figure 6C As shown, the overall calculation sequence of the single operator sub-model is as follows:
[0181] 4) Input Nb (first memory) -> Op3 -> Output Nb (first memory).
[0182] For example, Figure 6C As shown, the overall calculation sequence of the second sub-model is as follows:
[0183] 5) Input Nb2-0 (first memory) -> Op4 -> intermediate calculation result 5-1 (target buffer) -> Op5 -> output Nb2-0 (first memory);
[0184] 6) Input Nb2-1 (first memory) -> Op4 -> intermediate calculation result 6-1 (target buffer) -> Op5 -> output Nb2-1 (first memory);
[0185] 7) Input Nb2-2 (first memory) -> Op4 -> intermediate calculation result 7-1 (target buffer) -> Op5 -> output Nb2-2 (first memory).
[0186] It should be noted that Figure 6A to Figure 6C The optimization method of the artificial intelligence model shown is only exemplary, and the architecture of the artificial intelligence model, the types and number of operators included, the number and size of the target batches and first sub-batches obtained by segmentation, the specific implementation method of the target buffer and the first memory, etc. can all be selected according to actual needs, and the embodiments of the present disclosure do not limit this.
[0187] At least one embodiment of the present disclosure provides an optimization method for an artificial intelligence model. By dividing the artificial intelligence model into multiple sub-models and further dividing the target batch into multiple sub-batches, the intermediate calculation results of multiple operators calculated based on sub-batches can be stored in a target buffer. Therefore, when the artificial intelligence model contains operators that do not meet the size requirements of the target buffer, the target buffer can also be used. Therefore, the target buffer can be effectively used for optimization in more models, which expands the applicable scenarios of the optimization method, further improves the effective utilization rate of the target buffer, and enhances the performance of the artificial intelligence model.
[0188] For example, Figure 7 As shown, at least one embodiment of the present disclosure provides an optimization device 200 for an artificial intelligence model, wherein the artificial intelligence model includes multiple operators, and the input data of the artificial intelligence model includes input batches, and each batch in the input batches corresponds to a data tensor of the input data. For example, the optimization device 200 for the artificial intelligence model includes a segmentation module 210, a calculation module 220, and a storage module 230.
[0189] For example, the splitting module 210 is configured to, in response to the size of the storage space occupied by the input data and output data of the multiple operators being larger than the size of the target buffer, split the input batch of the input data of the artificial intelligence model into multiple target batches. For example, the size of each target batch in the multiple target batches satisfies the alignment requirement of each operator in the multiple operators. That is, the splitting module 210 can be configured to perform, for example Figure 2 Step S110 is shown.
[0190] For example, the computing module 220 is configured to use multiple operators to compute the input data of the artificial intelligence model based on the size of each target batch and the size of the data tensor of the input data and output data of each operator to obtain the intermediate calculation results of the multiple operators. That is, the computing module 220 can be configured to perform, for example Figure 2 Step S120 is shown.
[0191] For example, the storage module 230 is configured to store the intermediate calculation results of multiple operators in the target buffer. That is, the storage module 230 can be configured to perform, for example Figure 2 Step S130 is shown.
[0192] In some examples, the input data of the artificial intelligence model is stored in the first memory. For example, the computing module 220 is also configured to use multiple operators to calculate the input data of the artificial intelligence model based on the size of each target batch and the size of the data tensor of the input data and output data of each operator to obtain the output data of the artificial intelligence model; that is, the computing module 220 can be configured to perform step S140. For example, the storage module 230 is also configured to store the output data of the artificial intelligence model in the first memory; that is, the storage module 230 can be configured to perform step S150.
[0193] In some examples, the computing module 220 is further configured to, in response to the size of the storage space occupied by the input data and output data of the multiple operators being less than or equal to the size of the target buffer, use the multiple operators to compute the input data of the artificial intelligence model to obtain the intermediate calculation results of the multiple operators. That is, the computing module 220 can be configured to execute step S160.
[0194] For example, the storage module 230 is further configured to store the intermediate calculation results of the multiple operators in the target buffer. That is, the storage module 230 can be configured to execute step S170.
[0195] In some examples, the computing module 220 is further configured to calculate the buffer space size requirement of each operator based on the size of each target batch and the size of the data tensors of the input data and output data of each operator; in response to the buffer space size requirement of each operator being less than or equal to the size of the target buffer, use multiple operators to calculate the data corresponding to each target batch, and obtain intermediate calculation results of the multiple operators corresponding to each target batch.
[0196] In some examples, the plurality of operators include the 1st operator to the Nth operator, where N is an integer greater than 1, i=1, 2, ..., N. For example, the calculation module 220 is further configured to calculate the buffer space size requirement R (Op ) of the input data of the ith operator based on the size Nb of each target batch and the size Ta of the data tensor of the input data of the ith operator among the plurality of operators. i-input ), where R(Op i-input )=Nb×Ta×α, α is the byte size of the data type; based on the size Nb of each target batch and the size To of the data tensor of the output data of the ith operator, calculate the buffer space size requirement R(Op i-output ), where R(Op i-output )=Nb×To×α;Based on the buffer space requirement R(Op i-input ) and the buffer space requirement for output data R(Op i-output), calculate the buffer space requirement R(Op i ), where R(Op i )=R(Op i-input )+R(Op i-output ).
[0197] In some examples, the multiple target batches include the first target batch to the Mth target batch, where M is an integer greater than 1. For example, the calculation module 220 is further configured to use multiple operators to sequentially calculate data corresponding to the first target batch to data corresponding to the Mth target batch.
[0198] In some examples, the multiple operators include the 1st operator to the Nth operator in a first calculation order, where N is an integer greater than 1. For example, in response to the buffer space size requirement of each operator being less than or equal to the size of the target buffer, the intermediate calculation results of the multiple operators include output data of the 1st operator to output data of the N-1th operator, wherein, among the multiple operators, the output data of the kth operator is used as input data of the k+1th operator, k=1, 2, ..., N-1.
[0199] In some examples, the segmentation module 210 is further configured to, in response to a buffer space size requirement of at least one operator among the multiple operators being greater than a size of a target buffer, segment the artificial intelligence model into a plurality of first-level sub-models according to a preset segmentation rule.
[0200] In some examples, multiple first-level sub-models include the 1st sub-model to the Qth sub-model in a second calculation order, where Q is an integer greater than 1; among the multiple first-level sub-models, the output data of the jth sub-model serves as the input data of the j+1th sub-model, j=1, 2,…, Q-1; the input data and output data of each sub-model in the multiple first-level sub-models are stored in a first memory.
[0201] In some examples, the plurality of first-level sub-models include a multi-operator sub-model, and the multi-operator sub-model includes at least two operators. For example, the splitting module 210 is further configured to, in response to the minimum batch value that satisfies the alignment requirement of each operator in the multi-operator sub-model being less than the size of the target batch, split the plurality of target batches input into the multi-operator sub-model into a plurality of first sub-batches. For example, the size of each of the plurality of first sub-batches is equal to the minimum batch value that satisfies the alignment requirement of each operator in the multi-operator sub-model.
[0202] In some examples, the computing module 220 is further configured to calculate the buffer space size requirement of each operator in the multi-operator sub-model based on the size of each first sub-batch and the size of the data tensor of the input data and output data of each operator in the multi-operator sub-model. For example, the storage module 230 is further configured to store the intermediate calculation results of the multiple operators in the multi-operator sub-model corresponding to each first sub-batch in the target buffer.
[0203] In some examples, the calculation module 220 is further configured to, in response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, use the multi-operator sub-model to calculate the data corresponding to each first sub-batch, and obtain the output data of the multi-operator sub-model corresponding to each first sub-batch. For example, the storage module 230 is further configured to store the output data of the multi-operator sub-model corresponding to each first sub-batch in the first memory.
[0204] In some examples, the plurality of first sub-batches include the 1st first sub-batch to the Pth first sub-batch, where P is an integer greater than 1. For example, the calculation module 220 is further configured to, in response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, sequentially calculate data corresponding to the 1st first sub-batch to data corresponding to the Pth first sub-batch using the multi-operator sub-model.
[0205] In some examples, the segmentation module 210 is further configured to, in response to a buffer space size requirement of at least one operator in the multi-operator sub-model being greater than a size of a target buffer, segment the multi-operator sub-model into a plurality of second-level sub-models according to a preset segmentation rule.
[0206] In some examples, the multiple operators of the artificial intelligence model include the 1st operator to the Nth operator according to the first calculation order, where N is an integer greater than 1. For example, the segmentation module 210 is further configured to: in response to the buffer space size requirement of at least one operator among the multiple operators being greater than the size of the target buffer, select a segmentation operator from at least one operator, and use the segmentation operator as a single operator sub-model; according to the first calculation order, use the 1st operator to the operator before the segmentation operator as the first sub-model, and use the operator after the segmentation operator to the Nth operator as the second sub-model. For example, the single operator sub-model, the first sub-model, and the second sub-model are sub-models of the first level.
[0207] For example, the splitting operator is the operator with the largest buffer space requirement among the at least one operator. For example, in response to at least two operators with the largest buffer space requirements being included in the at least one operator, the splitting operator is the operator that comes first in the first calculation order among the at least two operators with the largest buffer space requirements. For example, at least one of the first sub-model and the second sub-model is a multi-operator sub-model.
[0208] Since in the above description, for example Figure 2 to Figure 6C In the process of the optimization method of the artificial intelligence model shown, the details of the contents involved in the operation of the optimization device 200 of the above-mentioned artificial intelligence model have been introduced. Therefore, for the sake of brevity, they will not be repeated here. For relevant details, please refer to the description of the optimization method of the artificial intelligence model provided in at least one embodiment of the present disclosure above.
[0209] It should be noted that Figure 7 The above modules in the optimization device 200 of the artificial intelligence model shown can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, these modules can correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. Figure 7 The described device may be a PC computer, a tablet device, a personal digital assistant, a smart phone, a web application, or other device capable of executing program instructions, but is not limited thereto.
[0210] In addition, although the optimization device 200 for the artificial intelligence model is described above as being divided into modules for respectively performing corresponding processing, it is clear to those skilled in the art that the processing performed by each module may also be performed without any specific module division in the device or without clear demarcation between the modules. Figure 7 The described artificial intelligence model optimization device 200 is not limited to including the modules described above, but some other modules (for example, a reading module, a control module, etc.) may be added as needed, or the above modules may also be combined.
[0211] At least one embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory; the memory includes one or more computer program modules; the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include an optimization method for implementing the artificial intelligence model provided by the embodiment of the present disclosure described above.
[0212] Figure 8 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.
[0213] For example, Figure 8As shown, the electronic device 300 includes a processor 310 and a memory 320. For example, the memory 320 is used to store non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 310 is used to run non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are run by the processor 310, one or more steps of the optimization method according to the artificial intelligence model described above can be executed. The memory 320 and the processor 310 can be interconnected through a bus system and / or other forms of connection mechanisms (not shown).
[0214] For example, the processor 310 may be a central processing unit (CPU), a graphics processing unit (GPU), a general purpose graphics processing unit (GPGPU), a digital signal processor (DSP), or other forms of processing units with performance prediction capabilities and / or program execution capabilities, such as a field programmable gate array (FPGA), etc.; for example, the central processing unit (CPU) may be an X86, RISC-V, or ARM architecture, etc. The processor 310 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 300 to perform desired functions.
[0215] For example, the memory 320 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 310 may run one or more computer program modules to implement various functions of the electronic device 300. Various applications and various data, as well as various data used and / or generated by the application, etc. may also be stored in the computer-readable storage medium.
[0216] It should be noted that in the embodiments of the present disclosure, the specific functions and technical effects of the electronic device 300 can refer to the above description of the optimization method of the artificial intelligence model provided by at least one embodiment of the present disclosure, and will not be repeated here.
[0217] Fig. 9 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.
[0218] For example, Fig. 9 As shown, the electronic device 400 is suitable for implementing the optimization method of the artificial intelligence model provided by the embodiment of the present disclosure. It should be noted that, Fig. 9 The electronic device 400 shown is only an example and does not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0219] For example, Fig. 9 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 41, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 42 or a program loaded from a storage device 48 into a random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44. Generally, the following devices can be connected to the I / O interface 45: an input device 46 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 47 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 48 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 49. The communication device 49 can allow the electronic device 400 to communicate with other electronic devices wirelessly or by wire to exchange data.
[0220] Although Fig. 9 The electronic device 400 is shown to have various devices, but it should be understood that it is not required to implement or possess all the devices shown, and the electronic device 400 may alternatively implement or possess more or fewer devices.
[0221] For the detailed description and technical effects of the electronic device 400, please refer to the above description of the optimization method of the artificial intelligence model provided in at least one embodiment of the present disclosure, which will not be repeated here.
[0222] Fig.10 A schematic diagram of a storage medium provided for at least one embodiment of the present disclosure.
[0223] For example, Fig.10 As shown, the storage medium 500 stores non-transitory computer-readable instructions 510. For example, when the non-transitory computer-readable instructions 510 are executed by a computer, one or more steps in the optimization method according to the artificial intelligence model described above are performed.
[0224] For example, the storage medium 500 can be applied to Figure 8 For example, the storage medium 500 may be the memory 320 in the electronic device 300. For example, the description of the storage medium 500 may refer to Figure 8 The corresponding description of the memory 320 in the electronic device 300 is shown and will not be repeated here.
[0225] There are a few points to note about this disclosure:
[0226] (1) In the drawings of the embodiments of the present disclosure, only the structures related to the embodiments of the present disclosure are involved, and other structures can refer to the general design.
[0227] (2) In the absence of conflict, features in the same embodiment or in different embodiments of the present disclosure may be combined with each other.
[0228] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. An optimization method for artificial intelligence models based on processor storage space, in, The artificial intelligence model includes a plurality of operators, the input data of the artificial intelligence model includes input batches, each batch in the input batches corresponds to a data tensor of the input data, The optimization method of the artificial intelligence model includes: In response to the size of the storage space occupied by the input data and output data of the multiple operators being larger than the size of the target buffer, the input batch of the input data of the artificial intelligence model is divided into multiple target batches, wherein the size of each target batch in the multiple target batches satisfies the alignment requirement of each operator in the multiple operators, each operator has a corresponding specific value, and the alignment requirement of each operator includes: the size of each target batch is an integer multiple of the specific value corresponding to each operator, and the specific value corresponding to each operator is determined by the own attribute of the corresponding operator or the specific implementation of the corresponding operator; Based on the size of each target batch and the size of the data tensor of the input data and output data of each operator, use the multiple operators to calculate the input data of the artificial intelligence model to obtain intermediate calculation results of the multiple operators; storing the intermediate calculation results of the plurality of operators in the target buffer, Wherein, based on the size of each target batch and the size of the data tensor of the input data and output data of each operator, using the multiple operators to calculate the input data of the artificial intelligence model to obtain the intermediate calculation results of the multiple operators includes: Based on the size of each target batch and the size of the data tensor of the input data and output data of each operator, the buffer space size requirement of each operator is calculated, The optimization method of the artificial intelligence model further includes: In response to a buffer space size requirement of at least one of the multiple operators being greater than a size of the target buffer, the artificial intelligence model is divided into a plurality of first-level sub-models according to a preset segmentation rule, The multiple operators of the artificial intelligence model include the 1st operator to the Nth operator in a first calculation order, where N is an integer greater than 1. In response to the buffer space size requirement of at least one operator among the multiple operators being greater than the size of the target buffer, the artificial intelligence model is divided into a plurality of first-level sub-models according to a preset segmentation rule, including: In response to a buffer space size requirement of at least one operator among the multiple operators being greater than a size of the target buffer, selecting a segmentation operator from the at least one operator, and using the segmentation operator as a single operator sub-model; According to the first calculation order, the operators from the first operator to the operator before the splitting operator are used as the first sub-model, and the operators after the splitting operator to the Nth operator are used as the second sub-model, Among them, the single operator sub-model, the first sub-model and the second sub-model are sub-models of the first level.
2. The optimization method of the artificial intelligence model according to claim 1, in, The input data of the artificial intelligence model is stored in a first memory, The optimization method of the artificial intelligence model also includes: Based on the size of each target batch and the size of the data tensor of the input data and the output data of each operator, use the multiple operators to calculate the input data of the artificial intelligence model to obtain the output data of the artificial intelligence model; The output data of the artificial intelligence model is stored in a first memory.
3. The optimization method of the artificial intelligence model according to claim 1, further comprising: include: In response to the size of the storage space occupied by the input data and output data of the multiple operators being less than or equal to the size of the target buffer, using the multiple operators to calculate the input data of the artificial intelligence model to obtain intermediate calculation results of the multiple operators; The intermediate calculation results of the multiple operators are stored in the target buffer.
4. The optimization method of the artificial intelligence model according to claim 1, in, The step of calculating the input data of the artificial intelligence model using the multiple operators based on the size of each target batch and the size of the data tensor of the input data and the output data of each operator to obtain the intermediate calculation results of the multiple operators includes: In response to the buffer space size requirement of each operator being less than or equal to the size of the target buffer, the multiple operators are used to calculate the data corresponding to each target batch to obtain intermediate calculation results of the multiple operators corresponding to each target batch.
5. The optimization method of the artificial intelligence model according to claim 4, in, The multiple operators include the 1st operator to the Nth operator, N is an integer greater than 1, i=1, 2, ..., N, The calculating the buffer space size requirement of each operator based on the size of each target batch and the size of the data tensor of the input data and the output data of each operator includes: Based on the size Nb of each target batch and the size Ta of the data tensor of the input data of the ith operator among the multiple operators, calculate the buffer space size requirement R (Op i-input ), where R(Op i-input ) = Nb×Ta×α, where α is the byte size of the data type; Based on the size Nb of each target batch and the size To of the data tensor of the output data of the i-th operator, the buffer space size requirement R (Op i-output ), where R(Op i-output ) = Nb × To × α; Based on the buffer space size requirement R(Op i-input ) and the buffer space requirement for output data R(Op i-output ), calculate the buffer space requirement R(Op i ), where R(Op i )=R(Op i-input )+R(Op i-output ).
6. The optimization method of the artificial intelligence model according to claim 4, in, The multiple target batches include the 1st target batch to the Mth target batch, where M is an integer greater than 1. In response to the buffer space size requirement of each operator being less than or equal to the size of the target buffer, using the multiple operators to calculate the data corresponding to each target batch includes: The multiple operators are used to sequentially calculate data corresponding to the first target batch to data corresponding to the Mth target batch.
7. The optimization method of the artificial intelligence model according to claim 4, in, The multiple operators include the 1st operator to the Nth operator according to the first calculation order, where N is an integer greater than 1, In response to the buffer space size requirement of each operator being less than or equal to the size of the target buffer, the intermediate calculation results of the multiple operators include the output data of the 1st operator to the output data of the N-1th operator, wherein, among the multiple operators, the output data of the kth operator serves as the input data of the k+1th operator, k=1, 2,…, N-1.
8. The optimization method of the artificial intelligence model according to claim 1, in, The plurality of first-level sub-models include the 1st sub-model to the Qth sub-model according to the second calculation order, where Q is an integer greater than 1, Among the multiple first-level sub-models, the output data of the j-th sub-model is used as the input data of the j+1-th sub-model, j=1, 2, ..., Q-1, The input data and output data of each sub-model in the plurality of first-level sub-models are stored in the first memory.
9. The optimization method of an artificial intelligence model according to claim 1, wherein the plurality of first-level sub-models include a multi-operator sub-model, and the multi-operator sub-model includes at least two operators. The optimization method of the artificial intelligence model also include: In response to the minimum batch value satisfying the alignment requirement of each operator in the multi-operator sub-model being smaller than the size of the target batch, multiple target batches input into the multi-operator sub-model are divided into multiple first sub-batches, The size of each first sub-batch in the multiple first sub-batches is equal to the minimum batch value that satisfies the alignment requirement of each operator in the multi-operator sub-model.
10. The method for optimizing the artificial intelligence model according to claim 9, further comprising: include: Calculating a buffer space size requirement of each operator in the multi-operator sub-model based on the size of each first sub-batch and the size of the data tensor of the input data and the output data of each operator in the multi-operator sub-model; In response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, using the multi-operator sub-model to calculate data corresponding to each first sub-batch, and obtaining intermediate calculation results of multiple operators in the multi-operator sub-model corresponding to each first sub-batch; The intermediate calculation results of the multiple operators in the multi-operator sub-model corresponding to each first sub-batch are stored in the target buffer.
11. The method for optimizing the artificial intelligence model according to claim 10, further comprising: include: In response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, using the multi-operator sub-model to calculate data corresponding to each first sub-batch, to obtain output data of the multi-operator sub-model corresponding to each first sub-batch; The output data of the multi-operator sub-model corresponding to each first sub-batch is stored in a first memory.
12. The method for optimizing an artificial intelligence model according to claim 10, in, The plurality of first sub-batches include the 1st first sub-batches to the Pth first sub-batches, where P is an integer greater than 1. In response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, using the multi-operator sub-model to calculate the data corresponding to each first sub-batch, comprising: In response to the buffer space size requirement of each operator in the multi-operator sub-model being less than or equal to the size of the target buffer, the multi-operator sub-model is used to sequentially calculate data corresponding to the 1st first sub-batch to data corresponding to the Pth first sub-batch.
13. The method for optimizing the artificial intelligence model according to claim 10, further comprising: include: In response to the buffer space size requirement of at least one operator in the multi-operator sub-model being greater than the size of the target buffer, the multi-operator sub-model is divided into a plurality of second-level sub-models according to the preset segmentation rule.
14. The method for optimizing an artificial intelligence model according to claim 1, in, The splitting operator is an operator having the largest buffer space requirement among the at least one operator.
15. The method for optimizing an artificial intelligence model according to claim 1, in, In response to the at least one operator including at least two operators with the largest buffer space requirements, the splitting operator is an operator that comes first in the first calculation order among the at least two operators with the largest buffer space requirements.
16. The method for optimizing an artificial intelligence model according to claim 1, in, At least one of the first sub-model and the second sub-model is a multi-operator sub-model.
17. An electronic device, include: processor; a memory including one or more computer program modules; Wherein, the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the optimization method of the artificial intelligence model based on the processor storage space as described in any one of claims 1-16.
18. A storage medium storing non-temporary computer-readable instructions, which, when executed by a computer, implements the optimization method of an artificial intelligence model based on processor storage space as described in any one of claims 1 to 16.
