Method for establishing and executing artificial intelligence model

Abstract

The application discloses a method for establishing and executing an artificial intelligence model. The method for establishing the artificial intelligence model is applied to an intelligent processing unit, the intelligent processing unit comprises a computing circuit and a memory, the artificial intelligence model comprises a plurality of operators, the computing circuit generates an intermediate tensor in the process of executing each operator, and the method comprises the following steps: dividing the operators according to a batch threshold value, a life cycle of the intermediate tensor, a size of the intermediate tensor and a capacity of the memory; calculating a bandwidth requirement of the intelligent processing unit to an external memory when the artificial intelligence model is executed; and storing a correspondence between the batch threshold value and the bandwidth requirement. The artificial intelligence model simultaneously performs the operation of the same operator on N batches of input data, and N is a positive integer.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to methods for establishing and executing artificial intelligence models. Background Technology

[0002] Figure 1 This is a common artificial intelligence (AI) model in existing technologies. The AI ​​model 100 includes operators OP0, OP1, OP2, and OP3. When multiple batches of input data (also called input feature maps or tensors) are input to the AI ​​model 100, the current batch of input data (e.g., input data BT0) undergoes operations from operators OP0 to OP3 in sequence before the next batch of input data (e.g., input data BT1) undergoes the same operations. This process continues, performing operations from operators OP0 to OP3 on input data BT2 and input data BT3.

[0003] The artificial intelligence model 100 is executed by a processing unit, which typically includes internal memory (e.g., Static Random Access Memory (SRAM)). A drawback of the artificial intelligence model 100 is that, because the processing unit can only process one batch of input data at a time, it cannot fully utilize the internal memory, resulting in wasted hardware resources. Furthermore, when the internal memory is not fully utilized, the processing unit needs to frequently read data from external memory (e.g., Dynamic Random Access Memory (DRAM)), leading to increased bandwidth requirements and consequently, a decrease in the performance of the circuit system. Summary of the Invention

[0004] Given the shortcomings of existing technologies, one of the objectives of this application is to provide a method for establishing an artificial intelligence model and a method for executing an artificial intelligence model, so as to improve the shortcomings of existing technologies.

[0005] In some embodiments of this application, a method for establishing an artificial intelligence model is provided, applied to an intelligent processing unit. The intelligent processing unit includes a computing circuit and a memory. The artificial intelligence model includes multiple operators. The computing circuit generates an intermediate tensor during the execution of each operator. The method includes:

[0006] (A) The operator is divided according to a batch threshold, a lifetime of the intermediate tensor, the size of the intermediate tensor, and a capacity of the memory;

[0007] (B) Calculate the bandwidth requirement of the intelligent processing unit for an external memory when executing the artificial intelligence model; and

[0008] (C) Store a one-to-one correspondence between the batch threshold and the bandwidth requirement;

[0009] The artificial intelligence model performs the same operator operation on N batches of input data simultaneously, where N is a positive integer.

[0010] In some embodiments of this application, a method for executing an artificial intelligence model is provided, applied to an electronic device, the electronic device including an intelligent processing unit and a memory, wherein the intelligent processing unit does not include the memory, the artificial intelligence model simultaneously performs the same operator operation on M batches of input data, where M is a positive integer, the method comprising:

[0011] (A) Based on multiple batch thresholds and the multiple bandwidth requirements of the intelligent processing unit for the memory, a batch to be processed is divided into multiple sub-batch numbers, wherein the batch thresholds and the bandwidth requirements correspond to each other; and

[0012] (B) Using one of the number of sub-batches as M, execute the artificial intelligence model.

[0013] The technical means embodied in the embodiments of the present invention can improve at least one of the shortcomings of the prior art. Therefore, the present invention can make full use of hardware resources and reduce bandwidth compared with the prior art.

[0014] The features, implementation, and effects of this application are described in detail below with reference to the accompanying drawings, showing preferred embodiments. Attached Figure Description

[0015] Figure 1 It is an artificial intelligence model in existing technology;

[0016] Figure 2 This is a functional block diagram of one embodiment of the electronic device of this application;

[0017] Figure 3 This is a schematic diagram of one embodiment of the artificial intelligence model of this application;

[0018] Figure 4 This is a flowchart of an embodiment of the method for establishing the artificial intelligence model of this application;

[0019] Figure 5This is a diagram showing the relationship between the bandwidth requirements of this application and the batch threshold N in one embodiment;

[0020] Figure 6 yes Figure 4 A flowchart of an embodiment of step S420;

[0021] Figures 7A-7B This is a schematic diagram illustrating the process of dividing an artificial intelligence model;

[0022] Figure 8 This is a schematic diagram of another embodiment of the artificial intelligence model of this application;

[0023] Figure 9A yes Figure 4 A flowchart of another embodiment of step S420;

[0024] Figure 9B This is a schematic diagram of another embodiment of the artificial intelligence model of this application;

[0025] Figure 10 This is a functional block diagram of one embodiment of the electronic device of this application;

[0026] Figures 11A-11B yes Figure 4 A flowchart of another embodiment of step S420;

[0027] Figures 12A-12B This is a schematic diagram illustrating the process of dividing artificial intelligence models;

[0028] Figure 13 This is a flowchart of an embodiment of the artificial intelligence execution method of this application; and

[0029] Figure 14 yes Figure 13 A flowchart of an embodiment of step S1310.

[0030] Figure label:

[0031] 100, 300, 900: Artificial intelligence models;

[0032] BT0, BT1, BT2, BT3: Input data;

[0033] OP0, OP1, OP2, OP3, OP4, OP5, OP6: Operators;

[0034] 200, 1000: Electronic devices;

[0035] 210, 1010: Intelligent processing unit;

[0036] 212, 1012: Operational circuits;

[0037] 214, 1014, 1018: Internal memory;

[0038] 216, 1016: Direct Memory Access Circuit;

[0039] 220, 1020: External memory;

[0040] BT0_1, BT0_2, BT0_3, BT0_4, BT1_1, BT1_2, BT1_3, BT1_4, BT2_1, BT2_2, BT2_3, BT2_4, BT3_1, BT3_2, BT3_3, BT3_4, BT_A, BT_B, BT_C, BT_D, TS0, TS1, TS2, TS3, TS4: intermediate tensor;

[0041] Lr0, Lr1, Lr2, Lr3: Operational layers;

[0042] 400: Methods for building artificial intelligence models;

[0043] S410, S420, S430, S440, S450, S610, S620, S630, S640, S650, S910, S1110, S1120, S1130, S1140, S1150, S1160, S1170, S1180, S1190, S1195, S1310, S1320, S1410, S1420, S1430, S1440, S1450, S1460: Steps;

[0044] DMt: Temporary set;

[0045] DM0, DM0_0, DM0_1: Operator set;

[0046] DM1_0, DM1_1, DM1_2, DM1_3: Operator groups. Detailed Implementation

[0047] The technical terms used in the following specific embodiments refer to the conventional terms in this technical field. If this specification redefines or explains some terms, the explanation of those terms shall be based on the description or definition in this specification.

[0048] This application discloses methods for establishing and executing artificial intelligence models. Since some components of the intelligent processing unit in this application may be known individually, the details of these known components are simplified in the following description without affecting the full disclosure and implementability of the device. Furthermore, some or all of the processes of the methods for establishing and executing artificial intelligence models in this application can be in the form of software and / or firmware. Without affecting the full disclosure and implementability of the methods, the following description will focus on the steps rather than the hardware.

[0049] Please see Figure 2 , Figure 2 This is a functional block diagram of one embodiment of the electronic device of this application. The electronic device 200 includes an intelligent processing unit 210 and an external memory 220. The intelligent processing unit 210 and the external memory 220 are coupled to each other. The intelligent processing unit 210 includes an arithmetic circuit 212, an internal memory 214, and a Direct Memory Access (DMA) circuit 216, but does not include the external memory 220. The arithmetic circuit 212, the internal memory 214, and the DMA circuit 216 are coupled to each other. The DMA circuit 216 is connected to the external memory 220. The intelligent processing unit 210 reads data from the external memory 220 or writes data to the external memory 220 through the DMA circuit 216.

[0050] In some embodiments, external memory 220 may be dynamic random access memory, while internal memory 214 may be static random access memory.

[0051] External memory 220 stores the input data and weights (or convolution kernels) of an artificial intelligence model. Internal memory 214 stores multiple tensors or tiles (a tile is a part of a tensor) of the input data and at least a portion of the weight data. The computation circuitry 212 may include multiple engines (including, but not limited to, vector engines and convolution engines) that read the tensors (or tiles) and / or weights from internal memory 214, process the tensors or tiles (e.g., vector operations or convolution operations), and then store the processing results (e.g., an output feature map) in internal memory 214.

[0052] Figure 3 This diagram illustrates one embodiment of the artificial intelligence model of this application. The artificial intelligence model 300 includes multiple layers of computation (…). Figure 3The implementation includes four operational layers: Lr0-Lr3), and the operations of each layer process the input data BT0-BT3 or data derived therefrom (intermediate tensors).

[0053] Following the foregoing, the arithmetic circuit 212 of this application processes the operations of arithmetic layer Lr0, arithmetic layer Lr1, arithmetic layer Lr2, and arithmetic layer Lr3 sequentially. The operations of arithmetic layer Lr0 include performing operator OP0 on input data BT0, BT1, BT2, and BT3. Similarly, the operations of arithmetic layer Lr1 (or Lr2 or Lr3) include performing operator OP1 (or OP2 or OP3) on the intermediate tensors derived from the input data BT0–BT3 (more specifically, the outputs of their corresponding operators OP0–OP2).

[0054] For input data BT0, intermediate tensor BT0_1 is the output of operator OP0 and the input of operator OP1. Intermediate tensor BT0_2 is the output of operator OP1 and the input of operator OP2. Intermediate tensor BT0_3 is the output of operator OP2 and the input of operator OP3. Intermediate tensor BT0_4 is the output of operator OP3.

[0055] Similarly, for the input data BT1 (BT2 or BT3), the intermediate tensor BT1_1 (BT2_1 or BT3_1) is the output of operator OP0 and also the input of operator OP1. The intermediate tensor BT1_2 (BT2_2 or BT3_2) is the output of operator OP1 and also the input of operator OP2. The intermediate tensor BT1_3 (BT2_3 or BT3_3) is the output of operator OP2 and also the input of operator OP3. The intermediate tensor BT1_4 (BT2_4 or BT3_4) is the output of operator OP3.

[0056] In some embodiments, the arithmetic circuit 212 may include K cores (K greater than or equal to 1). In the operation of a certain arithmetic layer, the K cores cyclically process all input data (i.e., Figure 3 The input data in the embodiment is BT0-BT3) or the intermediate tensor is BTx_y (0≤x≤3, 1≤y≤3).

[0057] Compared to Figure 1 Artificial intelligence model 100, because Figure 3The AI ​​model 300 can actually process multiple batches of input data simultaneously, so the internal memory 214 can store more tensors or blocks at the same time, meaning the internal memory 214 is utilized more fully. One advantage of this design is that it saves bandwidth requirements of the intelligent processing unit 210 on the external memory 220, because the intelligent processing unit 210 (more specifically, the arithmetic circuit 212) only needs to read the corresponding weight data from the external memory 220 once when processing the operational layer Lr0 (Lr1, Lr2, or Lr3) (because the same operational layer processes the same operator). In comparison, although the AI ​​model 100 performs 4 operations for each operator (OP0, OP1, OP2, or OP3) (assuming 4 batches of input data), the AI ​​model 100 does not actually perform the same operator operation on 4 batches of input data simultaneously. Therefore, the AI ​​model 100 must frequently read the corresponding weight data from the external memory, which increases bandwidth requirements.

[0058] Based on the aforementioned operating principles, this application proposes a method for establishing an artificial intelligence model. Please refer to [link / reference]. Figure 4 , Figure 4 This is a flowchart of one embodiment of the method for establishing an artificial intelligence model according to this application. The method for establishing an artificial intelligence model 400 includes the following steps.

[0059] Step S410: Determine the batch threshold N. When step S410 is executed for the first time, the batch threshold N is set to a preset value (e.g., 1). When step S410 is executed again, the batch threshold N is changed according to a preset rule.

[0060] Step S420: Based on the batch threshold N, the lifecycle of the intermediate tensor, the size of the intermediate tensor, and the capacity of the internal memory 214 of the intelligent processing unit 210, divide an artificial intelligence model into multiple operators. Step S420 will be combined with the following... Figure 6 Please provide a detailed explanation.

[0061] Step S430: Calculate the bandwidth requirement of the intelligent processing unit 210 on the external memory 220 when executing the artificial intelligence model. The bandwidth requirement corresponds to the current batch threshold N and the partitioning result of step S420. The lower (higher) the bandwidth requirement, the higher (lower) the performance of the intelligent processing unit 210 when executing the artificial intelligence model.

[0062] Step S440: Storing the correspondence between the batch threshold N and the bandwidth requirement.

[0063] Step S450: Determine if the trend in bandwidth demand has changed. Please refer to [link / reference]. Figure 5 , Figure 5This is a diagram showing the relationship between bandwidth requirements and batch threshold N in one embodiment of this application. When the batch threshold N = 1, 2, 4, 8, 16, 32, the bandwidth usage is approximately 100%, 75%, 60%, 50%, 30%, and 40%, respectively (i.e., step S440 stores the correspondence between 1, 2, 4, 8, 16, 32 and 100%, 75%, 60%, 50%, 30%, 40%). When the batch threshold N is less than or equal to 16, the bandwidth usage shows a decreasing trend (representing that the internal memory 214 is not yet fully utilized), while when the batch threshold N equals 32, the trend of bandwidth requirements changes (representing that the internal memory 214 has reached its maximum utilization rate before the batch threshold 32). In other words, in Figure 5 In this embodiment, when the batch threshold N = 16, the bandwidth requirement of the intelligent processing unit 210 for the external memory 220 is relatively low (that is, the utilization rate of the internal memory 214 is relatively high). When the result of step S450 is yes, the artificial intelligence model building method 400 ends. When the result of step S450 is no, the process returns to step S410 to determine the next batch threshold N.

[0064] exist Figure 5 In one embodiment, the batch threshold N increases based on a geometric progression (e.g., a power of 2). In other embodiments, the batch threshold N may be changed based on other rules (e.g., an arithmetic progression).

[0065] In some embodiments, the method 400 for establishing an artificial intelligence model is performed by a development device (e.g., a general-purpose computer) during the development phase of the electronic device 200 (more specifically, the intelligent processing unit 210).

[0066] Please see Figure 6 , Figure 6 This application Figure 4 A flowchart of one embodiment of step S420 includes the following sub-steps. The following description also refers to... Figure 7A ˉ Figure 7B , Figure 7A ˉ Figure 7B This is a diagram illustrating the process of dividing an artificial intelligence model. It is important to note that... Figure 7A ˉ Figure 7B This is a simplified diagram; the actual operational details are as follows: Figure 3 As shown (i.e., essentially performing operations on multiple input data simultaneously), rather than as... Figure 1 As shown (i.e., operations are performed on multiple input data in sequence).

[0067] Step S610: Based on the directed acyclic graph (DAG) of multiple operators in the artificial intelligence model, select one of the operators as a target operator. (Refer to...) Figure 7A If the current temporary set DMt only includes operator OP1, then step S610 selects the next operator (i.e. operator OP2) of the temporary set DMt as the target operator.

[0068] Step S620: Based on the lifetime and size of the intermediate tensor, calculate the temporary sum of the set data size and the data size of the intermediate tensor generated by the target operator. For example... Figure 7A As shown, when the arithmetic circuit 212 executes operator OP2, the intermediate tensor BT_A (i.e., the output of operator OP0, which is also the input of operator OP1 or temporary set DMt) is non-alive (i.e., it does not occupy internal memory 214), while the intermediate tensor BT_B (i.e., the output of operator OP1 or temporary set DMt, which is also the input of operator OP2) and the intermediate tensor BT_C (i.e., the output of operator OP2, which is also the input of operator OP3) are active (i.e., they occupy internal memory 214). Therefore, the temporary sum is equal to the sum of the data size of the intermediate tensor BT_B and the data size of the intermediate tensor BT_C. Thus, the temporary sum is related to the peak value of the intermediate tensors (i.e., the total data size of the most simultaneously active intermediate tensors).

[0069] Step S630: Determine whether the temporary sum is greater than the capacity of the internal memory 214. If yes (meaning the target operator cannot be added to the current temporary set DMt), then execute step S640; if no (meaning the target operator can be added to the current temporary set DMt), then execute step S650.

[0070] Step S640: Determine that the temporary set is an operator set, reset the temporary set (i.e., set the number of operators in the temporary set to 0), and then return to step S610.

[0071] Step S650: Add the target operator to the temporary set DMt (e.g. Figure 7B (as shown), then return to step S610 (in Figure 7B In this embodiment, the target operator is operator OP3.

[0072] It should be noted that the intermediate tensor BT_AˉBT_D is related to the number of batches M processed simultaneously by the intelligent processing unit 210. Please refer to [link / reference]. Figure 3For example, when the number of batches M = 2, the intermediate tensor BT_A includes intermediate tensor BT0_1 and intermediate tensor BT1_1, the intermediate tensor BT_B includes intermediate tensor BT0_2 and intermediate tensor BT1_2, and so on. When the number of batches M = 4, the intermediate tensor BT_A includes intermediate tensor BT0_1, intermediate tensor BT1_1, intermediate tensor BT2_1 and intermediate tensor BT3_1, the intermediate tensor BT_B includes intermediate tensor BT0_2, intermediate tensor BT1_2, intermediate tensor BT2_2 and intermediate tensor BT3_2, and so on.

[0073] Figure 8 A schematic diagram showing another embodiment of the artificial intelligence model of this application is displayed. Figure 8 In the embodiment, operators OP1, OP2, and OP3 are divided into the same operator set DM0.

[0074] When the arithmetic circuit 212 executes the operator set DM0, the data of the input operator set DM0 (i.e., intermediate tensors TS0 and TS1) and the data of the output operator set DM0 (i.e., intermediate tensor TS2) must switch according to the batch number. More specifically, the internal memory 214 may include multiple tensor buffers, each storing data corresponding to a batch number. For example, when the input data switches from input data BT0 to input data BT1, the arithmetic circuit 212 must retrieve intermediate tensors TS0 and TS1 from the tensor buffer corresponding to input data BT1 in the internal memory 214, and store intermediate tensor TS2 into the corresponding tensor buffer.

[0075] Intermediate tensors TS3 and TS4 are temporary data generated by the arithmetic circuit 212 when processing the operator set DM0. These temporary data do not need to be retained for a long time, and there is no need to switch the corresponding tensor temporary storage area.

[0076] As can be seen from the above description, the method for establishing the artificial intelligence model of this application enables the intelligent processing unit 210 to effectively utilize the internal memory 214 and reduce the bandwidth requirements of the external memory 220 when executing the artificial intelligence model.

[0077] Please see Figure 9A , Figure 9A This application Figure 4 A flowchart of another embodiment of step S420 is shown below in conjunction with... Figure 9B This application explains how it can further improve the utilization of internal memory 214. Figure 9BThis diagram illustrates another embodiment of the artificial intelligence model of this application. The artificial intelligence model 900 includes operators OP0-OP6. Operator OP0 belongs to operator set DM0_0, while operators OP1-OP6 belong to another operator set DM0_1. In this embodiment, the development device executes... Figure 9A In step S910, one of these operators is selected as the target operator based on the depth-first-search (DFS) algorithm. Figure 9A Steps S620-S650 are respectively with Figure 6 The steps S620-S650 are the same.

[0078] The advantage of using a depth-first search algorithm to determine the target operator is that it can shorten the lifetime of intermediate tensors, reducing the time that intermediate tensors occupy internal memory (2^14). Figure 9B For example, according to the depth-first search algorithm, operators OP1–OP5 are selected in the following order: OP1→OP2→OP3→OP4→OP5. In this case, the lifetime of the intermediate tensor TS1 ends in the third step (when operator OP3 is selected) (i.e., at this point, the intermediate tensor TS1 no longer occupies memory). In contrast, according to the breadth-first search (BFS) algorithm, operators OP1–OP5 are selected in the following order: OP1→OP4→OP2→OP5→OP3. In this case, the lifetime of the intermediate tensor TS1 ends in the fifth step (when operator OP3 is selected).

[0079] The shorter the lifetime of the intermediate tensor, the shorter the time it occupies the internal memory 214. Therefore, the internal memory 214 can be used more effectively, and the intelligent processing unit 210 can reduce the frequency of accessing the external memory 220 (i.e., reduce the bandwidth requirement), which helps to improve the overall performance of the electronic device 200.

[0080] Figure 10 This is a functional block diagram of one embodiment of the electronic device of this application. Electronic device 1000 is similar to electronic device 200, wherein the intelligent processing unit 1010, arithmetic circuit 1012, internal memory 1014, direct memory access circuit 1016, and external memory 1020 can respectively correspond to intelligent processing unit 210, arithmetic circuit 212, internal memory 214, direct memory access circuit 216, and external memory 220. Intelligent processing unit 1010 also includes internal memory 1018. Intelligent processing unit 1010 includes two internal memories. Direct memory access circuit 1016 updates internal memory 1014 with data from external memory 1020, and updates internal memory 1018 with data from internal memory 1014.

[0081] Based on the intelligent processing unit 1010 including two internal memories, this application proposes another method for partitioning operators. (See reference...) Figure 11A ˉ Figure 11B , Figure 11A ˉ Figure 11B This application Figure 4 A flowchart of another embodiment of step S420. Figure 11A The process involves dividing multiple operators in an artificial intelligence model into multiple operator groups based on the capacity of the internal memory 1018. Figure 11B The process involves dividing multiple operator groups into multiple operator sets based on the capacity of the internal memory 1014. Figure 11A and Figure 6 Similarly (steps S1110–S1150 correspond to steps S610–S650 respectively), and the first internal memory in step S1130 is internal memory 1018. See reference... Figure 12A In one embodiment, Figure 11A After the process is completed, the operators OP0-OP6 are divided into 4 operator groups: DM1_0-DM1_3.

[0082] The development device was completed Figure 11A After the process, continue execution. Figure 11B . Figure 11B It includes the following steps.

[0083] Step S1160: Select one of these operator groups as a target operator group. (See also...) Figure 12B The following explanation uses the example where the temporary set DMt only includes operator group DM1_1, and the target operator group is operator group DM1_2. Similar to step S910, step S1160 can select the target operator group based on a depth-first search algorithm to shorten the lifetime of the intermediate tensor.

[0084] Step S1170: Based on the lifecycle and size of the intermediate tensors, calculate the temporary sum of the set data volume of the temporary set DMt (i.e., the peak value of the intermediate tensors in the temporary set DMt) and the target group data volume of the target operator group (i.e., the peak value of the intermediate tensors in the target operator group). Continuing from the previous content, at this point, the set data volume is the total data volume of the most simultaneously active intermediate tensors in the temporary set DMt, and the target group data volume is the total data volume of the most simultaneously active intermediate tensors in the operator group DM1_2.

[0085] Step S1180: Determine whether the temporary sum is greater than the capacity of the second internal memory (i.e., internal memory 1014). This step is similar to step S630, so it will not be described again.

[0086] Steps S1190 and S1195 are similar to steps S640 and S650, respectively, so they will not be described again.

[0087] When the intelligent processing unit 1010 executes the artificial intelligence model, the arithmetic circuit 1012 executes each operator set in the order of the operator sets. Within each operator set, each operator group is executed in the order of the operator groups. Within each operator group, each operator is executed in the order of the operators themselves. In this way, the intelligent processing unit 1010 can effectively utilize the internal memory 1014 and internal memory 1018.

[0088] Compared to Figure 2 Implementation examples, Figure 10 The intelligent processing unit 1010 has an additional internal memory, allowing operators to be first divided into operator groups, and then further divided into operator sets. The advantage of adding operator groups is that when tensors in the internal memory 1018 need to be rearranged (e.g., by changing dimensions), the direct memory access circuit 1016 only needs to move data between the internal memory 1018 and internal memory 1014 to complete the tensor rearrangement. In comparison, for... Figure 2 For the intelligent processing unit 210, the rearrangement of tensors requires writing and reading data from the external memory 220, which increases the bandwidth requirements of the intelligent processing unit 210 on the external memory 220.

[0089] In some embodiments, the capacity of internal memory 1014 is greater than the capacity of internal memory 1018. It should be noted that when data in internal memory 1018 needs to be updated, the bandwidth requirement of external memory 1020 will not increase because direct memory access circuitry 1016 retrieves data from internal memory 1014 rather than from external memory 1020.

[0090] Back Figure 4 After obtaining Figure 5 After the relationship diagram between bandwidth requirements and batch threshold N (i.e., after the artificial intelligence model establishment method 400 is completed) is generated, the number of batches to be processed can be split based on the relationship diagram during actual operation, making the execution of the artificial intelligence model more efficient (e.g., reducing bandwidth requirements). For example, the developer or manufacturer of electronic device 200 (or electronic device 1000) can split the number of batches to be processed by calling one of the functions in the software development kit (SDK) provided by the developer or manufacturer of intelligent processing unit 210 (or intelligent processing unit 1010).

[0091] Please see Figure 13 , Figure 13This is a flowchart of an embodiment of the artificial intelligence execution method of this application, which includes the following steps.

[0092] Step S1310: Based on the correspondence between the number of batches and the bandwidth requirement, the number of batches to be processed, P_top, is split into multiple sub-batch numbers. The function described above corresponds to step S1310.

[0093] Step S1320: Execute the artificial intelligence model using one of these sub-batch numbers as the batch number of input data for the artificial intelligence model. As discussed above, when the artificial intelligence model is executed based on the input data according to the sub-batch number, the intelligent processing unit can effectively utilize the internal memory.

[0094] Please see Figure 14 , Figure 14 This application Figure 13 A flowchart of one embodiment of step S1310 includes the following sub-steps.

[0095] Step S1410: First, let the remaining number of batches P_rem be equal to the number of batches to be processed P_top.

[0096] Step S1420: Determine the target batch threshold N_tar. Figure 5 For example, since the batch threshold N includes 16, 8, 4, 2, and 1, the above function determines the target batch threshold N_tar according to a preset order (e.g., power-down order) (e.g., the target batch threshold N_tar is equal to 16, 8, 4, 2, and 1 in sequence).

[0097] Step S1430: Determine whether the remaining batch number P_rem is greater than or equal to the target batch threshold N_tar. If yes, proceed to step S1440; otherwise, return to step S1420 to select the next batch threshold N as the target batch threshold N_tar.

[0098] Step S1440: Use the target batch threshold N_tar as the number of sub-batches.

[0099] Step S1450: Update the remaining batch number P_rem by subtracting the target batch threshold N_tar from the remaining batch number P_rem.

[0100] Step S1460: Determine if the remaining batch number P_rem is 0. If yes, end step S1310; otherwise, return to step S1420.

[0101] In some embodiments, electronic device 200 (or electronic device 1000) is an image processing device, and a batch of input data can correspond to a single face. Assuming that electronic device 200 (or electronic device 1000) needs to process 37 batches of input data (i.e., the number of batches to be processed P_top = 37), the function can be based on... Figure 5 The relationship diagram (more specifically, based on the batch threshold N) divides 37 into four sub-batch numbers: 16, 16, 4, and 1. That is, when electronic device 200 (or electronic device 1000) actually executes the aforementioned artificial intelligence model, the model essentially performs the same operator operation simultaneously on the N batches of input data (i.e., one of the batch thresholds) (see the section on...). Figure 3 (Explanation).

[0102] In this way, regardless of the actual application, the intelligent processing unit 210 (or intelligent processing unit 1010) can reduce the bandwidth requirements of the external memory 220 (or external memory 1020).

[0103] Although the embodiments of this application are described above, these embodiments are not intended to limit this application. Those skilled in the art can make changes to the technical features of this application based on the express or implied content of this application. All such changes may fall within the scope of patent protection sought by this application. The scope of patent protection of this application shall be defined by the patent application scope in this specification.

Claims

1. A method for establishing an artificial intelligence model, characterized in that, The method is applied to an intelligent processing unit, the intelligent processing unit including a computing circuit and a memory, the artificial intelligence model including multiple operators, the computing circuit generating an intermediate tensor during the execution of each operator, the method comprising: (A) The operator is divided according to a batch threshold, a lifetime of the intermediate tensor, the size of the intermediate tensor, and a capacity of the memory; (B) Calculate the bandwidth requirement of the intelligent processing unit for an external memory when executing the artificial intelligence model; (C) Store the one-to-one correspondence between the batch threshold and the bandwidth requirement; and (D) Adjust the batch threshold and repeat steps (A) to (C) to obtain multiple batch thresholds until the trend of the bandwidth demand changes; The artificial intelligence model performs the same operator operation on N batches of input data simultaneously, where N is a positive integer and is one of the batch thresholds.

2. The method for establishing an artificial intelligence model as described in claim 1, characterized in that, Step (A) includes: (A1) Based on a directed acyclic graph of the operators, select one of the operators as a target operator; (A2) Add a set of data from a temporary set to a set of data from the intermediate tensor generated by the target operator to obtain a temporary sum, wherein the temporary set includes at least one of the operators; (A3) When the temporary sum is not greater than the capacity of the memory, the target operator becomes part of the temporary set; (A4) Repeat steps (A1) through (A3) until the temporary sum is greater than the capacity of the memory; and (A5) Determine that the temporary set is an operator set.

3. The method for establishing an artificial intelligence model as described in claim 2, characterized in that, Step (A1) is to select the target operator based on a depth-first search algorithm.

4. The method for establishing an artificial intelligence model as described in claim 1, characterized in that, The memory is a first memory, the capacity is a first capacity, and the intelligent processing unit further includes a second memory. Step (A) includes: (A1) Based on a directed acyclic graph of the operators, select one of the operators as a target operator; (A2) Add a group of data from a temporary group to a data volume of the intermediate tensor generated by the target operator to obtain a first temporary sum, wherein the temporary group includes at least one of the operators; (A3) When the first temporary sum is not greater than the first capacity of the first memory, the target operator becomes part of the temporary group; (A4) Repeat steps (A1) to (A3) until the first temporary sum is greater than the first capacity of the first memory; (A5) Determine that the temporary group is an operator group; (A6) Repeat steps (A1) through (A5) to obtain multiple operator groups; (A7) Select one of the operator groups as a target operator group; (A8) Add the set data of a temporary set to the target group data of the target operator group to obtain a second temporary sum, wherein the temporary set includes at least one of the operator groups; (A9) When the second temporary sum is not greater than a second capacity of the second memory, the target operator group becomes part of the temporary set; (A10) Repeat steps (A7) to (A9) until the second temporary sum is greater than the second capacity of the second memory; and (A11) Determines that the temporary set is a set of operators.

5. The method for establishing an artificial intelligence model as described in claim 4, characterized in that, Step (A1) is to select the target operator based on a depth-first search algorithm.

6. The method for establishing an artificial intelligence model as described in claim 5, characterized in that, Step (A7) is to select the target operator group based on the depth-first search algorithm.

7. The method for establishing an artificial intelligence model as described in claim 1, further comprising: (D) Increase the batch threshold, and repeat steps (A) to (C) to obtain multiple batch thresholds until the bandwidth requirement increases, where N is one of the batch thresholds.

8. A method for executing an artificial intelligence model, characterized in that, An artificial intelligence model is applied to an electronic device, the electronic device including an intelligent processing unit and a memory, wherein the intelligent processing unit does not include the memory, the artificial intelligence model is established using the method for establishing an artificial intelligence model as described in any one of claims 1 to 7, the artificial intelligence model simultaneously performs the same operator operation on M batches of input data, where M is a positive integer, the method comprising: (A) Based on multiple batch thresholds and the multiple bandwidth requirements of the intelligent processing unit for the memory, a batch to be processed is divided into multiple sub-batch numbers, wherein the batch thresholds and the bandwidth requirements correspond to each other; and (B) Using one of the number of sub-batches as M, execute the artificial intelligence model.

9. The method for executing the artificial intelligence model as described in claim 8, characterized in that, Step (A) includes: (A1) Let the number of remaining batches be equal to the number of batches to be processed; (A2) Determine a target batch threshold; (A3) When the remaining number of batches is greater than or equal to the target batch threshold, the target batch threshold is used as the number of sub-batches, and the remaining number of batches is equal to the remaining number of batches minus the target batch threshold; and (A4) Repeat steps (A2) to (A3) until the remaining number of batches is equal to 0.

Images