Processing circuit of artificial intelligence model and operation scheduling method

Abstract

The present application discloses a processing circuit of an artificial intelligence model and an operation scheduling method. The processing circuit is coupled to an external memory and includes a memory, a memory management circuit, and an operation circuit. The memory management circuit reads a tensor from the external memory and stores the tensor to the memory. The operation circuit is configured to perform the following operations: performing a first type of operation on a first sub-tensor of the tensor to generate a first intermediate data; performing the first type of operation on a second sub-tensor of the tensor to generate a second intermediate data; performing a second type of operation on the first intermediate data and the second intermediate data to generate a third intermediate data; performing the first type of operation on a third sub-tensor of the tensor to generate a fourth intermediate data; and performing the second type of operation on the first intermediate data, the second intermediate data, and the fourth intermediate data to generate a fifth intermediate data.

Description

Technical Field

[0001] This invention relates to artificial intelligence models, and more particularly to processing circuits and computation scheduling methods for artificial intelligence models. Background Technology

[0002] In a system-on-a-chip (SoC), the total memory bandwidth is typically fixed and shared by multiple modules. When a module consumes too much memory bandwidth, it can cause other modules to be blocked from accessing memory, leading to a decrease in system performance. Artificial intelligence (AI) models, as modules within a SoC, often need to process large amounts of data, resulting in high memory bandwidth requirements; therefore, reducing the bandwidth requirements of AI models has become an important challenge. Summary of the Invention

[0003] In view of the shortcomings of the prior art, one object of the present invention is to provide a processing circuit and operation scheduling method for an artificial intelligence model to improve the shortcomings of the prior art.

[0004] An embodiment of the present invention provides a processing circuit for an artificial intelligence model. The processing circuit is coupled to an external memory and includes a memory, a memory management circuit, and an arithmetic circuit. The memory management circuit reads a tensor from the external memory and stores the tensor in the memory. The arithmetic circuit is configured to perform the following operations: perform a first type of operation on a first sub-tensor of the tensor to generate first intermediate data; perform the first type of operation on a second sub-tensor of the tensor to generate second intermediate data; perform a second type of operation on the first and second intermediate data to generate third intermediate data; perform the first type of operation on a third sub-tensor of the tensor to generate fourth intermediate data; and perform the second type of operation on the first, second, and fourth intermediate data to generate fifth intermediate data.

[0005] Another embodiment of the present invention provides a processing circuit for an artificial intelligence model. The processing circuit is coupled to an external memory and includes a memory. The processing circuit performs the following operations: reading a tensor and a plurality of core parameters from the external memory, and storing the tensor and the core parameters into the memory, wherein the tensor includes a first sub-tensor and a second sub-tensor, and the core parameters include a vector core parameter; performing a first vector operation on the first sub-tensor based on a first portion of the vector core parameter to generate first intermediate data; and performing a second vector operation on the second sub-tensor based on a second portion of the vector core parameter to generate second intermediate data. The first portion of the vector core parameter is not equal to the second portion of the vector core parameter.

[0006] Another embodiment of the present invention provides a method for scheduling operations of an artificial intelligence model. The artificial intelligence model includes a first operator and a second operator. The method for scheduling operations includes: dividing a tensor into H sub-tensors, where H is an integer greater than 1; dividing the first operator into H first sub-operators; dividing the second operator into H second sub-operators; determining a dependency relationship between the H first sub-operators and the H second sub-operators; sorting the H first sub-operators and the H second sub-operators according to the dependency relationship to obtain an operation order; and determining, according to the operation order, when a processing circuit executing the artificial intelligence model should delete a target data from a memory contained in the processing circuit, the target data being the output data of one of the H first sub-operators and the H second sub-operators.

[0007] The technical means embodied in the embodiments of the present invention can improve at least one of the disadvantages of the prior art. Therefore, the present invention can reduce memory usage and / or reduce the bandwidth requirements of memory compared to the prior art.

[0008] The features, implementation, and effects of this application are described in detail below with reference to the accompanying drawings, using preferred embodiments. Attached Figure Description

[0009] Figure 1 This shows an example of an AI network;

[0010] Figure 2 A flowchart of an embodiment of the computation scheduling method for the artificial intelligence model of the present invention;

[0011] Figure 3 yes Figure 1 The result of segmenting tensors and operators;

[0012] Figure 4 A topological graph showing the connections between suboperators;

[0013] Figure 5 yes Figure 2 A detailed flow of one embodiment of step S230;

[0014] Figure 6 Display the dependencies between multiple suboperators;

[0015] Figure 7 yes Figure 2 A detailed flow of one embodiment of step S240;

[0016] Figure 8A and Figure 8B A schematic diagram showing an embodiment of the queue of the present invention;

[0017] Figure 9 This is a functional block diagram of an embodiment of the electronic device of the present invention;

[0018] Figure 10 A schematic diagram showing an embodiment of the lifecycle list of the present invention;

[0019] Figure 11 Display a flowchart of memory allocation;

[0020] Figure 12A and Figure 12B This is a schematic diagram of an embodiment of the active list of the present invention;

[0021] Figure 13 This is a schematic diagram of another AI model;

[0022] Figure 14 This is a schematic diagram of another AI model;

[0023] Figure 15A and Figure 15B This is a flowchart of an embodiment of the execution method of the artificial intelligence model of the present invention;

[0024] Figure 16 This is a schematic diagram of an embodiment of the cache circuit and the stored content of the memory of the present invention;

[0025] Figure 17 yes Figure 15A Detailed process of step S1510 or step S1520;

[0026] Figure 18 yes Figure 15A The detailed process of step S1530;

[0027] Figure 19 This is a schematic diagram of another embodiment of the cache circuit and memory storage content of the present invention;

[0028] Figure 20 yes Figure 15BThe detailed process of step S1560;

[0029] Figure 21 This is a functional block diagram of an embodiment of the memory management circuit of the present invention;

[0030] Figure 22 This is a schematic diagram of the multi-stage production line of the present invention; and

[0031] Figure 23 This is a flowchart of an embodiment of the multi-stage pipeline operation of the present invention. Detailed Implementation

[0032] The technical terms used in the following description are based on the customary terms in this technical field. If this specification provides explanations or definitions for certain terms, the interpretation of those terms shall be based on the explanations or definitions in this specification.

[0033] The disclosure of this invention includes processing circuitry and computation scheduling methods for artificial intelligence models. Since some components of the processing circuitry for the artificial intelligence model of this invention may be known individually, details of known components will be omitted in the following description without affecting the full disclosure and implementability of the device invention.

[0034] Figure 1 This is an example of an AI network, which can be viewed as a simple AI model or part of a complex AI model. AI network 100 is used to perform operations on the input data Din to produce the output data Dout. Figure 1 The AI ​​network 100 includes three operators: a subtraction operator 110 ("SUB"), a convolution operator 120 ("CONV"), and an addition operator 130 ("ADD"). The subtraction operator 110 performs a subtraction operation on tensor TS1 (i.e., input data Din) to produce tensor TS2. The convolution operator 120 performs a convolution operation on tensor TS2 to produce tensor TS3. The addition operator 130 performs an addition operation on tensor TS3 to produce tensor TS4 (i.e., output data Dout). Figure 1 In the example, the sizes (dimensional information) of tensors TS1, TS2, TS3 and TS4 are all [1,3,224,224].

[0035] Figure 2 This is a flowchart of an embodiment of the computation scheduling method for the artificial intelligence model of the present invention. Figure 2 The process is performed by chip development tools (e.g., a computer) and includes the following steps.

[0036] Step S210: Divide the tensor into H sub-tensors (or tiles), where H can be any dimension of the tensor (H is an integer greater than 1). More specifically, this step determines the value of H based on one dimension of the output tensor of the last operator of the AI ​​network 100, and then divides the tensor into H sub-tensors. Figure 1 Taking AI Network 100 as an example, since the size of the output tensor (i.e., tensor TS4) of the last operator (adder 130) is [1, 3, 224, 224], H can be either 3 or 224. Details on tensor splitting will be provided below. Figure 3 Provide an explanation.

[0037] Step S220: Divide the operator into H suboperators. This step will be used in conjunction with step S210 below. Figure 3 Provide an explanation.

[0038] Step S230: Determine the dependencies between multiple suboperators. This step will be discussed in conjunction with the following steps. Figure 5 Provide an explanation.

[0039] Step S240: Sort the suboperators according to their dependencies to obtain the operation order. This step will be explained below. Figure 7 Provide an explanation.

[0040] Step S250: Based on this operation sequence, determine when the electronic device executing the artificial intelligence model (more specifically, the processing circuitry of the electronic device) deletes target data from memory, which is the output data of one of the suboperators (i.e., the intermediate data of the AI ​​network 100). This step will be discussed below. Figure 10 , Figure 11 , Figure 12A and Figure 12B Provide an explanation.

[0041] Please see Figure 3 , Figure 3 yes Figure 1 The result of segmenting tensors and operators. Figure 3In this embodiment, the tensor segmentation operation (i.e., step S210) is based on the second dimension of tensor TS4 (i.e., H = 3). Therefore, subtraction operator 110 is segmented into subtraction sub-operator 110_1 ("SUB1"), subtraction sub-operator 110_2 ("SUB2"), and subtraction sub-operator 110_3 ("SUB3"); convolution operator 120 is segmented into convolution sub-operator 120_1 ("CONV1"), convolution sub-operator 120_2 ("CONV2"), and convolution sub-operator 120_3 ("CONV3"); addition operator 130 is segmented into addition sub-operator 130_1 ("ADD1"), addition sub-operator 130_2 ("ADD2"), and addition sub-operator 130_3 ("ADD3"). Tensor TS1 is divided into subtensors TS1_i1, TS1_i2, and TS1_i3 (which are the input subtensors of suboperators 110_1, 110_2, and 110_3, respectively, all with a size of [1, 1, 224, 224], and corresponding to the same dimension of tensor TS1 (e.g., the second dimension)). Tensor TS4 is divided into subtensors TS3_o1, TS3_o2, and TS3_o3 (which are the output subtensors of suboperators 130_1, 130_2, and 130_3, respectively, all with a size of [1, 1, 224, 224], and corresponding to the same dimension of tensor TS4). Tensor TS3 is divided into subtensors TS3_i1, TS3_i2, and TS3_i3 (which are the input subtensors of adders 130_1, 130_2, and 130_3, respectively, all of size [1, 1, 224, 224], and corresponding to the same dimension of tensor TS3). The individual output subtensors of convolution operators 120_1, 120_2, and 120_3 (i.e., subtensors TS2_o1, TS2_o2, and TS2_o3) are equal to subtensors TS3_i1, TS3_i2, and TS3_i3, respectively.

[0042] It should be noted that because of the visual field magnification... The problem with the enlargement is that the subtensor TS1_o1 (subtensor TS1_o2 or subtensor TS1_o3) output by the sub-operator 110_1 (sub-operator 110_2 or sub-operator 110_3) is not equal to the input subtensor TS2_i1 (subtensor TS2_i2 or subtensor TS2_i3) of the convolution sub-operator 120_1 (convolution sub-operator 120_2 or convolution sub-operator 120_3); more specifically, the sizes of subtensors TS1_o1, TS1_o2, and TS1_o3 are all [1,1,224,224], but the sizes of subtensors TS2_i1 and TS2_i3 are both [1,2,224,224], while the size of subtensor TS2_i2 is [1,3,224,224].

[0043] Depend on Figure 3 It can be seen that since subtensors TS1_o1, TS1_o2, and TS1_o3 correspond to subtensors TS1_i1, TS1_i2, and TS1_i3 respectively, and subtensors TS1_i1, TS1_i2, and TS1_i3 correspond to the same dimension of tensor TS1 (e.g., the second dimension), subtensors TS1_o1, TS1_o2, and TS1_o3 also correspond to the same dimension of tensor TS1.

[0044] Figure 2 This process can effectively manage the duration of target data in the memory of electronic devices, helping to reduce memory usage and / or lower bandwidth requirements. Details will follow below. Figure 9 , Figure 10 , Figure 11 , Figure 12A and Figure 12B Provide an explanation.

[0045] Based on the overlapping relationships between subtensors segmented from the same tensor (see [link to relevant documentation]). Figure 3 ), can yield the following Figure 4 The diagram shows the topological graph of connections between suboperators. Specifically, this topological graph is obtained based on the overlap between the input subtensors of each suboperator segmented from the same operator and the output subtensors of the source operators they depend on. For example... Figure 4As shown, the input subtensor TS2_i1 of convolution operator 120_1 includes subtensors TS1_o1 and TS1_o2; that is, convolution operator 120_1 can only begin after subtraction operators 110_1 and 110_2 have finished. Similarly, convolution operator 120_2 can only begin after subtraction operators 110_1, 110_2, and 110_3 have finished; convolution operator 120_3 can only begin after subtraction operators 110_2 and 110_3 have finished. Adder operators 130_1, 130_2, and 130_3 can only begin after convolution operators 120_1, 120_2, and 120_3 have all finished.

[0046] Please see Figure 5 , Figure 5 yes Figure 2 A detailed process of one embodiment of step S230 includes the following steps. (The following is in conjunction with...) Figure 4 illustrate Figure 5 Details.

[0047] Step S510: Determine the target sub-operator. For example, select convolution sub-operator 120_1 as the target sub-operator.

[0048] Step S520: Determine the source suboperator of the target suboperator. Continuing the previous example, since the source of the input subtensor TS2_i1 of the convolution suboperator 120_1 includes subtensors TS1_o1 and TS1_o2, the source suboperators of the convolution suboperator 120_1 are subtraction suboperators 110_1 and 110_2 (that is, the output subtensor TS1_o1 of subtraction suboperator 110_1 and the output subtensor TS1_o2 of subtraction suboperator 110_2 are the input subtensors of the convolution suboperator 120_1). Similarly, the source suboperators of convolution suboperator 120_2 are subtraction suboperators 110_1, 110_2, and 110_3; the source suboperators of convolution suboperator 120_3 are subtraction suboperators 110_2 and 110_3; and the source suboperator of addition suboperator 130_1 is convolution suboperator 120_1.

[0049] Step S530: Determine that the target suboperator depends on the source suboperator; that is, the source suboperator is a dependent suboperator of the target suboperator. For example, suboperators 110_1, 110_2, and 110_3 are dependent suboperators of convolution suboperator 120_2.

[0050] Take turns Figure 4 Each suboperator is used as the target suboperator and repeated Figure 5 The process can determine the dependencies between multiple suboperators, such as... Figure 6 As shown. Convolution operator 120_1 depends on subtraction operators 110_1 and 110_2. Convolution operator 120_2 depends on subtraction operators 110_1, 110_2, and 110_3. Convolution operator 120_3 depends on subtraction operators 110_2 and 110_3. Addition operators 130_1, 130_2, and 130_3 depend on convolution operators 120_1, 120_2, and 120_3, respectively.

[0051] Please see Figure 7 , Figure 7 yes Figure 2 A detailed process of one embodiment of step S240 includes the following steps. Figure 7 The process is based on the depth-first search algorithm.

[0052] Step S710: Find suboperators with an indegree of 0 and mark them as target suboperators that have been visited. Suboperators with an indegree of 0 are suboperators that have no dependencies. Figure 6 For example, adder suboperators 130_1, 130_2, and 130_3 are suboperators with an in-degree of 0, which are the top-level suboperators.

[0053] Step S720: Determine if a suboperator with an in-degree of 0 has been found. If yes, proceed to step S730; otherwise, proceed to step S795. The following explanation uses the addition suboperator 130_1 as an example.

[0054] Step S730: Find unvisited dependent suboperators of the target suboperator. For example... Figure 6 As shown, since convolution operator 120_1 is a dependent sub-operator of adder operator 130_1 (i.e., adder operator 130_1 depends on convolution operator 120_1), step S730 finds convolution operator 120_1.

[0055] Step S740: Determine whether a dependent suboperator has been found. If yes, proceed to step S750; otherwise, proceed to step S760.

[0056] Step S750: Mark the dependent sub-operator as the target sub-operator and acknowledge its visit, then proceed to step S730. Continuing the example above, in step S750, the convolution sub-operator 120_1 is marked as the target sub-operator and acknowledged. Then, when step S730 is executed again, the dependent sub-operator of the convolution sub-operator 120_1 is found (assuming the subtraction sub-operator 110_1 is found). Then, steps S730 and S740 are executed again; at this time, because the subtraction sub-operator 110_1 does not depend on any sub-operator (i.e., there is no dependent sub-operator, step S740 is negative), the process proceeds to step S760.

[0057] Step S760: Add the target suboperator to queue 800. Continuing with the previous example, this time add the subtraction suboperator 110_1 to queue 800. Please refer to... Figure 8A and Figure 8B , Figure 8A and Figure 8B Display changes to the contents of queue 800 ( Figure 8B Continued Figure 8A ).like Figure 8A As shown in the first column, queue 800 at this time contains only subtraction operator 110_1 ("SUB1").

[0058] Step S770: Determine whether the target suboperator is the top-level suboperator (i.e., the suboperator with an in-degree of 0). If yes, proceed to step S710; otherwise, proceed to step S780. Continuing with the previous example, since the subtraction suboperator 110_1 is not the top-level suboperator, the result of step S770 is no.

[0059] Step S780: Determine one of the upper-level suboperators that depends on the target suboperator (i.e., the previous-level suboperator) and mark that upper-level suboperator as the target suboperator. Continuing with the previous example, the process will now fall back to convolution suboperator 120_1.

[0060] Step S790: Determine if there are any unmarked dependent suboperators. Continuing the previous example, since there are still unmarked suboperators (i.e., subtraction suboperator 110_2) among the dependent suboperators (subtraction suboperator 110_1 and subtraction suboperator 110_2) of the target suboperator (convolution suboperator 120_1), the result of step S790 is yes; then, the process executes the following steps: Step S730 (find subtraction suboperator 110_2) → Step S740 (determined result is yes) → Step S750 (mark subtraction suboperator 110_2 as visited) → Step S730 (not found dependent suboperators of subtraction suboperator 110_2) → Step S740 (determined result is no) → Step S760 (add subtraction suboperator 110_2 to queue 800, etc.) Figure 8A(As shown in the second column) → Step S770 (Judgment result is negative) → Step S780 (Mark convolution sub-operator 120_1 as target sub-operator) → Step S790. At this time, because all dependent sub-operators of the target sub-operator (i.e., convolution sub-operator 120_1) (i.e., subtraction sub-operator 110_1 and subtraction sub-operator 110_2) have been accessed, the judgment result of step S790 is negative, and therefore convolution sub-operator 120_1 is added to queue 800 in the following step S760 (e.g. Figure 8A (As shown in the third column). After continuing to execute steps S770, S780 (marking the adder suboperator 130_1 as the target suboperator), S790, and S760 (adder suboperator 130_1 is added to queue 800), the judgment result of step S770 is yes (because adder suboperator 130_1 is the top-level suboperator), and the process returns to step S710 to select the next suboperator with an in-degree of 0 (e.g., adder suboperator 130_2).

[0061] Steps S710 to S790 described above will be repeated. Figure 6 The process of adding all suboperators to queue 800 is as follows: Figure 8A and Figure 8B As shown (not repeated here), until all suboperators with an in-degree of 0 have been visited (i.e., if the result of step S720 is no, the process proceeds to step S795).

[0062] Step S795: Sequentially retrieve all suboperators from queue 800. Figure 8B For example, the order in which suboperators are removed from queue 800 (that is, the order of suboperator operations) is: SUB1→SUB2→CONV1→…→CONV3→ADD3 (that is, the reverse of the order in which suboperators are added to the queue).

[0063] Figure 9 This is a functional block diagram of an embodiment of the electronic device of the present invention. The electronic device 900 includes a chip 901 and external memory 902 (e.g., Dynamic Random Access Memory (DRAM)). Chip 901 and external memory 902 are coupled or electrically connected to each other. Chip 901 includes processing circuitry 910 and processor 920. Processing circuitry 910 and processor 920 are coupled or electrically connected to each other.

[0064] The processor 920 controls the processing circuit 910 to jointly implement the functions of the chip 901. The processor 920 can be a circuit or electronic component with program execution capability, such as a central processing unit, microprocessor, microprocessor unit, digital signal processor, application specific integrated circuit (ASIC), or equivalent circuit.

[0065] The processing circuit 910 can be an intelligence processing unit (IPU) or a neural network processing unit (NPU). The processing circuit 910 includes an arithmetic circuit 912 (e.g., including but not limited to a convolution engine and a vector engine), a cache circuit 914 (e.g., including but not limited to multiple caches), a memory management circuit 916 (e.g., Direct Memory Access (DMA)), and memory 918 (e.g., Static Random Access Memory (SRAM)). The cache circuit 914 is used to store data required by the arithmetic circuit 912 when performing convolution or vector operations. Memory 918 can store... Figure 3 The output subtensors of each suboperator.

[0066] External memory 902 stores input data Din, core parameter Kp, and output data Dout. Memory management circuit 916 reads input data Din and core parameter Kp from external memory 902 and stores them in memory 918, reads at least a portion of input data Din and at least a portion of core parameter Kp from memory 918 and stores them in cache circuit 914, and stores output data Dout generated by arithmetic circuit 912 in external memory 902.

[0067] The following combinations Figure 9 , Figure 10 , Figure 11 , Figure 12A and Figure 12B illustrate Figure 2 Details of step S250. Figure 10 show Figure 3This is a list of the lifespans of the output subtensors of the suboperators. The horizontal axis corresponds to the order of operations of the aforementioned suboperators (not necessarily to the actual time length); more specifically, the output subtensor TS1_o1 of the subtraction suboperator 110_1 is generated at time point 0 of the operation order and ends at time point 5 of the operation order (i.e., it will not be used by other suboperators). Note that subtensors TS3_o1, TS3_o2, and TS3_o3 are generated at time points 3, 6, and 8 of the operation orders, respectively; however, because subtensors TS3_o1, TS3_o2, and TS3_o3 are not deleted from memory 918 in advance (because they are each part of the output data Dout), Figure 10 The lifecycle list does not show these three sub-tensors.

[0068] Figure 2 Details of step S250 are included in the reference. Figure 10 The lifecycle list allocates memory 918, and the process of allocating memory 918 is as follows: Figure 11 As shown. Figure 12A and Figure 12B This is a schematic diagram of one embodiment of the active list of the present invention. Please refer to the following description as well. Figure 10 , Figure 11 , Figure 12A and Figure 12B The active list is used to display the activity of the sub-tensor in memory 918 (more specifically, to display the time when the sub-tensor was stored in memory 918 and deleted from memory 918). Figure 11 It includes the following steps.

[0069] Step S1110: Create a lifecycle list. An example of a lifecycle list is as follows: Figure 10 As shown.

[0070] Step S1120: Search the lifecycle list to find the subtensors that are currently active. For example, subtensor TS1_o1 becomes active at time point 0 of operation order, and the active period of subtensor TS1_o1 is from time point 0 of operation order to time point 5 of operation order.

[0071] Step S1130: Add the active subtensor to the active list. For example... Figure 12A As shown, the subtensor TS1_o1 is added to the active list when its lifetime is 0.

[0072] Step S1140: Allocate memory for the active subtensor, that is, arrange the corresponding storage space in memory 918. Continuing the previous example, as follows... Figure 12A As shown, a portion of memory 918 is allocated to the subtensor TS1_o1 when its lifetime is 0.

[0073] Step S1150: Delete inactive sub-tensors from the active list. For example, because... Figure 10 The neutron tensor TS2_o1 becomes inactive after the time point with operation sequence 3, therefore... Figure 12A The neutron tensor TS2_o1 was deleted when its lifetime was 3.

[0074] Step S1160: Release the memory corresponding to the no longer active sub-tensor. In response to the previous step of deleting the sub-tensor from the active list, this step releases the corresponding storage space in memory 918, so that memory 918 can be used more promptly and flexibly.

[0075] Step S1170: Increment the lifecycle by 1.

[0076] If there are no inactive subtensors in the current lifecycle, skip steps S1150 and S1160 and proceed directly to step S1170.

[0077] Step S1180: Determine if the lifecycle has ended (i.e., determine...) Figure 10 (Whether the operation sequence has ended). If so, then end. Figure 11 If not, proceed to step S1120 to continue finding active subtensors.

[0078] As mentioned above, refer to Figure 10 Lifecycle list and Figure 11 The process can be obtained Figure 12A and Figure 12B The list of active users. Figure 12A and Figure 12B The lifecycle corresponds to Figure 10 The order of operations. For example, in Figure 10 In the process, the subtensor TS1_o1 is generated at time point 0 of the operation sequence and ends at time point 5 of the operation sequence; therefore, in Figure 12A The lifetime of the neutron tensor TS1_o1 is 0 to 4. Similarly, in Figure 10 The neutron tensor TS2_o2 exists between time points with operation sequence 5 and operation sequence 6; therefore, in Figure 12B In this context, the subtensor TS2_o2 only exists within lifetime 5. This allows the developers or designers of the 901 chip to refer to... Figure 12A and Figure 12B The active list is used to design or manage memory 918; therefore, the bandwidth requirements of external memory 902 can be reduced (to save costs) without increasing memory 918 (which can improve the overall performance of external memory 902), or memory 918 can be further saved without increasing the memory bandwidth of external memory 902.

[0079] For comparison, if there is no... Figure 1 If the operators and tensors are segmented, the chip developer must pre-allocate one storage block of memory 918 to tensor TS2 (the total data volume is equivalent to the sum of the data volumes of sub-tensors TS1_o1, TS1_o2, and TS1_o3), and this storage block cannot be released until the convolution operator 120 finishes. Furthermore, if the sub-operations after segmentation are not ordered, the same sub-tensor, such as sub-tensor TS1_o1, must be repeatedly operated on, increasing the data volume of the entire AI model and leading to increased costs (due to increased demand on memory 918) or decreased performance (due to increased bandwidth demand on external memory 902). In practice, due to the large number of operators and tensor sizes, the effectiveness of this invention is quite significant.

[0080] This invention can be extended to AI models that include more operators. See also... Figure 13 , Figure 13 This is a schematic diagram of another AI model. Figure 13 The left side shows that the AI ​​model contains N operators (operator 1, operator 2, ..., operator N). Figure 13 The right side shows an operator divided into H suboperators. The arrows indicate the dependencies between the suboperators. For example, the first suboperator of operator 2 can only be executed after the third suboperator of operator 1 has finished executing.

[0081] Figure 14 This is a schematic diagram of another AI model. AI network 1400 includes an adder 1410 and a convolution operator 1420, with tensor sizes of [1, 56, 56, 224]. After segmentation (e.g.) Figure 14 As shown in the lower half, the adder 1410 and convolution operator 1420 are each divided into 56 suboperators ("ADD1" to "ADD56" and "CONV1" to "CONV56"). The size of the sub-tensor output by each adder suboperator ("ADD1" to "ADD56") is [1, 1, 56, 224]. However, not all convolution suboperators ("CONV1" to "CONV56") have the same input sub-tensor size. More specifically, the input sub-tensor size of the first convolution suboperator ("CONV1") is [1, 2, 56, 224], while the input sub-tensor size of the second convolution suboperator ("CONV2") is [1, 3, 56, 224] (i.e., the aforementioned problem of magnified field of view).

[0082] Figure 15A and Figure 15BThis is a flowchart of an embodiment of the execution method of the artificial intelligence model of the present invention. Figure 15A and Figure 15B It includes the following steps.

[0083] Step S1505: The memory management circuit 916 reads a tensor (i.e., input data Din) and a plurality of core parameters Kp from the external memory 902, and stores the tensor and core parameters Kp into the memory 918.

[0084] Step S1510: The processing circuit 910 (more specifically, the arithmetic circuit 912) performs a first-type operation on a first sub-tensor of the tensor to generate first intermediate data, and the memory management circuit 916 stores the first intermediate data into memory 918. Figure 4 For example, the first subtensor can be the input subtensor of the subtraction suboperator 110_1 (i.e., subtensor TS1_i1), the first type of operation can be subtraction (a type of vector operation), and the first intermediate data can be the output subtensor of the subtraction suboperator 110_1 (i.e., subtensor TS1_o1). Figure 14 For example, the first subtensor can be the input subtensor of the adder (e.g., "ADD1"), the first type of operation can be addition (a type of vector operation), and the first intermediate data can be the output subtensor of the adder (e.g., "ADD1").

[0085] Step S1520: The processing circuit 910 (more specifically, the arithmetic circuit 912) performs the first type of operation on a second sub-tensor of the tensor to generate second intermediate data, and the memory management circuit 916 stores the second intermediate data into memory 918. Figure 4 For example, the second subtensor can be the input subtensor of the subtraction suboperator 110_2 (i.e., subtensor TS1_i2), the first type of operation can be subtraction, and the second intermediate data can be the output subtensor of the subtraction suboperator 110_2 (i.e., subtensor TS1_o2). Figure 14 For example, the second subtensor can be the input subtensor of the adder (e.g., "ADD2"), the first type of operation can be addition, and the second intermediate data can be the output subtensor of the adder (e.g., "ADD2").

[0086] Step S1530: The processing circuit 910 (more specifically, the arithmetic circuit 912) performs a second type of operation on the first intermediate data and the second intermediate data to generate a third intermediate data, and the memory management circuit 916 stores the third intermediate data into memory 918. Figure 4 and Figure 14For example, the second type of operation can be a convolution operation (e.g., convolution sub-operator 120_1 or "CONV1"), and the third intermediate data can be the result of that convolution operation (e.g., Figure 4 Son of tensor TS2_o1).

[0087] Step S1540: The memory management circuit 916 deletes the third intermediate data from the memory 918. For example... Figure 10 As shown, because operations after time point 3 in the operation sequence will no longer use subtensor TS2_o1 (i.e., subtensor TS2_o1 at time point 3). Figure 12A Since the lifecycle of the tensor begins to become inactive (after 3), the subtensor TS2_o1 can be removed from memory 918 to free up part of memory 918.

[0088] Step S1550: The processing circuit 910 (more specifically, the arithmetic circuit 912) performs the first type of operation on a third subtensor of the tensor to generate a fourth intermediate data, and the memory management circuit 916 stores the fourth intermediate data into memory 918. Figure 4 For example, the third subtensor can be the input subtensor of the subtraction suboperator 110_3 (i.e., subtensor TS1_i3), the first type of operation can be subtraction, and the fourth intermediate data can be the output subtensor of the subtraction suboperator 110_3 (i.e., subtensor TS1_o3). Figure 14 For example, the third subtensor can be the input subtensor of the adder (e.g., "ADD3"), the first type of operation can be addition, and the fourth intermediate data can be the output subtensor of the adder (e.g., "ADD3").

[0089] Step S1560: The processing circuit 910 (more specifically, the arithmetic circuit 912) performs the second type of operation on the first intermediate data, the second intermediate data, and the fourth intermediate data to generate a fifth intermediate data, and the memory management circuit 916 stores the fifth intermediate data into memory 918. Figure 4 and Figure 14 For example, the second type of operation can be a convolution operation (e.g., convolution sub-operator 120_2 or "CONV2"), and the fifth intermediate data can be the result of that convolution operation (e.g., Figure 4 Son of tensor TS2_o2).

[0090] Step S1570: The memory management circuit 916 deletes the first intermediate data from the memory 918. For example... Figure 10 As shown, because operations after time point 5 in the operation sequence will no longer use the subtensor TS1_o1 (i.e., the subtensor TS1_o1 is used at time point 5). Figure 12BSince the lifecycle of the tensor begins to become inactive (after 5), the subtensor TS1_o1 can be removed from memory 918 to free up part of memory 918.

[0091] Step S1580: The memory management circuit 916 deletes the fifth intermediate data from memory 918. For example... Figure 10 As shown, because operations after time point 6 in the operation sequence will no longer use the subtensor TS2_o2 (i.e., the subtensor TS2_o2 is used at time point 6). Figure 12B Since the lifecycle of the tensor TS2_o2 begins to become inactive (6), the subtensor TS2_o2 can be removed from memory 918 to free up part of memory 918.

[0092] like Figure 15A and Figure 15B As the discussion shows, the developer of chip 901 can pre-arrange the instructions executed by processing circuit 910 (more specifically, memory management circuit 916) based on the usage status of memory 918 (provided to processing circuit 910 by processor 920 in some embodiments) to make full use of memory 918.

[0093] Please see Figure 16 , Figure 16 This is a schematic diagram of an embodiment of the cache circuit 914 and the stored content of the memory 918 of the present invention. Figure 16 In the example, the core parameter Kp stored in memory 918 includes the subtraction core parameter Kp_s (a type of vector core parameter), the convolution core parameter Kp_c, and the addition core parameter Kp_a (a type of vector core parameter).

[0094] The core parameter Kp_s for subtraction includes subparameters Kp_s1, Kp_s2, and Kp_s3. Kp_s is the parameter required by the subtraction operator 110 when performing subtraction on tensor TS1, while subparameters Kp_s1, Kp_s2, and Kp_s3 can be respectively mapped to... Figure 4 The subtraction suboperators 110_1, 110_2, and 110_3 are specified. That is, when the arithmetic circuit 912 executes subtraction suboperators 110_1 (110_2 or 110_3), it performs operations on the subtensor TS1_i1 (TS1_i2 or TS1_i3) based on the sub-parameter Kp_s1 (Kp_s2 or Kp_s3). In some embodiments, the sub-parameter Kp_s1 is not equal to the sub-parameters Kp_s2 and Kp_s3, and the sub-parameter Kp_s2 is not equal to the sub-parameter Kp_s3.

[0095] The addition core parameter Kp_a includes sub-parameters Kp_a1, Kp_a2, and Kp_a3. The addition core parameter Kp_a is the parameter required by the adder operator 130 when performing addition operations on the tensor TS3, while the sub-parameters Kp_a1, Kp_a2, and Kp_a3 can be respectively mapped to... Figure 4 The addition suboperators 130_1, 130_2, and 130_3 are specified. That is, when the arithmetic circuit 912 executes the addition suboperator 130_1 (130_2 or 130_3), it performs operations on the subtensor TS3_i1 (TS3_i2 or TS3_i3) based on the sub-parameter Kp_a1 (Kp_a2 or Kp_a3). In some embodiments, the sub-parameter Kp_a1 is not equal to either the sub-parameters Kp_a2 or Kp_a3, and the sub-parameter Kp_a2 is not equal to the sub-parameter Kp_a3.

[0096] The convolution kernel parameter Kp_c includes sub-parameters Kp_c1, Kp_c2, and Kp_c3. Unlike subtraction and addition operations, although the operators and tensors are segmented, convolution sub-operators 120_1, 120_2, and 120_3 still require the complete convolution kernel parameter Kp_c when performing convolution operations.

[0097] Please see Figure 17 , Figure 17 yes Figure 15A The detailed process of step S1510 or step S1520 includes the following steps. Please refer to the following explanations as well. Figure 16 .

[0098] Step S1710: The memory management circuit 916 reads a target portion of the vector core parameters from the memory 918 and stores the target portion in the cache circuit 914. More specifically, for step S1510, the target portion can be the sub-parameter Kp_s1. For step S1520, the target portion can be the sub-parameter Kp_s2. Figure 16As shown, the cache circuit 914 stores sub-parameters Kp_s1 and / or Kp_s2 and other data (e.g., the sub-tensor to be operated on). Since the first type of operation (e.g., vector operation) in steps S1510 and S1520 only uses a portion of the subtraction core parameter Kp_s (i.e., the target portion), the cache circuit 914 does not need to store the complete subtraction core parameter Kp_s to save storage space. In some embodiments, the memory management circuit 916 stores the sub-parameters Kp_s1 (Kp_s2) in the cache circuit 914 before step S1510 (S1520) and deletes the sub-parameters Kp_s1 (Kp_s2) from the cache circuit 914 after step S1510 (S1520) is completed to save storage space.

[0099] Step S1720: The arithmetic circuit 912 performs target vector operations on the target sub-tensor based on the target portion of the vector core parameters to generate target intermediate data. More specifically, for step S1510, the target sub-tensor can be sub-tensor TS1_i1, the target vector operation can be sub-operator 110_1, and the target intermediate data can be sub-tensor TS1_o1. For step S1520, the target sub-tensor can be sub-tensor TS1_i2, the target vector operation can be sub-operator 110_2, and the target intermediate data can be sub-tensor TS1_o2. Figure 14 In this context, the target part can be the sub-parameter Kp_a1, the target sub-tensor can be the input sub-tensor of an adder (e.g., "ADD1"), the target vector operation can be the adder, and the first intermediate data can be the output sub-tensor of the adder.

[0100] As mentioned above, since the original tensor is divided into multiple sub-tensors, vector operations on the sub-tensors only require some of the core parameters.

[0101] Those with ordinary knowledge in this field can... Figure 17 Explanation and understanding Figure 15B The details of step S1550 will not be elaborated here. For example, for step S1550, the target part can be the sub-parameter Kp_s3, the target sub-tensor can be the sub-tensor TS1_i3, the target vector operation can be the sub-operator 110_3, and the target intermediate data can be the sub-tensor TS1_o3.

[0102] Please see Figure 18 , Figure 18 yes Figure 15A The detailed process of step S1530 includes the following steps. Please refer to the following instructions. Figure 19 .

[0103] Step S1810: The memory management circuit 916 reads the convolution kernel parameters Kp_c from the memory 918 and stores the convolution kernel parameters Kp_c in the cache circuit 914. For example... Figure 19 As shown, the cache circuit 914 stores the convolution kernel parameters Kp_c and other data (e.g., the sub-tensors to be operated on) before the convolution operation begins. Since the second type of operation in step S1530 (e.g., the convolution operation) requires the complete convolution kernel parameters Kp_c, the cache circuit 914 needs to store the complete convolution kernel parameters Kp_c at this time.

[0104] Step S1820: The computation circuit 912 performs the second type of operation on the first intermediate data and the second intermediate data according to the convolution kernel parameter Kp_c to generate the third intermediate data. Performing convolution operations on tensors according to the convolution kernel parameter Kp_c is well known to those skilled in the art and will not be described further. In some embodiments, the memory management circuit 916 deletes the convolution kernel parameter Kp_c from the cache circuit 914 after step S1820.

[0105] Please see Figure 20 , Figure 20 yes Figure 15B The detailed process of step S1560 includes the following steps. Please refer to the following instructions. Figure 19 .

[0106] Step S2010: Step S2010 is similar to step S1810, so it will not be described again. Please note that if the memory management circuit 916 does not delete the convolution kernel parameter Kp_c from the cache circuit 914 after step S1530, step S2010 can be skipped.

[0107] Step S2020: The computation circuit 912 performs the second type of operation on the first intermediate data, the second intermediate data, and the fourth intermediate data according to the convolution kernel parameter Kp_c to generate the fifth intermediate data. Step S2020 is similar to step S1820, so it will not be described again.

[0108] Please see Figure 21 , Figure 21 This is a functional block diagram of an embodiment of the memory management circuit 916 of the present invention. The memory management circuit 916 includes at least two channels (channel 916a and channel 916b), each of which can operate independently, and multiple channels can operate simultaneously. Based on this characteristic, the present invention further divides suboperators and subtensors into several small blocks, enabling the processing circuit 910 to perform calculations on the AI ​​model using a multistage pipeline, thereby improving the performance of the chip 901.

[0109] Please see Figure 22 , Figure 22 This is a schematic diagram of the multi-stage production line of the present invention. Figure 22 The following explanation uses subtraction operator 110_1, subtraction operator 110_2, and convolution operator 120_1 as examples. Figure 22 In the example, the subtraction operator 110_1 is further divided into operation block 110_1a ("SUB1a") and operation block 110_1b ("SUB1b"), and the subtensor TS1_i1 is further divided into data block TS1_i1a and data block TS1_i1b; the subtraction operator 110_2 is further divided into operation block 110_2a ("SUB2a") and operation block 110_2b ("SUB2b"), and the subtensor TS1_i2 is further divided into data block TS1_i2a and data block TS1_i2b; the convolution operator 120_1 is further divided into operation block 120_1a ("CONV1a") and operation block 120_1b ("CONV1b"), and the subtensor TS2_i1 is further divided into data block TS2_i1a and data block TS2_i1b. In this way, while channel 916a is performing operations related to computation block 110_1a (between time points T0 and T3), channel 916b can essentially perform operations related to computation block 110_1b simultaneously (between time points T1 and T4). Compared to a single-stage pipeline (i.e., one that does not use multiple channels simultaneously to compute the AI ​​model), using two channels simultaneously saves approximately half the time. Similarly, using N channels simultaneously requires approximately 1 / N of the processing time of a single-stage pipeline.

[0110] Please see Figure 23 , Figure 23 This is a flowchart of an embodiment of the multi-stage pipeline operation of the present invention, which includes the following steps.

[0111] Step S2310: The memory management circuit 916 uses a first channel (e.g., channel 916a) to read a first data block (e.g., data block TS1_i1a) of the first sub-tensor (e.g., sub-tensor TS1_i1) from the memory 918, and stores the first data block into the cache circuit 914. For example, step S2310 can correspond to... Figure 22 Between time point T0 and time point T1 (i.e., the "SUB1a load" operation).

[0112] Step S2320: The arithmetic circuit 912 performs a first type of operation (e.g., subtraction) on the first data block (e.g., data block TS1_i1a) to produce a first portion of the first intermediate data (e.g., a portion of the subtensor TS1_o1). For example, step S2320 can correspond to... Figure 22 Between time point T1 and time point T2 (i.e., the "SUB1a compute" operation).

[0113] Step S2330: The memory management circuit 916 uses the second channel (e.g., channel 916b) to read the second data block (e.g., data block TS1_i1b) of the first sub-tensor from the memory 918 and stores the second data block into the cache circuit 914. For example, step S2330 can correspond to... Figure 22 Between time point T1 and time point T2 (i.e., the "SUB1b loading" operation). In other words, steps S2320 and S2330 are executed at least partially simultaneously.

[0114] Step S2340: The memory management circuit 916 uses a first channel (e.g., channel 916a) to store the first portion of the first intermediate data (e.g., a portion of the sub-tensor TS1_o1) into memory 918. For example, step S2340 can correspond to... Figure 22 Between time point T2 and time point T3 (i.e., the "SUB1a store" operation).

[0115] Step S2350: The arithmetic circuit 912 performs the first type of operation (e.g., subtraction) on the second data block (e.g., data block TS1_i1b) to produce a second part of the first intermediate data (e.g., a part of the subtensor TS1_o1). For example, step S2350 can correspond to... Figure 22 Between time point T2 and time point T3 (i.e., the "SUB1b calculation" operation). In other words, steps S2340 and S2350 are executed at least partially simultaneously.

[0116] Step S2360: The memory management circuit 916 uses the second channel (e.g., channel 916b) to store the second portion of the first intermediate data (e.g., a portion of the sub-tensor TS1_o1) into memory 918. For example, step S2360 can correspond to... Figure 22 Between time point T3 and time point T4 (i.e., the "SUB1b storage" operation).

[0117] Those with general knowledge in this technical field may refer to... Figure 23 Explanation and understanding Figure 22 Other operations after time point T4 will not be elaborated upon.

[0118] While the embodiments of the present invention have been described above, these embodiments are not intended to limit the present invention. Those skilled in the art can make changes to the technical features of the present invention based on the explicit or implicit content of the present invention. All such changes may fall within the scope of patent protection sought by the present invention. In other words, the scope of patent protection of the present invention shall be determined by the scope of the patent application in this specification.

[0119] [Symbol Explanation]

[0120] 100,1400: AI Network

[0121] Din: Input data

[0122] Dout: Output data

[0123] 110,SUB: Subtraction operator

[0124] 120, 1420, CONV: Convolution operator

[0125] 130,1410,ADD: Addition operator

[0126] TS1, TS2, TS3, TS4: Tensors

[0127] 110_1,110_2,110_3,SUB1,SUB2,SUB3: Subtraction operators

[0128] 120_1,120_2,120_3,CONV1,CONV2,CONV3,CONV56: Convolution operators

[0129] 130_1,130_2,130_3,ADD1,ADD2,ADD3,ADD56: Addition operators

[0130] TS1_i1,TS1_i2,TS1_i3,TS3_o1,TS3_o2,TS3_o3,TS3_i1,TS3_i2,TS3_i3,TS2_o1,TS2_o2,TS2_o3,TS1_o1,TS1_o2,TS1_o3,TS2_i1,TS2_i2,TS2_i3: sub-tensor

[0131] 800: Queue

[0132] 900: Electronic Devices

[0133] 901: Chip

[0134] 902: External Memory

[0135] 910: Processing Circuit

[0136] 920: Processor

[0137] 912: Operational Circuit

[0138] 914: Buffer Circuit

[0139] 916: Memory Management Circuit

[0140] 918: Memory

[0141] Kp: Core parameter

[0142] Kp_s: Core parameter for subtraction

[0143] Kp_c: Convolution kernel parameter

[0144] Kp_a: Core parameter for addition

[0145] Kp_s1, Kp_s2, Kp_s3, Kp_a1, Kp_a2, Kp_a3, Kp_c1, Kp_c2, Kp_c3: Sub-parameters

[0146] 916a, 916b: Channels

[0147] 110_1a,110_1b,110_2a,110_2b,120_1a,120_1b,SUB1a,SUB1b,SUB2a,SUB2b,CONV1a,CONV1b: Operation blocks

[0148] TS1_i1a, TS1_i1b, TS1_i2a, TS1_i2b, TS2_i1a, TS2_i1b: Data blocks

[0149] T0, T1, T2, T3, T4: Time points

[0150] S210,S220,S230,S240,S250,S510,S520,S530,S710,S720,S730,S740,S750,S 760,S770,S780,S790,S795,S1110,S1120,S1130,S1140,S1150,S1160,S1170,S 1180, S1505, S1510, S1520, S1530, S1540, S1550, S1560, S1570, S1580, S1710, S1720, S1810, S1820, S2010, S2020, S2310, S2320, S2330, S2340, S2350, S2360: Steps

Claims

1. A processing circuit for an artificial intelligence model, characterized in that, The processing circuitry is coupled to an external memory and includes: One memory; A memory management circuit reads a tensor from the external memory and stores the tensor into the external memory; and An operational circuit is configured to perform the following operations: Perform a first-type operation on a first subtensor of the tensor to generate a first intermediate data; Perform the first type of operation on a second subtensor of the tensor to generate a second intermediate data; A second type of operation is performed on the first intermediate data and the second intermediate data to generate a third intermediate data; Perform the first type of operation on a third subtensor of the tensor to produce a fourth intermediate data; and Perform the second type of operation on the first intermediate data, the second intermediate data, and the fourth intermediate data to generate a fifth intermediate data; The memory management circuit stores the first intermediate data and the second intermediate data into the memory. The arithmetic circuit determines an arithmetic order based on the input-output dependencies between the suboperators corresponding to the first intermediate data, the second intermediate data, the third intermediate data, the fourth intermediate data, and the fifth intermediate data. Based on the arithmetic order, the first intermediate data is deleted from the memory after it no longer participates in subsequent arithmetic operations.

2. The processing circuit as described in claim 1, characterized in that, The first type of operation is an addition operation or a subtraction operation, and the second type of operation is a convolution operation.

3. The processing circuit as described in claim 1, characterized in that, The first intermediate data, the second intermediate data, and the fourth intermediate data correspond to the same dimension of the tensor.

4. The processing circuit as described in claim 3, characterized in that, The fourth intermediate data is generated after the third intermediate data is generated.

5. The processing circuit as described in claim 1, characterized in that, The processing circuit further includes: A buffer circuit; When the arithmetic circuit performs the first type of operation, the memory management circuit reads at least a portion of a core parameter from the memory to the cache circuit, and the arithmetic circuit performs the first type of operation only based on the at least a portion of the core parameter. The first type of operation is either a subtraction operation or an addition operation.

6. The processing circuit as described in claim 1, characterized in that, The first intermediate data, the second intermediate data, and the fourth intermediate data are all the same size.

7. The processing circuit as described in claim 1, characterized in that, The memory management circuit includes a first channel and a second channel, and the operation of the first type on the first sub-tensor of the tensor includes the following steps: (A) Using the first channel, read a first data block of the first sub-tensor from the memory; (B) Perform the first type of operation on the first data block to produce a first portion of the first intermediate data; (C) Using the second channel, read a second data block of the first sub-tensor from the memory; (D) Use the first channel to store the first portion of the first intermediate data into the memory; (E) Perform the first type of operation on the second data block to produce a second portion of the first intermediate data; as well as (F) Use the second channel to store the second portion of the first intermediate data into the memory; Wherein, steps (B) and (C) are performed at least partially simultaneously, and steps (D) and (E) are performed at least partially simultaneously.

8. A processing circuit for an artificial intelligence model, characterized in that, The processing circuit is coupled to and includes an external memory, and the processing circuit performs the following operations: Read a tensor and a plurality of core parameters from the external memory, and store the tensor and each of the core parameters into the memory, wherein the tensor includes a first sub-tensor and a second sub-tensor, and each of the core parameters includes a vector core parameter; A first vector operation is performed on the first subtensor based on a first portion of the vector core parameters to generate first intermediate data, and the first intermediate data is stored in the memory; and A second vector operation is performed on the second sub-tensor based on a second part of the vector core parameters to generate a second intermediate data; An operation order is determined based on the input-output dependency relationship between the suboperators corresponding to the first intermediate data and the second intermediate data, and the memory management circuit is instructed to delete the first intermediate data from the memory after the first intermediate data no longer participates in the subsequent operation according to the operation order. Wherein, the first part of the vector core parameter is not equal to the second part of the vector core parameter.

9. The processing circuit as described in claim 8, characterized in that, The tensor further includes a third subtensor, and each of the core parameters further includes a convolution core parameter for a convolution operation. The processing circuit further performs the following operations: The convolution operation is performed on the first intermediate data and the second intermediate data according to the convolution kernel parameters to generate a third intermediate data; and After the convolution operation, a third vector operation is performed on the third sub-tensor based on a third part of the vector core parameters to generate a fourth intermediate data.

10. The processing circuit as described in claim 9, characterized in that, The convolution operation is a first convolution operation, and the processing circuit further performs the following operations: A second convolution operation is performed on the first intermediate data, the second intermediate data, and the fourth intermediate data based on the convolution kernel parameters.

11. The processing circuit as described in claim 8, characterized in that, The first subtensor and the second subtensor correspond to the same dimension of the tensor.

12. The processing circuit as described in claim 8, characterized in that, The first vector operation and the second vector operation are one of an addition operation and a subtraction operation.

13. The processing circuit as described in claim 8, characterized in that, The processing circuit further includes a memory management circuit, which includes a first channel and a second channel. The step of performing the first vector operation on the first sub-tensor to generate the first intermediate data includes the following steps: (A) Using the first channel, read a first data block of one of the first sub-tensors from the memory; (B) Perform an operation on the first data block to generate a first portion of the first intermediate data; (C) Using the second channel, read a second data block of the first sub-tensor from the memory; (D) Use the first channel to store the first portion of the first intermediate data into the memory; (E) Perform the operation on the second data block to produce a second portion of the first intermediate data; as well as (F) Use the second channel to store the second portion of the first intermediate data into the memory; Wherein, steps (B) and (C) are performed at least partially simultaneously, and steps (D) and (E) are performed at least partially simultaneously.

14. A computation scheduling method for an artificial intelligence model, characterized in that, The artificial intelligence model includes a first operator and a second operator, wherein the output of the first operator is the input of the second operator, and the operation scheduling method includes: Divide a single tensor into H subtensors, where H is an integer greater than 1; Divide the first operator into H first suboperators; Divide the second operator into H second suboperators; An input-output dependency relationship between the H first suboperators and the H second suboperators is determined based on the input of each of the H second suboperators; The H first suboperators and the H second suboperators are ordered according to the dependencies to obtain an operation order; and According to the operation sequence, it is determined when a processing circuit executing the artificial intelligence model deletes a target data from a memory contained in the processing circuit, the target data being the output data of one of the H first sub-operators and the H second sub-operators.

15. The scheduling method as described in claim 14, characterized in that, The step of determining the dependency relationship between the H first suboperators and the H second suboperators includes: Determine an object suboperator; A source suboperator is determined based on the input of the target suboperator, wherein the output of the source suboperator is the input of the target suboperator; and The target suboperator is determined to depend on the source suboperator.

16. The scheduling method as described in claim 14, characterized in that, The step of sorting the H first sub-operators and the H second sub-operators according to the dependency relationship to obtain the operation order includes: (A) Determine a target suboperator; (B) Determine a source suboperator on which the target suboperator depends; (C) When the source suboperator does not depend on any suboperator, add the source suboperator to a queue; (D) Repeat steps (B) to (C) until all source suboperators on which the target suboperator depends have been added to the queue; and (E) Add the target suboperator to the queue.

17. The scheduling method as described in claim 16, characterized in that, The step of sorting the H first sub-operators and the H second sub-operators according to the dependency relationship to obtain the operation order further includes: (F) Determine a higher-level suboperator that depends on the target suboperator; (G) Using the upper-level suboperator as the target suboperator, repeat steps (B) to (E) and; (H) Repeat steps (F) and (G) until the target suboperator is a top-level suboperator.

18. The scheduling method as described in claim 16, characterized in that, The step of determining when to delete the target data from the memory according to the operation sequence includes: The last suboperator to use the target data is determined based on the queue, and the suboperator is one of the H first suboperators and the H second suboperators; The target data is deleted after the suboperation ends.

Images