AI operator fusion method, data processing system and related equipment

Abstract

The invention discloses an AI operator fusion method, a data processing system and related equipment, and relates to the technical field of artificial intelligence. The processor obtains input data of an operator set and available capacity of a cache corresponding to at least one acceleration unit, wherein the operator set comprises a plurality of continuous AI operators in an AI model; the processor generates a plurality of AI calculation tasks according to the available capacity of the cache, the input data and the operator set and distributes the AI calculation tasks to at least one acceleration unit, and each AI calculation task is generated based on part of data in the input data; the at least one acceleration unit executes a plurality of AI calculation tasks according to the kernel of each AI operator, and data generated in the process that each acceleration unit executes the respective AI calculation task is stored in the cache corresponding to each acceleration unit. Therefore, the capacity of the cache occupied in the process of executing the multiple AI computing tasks can be smaller than the available capacity of the cache, so that codes do not need to be regenerated and compiled for the multiple AI operators, and the overhead for fusing the multiple AI operators is reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an AI operator fusion method, a data processing system and related equipment. Background Technology

[0002] Artificial intelligence (AI) models are mathematical models based on artificial intelligence technology that can automatically process and analyze input data and output corresponding results. For example, an AI model can be a language model (LM), which can output text in language B based on input text in language A, achieving translation between texts in different languages. An AI model can include multiple operators, each of which is a basic unit that performs a specific mathematical or logical operation, such as a convolution operator used to perform convolution operations.

[0003] Typically, during AI model inference, multiple consecutive AI operators within the model can be fused. For AI operator 1 and AI operator 2 involved in the fusion, AI operator 1 stores its calculation result in a cache so that AI operator 2 can directly retrieve the result from the cache and use it as input data to continue calculation. Thus, throughout the entire inference process, AI operator 1 does not need to store the cached calculation result in memory, and AI operator 2 does not need to read the calculation result from memory. This reduces the transmission overhead between the cache and memory, improves the execution efficiency of these multiple operators, and ultimately enhances the overall inference efficiency of the AI ​​model.

[0004] However, in the era of large-scale models, the dimensional information of the input data for AI models, such as the shape of the input data, often changes dynamically. This dimensional information can include the number of dimensions and the number of elements in each dimension. For example, when the input data for an AI model is an image, the dimensional information could be the number of rows (i.e., the number of dimensions) and columns (i.e., the number of elements) of pixels in the image. Correspondingly, when images of different sizes are input into the AI ​​model, the dimensional information of the input data changes.

[0005] Currently, when the dimensionality of the input data to an AI model changes dynamically, code is regenerated and compiled for multiple consecutive AI operators involved in the fusion process during each inference step. This aims to improve the overall inference efficiency of the AI ​​model by accelerating the inference efficiency of these multiple AI operators. However, this method of fusing multiple AI operators requires regenerating and compiling code, which incurs significant optimization overhead. Summary of the Invention

[0006] This application provides an AI operator fusion method aimed at reducing the overhead of fusing multiple AI operators. Furthermore, this application also provides a corresponding data processing system, processor, computer-readable storage medium, and computer program product.

[0007] Firstly, this application provides an AI operator fusion method, which can be applied to a data processing system. This data processing system includes a processor and at least one acceleration unit. For example, the processor may be a CPU, and each acceleration unit may be a computing unit located in a GPU for performing vector computation operations, matrix computation operations, or scalar computation operations. During the fusion of AI operators, the processor acquires input data for an operator set, which includes multiple consecutive AI operators in an AI model. The processor also acquires the available capacity of the cache corresponding to the at least one acceleration unit. The cache corresponding to each acceleration unit is used to cache the data generated during the execution of AI computation tasks by that acceleration unit. Then, the processor generates multiple AI computation tasks based on the available capacity of the cache corresponding to the at least one acceleration unit, the input data of the operator set, and the operator set. Each AI computation task is generated based on a portion of the input data, and the multiple AI computation tasks are allocated to the at least one acceleration unit. Correspondingly, the at least one acceleration unit executes the multiple AI computation tasks according to the kernel of each AI operator in the operator set. Data generated during the execution of each AI computation task by each acceleration unit is stored in the cache corresponding to that acceleration unit.

[0008] Since the processor generates AI computing tasks based on the available capacity of the cache and a portion of the input data, the cache capacity occupied by the scheduling acceleration unit during the execution of the AI ​​computing task based on the kernel of multiple AI operators can be less than the available capacity of the cache. Thus, when the available capacity of the cache is sufficient, there is no need to regenerate and compile the code for the multiple AI operators involved in the fusion. This can effectively reduce the overhead of fusing multiple AI operators and achieve lightweight optimization of the multiple AI operators.

[0009] In one possible implementation, when the processor generates multiple AI computing tasks based on the available capacity of the cache corresponding to at least one acceleration unit, the input data of the operator set, and the operator set, it may first divide the input data into multiple data slices based on the available capacity of the cache, and then generate the multiple AI computing tasks based on the multiple data slices and the operator set. Each AI computing task is generated based on at least one data slice from the multiple data slices, and each AI computing task is executed based on the kernel of one AI operator from the operator set. In this way, the processor generates one AI computing task based on one data slice of the input data, ensuring that the cache usage of multiple acceleration units during the execution of AI computing tasks does not exceed the cache's upper limit. Therefore, when the available cache capacity is sufficient, it is not necessary to regenerate and compile code for multiple AI operators involved in the fusion, reducing the overhead of fusing multiple AI operators.

[0010] In one possible implementation, when the processor partitions the input data based on the available capacity of the cache, it may first calculate the dimensionality of the input data and the output data of each AI operator in the operator set based on the dimensionality information of the input data of the operator set. For example, the dimensionality information of the input data may be, for instance, the shape of the input data. Then, based on the dimensionality information of the input data and the output data of each AI operator, the processor estimates the cache capacity required for multiple AI operators in the operator set to run based on the input data of the operator set. If the estimated cache capacity is greater than the available cache capacity, the processor partitions the input data of the operator set according to the available cache capacity. In this way, by estimating the cache capacity required to compute the input data based on multiple AI operators and partitioning the input data accordingly, the processor can avoid excessive cache usage when at least one acceleration unit executes multiple AI computing tasks based on the kernel of these multiple AI operators, thus ensuring the normal execution of multiple AI computing tasks.

[0011] In one possible implementation, when the processor partitions the input data according to the available capacity of the cache, it may specifically partition the input data of the operator set based on at least one dimension, including an element-wise dimension, a reduction dimension, or a sliding window dimension. Optionally, the processor may also partition the input data based on other dimensions.

[0012] In one possible implementation, the input data is divided based on the reduction dimension or the sliding window dimension. Then, when the processor generates multiple AI computing tasks based on the available capacity of the cache corresponding to at least one acceleration unit, the input data of the operator set, and the operator set, specifically, it can generate the multiple AI computing tasks based on the available capacity of the cache, the input data, the operator set, and a merging operator. The merging operator is an AI operator used to merge the calculation results corresponding to multiple parts of the input data. In this way, the processor can utilize an additional merging operator to merge the calculation results corresponding to different parts of the input data to ensure the accuracy of data calculation.

[0013] In one possible implementation, before the processor acquires the input data for the operator set, the processor may also acquire a configuration file, which instructs the fusion of consecutive AI operators in the AI ​​model; or, before acquiring the input data for the operator set, the processor uses an operator fusion strategy to traverse the AI ​​operators in the AI ​​model to determine the operator set. In this way, the processor can actively or passively determine the multiple AI operators in the AI ​​model that participate in the fusion, thereby improving the overall inference efficiency of the AI ​​model by fusing these multiple AI operators.

[0014] In one possible implementation, the kernels of the various AI operators run by the at least one acceleration unit are derived from the kernel library corresponding to the AI ​​model, and the kernel library includes the kernels of the AI ​​operators in the AI ​​model. Thus, during the fusion of multiple AI operators, the acceleration unit can reuse the kernels of AI operators from the already developed kernel library without generating new kernels for the fusion operators. This improves the efficiency of executing the multiple AI operators involved in the fusion, i.e., improves the inference efficiency of the AI ​​model during runtime.

[0015] In one possible implementation, the processor may prefetch a portion of the input data from the operator set into the cache before the at least one acceleration unit executes the plurality of AI computing tasks; and during the execution of the plurality of AI computing tasks by the at least one acceleration unit, the processor may invalidate data consumed by the AI ​​operators in the operator set in the cache and write the data in the cache back to memory in advance. Thus, by prefetching data from the cache, the efficiency of executing multiple AI computing tasks can be improved, and by invalidating data from the cache and writing back content in advance, the utilization rate of the cache can be improved.

[0016] In one possible implementation, when the processor allocates the plurality of AI computing tasks to the at least one acceleration unit, it may specifically determine the scheduling template corresponding to the operator set from multiple scheduling templates based on the number of parts into which the input data of the operator set is divided, and allocate the plurality of AI computing tasks to the at least one acceleration unit according to the scheduling template corresponding to the plurality of operator sets. The scheduling template is used to indicate the execution order of the plurality of AI computing tasks. In this way, the processor can use a pre-created scheduling template to schedule the execution order of multiple AI computing tasks by at least one acceleration unit, thereby reducing the operating overhead of scheduling the acceleration unit and improving scheduling efficiency.

[0017] In one possible implementation, the processor can further create initial AI computation tasks corresponding to each AI operator in the operator set before partitioning the input data of the operator set. These initial AI computation tasks include parameters unrelated to the dimensionality information of the AI ​​operators. Alternatively, when generating multiple AI computation tasks based on the available cache capacity, the input data, and the operator set, the processor can specifically add parameters related to the dimensionality information of a first AI operator to the initial AI computation task corresponding to that first AI operator, thus obtaining the AI ​​computation task corresponding to the first AI operator. The first AI operator is one of the AI ​​operators in the operator set, and the parameters related to the dimensionality information of the first AI operator include the number of partitions in the input data of the operator set. In this way, by adding parameters related to the dimensionality information of the AI ​​operator to the already created initial AI computation tasks, the processor can generate the AI ​​computation task corresponding to that AI operator, thereby effectively improving the efficiency of the processor in generating multiple AI computation tasks and thus improving the overall operating efficiency of the AI ​​model.

[0018] Secondly, this application provides an AI operator fusion method applied to a processor belonging to a data processing system, the data processing system further including at least one acceleration unit. The method includes: the processor acquiring input data of an operator set, the operator set including multiple consecutive AI operators participating in fusion in an AI model, and acquiring the available capacity of the cache corresponding to the at least one acceleration unit; then, the processor generating multiple AI computing tasks based on the available capacity of the cache, the input data, and the operator set, and allocating the multiple AI computing tasks to the at least one acceleration unit, each of the multiple AI computing tasks being generated based on a portion of the input data, the multiple AI computing tasks being executed by the at least one acceleration unit according to the kernel of each AI operator in the operator set, wherein the data generated by each acceleration unit during the execution of its respective AI computing task is stored in the cache corresponding to each acceleration unit.

[0019] In this way, the cache capacity occupied by the AI ​​computing tasks generated by the processor during execution can be less than the available cache capacity. Thus, when the available cache capacity is sufficient, there is no need to regenerate and compile code for multiple AI operators involved in the fusion. This can effectively reduce the overhead of fusing multiple AI operators and achieve lightweight optimization of these multiple AI operators.

[0020] Thirdly, this application provides a data processing system, which includes a processor and at least one acceleration unit. The processor is configured to acquire input data of an operator set, the operator set including multiple consecutive AI operators participating in fusion in an AI model; acquire the available capacity of the cache corresponding to the at least one acceleration unit; generate multiple AI computing tasks based on the available capacity of the cache, the input data, and the operator set, each of the multiple AI computing tasks being generated based on a portion of the input data; and allocate the multiple AI computing tasks to the at least one acceleration unit. The at least one acceleration unit is configured to execute the multiple AI computing tasks according to the kernel of each AI operator in the operator set, wherein the data generated by each acceleration unit during the execution of its respective AI computing task is stored in the cache corresponding to each acceleration unit.

[0021] In one possible implementation, the processor is configured to: divide the input data according to the available capacity of the cache to obtain multiple data shards; generate multiple AI computing tasks according to the multiple data shards and the set of operators, wherein each AI computing task is generated based on at least one data shard in the multiple data shards, and each AI computing task is executed based on the kernel of an AI operator in the set of operators.

[0022] In one possible implementation, the processor is configured to: calculate the dimension information of the input data and the dimension information of the output data of each AI operator in the operator set based on the dimension information of the input data of the operator set; estimate the cache capacity required for multiple AI operators in the operator set to run based on the input data of the operator set based on the dimension information of the input data and the dimension information of the output data of each AI operator; and when the estimated cache capacity is greater than the available cache capacity, divide the input data of the operator set according to the available cache capacity.

[0023] In one possible implementation, the processor is configured to partition the input data based on at least one dimension, including an element-wise dimension, a reduction dimension, or a sliding window dimension, according to the available capacity of the cache.

[0024] In one possible implementation, the input data is divided based on the reduction dimension or the sliding window dimension; the processor is configured to generate the plurality of AI computing tasks based on the available capacity of the cache, the input data, the operator set, and the merging operator, wherein the merging operator is an AI operator used to merge the computing results corresponding to multiple parts of the input data.

[0025] In one possible implementation, the processor is further configured to: obtain a configuration file before obtaining input data for the operator set, the configuration file being used to instruct the fusion of consecutive AI operators in the AI ​​model; or, before obtaining input data for the operator set, traverse the AI ​​operators in the AI ​​model using an operator fusion strategy to determine the operator set.

[0026] In one possible implementation, the kernel of each AI operator run by the at least one acceleration unit is derived from the kernel library corresponding to the AI ​​model, and the kernel library includes the kernels of the AI ​​operators in the AI ​​model.

[0027] In one possible implementation, the processor is further configured to: prefetch a portion of the input data of the operator set in the cache before the at least one acceleration unit executes the plurality of AI computing tasks; invalidate data consumed by AI operators in the operator set in the cache and write the data in the cache back to memory in advance during the execution of the plurality of AI computing tasks by the at least one acceleration unit;

[0028] In one possible implementation, the processor is configured to: determine a scheduling template corresponding to the operator set from multiple scheduling templates based on the number of parts into which the input data of the operator set is divided; and allocate the multiple AI computing tasks to the at least one acceleration unit according to the scheduling templates corresponding to the multiple operator sets, wherein the scheduling templates are used to indicate the order in which the multiple AI computing tasks are executed.

[0029] In one possible implementation, the processor is further configured to: before partitioning the input data of the operator set, create an initial AI computation task corresponding to each AI operator in the operator set, the initial AI computation task including parameters unrelated to the dimensional information of the AI ​​operator; specifically, the processor is configured to add parameters related to the dimensional information of the first AI operator to the initial AI computation task corresponding to the first AI operator to obtain the AI ​​computation task corresponding to the first AI operator, the first AI operator being an AI operator in the operator set, the parameters related to the dimensional information of the first AI operator including the number of partitions of the input data of the operator set.

[0030] The data processing system provided in the third aspect corresponds to the AI ​​operator fusion method provided in the first aspect. Therefore, the technical effects of the third aspect and any implementation thereof can be found in the relevant descriptions of the technical effects of the first aspect and the corresponding implementation thereof, and will not be repeated here.

[0031] Fourthly, this application provides a processor for executing the operational steps of the AI ​​operator fusion method described in the second aspect above.

[0032] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the operation steps of the AI ​​operator fusion method described in the first aspect or any implementation thereof, or to perform the operation steps of the AI ​​operator fusion method described in the second aspect.

[0033] In a sixth aspect, this application provides a computer program product containing instructions that, when run on a computing device, causes the computing device to perform the operation steps of the AI ​​operator fusion method described in the first aspect or any implementation thereof, or to perform the operation steps of the AI ​​operator fusion method described in the second aspect.

[0034] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0035] Figure 1 A schematic diagram of the structure of an exemplary data processing system provided in this application;

[0036] Figure 2 A flowchart illustrating an AI operator fusion method provided in this application;

[0037] Figure 3a A schematic diagram of the input data for multiple AI operators involved in fusion;

[0038] Figure 3b A schematic diagram of the input data for another type of AI operator involved in fusion;

[0039] Figure 4a This is a schematic diagram of an exemplary fusion graph including multiple AI operators;

[0040] Figure 4b This is another example of a fusion graph including multiple AI operators;

[0041] Figure 5 This diagram illustrates the use of different acceleration units to perform different AI computing tasks and the management of cache 210.

[0042] Figure 6 A schematic diagram for adding management operations for cache 210 to a fusion graph composed of multiple AI operators. Detailed Implementation

[0043] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, various non-limiting embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained based on the embodiments in this application and based on the above content are within the scope of protection of this application.

[0044] See Figure 1 This illustrates a data processing system. For example... Figure 1 As shown, the data processing system 10 includes a processor 100 and multiple accelerators. Figure 1 The following example illustrates the use of accelerators 20 and 30. The processor 100 can be connected to multiple accelerators via a bus, which may be, for example, a peripheral component interconnect express (PCIE) bus, or a unified bus (UB or Ubus).

[0045] For example, the processor 100 may be a central processing unit (CPU), or a processor implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a digital processing unit (DPU), an non-processor unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0046] Each accelerator can be a processor used to provide accelerated data processing services to processor 100, such as a graphics processing unit (GPU), neural network processing unit (NPU), tensor processing unit (TPU), or other types of accelerators. Furthermore, each accelerator can include at least one acceleration unit; for example, accelerator 20 can include acceleration units 201, 202, and 203. Additionally, each accelerator can also include a cache; for example, accelerator 20 can also include cache 210.

[0047] Each acceleration unit can be a scalar computation unit, a vector computation unit, or a matrix computation unit. A scalar computation unit refers to a computation unit primarily used for performing scalar computation operations; a vector computation unit refers to a computation unit primarily used for performing vector computation operations; and a matrix computation unit refers to a computation unit primarily used for performing matrix computation operations. Each acceleration unit can also be a hardware unit on the accelerator with data computation capabilities.

[0048] The cache 210 can be, for example, an L1 cache (first-level cache), an L2 cache (second-level cache), or an L3 cache (third-level cache), or it can be implemented in other ways.

[0049] Furthermore, the data processing system 10 may also include memory 300, and the processor 100, multiple accelerators, and memory 300 may be connected via a bus, such as... Figure 1 As shown.

[0050] The memory 300 can be random access memory (RAM) or read-only memory (ROM). RAM can be, for example, dynamic random access memory (DRAM), static random access memory (SRAM), or storage class memory (SCM). ROM can be, for example, programmable read-only memory (PROM) or erasable programmable read-only memory (EPROM). Alternatively, the memory 300 can also be a dual in-line memory module (DIMM), i.e., a module composed of dynamic random access memory (DRAM); or, the memory 300 can be a solid-state drive (SSD).

[0051] In practical applications, the data processing system 10 can run an AI model and utilize multiple acceleration units to execute multiple AI operators within the AI ​​model. During the execution of an AI operator, each acceleration unit first reads data from memory 300 into cache 210, performs calculations based on the data in cache 210, and obtains the calculation result. Then, the acceleration unit saves the calculation result to cache 210 and writes the calculation result from cache 210 back to memory 300. When other acceleration units execute other AI operators based on the calculation result, they read the calculation result from memory 300 into cache 210 and perform corresponding calculations based on the calculation result in cache 210. This process continues until the data processing system 10 has completed executing all AI operators in the AI ​​model. During the execution of multiple AI operators in the AI ​​model, each acceleration unit needs to frequently transfer data between cache 210 and memory 300, which leads to high data transfer overhead and reduces the performance of the data processing system 10.

[0052] Therefore, the data processing system 10 can typically fuse consecutive AI operators in the AI ​​model. This allows one or more acceleration units to directly continue the calculation based on the calculation results in the cache 210 during the execution of the fused AI operators (specifically, to continue the calculation corresponding to the current AI operator based on the calculation result of the previous AI operator). There is no need to write the calculation results from the cache 210 to the memory 300 and then read them from the memory 300 to the cache 210, thereby reducing the data transmission overhead generated during the execution of multiple AI operators involved in the fusion.

[0053] However, in practical applications, the capacity of cache 210 is usually small, and the dimensionality of the input data of AI models often changes dynamically. This makes it difficult for cache 210 to store all the data (such as calculation results) generated by the AI ​​operators involved in the fusion when the number of dimensions of the input data of the AI ​​model is large. In this case, if the code is regenerated and compiled for the multiple AI operators involved in the fusion (so that the regenerated code does not occupy the cache 210 capacity limit during operation), the process of generating and compiling the code will not only consume a lot of computing power of the data processing system 10, but also occupy a lot of memory storage space (to store the compiled code), thus generating a large resource overhead.

[0054] Based on this, in the data processing system 10 provided in this application, the data processing system 10 can execute multiple AI operators participating in the fusion based on each data part of the input data, thereby reducing the capacity requirements of the multiple AI operators on the cache 210.

[0055] Specifically, the processor 100 can first acquire input data from a set of operators, which includes multiple consecutive AI operators participating in the fusion within the AI ​​model. For two consecutive AI operators in the AI ​​model, the calculation result output by the previous AI operator serves as the input data for the next AI operator. Correspondingly, multiple consecutive AI operators refer to data being passed and calculated sequentially within these multiple AI operators. Furthermore, the processor 100 also acquires the available capacity of a cache 210, which is used to store relevant data generated during the execution of the AI ​​calculation tasks corresponding to the multiple AI operators. The available capacity refers to the size of the free storage area in the cache 210 that can be used to cache data.

[0056] Then, the processor 100 generates multiple AI computing tasks based on the available capacity of the cache 210, the input data of the operator set, and the operator set (i.e., the multiple AI operators participating in the fusion). Each AI computing task is generated based on a portion of the input data. For example, the processor 100 can first divide the input data of the operator set to obtain multiple data slices of the input data. For example, when the input data specifically includes tensor data of 16×H (H is a positive integer), the processor 100 can divide the input data into 8 data slices, each of which is tensor data of 2×H. Then, the processor 100 can generate multiple AI computing tasks based on these multiple data slices and the operator set. At this time, each AI computing task corresponds to one AI operator in the operator set, and each AI computing task is generated based on one data slice.

[0057] Therefore, the processor 100 can allocate multiple AI computing tasks to acceleration units 201 to 203. Accordingly, acceleration units 201 to 203 can execute the multiple AI computing tasks according to the kernel of each AI operator among the multiple AI operators; specifically, the corresponding acceleration unit can execute the AI ​​computing tasks according to the program code of the kernel. Data generated during the execution of each acceleration unit's respective AI computing task is stored in the cache corresponding to that acceleration unit. Figure 1 Acceleration units 201 to 203 share the same cache 210. Therefore, the data generated by acceleration units 201 to 203 during the execution of their respective AI computing tasks are all stored in the cache 210.

[0058] Since the processor 100 generates AI computing tasks based on the available capacity of the cache 210 and some data in the input data, the capacity of the cache 210 occupied by the scheduling acceleration units 201 to 203 during the execution of the generated AI computing tasks according to the kernel of the multiple AI operators can be less than the available capacity of the cache 210. Thus, when the available capacity of the cache 210 is sufficient, there is no need to regenerate and compile the code for the multiple AI operators involved in the fusion. This can effectively reduce the overhead of fusing multiple AI operators and achieve lightweight optimization of the multiple AI operators.

[0059] Furthermore, when a kernel library is configured, during the fusion of multiple AI operators, the data processing system 10 can reuse the kernels of the multiple AI operators in the developed kernel library without generating a new kernel for the fusion operator. This can improve the efficiency of executing the multiple AI operators, that is, improve the inference efficiency of the AI ​​model during runtime.

[0060] It is worth noting that the above description uses multiple acceleration units in accelerator 20 and cache 210 to execute multiple AI computing tasks as an example. In actual applications, processor 100 can generate multiple AI computing tasks based on the available capacity of the cache in multiple accelerators, and use the acceleration units in multiple accelerators to execute the multiple AI computing tasks. The specific implementation method can be referred to the relevant part of the above implementation method description, and will not be repeated here.

[0061] The above Figure 1 The data processing system 10 shown is merely an illustrative example and is not intended to be limiting. For instance, other possible data processing systems 10 may also include a client, which could be an application running on a user's device or a web browser. This allows the data processing system 10 to interact with the user, such as obtaining configuration information provided by the user or presenting the inference results output by the AI ​​model to the user. For example, Figure 1 The data processing system 10 shown uses an accelerator with three acceleration units as an example. In other data processing systems, the number of acceleration units in each accelerator can be different, and the number of acceleration units in different accelerators can be the same or different. Similarly, a data processing system can include any number of accelerators, such as three or more (including three) accelerators. For example... Figure 1 The data processing system 10 shown is an example of encapsulating multiple acceleration units onto a single accelerator. In other data processing systems, the system may directly include a processor and multiple acceleration units (i.e., without encapsulating multiple acceleration units). Furthermore, multiple acceleration units may share the same cache or use different caches. Additionally, in other possible data processing systems, hardware such as registers or disks (e.g., solid-state hardware) may be further included; there are no limitations on this.

[0062] For ease of understanding, the following description, in conjunction with the accompanying drawings, illustrates an embodiment of the problem operator for determining the loss value in an AI model provided in this application.

[0063] See Figure 2 , Figure 2 This is a flowchart illustrating an exemplary AI operator fusion method provided in an embodiment of this application. Figure 2 The AI ​​operator fusion method shown can be applied to Figure 1 The data processing system 10 shown can be applied to other possible data processing systems. For ease of understanding and description, it will be used in the following example. Figure 1 The data processing system 10 shown is illustrated, and AI computing tasks are performed using the acceleration unit and cache in the accelerator 20. Figure 2As shown, the AI ​​operator fusion method may specifically include the following steps.

[0064] S201: Processor 100 determines a set of operators, which includes multiple consecutive AI operators involved in the fusion of the AI ​​model.

[0065] For example, the AI ​​model may be a language model (LM), a generative pre-trained transformer (GPT), a bidirectional encoder representations from transformer (BERT) model, a vision transformer (ViT) model, a contrastive language-image pre-training (CLIP) model, or other types of AI models, without limitation.

[0066] In this embodiment, the AI ​​model can be deployed and run in the data processing system 10. The data processing system 10 can utilize multiple acceleration units to execute all the AI ​​operators of the AI ​​model, so as to provide corresponding inference services, such as text translation services and face recognition services, using the AI ​​model. During the execution of the AI ​​model, the processor 100 can fuse multiple consecutive AI operators (typically a subset of AI operators) in the AI ​​model to improve the efficiency of subsequent acceleration units in executing the fused AI operators.

[0067] As examples, processor 100 can determine the multiple AI operators involved in the fusion in several ways.

[0068] Example 1: Processor 100 can traverse the AI ​​operators in the AI ​​model using an operator fusion strategy to determine the multiple AI operators participating in the fusion. For example, processor 100 can be configured with at least one operator fusion template, which may be pre-configured in processor 100 by an engineer. Each operator fusion template includes multiple consecutive AI operators. Taking processor 100 configured with operator fusion template P and operator fusion template Q as an example, operator fusion template P is "AI operator 1 → AI operator 2 → AI operator 3", that is, the calculation result output by AI operator 1 is the input data of AI operator 2, and the calculation result output by AI operator 2 is the input data of AI operator 3. Operator fusion template Q can be "AI operator 4 → AI operator 5 → AI operator 6". Processor 100 can match each operator fusion template with the network structure or computation graph of the AI ​​model. The network structure of the AI ​​model is used to indicate the dependencies between multiple AI operators in the AI ​​model. When multiple consecutive AI operators in an AI model match multiple AI operators in the fusion template, the processor 100 can determine that these multiple AI operators are the multiple AI operators that need to be fused.

[0069] In Example 2, the processor 100 can provide a client interface (UI) that displays the network structure or computation graph of the AI ​​model. Based on the network structure or computation graph presented by the client, the user can specify multiple consecutive AI operators to be fused within the UI. Correspondingly, the client can generate a configuration file based on the user's specified operations for the AI ​​operators and send this configuration file to the processor 100. Thus, the processor 100 can determine the multiple consecutive AI operators to be fused within the AI ​​model based on the received configuration file.

[0070] In other embodiments, the processor 100 may also determine the multiple AI operators participating in the fusion in the operator set in other ways, without limitation.

[0071] In practical applications, for multiple consecutive AI operators participating in fusion, a fusion graph can be used to indicate the dependencies between these operators. This fusion graph is a subgraph within the computation graph corresponding to the AI ​​model (which indicates the dependencies between all AI operators in the AI ​​model) that includes the multiple AI operators participating in the fusion. Since the dimensionality of the data input to the AI ​​model (such as the number of dimensions of the input data or the number of elements in a single dimension) may change during each inference process, the dimensionality of the input and output data of each AI operator in the fusion graph will also change.

[0072] S202: Processor 100 acquires input data for the operator set.

[0073] The input data obtained refers to the input data required for multiple AI operators as a whole.

[0074] For example, suppose processor 100 can determine such as Figure 3a The AI ​​operators shown include matrix multiplication (matmul), mean, and addition. The input data for the matmul operator includes tensor 0 and tensor 1; the input data for the mean operator is tensor A, the output of the matmul operator (i.e., the calculation result output by the matmul operator); and the input data for the add operator is tensor B, the output of the mean operator. Therefore, the input data acquired by the processor 100 includes... Figure 3a The tensor 0 and tensor 1 are shown.

[0075] In practical applications, the input data for multiple consecutive AI operators can include not only the input data of the first AI operator among them, but also the input data of other AI operators. For example, ... Figure 3b As shown, the input data of the add operator includes tensor B, the output of the mean operator, as well as tensor 2. At this time, the input data acquired by the processor 100 includes... Figure 3b Tensor 0, tensor 1, and tensor 2 are shown.

[0076] Typically, the input data acquired by the processor 100 can be the input data of the AI ​​model, or the calculation results output by other AI operators before the multiple AI operators in the AI ​​model.

[0077] S203: Processor 100 obtains the available capacity of cache 210 corresponding to acceleration units 201 to 203.

[0078] The available capacity refers to the size of the storage area in cache 210 that can be used to cache data. Typically, cache 210 is characterized by its fast read and write speeds and high hardware costs. Consequently, the capacity of cache 210 is usually small and may not be sufficient to support storing all the data (including input data, output data, etc.) involved in the operation of multiple AI operators participating in the fusion.

[0079] The cache corresponding to the acceleration unit refers to the cache used to store the data generated by the acceleration unit during the execution of AI computing tasks (or the execution of AI operators). In this embodiment, acceleration units 201 to 203 share cache 210. Therefore, processor 100 can obtain the available capacity of cache 210 so as to fuse multiple AI operators according to the actual available capacity of cache 210. In this way, the capacity of cache 210 occupied when executing the fused multiple AI operators does not exceed the available capacity of cache 210.

[0080] For example, processor 100 can access cache 210 to obtain the available capacity of cache 210.

[0081] S204: The processor 100 generates multiple AI computing tasks based on the available capacity of the cache 210, the input data of the operator set, and the operator set. Each AI computing task is generated based on a portion of the input data.

[0082] In this embodiment, when multiple operators participating in the fusion perform calculations based on the complete input data, the cache 210 is heavily occupied, easily exceeding its available capacity. Therefore, for each AI operator in the operator set, the processor 100 can generate an AI computation task for that AI operator based on a portion of the input data, according to the available capacity of the cache 210. Thus, the processor 100 can generate multiple AI computation tasks for that AI operator based on different portions of the input data. In this way, for multiple AI operators in the operator set, the processor 100 can generate multiple AI computation tasks based on the available capacity of the cache 210 and the input data of the operator set.

[0083] In one possible implementation, the processor 100 can segment the acquired input data into multiple data slices based on the available capacity of the cache 210. Then, the processor 100 can generate multiple AI computing tasks based on the multiple data slices and the operator set. In this way, the multiple AI operators subsequently involved in the fusion can execute their corresponding computational logic on a data slice basis, thereby reducing the storage space occupied by the cache 210 at any given time.

[0084] The following describes an exemplary implementation of how the processor 100 segments the acquired input data based on the available capacity of the cache 210.

[0085] Specifically, the processor 100 can deduce the dimensionality information of the input data and output data of each AI operator participating in the fusion based on the dimensionality information of the input data of the operator set. The dimensionality information of the input data can be, for example, the shape of the input data, including the number of dimensions and the number of elements in each dimension. For example, for... Figure 3a The matmul and mean operators shown are given. Assuming tensor 0 and tensor 1, which are input data for the matmul operator, are both 8×8 matrices, the processor 100 can deduce that the output data tensor A of the matmul operator also includes 8×8 matrices. For example, the shape of the input data for the mean operator can be represented as [8,8], meaning the input data has 8 dimensions, and each dimension includes 8 elements. Assuming the mean operator is used to calculate the average for each dimension of the input data, the processor 100 can deduce that the shape of the output data after the mean operator calculates based on the 8×8 matrix data (i.e., the input data) is [8,1], meaning the output data has 8 dimensions, and each dimension includes 1 element. Similarly, the processor 100 can further deduce... Figure 3a The shape of the add operator in the model.

[0086] Then, the processor 100 can estimate the cache 210 capacity required for multiple AI operators to run based on the dimensional information of the input data and the dimensional information of the output data of each AI operator. (Continuing with...) Figure 3a Taking the multiple operators shown as an example, assuming each value in the input or output data is represented by 2 bytes, the storage area in cache 210 required for the input data of the matmul operator is 256 (i.e., 8×8×2 + 8×8×2) bytes, and the storage area in cache 210 required for the output data of the matmul operator (i.e., the input data of the mean operator) is 128 (i.e., 8×8×2) bytes. Similarly, the processor 100 can estimate the storage area in cache 210 required by each AI operator during runtime based on the dimensional information of the input and output data, and thus calculate the total capacity of cache 210 required by multiple AI operators during runtime.

[0087] Next, the processor 100 can determine whether the estimated required capacity of the cache 210 is greater than the available capacity of the cache 210. If so, it indicates that the available capacity of the cache 210 does not meet the capacity requirements of the multiple AI operators. In this case, the processor 100 can divide the acquired input data into multiple data slices based on the available capacity of the cache 210. For example, the processor 100 can estimate the upper limit of the shape of the input data when the cache 210 supports multiple AI operators based on the available capacity of the cache 210, and then determine the number of slices to be divided based on the upper limit of the shape and the actual shape of the acquired input data, and divide the acquired input data according to the number of slices. If not, it indicates that the available capacity of the cache 210 can meet the cache capacity requirements of the multiple AI operators, and the processor 100 can directly execute the multiple AI operators using the acceleration unit. Since in real-world applications, the capacity of cache 210 is often insufficient to support the operation of multiple AI operators participating in the fusion, this embodiment mainly uses the example of the estimated required capacity of cache 210 being greater than the available capacity of cache 210 for illustration.

[0088] For example, when partitioning the input data, the processor 100 may partition the input data according to at least one of the following dimensions: element-wise, reduction, and sliding window. Partitioning the input data according to the element-wise dimension means dividing the input data at the element level, such as dividing an 8×8 matrix into eight 1×8 matrices, each 1×8 matrix being a data slice, or dividing the 8×8 matrix into sixteen 1×4 matrices, each 1×4 matrix being a data slice, etc. Partitioning the input data according to the reduction dimension means dividing the input data according to the dimension of the reduction operation; for example, when the reduction operation is to aggregate all data in two dimensions, the processor 100 may partition the 8×8 matrix into four 2×8 matrices according to the reduction dimension, each 2×8 matrix being a data slice, and each 2×8 matrix can be aggregated into a single value (such as the average). The sliding window dimension refers to dividing the input data according to the sliding window. For example, using a 2×2 sliding window (with a step size of 2), an 8×8 matrix can be divided into 16 2×2 matrices, and each 2×2 matrix is ​​a data slice.

[0089] In practical applications, the element-wise dimension, reduction dimension, and sliding window dimension can be configured with priorities. The processor 100 can then prioritize partitioning the input data based on the higher-priority dimension until the number of data fragments obtained exceeds a threshold. This threshold can be determined, for example, based on the actual shape and its upper bound, such as the ratio (rounded up) between the actual shape and its upper bound. For instance, assuming the element-wise dimension has the highest priority and the sliding window dimension has the lowest priority, the processor 100 can determine whether the number of data fragments obtained by partitioning the input data according to the element-wise dimension exceeds the threshold. If so, the processor 100 can partition the input data into multiple data fragments based solely on the element-wise dimension. If not, the processor 100 can determine whether the number of data fragments obtained by partitioning the input data according to both the element-wise and reduction dimensions exceeds the threshold, and further determine whether the number of further partitions of the input data exceeds the threshold. If so, the processor 100 can partition the input data into multiple data fragments based on both the element-wise and reduction dimensions. If not, the processor 100 can divide the input data according to the element-wise dimension, reduction dimension, and sliding window dimension to obtain multiple data slices.

[0090] For example, suppose processor 100 determines that the input data needs to be divided into 16 data fragments based on the available capacity of cache 210 (i.e., the threshold mentioned above is 16), and processor 100 determines that the number of data fragments obtained after dividing the input data according to the element-wise dimension is 8 (less than the threshold). Then processor 100 will determine that the number of data fragments obtained after dividing the input data according to the element-wise dimension and the reduction dimension is 16 (equal to the threshold). Therefore, processor 100 can determine that the input data should be divided into 16 data fragments according to the element-wise dimension and the reduction dimension.

[0091] The following describes an exemplary implementation of how processor 100 generates multiple AI computing tasks based on multiple data slices and operator sets obtained from the partitioning.

[0092] Each AI computing task is a data computation process performed based on one AI operator in the set of operators. The input data for each AI computing task can be a single data slice, multiple data slices, or the computation result obtained based on at least one data slice. Different AI computing tasks include different input data.

[0093] In practical applications, after dividing the input data of multiple AI operators in the fusion graph, a new fusion graph can be generated based on the multiple AI operators and multiple data slices of the input data, such as... Figure 4a or Figure 4b The fusion graph shown is used to generate multiple AI computing tasks based on this new fusion graph.

[0094] In a first possible implementation, when the processor 100 partitions the input data of the operator set solely according to the element-wise dimension, the input data of each AI operator participating in the fusion can be a data slice or the computation result output by another AI operator based on that data slice. In this case, the processor 100 can generate n*m ​​AI computation tasks. Here, n is the number of partitions of the input data of the operator set, m is the number of AI operators participating in the fusion, and both n and m are positive integers greater than 1.

[0095] For example, such as Figure 4a As shown, assuming the multiple AI operators involved in the fusion are AI operator 1, AI operator 2, and AI operator 3, the processor 100 can divide the input data of the operator set into four data slices, namely data slice 1_1 to data slice 1_4. Then, for AI operator 1, the processor 100 can generate four AI computing tasks based on the four data slices, each AI computing task including one data slice from data slice 1_1 to data slice 1_4. AI operator 1 performs calculations based on the four data slices, generating four calculation results. These four calculation results are the four input data of AI operator 2, namely data slice 2_1 to data slice 2_4. Furthermore, for AI operator 2, the processor 100 can generate four AI computing tasks based on the four data slices, each AI computing task including one data slice from data slice 2_1 to data slice 2_4. Similarly, processor 100 can generate 12 AI computing tasks based on the four data slices of the input data and three AI operators in the operator set, such as... Figure 4a As shown.

[0096] In a second possible implementation, the processor 100 partitions the input data of the operator set based on a non-element-wise dimension (such as a reduction dimension or a sliding window dimension). Specifically, in generating AI computing tasks, the processor 100 can generate multiple AI computing tasks based on multiple data slices, multiple AI operators, and additional merging operators. The merging operator is an AI operator used to merge the computation results corresponding to multiple data slices. It is understood that when the input data of the operator set is partitioned based on a non-element-wise dimension, directly concatenating the results calculated based on each data slice will typically result in a difference between the concatenated result and the computation result obtained by the multiple AI operators based on the input data of the complete operator set. Therefore, the processor 100 can utilize additional merging operators to merge the computation results corresponding to multiple data slices to ensure the accuracy of data computation.

[0097] For example, such as Figure 4b As shown, assume that the multiple AI operators participating in the fusion are as follows: Figure 3a The matmul operator, mean operator, and add operator are shown. Furthermore, the processor 100 can partition the input data of the operator set based on the reduction dimension to obtain, as shown... Figure 4bThe four data slices shown are matmul_data slice 1 to matmul_data slice 4. For the matmul operator, the processor 100 can generate four AI computing tasks (AI computing task 1 to AI computing task 4) based on these four data slices. Each AI computing task includes one data slice from matmul_data slice 1 to matmul_data slice 4. The four computation results generated by the matmul operator based on the four data slices are used as four input data sets for the mean operator, namely mean_data slice 1 to mean_data slice 4, thus generating four AI computing tasks (AI computing tasks 5 to AI computing tasks 8) for the mean operator. After partitioning the input data for the mean operator, the processor 100 adds a mad operator (an AI operator combining addition and multiplication operations) as shown in Figure 4 after the mean operator (before the add operator) according to the reduction dimension used to partition the input data, reducing the multiple computation results generated by the mean operator based on the multiple data slices. For the newly added mad operator, the input data of the mad operator is mad_data slice 1. mad_data slice 1 includes two calculation results obtained by the mean operator based on mean_data slice 1 and mean_data slice 2, respectively. The value calculated by the mad operator based on mad_data slice 1 is used as a data slice of the input data for the subsequent AI operator (i.e., the add operator). Similarly, the mad operator also obtains mad_data slice 2, which includes two calculation results obtained by the mean operator based on mean_data slice 3 and mean_data slice 4, respectively. Therefore, two AI computation tasks (i.e., AI computation task 9 and AI computation task 10) can be generated for the new mad operator. Next, the add operator will perform calculations based on add_data slice 1 and add_data slice 2, obtaining two calculation results. For the add operator, two AI computation tasks (i.e., AI computation task 11 and AI computation task 12) can be generated. Since the calculation results of the add operator are concatenated based on add_data partition 1 and add_data partition 2 respectively, the resulting concatenation is not the result obtained by the matmul operator, mean operator, and add operator based on the input data of the complete set of operators. Therefore, the processor 100 will continue to add additional mad operators after the add operator according to the reduction dimension used to partition the input data (the mad operators added at different positions are the aforementioned merging operators), such as... Figure 4bAs shown. In this way, the merging operator can merge the computation results corresponding to multiple data slices, ensuring that the final computation result is consistent with the computation result obtained by the multiple AI operators participating in the fusion based on the input data of the complete operator set. Accordingly, the processor 100 can generate one AI computation task (i.e., AI computation task 13) based on the additional mad operator added after the add operator. Therefore, the processor 100 can generate 13 AI computation tasks based on the four data slices, three AI operators, and the merging operator of the input data of the operator set, as shown. Figure 4b As shown.

[0098] In this embodiment, for different reduction dimensions, the processor 100 can supplement the multiple AI operators participating in the fusion with different merging operators. For example, when the input data of the operator set is divided using the reduction dimension for the sum (cumulative summation) operator, the processor 100 can add the add operator; when the input data of the operator set is divided using the reduction dimension for the mean operator, the processor 100 can add the mad operator. In practical applications, the processor 100 can be pre-configured with multiple partitioning rules. Each partitioning rule is used to indicate at least one merging operator to be added to the multiple AI operators participating in the fusion after partitioning the input data of the operator set based on a non-element-wise dimension. Thus, after partitioning the input data of the operator set using a non-element-wise dimension, the processor 100 can match the non-element-wise dimension with the partitioning rule and add the corresponding merging operator to the multiple AI operators participating in the fusion according to the successfully matched partitioning rule.

[0099] Since the process of generating multiple AI computing tasks is time-consuming, in a further possible implementation, the processor 100 can also improve the efficiency of generating AI computing tasks by pre-creating AI computing tasks.

[0100] Specifically, before partitioning the input data, the processor 100 can first create an initial AI computing task corresponding to each AI operator participating in the fusion, and prepare parameters unrelated to the dimensional information of the AI ​​operators in the created initial AI computing tasks, such as the type of AI computing task and the identifiers of the AI ​​operators included in the AI ​​computing task. Parameters related to the dimensional information of the input data and the dimensional information of the output data of the AI ​​operators are not added during the creation of the AI ​​computing task. After the processor 100 partitions the input data of the operator set into multiple data fragments, for the first AI operator among the multiple AI operators, the processor 100 can add the parameters related to the dimensional information of the input data and the dimensional information of the output data of the first AI operator to the already created initial AI computing task corresponding to the first AI operator, thereby generating the AI ​​computing task corresponding to the first AI operator. Among these, the parameters related to dimensional information could be, for example, the number of times the AI ​​computing task is executed (the number of executions could be the number of parts into which the input data of the operator set is divided, etc.). Furthermore, parameters related to dimensional information may also include the shape of the input and output data of the AI ​​operator, the dependency relationship between the AI ​​computing task and other AI computing tasks (i.e., the dependency relationship between the AI ​​operator and other AI operators), the storage address for storing the input and output data, and the scale of the AI ​​computing task. For other AI operators besides the first AI operator (such as the second AI operator, the third AI operator, etc.), the processor 100 can generate AI computing tasks corresponding to the other AI operators based on a similar method described above.

[0101] In this way, after the processor 100 completes the division of the input data, it can generate the AI ​​computing task corresponding to the AI ​​operator by adding the parameters related to the shape information of the AI ​​operator to the pre-created initial AI computing task. This can effectively improve the efficiency of the processor 100 in generating AI computing tasks, thereby improving the overall running efficiency of the AI ​​model.

[0102] It is understood that the above implementation method for generating multiple AI computing tasks is only an example. In other embodiments, the processor 100 may also generate multiple AI computing tasks in other ways, which will not be elaborated here.

[0103] S205: The processor 100 assigns multiple AI computing tasks to acceleration units 201 to acceleration units 203.

[0104] S206: Acceleration units 201 to 203 execute multiple AI computing tasks according to the kernel of each AI operator in the operator set. The data generated by each acceleration unit in the process of executing its own AI computing task is stored in the cache corresponding to that acceleration unit.

[0105] Among them, the cache corresponding to each acceleration unit in acceleration units 201 to 203 is cache 210.

[0106] Since each AI computing task needs to be executed based on the computing logic indicated by the kernel of an AI operator, after generating multiple AI computing tasks, the processor 100 can allocate these multiple AI computing tasks to at least one acceleration unit, so that the at least one acceleration unit can execute the multiple AI computing tasks according to the kernels of the multiple AI operators. This embodiment takes the allocation of multiple AI computing tasks to acceleration units 201 to 203, and the execution of the multiple AI computing tasks by acceleration units 201 to 203, as an example.

[0107] In this context, the AI ​​operator's kernel refers to the program code of the AI ​​operator (i.e., the underlying implementation of the AI ​​operator). By executing the AI ​​operator's kernel, the corresponding calculations are performed using the AI ​​operator, that is, the AI ​​computing task corresponding to the AI ​​operator is executed. During the execution of multiple AI computing tasks, the AI ​​operator's kernel can run in the acceleration unit within the data processing system 10. For example, the kernel of the matmul operator can run in the acceleration unit within the data processing system 10 for performing matrix calculation operations; the kernel of the mean operator can run in the acceleration unit within the data processing system 10 for performing vector calculation operations. Therefore, when the processor 100 allocates multiple AI computing tasks, it can assign the AI ​​computing task to an acceleration unit running the kernel of the AI ​​operator in that AI computing task. Alternatively, the processor 100 can first assign the AI ​​computing task to an acceleration unit that is not running a kernel, so that the acceleration unit, upon receiving the AI ​​computing task, can load and run the AI ​​computing kernel within that AI computing task.

[0108] This embodiment provides several implementation examples of distributing multiple AI computing tasks to multiple acceleration units for execution.

[0109] In the first implementation example, the processor 100 can allocate multiple AI computing tasks to corresponding acceleration units based on the dependencies between multiple AI operators in the fusion graph. Each acceleration unit may run the kernel of the AI ​​operator in the AI ​​computing task before being allocated; alternatively, each acceleration unit may temporarily load and run the kernel of the AI ​​operator in the AI ​​computing task after being allocated, without limitation. Each acceleration unit receiving an AI computing task can execute the program code included in the kernel of the AI ​​operator in that AI computing task to perform corresponding computational operations based on the input data in the AI ​​computing task. For example, assuming acceleration unit 201 is used for matrix computation, the processor 100 can allocate the AI ​​computing task generated for the matmul operator to acceleration unit 201, so that acceleration unit 201 can execute the program code included in the kernel of the matmul operator to perform corresponding matrix multiplication operations on acceleration unit 201 based on the input data in the AI ​​computing task and obtain the corresponding computational result. In practical applications, the processor 100 can be pre-configured with a kernel library, which includes kernels for multiple AI operators. Each AI operator's kernel can support dynamic changes in the shape of the input and output data. That is, when the acceleration unit executes the program code included in the AI ​​operator's kernel, it can perform calculations on input data with different shapes and obtain calculation results (i.e., output data) with different shapes. In this way, the processor 100 can perform AI calculation tasks by reusing the kernels of existing AI operators in the kernel library without having to perform the operation of generating and compiling the program code in the kernel.

[0110] The processor 100 can allocate each AI computing task individually; or, the processor 100 can allocate all AI computing tasks in batches to avoid the kernel startup overhead caused by allocating AI computing tasks multiple times.

[0111] Furthermore, after the input data is divided, there may be different types of AI computing tasks (such as matrix computing tasks, vector computing tasks, etc.) that are independent of each other among the multiple AI operators in the fusion graph. At this time, the processor 100 can schedule the kernels corresponding to the multiple AI operators to execute the different types of AI computing tasks that are independent of each other in parallel according to the fusion graph, so as to improve the execution efficiency of the multiple AI computing tasks and realize the parallel fusion and vertical fusion of the multiple AI operators.

[0112] by Figure 4b Taking the fusion diagram shown as an example, as Figure 5 As shown, since the AI ​​computation task corresponding to the matmul operator is a matrix computation type AI computation task and the AI ​​computation task corresponding to the mean operator is a vector computation type AI computation task, there is no dependency between the AI ​​computation task A2 generated based on matmul_data partition 2 and the AI ​​computation task B1 generated based on mean_data partition 1. The processor 100 can schedule the acceleration unit used to perform matrix computation operations to execute AI computation task A2, and schedule the acceleration unit used to perform vector computation operations to execute AI computation task B1, so that AI computation task A2 and AI computation task B1 can be executed in parallel, thereby improving the overall efficiency of executing AI computation task B1 and AI computation task A2. In practical applications, during the stage of partitioning the input data of the operator set, the number of partitions of the input data can be determined according to the upper limit of the shape of the input data when supporting multiple AI operators to perform operations. Therefore, during the parallel execution of multiple AI computation tasks by multiple acceleration units, the capacity of the cache 210 required usually does not exceed the available capacity of the cache 210.

[0113] For AI computing tasks with dependencies, the processor 100 can insert wait events between multiple AI computing tasks to ensure that these tasks are executed in the correct order. For example, based on... Figure 4b As shown in the fusion diagram, the processor 100 can add a wait event after the AI ​​computing task B1 generated based on mean_data shard 1 to wait for the AI ​​computing task B2 generated based on mean_data shard 2 to complete. In this way, when the scheduling acceleration unit executes the AI ​​computing task C1 corresponding to the add operator, it can perform a summation operation based on the calculation results obtained from executing AI computing task B1 and AI computing task B2, ensuring that AI computing task C1 can be executed after AI computing tasks B1 and B2. This guarantees the accuracy of the calculations performed by the multiple AI operators participating in the fusion on the input data of the operator set.

[0114] In other implementations, the processor 100 can serially schedule each AI computing task to the corresponding acceleration unit for execution based on the fusion graph, without any limitation on this.

[0115] In the second implementation example, the data processing system 10 may further include a hardware scheduler, such as an accelerator, capable of automatically scheduling AI computing tasks. Then, after generating multiple AI computing tasks, the processor 100 can use the hardware scheduler to schedule multiple acceleration units to execute the AI ​​computing tasks. Specifically, the processor 100 can send a fusion graph generated by partitioning the input data of the operator set to the hardware scheduler. After receiving the fusion graph, the hardware scheduler can determine the dependencies between the multiple AI computing tasks based on the fusion graph, and allocate the multiple AI computing tasks sequentially to the corresponding acceleration units for execution according to these dependencies. Each acceleration unit, upon receiving an AI computing task, executes the program code included in the kernel of the AI ​​operator corresponding to that AI computing task, and performs calculations on the input data of the AI ​​operator according to the computational logic indicated by the program code, thereby executing the AI ​​computing task. The specific implementation of the hardware scheduler allocating multiple AI computing tasks to acceleration units can be found in the relevant description of the implementation of the processor 100 distributing multiple AI computing tasks to the corresponding acceleration units, and will not be repeated here.

[0116] In the third implementation example, before partitioning the input data of the operator set, the processor 100 can create multiple scheduling templates for various dimensional combinations of partitioning the input data. Each dimensional combination includes at least one possible dimension for partitioning the input data and the number of partitions based on each dimension. Each created scheduling template is used to indicate the execution order of multiple AI computing tasks. Since the number of dimensional combinations for partitioning the input data of the operator set is usually limited, only a small number of scheduling templates need to be created to cover all possible partitions of the input data, that is, to cover all possible combinations of multiple AI computing tasks generated by the processor 100. Accordingly, when sending multiple AI computing tasks to multiple acceleration units, the processor 100 can select a scheduling template that matches the partitioning result of the input data from the multiple created scheduling templates according to the number of partitions of the input data of the operator set, and sequentially schedule the acceleration units to execute the AI ​​computing tasks based on the kernels of the corresponding AI operators according to the execution order of the multiple AI computing tasks indicated by the scheduling template.

[0117] It is understood that, in addition to the various implementation methods described above, the processor 100 can also implement the scheduling acceleration units 201 to 203 to execute multiple AI computing tasks through other applicable implementation methods, and there is no limitation thereto. For example, the processor 100 can also pre-create multiple scheduling templates for combinations of multiple AI operators, and determine a suitable scheduling template from the multiple scheduling templates to schedule tasks based on the partitioning results of the multiple AI operators participating in the fusion and the input data of the operator set.

[0118] During the execution of multiple AI computing tasks by multiple acceleration units, each acceleration unit can read the input data indicated by the AI ​​computing task from cache 210, perform the corresponding computation operation of the AI ​​operator based on the input data, obtain the corresponding computation result, and write the computation result into cache 210. Based on... Figure 4a Taking the 12 AI computing tasks generated by the fusion graph shown as an example, assuming that acceleration units 201, 202, and 203 are used to execute AI operator 1, AI operator 2, and AI operator 3 respectively, when acceleration unit 201 executes the AI ​​computing task based on data slice 1_1 (i.e., the AI ​​computing task for AI operator 1), it can read data slice 1_1 from cache 210, perform calculations based on data slice 1_1, obtain calculation result 1, and write the calculation result 1 into cache 210. This calculation result 1 is also the data slice 2_1 that serves as the input data for AI operator 2. Correspondingly, when acceleration unit 202 executes the AI ​​computing task based on data slice 2_1 (i.e., the AI ​​computing task for AI operator 2), it can directly read the calculation result 1 from cache 210, perform calculations based on the calculation result 1, obtain calculation result 2, and write the calculation result 2 into cache 210. This calculation result 2 is also the input data 2_1 for AI operator 3. Similarly, each acceleration unit can read and write data based on cache 210 when executing various AI computing tasks, without having to perform the process of transferring data between cache 210 and memory 300. This can effectively improve the efficiency of executing multiple AI operators participating in the fusion and reduce resource consumption.

[0119] In a further possible implementation, the processor 100 may also manage the cache 210 to improve the utilization of the cache 210.

[0120] In specific implementation, before the acceleration units 201 to 203 execute multiple AI computing tasks, the processor 100 can prefetch a portion of the input data of the operator set from the cache 210, such as prefetching... Figure 3bThe diagram shows partial data fragments from multiple data fragments of Tensor 0, Tensor 1, and Tensor 2. For example, the processor 100 can prefetch the input data required for the AI ​​computing tasks to be executed first from memory 300 to cache 210 according to the execution order of the AI ​​computing tasks. In this way, the acceleration unit in the data processing system 10 can directly read the required data fragments from cache 210 during the execution of multiple AI computing tasks without waiting for the data fragments to be transferred from memory 300 to cache 210, thereby improving the efficiency of the acceleration unit in executing the AI ​​computing task. Furthermore, for the remaining data that has not been prefetched to cache 210, the processor 100 can also prefetch the remaining data to cache 210 during the execution of multiple AI computing tasks by acceleration units 201 to 203, so that the acceleration units can directly read the remaining data from cache 210 and perform calculations.

[0121] Furthermore, during the execution of multiple AI computing tasks, when data fragments or computation results stored in cache 210 are consumed by other AI operators, processor 100 can invalidate those data fragments or computation results in cache 210. For example, Figure 5 As shown, after matmul_data fragment 1 is consumed by the matmul operator, matmul_data fragment 1 in cache 210 can be invalidated. Furthermore, after the tensor A_0 output by the matmul operator (i.e., the calculation result calculated based on matmul_data fragment 1) is consumed by the mean operator, tensor A_0 in cache 210 can be invalidated. Similarly, the remaining consumed data fragments and calculation results can also be invalidated after being consumed by subsequent AI operators, such as... Figure 5 As shown. In this way, the cache 210 can use the storage space occupied by invalid data to store the input data required by the acceleration unit to perform other AI computing tasks, or to store the computing results generated by the acceleration unit to perform other AI computing tasks, thereby making full use of the cache 210 and improving the locality of data.

[0122] Furthermore, the final result calculated by the multiple AI operators involved in the fusion is typically first written to cache 210. When preset conditions are met (such as the available capacity of cache 210 being insufficient to store new data), the result is written from cache 210 to memory 300. Based on this, processor 100 can write the result in cache 210 to memory 300 in advance to free up the storage space occupied by the result in cache 210, thereby allowing cache 210 to have more available storage space to store the input or output data of other AI operators in the AI ​​model.

[0123] Thus, by performing management operations such as prefetching, invalidating, and writing data to memory 300 in advance on cache 210, not only can the efficiency of executing multiple AI operators be improved, but also multiple AI operators can be supported to perform calculations based on input data with a larger shape while making full use of the storage space of cache 210.

[0124] In practical applications, before partitioning the input data of the operator set, the processor 100 can add management operations for the cache 210 to the fusion graph composed of multiple AI operators. For example, Figure 6 As shown, the processor 100 can add prefetch operations for tensor 0 and tensor 1, and invalidation operations for tensor A and tensor B, to instruct that tensor A in cache 210 be invalidated after tensor A, the output of the matmul operator, is consumed by the mean operator, and tensor B, the output of the mean operator, is invalidated after tensor B, the output of the mean operator, is consumed by the mean operator. Additionally, the processor 100 can add a write-back operation for the final output calculation result in the fusion graph, to instruct that after the calculation result output by the add operator is written to cache 210, the calculation result be written from cache 210 to memory 300 in advance to release the storage space occupied by the calculation result. Accordingly, after the processor 100 partitions the input data, the management operations for each tensor in the fusion graph can be adaptively adjusted by the processor 100 to management operations for each data fragment of that tensor.

[0125] It is understood that the above implementation of processor 100 managing cache 210 is only one implementation example. In other embodiments, the operation of processor 100 managing cache 210 can be other types of operations. Alternatively, data processing system 10 may also include other hardware, so that cache 210 can be managed by other hardware during the execution of multiple AI computing tasks.

[0126] It is worth noting that the above description uses a set of operators participating in the fusion in the AI ​​model as an example. In actual applications, there may be multiple AI operators that can participate in the fusion in the AI ​​model. In this case, during the operation of the AI ​​model, the processor 100 can use the above method to achieve fusion for each AI operator participating in the fusion, so as to improve the overall operating efficiency of the AI ​​model.

[0127] in addition, Figure 2 The AI ​​operator fusion method shown is only an example; in practical applications, it can also be used... Figure 2 More steps can be added to the method flow shown, or the steps can be modified accordingly. Figure 2The method flow shown may include the substitution or omission of some steps, without any restrictions.

[0128] It is worth noting that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0129] In addition, embodiments of this application also provide a processor capable of performing the above-described... Figure 2 The AI ​​operator fusion method executed by the processor 100 in the illustrated embodiment.

[0130] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned AI operator fusion method.

[0131] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0132] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0133] The computer program product can be a software installation package. When any of the aforementioned AI operator fusion methods is required, the computer program product can be downloaded and executed on a computing device.

[0134] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0135] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment. The terms “first,” “second,” etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate, and this is merely a way of distinguishing objects with the same attributes in the embodiments of this application.

[0136] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0137] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for fusing artificial intelligence (AI) operators, characterized in that, The method is applied to a data processing system, the data processing system including a processor and at least one acceleration unit, and the method includes: The processor acquires input data for the operator set, which includes multiple consecutive AI operators in the AI ​​model; The processor obtains the available capacity of the cache corresponding to the at least one acceleration unit; The processor generates multiple AI computing tasks based on the available capacity of the cache, the input data, and the set of operators. Each AI computing task is generated based on a portion of the input data. The processor assigns the plurality of AI computing tasks to the at least one acceleration unit; The at least one acceleration unit executes the multiple AI computing tasks according to the kernel of each AI operator in the operator set, wherein the data generated by each acceleration unit in the process of executing its own AI computing task is stored in the cache corresponding to each acceleration unit.

2. The method according to claim 1, characterized in that, The processor generates multiple AI computing tasks based on the available capacity of the cache, the input data, and the set of operators, including: The processor divides the input data into multiple data fragments based on the available capacity of the cache. The processor generates the plurality of AI computing tasks based on the plurality of data shards and the set of operators. Each AI computing task is generated based on at least one data shard in the plurality of data shards, and each AI computing task is executed based on the kernel of one AI operator in the set of operators.

3. The method according to claim 2, characterized in that, The processor divides the input data according to the available capacity of the cache, including: The processor calculates the dimension information of the input data and the dimension information of the output data of each AI operator in the operator set based on the dimension information of the input data of the operator set. The processor estimates the cache capacity required for multiple AI operators in the operator set to run based on the input data and output data of each AI operator, according to the dimensional information of the input data of the operator set. When the estimated cache capacity is greater than the available cache capacity, the processor divides the input data of the operator set according to the available cache capacity.

4. The method according to claim 2, characterized in that, The processor divides the input data according to the available capacity of the cache, including: The processor divides the input data based on at least one dimension according to the available capacity of the cache, and the at least one dimension includes an element-wise dimension, a reduction dimension, or a sliding window dimension.

5. The method according to claim 4, characterized in that, The input data is divided based on the reduction dimension or the sliding window dimension; The processor generates multiple AI computing tasks based on the available capacity of the cache, the input data, and the set of operators, including: The processor generates the multiple AI computing tasks based on the available capacity of the cache, the input data, the set of operators, and the merging operator. The merging operator is an AI operator used to merge the calculation results corresponding to multiple parts of the input data.

6. The method according to any one of claims 1 to 5, characterized in that, Before the processor acquires the input data for the operator set, the method further includes: The processor acquires a configuration file, which is used to instruct the fusion of the multiple consecutive AI operators in the AI ​​model; Alternatively, the processor may use an operator fusion strategy to traverse the AI ​​operators in the AI ​​model to determine the set of operators.

7. The method according to any one of claims 1 to 6, characterized in that, The kernels of the AI ​​operators run by the at least one acceleration unit are derived from the kernel library corresponding to the AI ​​model, and the kernel library includes the kernels of the AI ​​operators in the AI ​​model.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Before the at least one acceleration unit executes the plurality of AI computing tasks, the processor prefetches a portion of the input data of the operator set from the cache; During the execution of the plurality of AI computing tasks by the at least one acceleration unit, the processor invalidates data consumed by AI operators in the operator set in the cache and writes the data in the cache back to memory in advance.

9. The method according to any one of claims 1 to 8, characterized in that, The processor assigns the plurality of AI computing tasks to the at least one acceleration unit, including: The processor determines the scheduling template corresponding to the operator set from multiple scheduling templates based on the number of parts into which the input data of the operator set is divided. The processor allocates the multiple AI computing tasks to the at least one acceleration unit according to the scheduling template corresponding to the multiple operator sets, wherein the scheduling template is used to indicate the order in which the multiple AI computing tasks are executed.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: Before partitioning the input data of the operator set, an initial AI computation task is created for each AI operator in the operator set. The initial AI computation task includes parameters that are independent of the dimensional information of the AI ​​operator. The processor generates multiple AI computing tasks based on the available capacity of the cache, the input data, and the set of operators, including: The processor adds parameters related to the dimensional information of the first AI operator to the initial AI computing task corresponding to the first AI operator to obtain the AI ​​computing task corresponding to the first AI operator. The first AI operator is an AI operator in the operator set. The parameters related to the dimensional information of the first AI operator include the number of parts into which the input data of the operator set is divided.

11. A method for fusing artificial intelligence (AI) operators, characterized in that, The method is applied to a processor, which belongs to a data processing system, and the data processing system further includes at least one acceleration unit. The method includes: The processor acquires input data for the operator set, which includes multiple consecutive AI operators participating in the fusion of the AI ​​model. The processor obtains the available capacity of the cache corresponding to the at least one acceleration unit; The processor generates multiple AI computing tasks based on the available capacity of the cache, the input data, and the set of operators. Each AI computing task is generated based on a portion of the input data. The processor assigns the plurality of AI computing tasks to the at least one acceleration unit. The plurality of AI computing tasks are executed by the at least one acceleration unit according to the kernel of each AI operator in the set of operators. The data generated by each acceleration unit in the process of executing its own AI computing task is stored in the cache corresponding to each acceleration unit.

12. A data processing system, characterized in that, The data processing system includes a processor and at least one acceleration unit; The processor is configured to: acquire input data of an operator set, the operator set including multiple consecutive AI operators participating in fusion in an AI model; acquire the available capacity of the cache corresponding to the at least one acceleration unit; generate multiple AI computing tasks based on the available capacity of the cache, the input data, and the operator set, each of the multiple AI computing tasks being generated based on a portion of the input data; and allocate the multiple AI computing tasks to the at least one acceleration unit. The at least one acceleration unit is used to execute the plurality of AI computing tasks according to the kernel of each AI operator in the operator set, wherein the data generated by each acceleration unit in executing its respective AI computing task is stored in the cache corresponding to each acceleration unit.

13. The data processing system according to claim 12, characterized in that, The processor is used for: Based on the available capacity of the cache, the input data is divided into multiple data fragments; Based on the multiple data shards and the set of operators, multiple AI computing tasks are generated. Each AI computing task is generated based on at least one data shard in the multiple data shards, and each AI computing task is executed based on the kernel of one AI operator in the set of operators.

14. The data processing system according to claim 13, characterized in that, The processor is used for: Based on the dimension information of the input data of the operator set, the dimension information of the input data and the dimension information of the output data of each AI operator in the operator set are deduced; Based on the dimensional information of the input data and the dimensional information of the output data of each AI operator, the cache capacity required for multiple AI operators in the operator set to run based on the input data of the operator set is estimated. When the estimated cache capacity is greater than the available cache capacity, the input data of the operator set is divided according to the available cache capacity.

15. The data processing system according to claim 14, characterized in that, The processor is configured to partition the input data based on at least one dimension according to the available capacity of the cache, wherein the at least one dimension includes an element-wise dimension, a reduction dimension, or a sliding window dimension.

16. The data processing system according to claim 15, characterized in that, The input data is divided based on the reduction dimension or the sliding window dimension; The processor is configured to generate the plurality of AI computing tasks based on the available capacity of the cache, the input data, the set of operators, and the merging operator. The merging operator is an AI operator used to merge the calculation results corresponding to multiple parts of the input data.

17. The data processing system according to any one of claims 12 to 16, characterized in that, The processor is also used for: Before obtaining the input data for the set of operators, a configuration file is obtained, which is used to instruct the fusion of the multiple consecutive AI operators in the AI ​​model; Alternatively, before obtaining the input data for the operator set, the AI ​​operators in the AI ​​model can be traversed using an operator fusion strategy to determine the operator set.

18. The data processing system according to any one of claims 12 to 17, characterized in that, The kernels of the AI ​​operators run by the at least one acceleration unit are derived from the kernel library corresponding to the AI ​​model, and the kernel library includes the kernels of the AI ​​operators in the AI ​​model.

19. The data processing system according to any one of claims 12 to 18, characterized in that, The processor is also used for: Before the at least one acceleration unit executes the plurality of AI computing tasks, a portion of the input data of the operator set is prefetched in the cache; During the execution of the plurality of AI computing tasks by the at least one acceleration unit, the data consumed by the AI ​​operators in the operator set is invalidated in the cache, and the data in the cache is written back to memory in advance.

20. The data processing system according to any one of claims 12 to 19, characterized in that, The processor is used for: Based on the number of parts into which the input data of the operator set is divided, the scheduling template corresponding to the operator set is determined from multiple scheduling templates; According to the scheduling template corresponding to the multiple operator sets, the multiple AI computing tasks are assigned to the at least one acceleration unit, and the scheduling template is used to indicate the order in which the multiple AI computing tasks are executed.

21. The data processing system according to any one of claims 12 to 20, characterized in that, The processor is also used for: Before partitioning the input data of the operator set, an initial AI computation task is created for each AI operator in the operator set. The initial AI computation task includes parameters that are independent of the dimensional information of the AI ​​operator. The processor is specifically configured to add parameters related to the dimensional information of the first AI operator to the initial AI computing task corresponding to the first AI operator, thereby obtaining the AI ​​computing task corresponding to the first AI operator. The first AI operator is an AI operator in the operator set, and the parameters related to the dimensional information of the first AI operator include the number of parts into which the input data of the operator set is divided.

22. A processor, characterized in that, The processor is used to perform the method as described in claim 11.

23. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computing device, cause the computing device to perform the method as described in any one of claims 1 to 11.

24. A computer program product containing instructions, characterized in that, When it is run on at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 1 to 11.

Images