Method, device, equipment, and medium for evaluating operator performance in AI chips

Abstract

The present invention discloses a method, device, equipment and medium for evaluating the performance of operators in AI chips. The method includes: obtaining the operator's splitting strategy between caches at all levels of the AI ​​chip based on the operator characteristics of data handling and computing operators in the AI ​​chip, the storage characteristics of caches at all levels, and the computing core characteristics; calculating the data handling time consumed by each splitting strategy of the operator when performing data handling between caches at all levels according to a bandwidth model adapted to the AI ​​chip; calculating the computational time consumed by the operator when performing computations on the first-level cache for each splitting strategy according to the operator computing core performance model adapted to the AI ​​chip; calculating the total operator time consumed by the operator for each splitting strategy based on the computational time and data handling time, combined with the data handling and computation hiding strategy included in the splitting strategy, and performing operator performance evaluation on the operator. The operator performance of the AI ​​chip can be quickly and comprehensively evaluated on the basis of minimal manpower cost.

Description

Technical Field

[0001] The present invention relates to the field of chip technology, and in particular to a method, device, equipment and medium for evaluating operator performance in an AI (artificial intelligence) chip. Background Art

[0002] In high-performance computing scenarios for AI chips, operator performance is a crucial component of chip performance evaluation. Operator performance largely determines the upper limit of chip performance, but many factors influence operator performance, including operator type, input shape, segmentation method, boundary points, and pipelining strategy. Therefore, determining the precise and optimal performance of operators, taking into account these numerous factors, remains a significant technical challenge.

[0003] In related technologies, operator managers typically build multiple operator templates specifically for their respective operators, combining various operator input shapes. Based on various factors affecting operator performance, they determine all possible operator segmentation strategies for different input shapes. Each segmentation strategy is then used to fully execute each operator template on the AI ​​chip, ultimately selecting the optimal operator template.

[0004] In the process of realizing the present invention, the inventors found that the existing technology has the following technical problems: 1. It is necessary to introduce a large number of professionals and invest a lot of time and cost to conduct a large number of actual tests on actual AI chips, which will consume a lot of manpower, equipment and time costs; 2. The performance of the operator is closely related to the hardware architecture of the AI ​​chip. When the AI ​​chip is upgraded, each operator in the AI ​​chip needs to be re-evaluated. Even a small chip performance upgrade will consume a lot of manpower and material resources to complete the evaluation; 3. The input shape of the operator is ever-changing, and each input shape may correspond to a high-performance operator template. It is almost impossible to implement these operator templates entirely by manpower. Summary of the Invention

[0005] Embodiments of the present invention provide a method, apparatus, device, and medium for evaluating the performance of operators in an AI chip, so as to quickly and comprehensively evaluate the operator performance of the AI ​​chip while minimizing the human cost.

[0006] According to one aspect of an embodiment of the present invention, a method for evaluating operator performance in an AI chip is provided, comprising:

[0007] Obtain multiple partitioning strategies for the operator to be evaluated across cache levels of the target AI chip based on at least one of the operator characteristics of the operator to be evaluated in the target AI chip, the storage characteristics of cache levels of the target AI chip, and the computing core characteristics;

[0008] Based on a pre-built bandwidth model adapted to the target AI chip, the data transfer time between cache levels is calculated for each splitting strategy of the operator to be evaluated.

[0009] The bandwidth model is used to predict bandwidth data when transferring data of a set data transfer type, a set data type, and a set tensor shape between two set caches in the target AI chip;

[0010] If the operator to be evaluated is a pure data handling operator, then the total operator time of the operator to be evaluated for each splitting strategy is calculated based on the data handling time and the first hidden strategy included in the splitting strategy, and the operator performance of the operator to be evaluated is evaluated based on the total operator time.

[0011] According to another aspect of an embodiment of the present invention, a device for evaluating operator performance in an AI chip is provided, comprising:

[0012] a partitioning strategy determination module, configured to obtain multiple partitioning strategies for the operator to be evaluated across cache levels of the target AI chip based on at least one of the operator characteristics of the operator to be evaluated in the target AI chip, the storage characteristics of the cache levels of the target AI chip, and the computing core characteristics;

[0013] The data transfer time calculation module is used to calculate the data transfer time when each splitting strategy of the operator to be evaluated executes data transfer between cache levels based on a pre-built bandwidth model adapted to the target AI chip;

[0014] The bandwidth model is used to predict bandwidth data when transferring data of a set data transfer type, a set data type, and a set tensor shape between two set caches in the target AI chip;

[0015] The first performance evaluation module is used to calculate the total operator time of the operator to be evaluated for each splitting strategy based on the time consumption of each data transfer and the first hidden strategy included in the splitting strategy if the operator to be evaluated is a pure data handling operator, and perform operator performance evaluation on the operator to be evaluated based on the total time consumption of each operator.

[0016] According to another aspect of an embodiment of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the operator performance evaluation method in the AI ​​chip described in any embodiment of the present invention.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the operator performance evaluation method in the AI ​​chip described in any embodiment of the present invention when executed.

[0021] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the steps of the operator performance evaluation method in an AI chip as described in any embodiment of the present invention.

[0022] The technical solution of the embodiment of the present invention obtains multiple splitting strategies of the operator to be evaluated between the caches of the target AI chip at all levels according to the operator characteristics of the operator to be evaluated in the target AI chip, the storage characteristics of the caches of the target AI chip at all levels, and the computing core characteristics; calculates the data transfer time of each splitting strategy of the operator to be evaluated when performing data transfer between the caches of all levels according to the pre-built bandwidth model adapted to the target AI chip; if the operator to be evaluated is a pure data transfer operator, then according to each data transfer time, combined with the first hidden strategy included in the splitting strategy, calculates the total operator time of the operator to be evaluated for each splitting strategy, and performs operator performance evaluation on the operator to be evaluated according to the total operator time, by constructing a bandwidth The model effectively performs data fitting and learning on various types of bandwidth data, realizes efficient evaluation of operator performance, significantly improves the evaluation speed while ensuring evaluation accuracy, and can effectively reduce dependence on and occupation of hardware resources in AI chips. In addition, the technical solution of the embodiment of the present invention does not require manpower to implement operator templates, and since the bandwidth model does not need to run on the AI ​​chip, it can save a lot of human resources and equipment resources, and effectively save evaluation time. At the same time, the solution of the embodiment of the present invention can provide optimization guidance for the operator implementation of the AI ​​chip before the AI ​​chip is mass-produced, provide an effective solution for optimizing the operator to the optimal performance, and save the trial and error cost in the process of implementing the operator on the new chip.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 This is a flowchart of a method for evaluating operator performance in an AI chip according to an embodiment of the present invention;

[0026] Figure 2 This is a flowchart of another method for evaluating operator performance in an AI chip provided by an embodiment of the present invention;

[0027] Figure 3 is a flowchart of another method for evaluating operator performance in an AI chip according to an embodiment of the present invention;

[0028] Figure 4 is a flowchart of another method for evaluating operator performance in an AI chip provided according to an embodiment of the present invention;

[0029] Figure 5 2 is a schematic structural diagram of an operator performance evaluation device in an AI chip provided according to an embodiment of the present invention;

[0030] Figure 6 2 is a schematic diagram of the structure of an electronic device that implements the operator performance evaluation method in the AI ​​chip of an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Figure 1 This is a flowchart of a method for evaluating operator performance in an AI chip provided in an embodiment of the present invention. This embodiment is applicable to situations where efficient and rapid operator performance evaluation is performed before mass production of AI chips, without constructing operator templates. The method can be performed by an operator performance evaluation device in the AI ​​chip, which can be implemented in the form of hardware and / or software and can generally be configured in various electronic devices with data computing functions, such as various terminal devices or servers.

[0034] Correspondingly, such as Figure 1 As shown, the method may include:

[0035] S110. Obtain multiple partitioning strategies for the operator to be evaluated between cache levels of the target AI chip based on at least one of the operator characteristics of the operator to be evaluated in the target AI chip, the storage characteristics of cache levels of the target AI chip, and the computing core characteristics.

[0036] The target AI chip can be understood as an AI chip currently in the development phase, and the operator to be evaluated can be understood as a function that needs to run in the target AI chip to perform specific mathematical or logical operations (e.g., dot multiplication, convolution, or pure data transfer). The term "operator to be evaluated" means that the operator has either not yet been implemented in the AI ​​chip and requires effective evaluation of its optimal performance during the chip development phase, or it has already been implemented in the AI ​​chip, but the performance of the operator implementation is suboptimal, requiring operator performance optimization during the operator optimization phase. Generally speaking, operators to be evaluated can fall into two categories: pure data transfer operators, which only require data transfer across multiple levels of cache and do not require actual computation using the target AI chip's compute cores; and data transfer and computation operators, which require data transfer across multiple levels of cache and then use the target AI chip's compute cores to perform computation. In this embodiment, the performance evaluation method for this type of operator is described in detail, using the example of a pure data transfer operator to be evaluated.

[0037] Operator characteristics can be understood as the various attributes of an operator, such as operator type, operator computation and handling type, tensor shape, and data type. The operator type can be understood as the specific processing logic implemented by the operator. For example, if the operator to be evaluated is a data handling and computation operator, the operator type can be a convolution operator, a differentiation operator, or a reduction operator. If the operator to be evaluated is a pure data handling operator, the operator type can be a slice handling operator or a transpose handling operator. The computation and handling type is a characteristic of data handling and computation operators and can be categorized as L2 or L3 operators, depending on the data handling format used during computation. The tensor shape can be understood as the dimensional structure of the operator's input and output data, that is, the input and output shapes of the operator. The data type can be understood as the numerical precision of the data processed (computational or handling) by the operator. Specifically, it refers to the numerical representation format of each element in the operator's input and output tensors, such as INT8, FP16, or FP32.

[0038] The storage characteristics of each level of cache in the target AI chip can be understood as the attribute information of each level of cache under the specific cache architecture of the target AI chip, such as cache size and the pipeline strategy that can be used on the cache. When the operator to be evaluated is a data handling and computational operator, it is often necessary to determine the corresponding segmentation strategy in combination with the computational core characteristics of the target AI chip. The computational core characteristics can be understood as the attribute information of the computational cores specifically used to perform the calculations in the target AI chip, such as computational core constraints, the total number of computational cores included in the target AI chip, and the number of computational cores required for the target AI chip to perform operator calculations matching the operator to be evaluated. The computational core specifically refers to the physical circuit module in the target AI chip dedicated to performing tensor-level mathematical operations.

[0039] Generally speaking, when the operator to be evaluated is a data handling and computing operator, after completing the data handling, it needs to use the computing core operator that matches the operator type of the operator to be evaluated to perform the calculation in the computing core. The two operators have a one-to-one correspondence. For example, for an operator to be evaluated that is used to implement convolution calculation logic, after it transfers the data to the first-level cache of the target AI chip, it needs to perform matching convolution calculations in the computing core based on the computing core operator used to implement convolution calculations.

[0040] Generally speaking, the input data required to be moved and calculated by the operator to be evaluated is often large in scale, while the tensor shape of the computational data that can be processed by the corresponding computing core operator is generally much smaller than this input data scale. Therefore, when executing a specific data calculation process, the input data stored in the target AI chip's memory needs to be segmented and moved to the target AI chip's first-level cache level by level based on the target AI chip's cache architecture. One or more computing cores on the target AI chip then perform the calculation based on the corresponding computing core operator.

[0041] For example, assuming the target AI chip has a three-level cache architecture, and in order of distance from the computing core in the target AI chip, the three-level cache architecture is respectively a first-level cache, a second-level cache, and a third-level cache. Then, based on the input tensor shape that matches the operator to be evaluated, a split on the third-level cache (the split on the third-level cache can generally directly use the input tensor shape), a split on the second-level cache, and a split on the first-level cache can be constructed. Based on the split on each level of cache, the input tensor is moved to the first-level cache one by one and level by level for calculation. Accordingly, the split on the third-level cache, the split on the second-level cache, and the split on the first-level cache can serve as important information in a split strategy for the operator to be evaluated, that is, the split of data in each level of cache.

[0042] In this embodiment, by comprehensively utilizing the operator characteristics of the operator to be evaluated in the target AI chip, the storage characteristics of the cache at each level of the target AI chip, and at least one of the computing core characteristics, all partitioning strategies for the operator to be evaluated between the caches at each level of the target AI chip can be traversed.

[0043] In an optional implementation of this embodiment, when the operator to be evaluated is a pure data transfer operator, the partitioning strategy specifically includes: partitioning of data in caches at all levels, and a first hiding strategy in which data is hidden from each other when transferred between caches at different levels;

[0044] When the operator to be evaluated is a data handling and computing operator, the splitting strategy specifically includes: data splitting in caches at all levels, the number of computing cores used when executing operator calculations, and a second hiding strategy in which data is hidden from each other when being moved between caches at different levels and when performing calculations.

[0045] S120. Based on a pre-built bandwidth model adapted to the target AI chip, calculate the data transfer time consumed by each splitting strategy of the operator to be evaluated when performing data transfer between caches at various levels.

[0046] Among them, the bandwidth model is used to predict the bandwidth data when data of a set data transfer type, a set data type, and a set tensor shape is transferred between two set caches in the target AI chip.

[0047] Currently, the performance of an operator is mainly evaluated by its time consumption, which generally consists of two parts: one is the data transfer time of the DMA (Direct Memory Access) device, and the other is the computational time when the computing core performs the calculation.

[0048] Specifically, when the operator to be evaluated is a pure data handling operator, the operator's time consumption can be evaluated using only the data handling time consumption. However, when the operator to be evaluated is a data handling and computing operator, the operator's time consumption needs to be evaluated using both the data handling time consumption and the computing time consumption when the computing core performs the calculation.

[0049] As mentioned above, in the related technology, multiple operator templates corresponding to different tensor shapes are artificially constructed, and the operator templates are actually run on the target AI chip based on different segmentation strategies to obtain the measured operator time consumption.

[0050] Because the implementation of related technologies requires a significant amount of manpower and chip computing power, and the existing partitioning space pruning method may eliminate the optimal partitioning strategy, the inventors creatively proposed a new implementation method that introduces a machine learning model to efficiently evaluate the data transfer time or computation time of different partitioning strategies on a general-purpose computing device. The general-purpose computing device here refers to an electronic device that can run a machine learning model. In other words, a new method is proposed to predict data transfer time or computation time without consuming the computing power of the target AI chip.

[0051] Specifically, the machine learning model used to evaluate the time consumption of data transfer can be a bandwidth model used to predict bandwidth data. Bandwidth data can be understood as the amount of data transferred between two caches per unit time. Accordingly, the bandwidth model can be understood as the bandwidth data obtained by training a set machine learning model on a small amount of measured data of the target AI chip, and is used to predict the bandwidth data when transferring data of a set, set data type, and set tensor shape between two caches in the target AI chip.

[0052] The data transfer type can be understood as the characteristics of the data reorganization method when data is transferred between different levels of cache. The data transfer type may include: in-situ transfer (Copy), transpose transfer (Transpose), slice transfer (Slice) and compression transfer (Compress).

[0053] In a specific example, suppose that the bandwidth data for moving data from the third-level cache to the second-level cache under a splitting strategy A is evaluated through a bandwidth model. Combined with the data splitting for the third-level cache in the splitting strategy A (that is, the tensor shape of the data blocks transmitted between different caches), the data transfer time between the third-level cache and the second-level cache under the splitting strategy A can be calculated, that is, the result obtained by dividing the amount of data matching the split tensor shape by the bandwidth data.

[0054] S130. If the operator to be evaluated is a pure data handling operator, then the total operator time consumption of the operator to be evaluated for each splitting strategy is calculated based on the data handling time consumption and the first hidden strategy included in the splitting strategy, and the operator performance of the operator to be evaluated is evaluated based on the total operator time consumption.

[0055] As mentioned above, if the operator to be evaluated is a pure data-moving operator, then the total operator time for each partitioning strategy can be estimated simply by calculating the data-moving time between cache levels for each partitioning strategy. In other words, after calculating the data-moving time between cache levels for each partitioning strategy, the total operator time for each partitioning strategy can theoretically be calculated. For example, in the most extreme case, the total data-moving time can be accumulated.

[0056] However, when moving data between different levels of cache, if different DMA hardware on the target AI chip is used, the time required for each of these data movement operations can theoretically be hidden from each other. For example, the time required to move data from the L3 cache to the L2 cache can be hidden from the time required to move data from the L2 cache to the L1 cache.

[0057] Correspondingly, the first hiding strategy included in the splitting strategy specifically refers to the mutual hiding strategy when data is moved between different levels of cache. After the structure of the DMA hardware used in the target AI chip is clear, the corresponding first hiding strategy can also be determined, and this first hiding strategy also needs to be added to each splitting strategy. Of course, it is understandable that when data is moved on different levels of cache, if the same DMA hardware on the target AI chip is used, the first hiding strategy can be empty, that is, when data is moved between different levels of cache, no time hiding can be performed.

[0058] In this embodiment, after evaluating the data transfer time of each splitting strategy when performing data transfer between caches at all levels for the pure data transfer operator, and combining the mutual hiding strategy when the data in the splitting strategy is transferred between caches at different levels, the total time consumption of the operator corresponding to each splitting strategy can be uniquely determined.

[0059] The technical solution of the embodiment of the present invention obtains multiple splitting strategies of the operator to be evaluated between the caches of the target AI chip at all levels according to the operator characteristics of the operator to be evaluated in the target AI chip, the storage characteristics of the caches of the target AI chip at all levels, and the computing core characteristics; calculates the data transfer time of each splitting strategy of the operator to be evaluated when performing data transfer between the caches of all levels according to a pre-built bandwidth model adapted to the target AI chip; if the operator to be evaluated is a pure data transfer operator, then according to each data transfer time, combined with the first hidden strategy included in the splitting strategy, calculates the total operator time of the operator to be evaluated for each splitting strategy, and performs operator performance evaluation on the operator to be evaluated according to the total operator time, by constructing a bandwidth model adapted to the target AI chip The wide model effectively performs data fitting and learning on various types of bandwidth data, realizes efficient evaluation of operator performance, significantly improves the evaluation speed while ensuring evaluation accuracy, and can effectively reduce dependence on and occupation of hardware resources in AI chips. In addition, the technical solution of the embodiment of the present invention does not require manpower to implement operator templates, and the bandwidth model does not need to run on the AI ​​chip, which can save a lot of human resources and equipment resources and effectively save evaluation time. At the same time, the solution of the embodiment of the present invention can provide optimization guidance for the operator implementation of the AI ​​chip before the AI ​​chip is mass-produced, provide an effective solution for optimizing the operator to the optimal performance, and save the trial and error cost of implementing the operator on the new chip.

[0060] Based on the above embodiments, before calculating the data transfer time consumed by each splitting strategy of the operator to be evaluated when performing data transfer between cache levels according to a pre-built bandwidth model adapted to the target AI chip, the following steps may also be included:

[0061] Run a bandwidth test program adapted for the target AI chip on the target AI chip to measure data under multiple data transfer types, multiple data types, and multiple tensor shape combinations, and the actual bandwidth data when performing data transfer between various levels of cache on the target AI chip; based on the measured bandwidth data, construct a bandwidth model of the target AI chip.

[0062] In this optional implementation, a corresponding bandwidth test program can be written based on the minimum granularity DMA operation of the target AI chip to obtain actual bandwidth data under multiple data transfer types, multiple data types, and multiple tensor shape combinations, and to perform data transfer between cache levels of the target AI chip.

[0063] It's understandable that once the tensor shape, data type, and data transfer type of the data being transferred are determined, the bandwidth required to transfer that data between two predefined levels of cache can be measured. By constructing a bandwidth test program based on this information and running it on the target AI chip, the corresponding bandwidth data, also known as the measured bandwidth data, can be measured and obtained.

[0064] After obtaining the aforementioned multiple measured bandwidth data, a general bandwidth model can be trained to output matching predicted bandwidth data based on the input set data transfer type, set data type, set tensor shape, and two set buffers for data transfer. Alternatively, to ensure model calculation accuracy, multiple bandwidth models can be trained corresponding to the set data transfer type, set data type, and two set buffers for performing data transfer. Different types of bandwidth models can then be used to obtain predicted bandwidth data for different tensor shapes.

[0065] Figure 2 This is a flowchart of another method for evaluating operator performance in an AI chip provided by an embodiment of the present invention. In this embodiment, when the operators to be evaluated are specifically data handling and calculation operators, a method for evaluating the total operator time consumption of such operators to be evaluated is described in detail.

[0066] Correspondingly, such as Figure 2 As shown, the method may include:

[0067] S210. Obtain multiple partitioning strategies for the operator to be evaluated between caches at various levels of the target AI chip based on at least one of the operator characteristics of the operator to be evaluated in the target AI chip, storage characteristics of caches at various levels of the target AI chip, and computing core characteristics.

[0068] In this embodiment, the operator to be evaluated is concretized as a data handling and calculation operator. At this time, each splitting strategy further includes the number of computing cores used when executing the operator calculation. Since the data handling and calculation operators need to use the computing cores in the target AI chip to perform the corresponding operator calculations, the number of computing cores used when implementing the operator calculations is also closely related to the operator performance. For an operator to be evaluated, the various optional numbers of computing cores that can be used by the operator to be evaluated when executing the operator calculation can be determined based on the operator characteristics of the operator, the computing core characteristics of the target AI chip, or even based on a complete traversal method, and the above-mentioned various optional numbers of computing cores can be added to the splitting strategy respectively.

[0069] S220. Based on a pre-built bandwidth model adapted to the target AI chip, calculate the data transfer time consumed by each splitting strategy of the operator to be evaluated when performing data transfer between caches at various levels.

[0070] Among them, the bandwidth model is used to predict the bandwidth data when data of a set data transfer type, a set data type, and a set tensor shape is transferred between two set caches in the target AI chip.

[0071] S230. If the operator to be evaluated is a data handling and computing operator, the computational time consumed by the operator to be evaluated when executing the computation for each partitioning strategy on the first-level cache is calculated based on a pre-built operator computing core performance model adapted to the target AI chip.

[0072] As mentioned above, when the operator to be evaluated is a data handling and computing operator, in order to evaluate the total operator time consumption of the operator to be evaluated, in addition to calculating the data handling time consumption corresponding to the operator to be evaluated, the computing time consumption corresponding to the operator to be evaluated also needs to be calculated.

[0073] The L1 cache is located close to the compute cores in the target AI chip. For example, if the target AI chip has eight compute cores, each core will have a corresponding L1 cache, resulting in a total of eight L1 caches, each of which is located close to its corresponding compute core.

[0074] Generally speaking, regardless of the target AI chip's cache architecture, it contains a level-one cache (also known as L1 cache) located close to the compute cores. All computational data stored in memory must be moved to this L1 cache through a level-by-level partitioning process before one or more compute cores can perform the corresponding calculations using a compute core operator that matches the operator type being evaluated. Accordingly, the computational time mentioned above specifically refers to the computational time it takes for the operator being evaluated to execute calculations in the L1 cache for each partitioning strategy.

[0075] Among them, the operator computing core performance model is used to predict the computational time required to perform a set type of operator calculation on data of a set data type and a set tensor shape in the target AI chip.

[0076] In this embodiment, the machine learning model used to evaluate computational time can be an operator computation core performance model. Accordingly, the operator computation core performance model can be understood as a model obtained by training a set machine learning model on a small amount of measured data from a target AI chip, and is used to predict the computational time required to execute a set type of operator computation on data of a set data type and a set tensor shape in the target AI chip.

[0077] S240 , based on each calculation time consumption and each data transfer time consumption, combined with the second hidden strategy included in the segmentation strategy, calculate the total operator time consumption of the operator to be evaluated for each segmentation strategy, and perform operator performance evaluation on the operator to be evaluated based on the total operator time consumption.

[0078] In this embodiment, the second hiding strategy specifically refers to the mutual hiding strategy of data when moving between different levels of cache and performing calculations. As mentioned above, the hardware that performs data movement and the hardware that performs data calculations in the target AI chip are generally different. Therefore, in addition to the data movement operations at different levels being able to hide from each other, the operations of data movement and data calculation can also be hidden from each other. In a specific example, the data movement time of moving data from the L3 cache to the L2 cache can be hidden from each other with the data movement time of moving data from the L2 cache to the L1 cache plus the time for the data to be calculated on the L1 cache.

[0079] In an optional implementation of this embodiment, performing operator performance evaluation on the operator to be evaluated based on the total time consumption of each operator may include:

[0080] Obtain an optimal splitting strategy corresponding to the minimum total operator time consumption; and provide optimization guidance to the operator to be evaluated based on the various strategy information contained in the optimal splitting strategy so that the operator to be evaluated achieves the best performance.

[0081] It can be understood that the optimal splitting strategy corresponding to the minimum total operator time consumption is the optimal performance that the operator to be evaluated can achieve on the target AI chip. Furthermore, the various strategy information contained in the optimal splitting strategy, such as specific splitting information, pipeline arrangement information, and the optimal number of computing cores used when executing operator calculations when the operator to be evaluated is a data handling and calculation operator, can guide the operator to be evaluated to be optimized to the corresponding optimal performance.

[0082] In other words, the solution of this embodiment of the present invention can select the optimal performance based on the total time consumed by all operators. The partitioning strategy corresponding to this performance is the optimal performance of the operator to be evaluated relative to the input on the target AI chip. Accordingly, the specific partitioning information, pipeline arrangement, and optimal number of computing cores used to execute the operator calculation output in the optimal partitioning strategy can guide the operator to be evaluated to achieve the corresponding optimal performance on the target AI chip.

[0083] Furthermore, for the new generation of AI chips, it is only necessary to update the corresponding bandwidth measured data and computing time measured data, retrain the corresponding bandwidth model and operator computing core performance model, and then fine-tune the above two models according to the new generation of AI chip architecture. The performance of each operator corresponding to the new generation of AI chip can be recalculated on the new generation of AI chip using the methods of the various embodiments of the present invention.

[0084] The technical solution of the embodiment of the present invention effectively fits and learns various bandwidth data and computing time data by constructing a bandwidth model and an operator computing core performance model, thereby realizing efficient evaluation of operator performance, significantly improving the evaluation speed while ensuring evaluation accuracy, and effectively reducing dependence on and occupation of hardware resources in the AI ​​chip. In addition, the technical solution of the embodiment of the present invention does not require manpower to implement the operator template, and the bandwidth model does not need to run on the AI ​​chip, which can save a lot of human resources and equipment resources and effectively save evaluation time. At the same time, the solution of the embodiment of the present invention can provide optimization guidance for the operator implementation of the AI ​​chip before the AI ​​chip is mass-produced, provide an effective solution for optimizing the operator to the optimal performance, and save the trial and error cost of implementing the operator on the new chip.

[0085] Based on the above embodiments, before calculating the computation time of the operator to be evaluated when executing the computation for each slicing strategy on the L1 cache based on a pre-built operator computing core performance model adapted to the target AI chip, the following steps may also be included:

[0086] Obtain a computing core test program corresponding to each operator on the target AI chip; run each computing core test program on the target AI chip respectively, and measure the actual computing time consumed by the computing core of the target AI chip when performing various types of operator calculations on data of multiple data types and multiple tensor shapes on the first cache; and construct an operator computing core performance model of the target AI chip based on the actual computing time consumed.

[0087] In this optional implementation, a computing core test program corresponding to each computing core operator can be written based on the target AI chip's computing core minimum granularity specifications. Different computing core operators correspond to different data handling and computation operators, which are used to implement computational logic adapted to the operator type of the data handling and computation operator.

[0088] Accordingly, different computing core operators need to implement different computing logic. The computing core test program described above is the program code used to simulate the computing logic of different computing core operators. Similarly, once the data type and tensor shape of the required computational data and the required computational logic (i.e., operator type) are determined, by calling the computing core test program that matches the computational logic, the computation is performed on the target AI chip for data of that data type and tensor shape. The actual computation time taken by the computing core of the target AI chip to perform the set type of operator computation on data of that data type and tensor shape in the first cache can be measured. Through the above settings, multiple different computational times can be measured, i.e., the actual measured computation times.

[0089] After obtaining the above-mentioned multiple measured computing times, a general operator computing core performance model can be trained to output the matching predicted computing time according to the input set operator type, the set operator data type and the set tensor shape. Alternatively, in order to ensure the accuracy of the model calculation, multiple operator computing core performance models corresponding to the set operator type and the set data type can be trained separately, and then different types of operator computing core performance models can be selected to obtain the predicted computing time under different tensor shapes.

[0090] Figure 3 This is a flowchart of another method for evaluating operator performance in an AI chip, provided by an embodiment of the present invention. This embodiment is optimized based on the above embodiments. Specifically, when the operator to be evaluated is a data handling and computation operator, the operations of "constructing a bandwidth model of the target AI chip based on the measured bandwidth data" and "constructing a performance model of the operator core of the target AI chip based on the measured computation time" are specified.

[0091] Correspondingly, such as Figure 3 As shown, the method may specifically include:

[0092] S310. Run a bandwidth test program adapted for the target AI chip on the target AI chip to measure data under multiple data transfer types, multiple data types, and multiple tensor shape combinations, and measure the actual bandwidth data when performing data transfer between cache levels of the target AI chip.

[0093] In this embodiment, a general bandwidth test program template can be pre-written, which takes a data transfer type, a data type, a tensor shape, and two predetermined buffers as input variables. Furthermore, when the general bandwidth test program template is reconstructed using the predetermined data transfer type, data type, tensor shape, and two predetermined buffers, and the reconstructed bandwidth test program is run on the target AI chip, data with the predetermined data transfer type, data type, and tensor shape can be obtained, along with actual bandwidth data when executing data transfer between the two predetermined buffers of the target AI chip.

[0094] Optionally, when constructing a specific bandwidth test program, data of multiple different tensor shapes of the same data transfer type and the same data type can be obtained simultaneously during the execution of a bandwidth test program, and the actual bandwidth data when performing data transfer between the same two caches of the target AI chip can be obtained to improve test efficiency.

[0095] S320: Acquire respectively data of the same data transfer type and the same data type, and perform measured bandwidth data when data transfer is performed between the same two caches, and construct a plurality of measured bandwidth data groups.

[0096] Specifically, each measured bandwidth data group contains data of multiple different tensor shapes under the same data transfer type and the same data type, and the measured bandwidth data when performing data transfer between the same two caches of the target AI chip.

[0097] Accordingly, each measured bandwidth data group may be named based on the set data transport type, the set data type, and the set two buffers.

[0098] For example, if measured bandwidth data group 1 is named "L3->L2_fp16_slice," this indicates that it contains the measured bandwidth data for data of various tensor shapes, with a slice transfer type and fp16 data type, when moving from the L3 cache (L3) to the L2 cache (L2) on the target AI chip. Alternatively, the tensor shape mentioned above can be understood as the tensor shape of the data in the L2 cache.

[0099] After obtaining the aforementioned measured bandwidth data groups, different bandwidth sub-models can be trained based on each measured bandwidth data group. Accordingly, the data in each measured bandwidth data group can be preprocessed, for example, by removing unreasonable data or distributing and shuffling the data. Afterward, the measured bandwidth data in each measured bandwidth data group can be divided into a training set and a test set.

[0100] S330: Add hardware attribute information of the matching direct access storage hardware in the target AI chip to each measured bandwidth data in each measured data group.

[0101] In this embodiment, to ensure a higher degree of correlation and compatibility between the final trained bandwidth model and the data handling hardware of the actual target AI chip, and thus to increase the accuracy of the resulting bandwidth data, hardware attribute information of the direct access storage hardware matching the target AI chip is added to the training and test samples in each measured bandwidth data group (i.e., each piece of measured bandwidth data).

[0102] In a specific example, the hardware attribute information of the DMA hardware may include data such as bus bandwidth, bus frequency, cache capacity, and cache level of the DMA hardware, which is not limited in this embodiment.

[0103] S340: Use each measured bandwidth data group to train a preset machine learning model to obtain multiple bandwidth sub-models, and organize the bandwidth sub-models to obtain the bandwidth model.

[0104] In this embodiment, the inventors have found through multiple experiments that when the Extreme Gradient Boosting (XGBoost) model or the Multi-Layer Perceptron (MLP) model is selected for model training to obtain a bandwidth model, the complex nonlinear relationship between input and output can be better fitted. Furthermore, the above-mentioned preset machine learning model can choose to use the XGBoost model or the MLP model.

[0105] As mentioned above, each measured bandwidth data group can be used to train the set XGBoost model or MLP model respectively to obtain a bandwidth sub-model corresponding to each measured bandwidth data group.

[0106] Furthermore, bandwidth sub-models can be constructed that correspond to a specific data transfer type, a specific data type, and two caches. For example, a bandwidth sub-model named L3->L2_fp16_slice_model can be used to predict the bandwidth of data with a specific tensor shape, whose data type is slice and whose data type is fp16, when moving from the third-level cache (L3) to the second-level cache (L2) on the target AI chip.

[0107] That is, after selecting the above L3->L2_fp16_slice_model based on a data transfer behavior of the bandwidth data to be predicted, we can input only the tensor shape in the data transfer behavior into the model, and then we can obtain the corresponding bandwidth data (that is, the prediction result of the actual bandwidth data).

[0108] The bandwidth sub-model corresponds to two buffers for setting a data transfer type, setting a data type, and performing data transfer.

[0109] S350. Obtain a computing core test program corresponding to each operator on the target AI chip.

[0110] As mentioned above, in order to predict computational time, it is necessary to first obtain the measured computational time data for various computational core operators. Furthermore, it is necessary to first build a computational core test program corresponding to each computational core operator on the target AI chip. This computational core test program can be understood as a program used to perform software simulation of the computational logic of a computational core operator of a given operator type.

[0111] S360. Run each computing core test program on the target AI chip respectively to measure the actual computing time consumed by the computing core of the target AI chip when performing various types of operator calculations on data of multiple data types and multiple tensor shapes on the first cache.

[0112] Similarly, this embodiment can pre-construct a universal computing core test program template corresponding to each computing core operator of each operator type, with the computing core test program template taking the data type and tensor shape as input variables. Furthermore, after reconstructing the computing core test program template corresponding to the target operator type using the set data type and set tensor shape, the reconstructed computing core test program, when run on the target chip, can obtain the computational time required to execute the computational logic corresponding to the target operator type on data of the set data type and set tensor shape.

[0113] S370: Obtain the measured calculation time consumption of the same operator type under the same data type respectively, and construct multiple groups of the measured calculation time consumption.

[0114] Specifically, each measured computation time group contains data of multiple tensor shapes with the same data type, and the computation time when performing calculations of the same operator type on the first-level cache.

[0115] Accordingly, each measured calculation time-consuming group can be named based on the set data type and the set operator type.

[0116] For example, if the measured computing time group 1 is named matmul_fp16, it means that the measured computing time group 1 stores data of various tensor shapes with a data type of fp16, and the measured computing time data when the operator type is matmul is executed on the first-level cache of the target AI chip.

[0117] After obtaining the aforementioned measured computation time groups, different performance sub-models can be trained based on each measured computation time group. Accordingly, the data in each measured computation time group can be preprocessed, for example, by removing unreasonable data or distributing and shuffling the data. Afterwards, the measured computation time data in each measured computation time group can be divided into training and test sets.

[0118] S380. Add hardware attribute information of the computing hardware in the target AI chip to each measured computing time of each measured computing time group.

[0119] In this embodiment, in order to ensure that the operator computing core performance model ultimately obtained through training is more closely aligned with and compatible with the actual target AI chip's computing hardware, and thus to achieve higher accuracy in the resulting computing time data, hardware attribute information of the target AI chip's computing hardware is added to each training sample and test sample in each measured computing time data group (i.e., each piece of measured computing time data).

[0120] In a specific example, the hardware attribute information of the computing hardware may include data such as the number of registers of each computing core (computing unit) and the chip register width, which is not limited in this embodiment.

[0121] S390. Use each measured computing time group to train the preset machine learning model to obtain multiple performance sub-models, and organize each performance sub-model to obtain the operator computing core performance model.

[0122] In this embodiment, the above-mentioned preset machine learning model can also be selected to use an XGBoost model or an MLP model. Accordingly, each measured computation time group can be used to train the set XGBoost model or MLP model respectively to obtain a performance sub-model corresponding to each measured computation time data group.

[0123] Furthermore, a performance sub-model can be constructed that corresponds to both a specified data type and a specified operator type. For example, a performance sub-model named matmul_fp16_model can be used to predict the computational time required to perform a matmul operation on a specific tensor shape with an fp16 data type in the target AI chip's L1 cache.

[0124] That is, after selecting the aforementioned matmul_fp16_model based on a data computation behavior for which the computational time data is to be predicted, we can input only the tensor shape of that data computation behavior (typically, data segmentation on L1) into the model to obtain the corresponding computational time data (that is, the predicted result of the actual computational time data).

[0125] The performance sub-model corresponds to the set operator type and the set data type.

[0126] S3100: Obtain multiple partitioning strategies for the operator to be evaluated between caches at various levels of the target AI chip based on operator characteristics of the operator to be evaluated in the target AI chip, storage characteristics of caches at various levels of the target AI chip, and at least one of computing core characteristics.

[0127] In this embodiment, the operator to be evaluated is taken as a data handling and calculation operator as an example to provide a detailed introduction to the total time consumption of the operator.

[0128] S3110. Based on a pre-built bandwidth model adapted to the target AI chip, calculate the data transfer time consumed by each splitting strategy of the operator to be evaluated when performing data transfer between caches at all levels.

[0129] S3120. Calculate the computational time of the operator to be evaluated when executing calculations for each splitting strategy on the first-level cache based on a pre-built operator computing core performance model adapted to the target AI chip.

[0130] S3130. Based on the calculation time and data transfer time, combined with the data transfer between different levels of cache included in the splitting strategy and the mutual hiding strategy when executing the calculation, calculate the total operator time of the operator to be evaluated for each splitting strategy, and perform operator performance evaluation on the operator to be evaluated based on the total operator time.

[0131] The technical solution of the embodiment of the present invention performs data fitting and learning on hardware characteristics by constructing a deep learning model, thereby realizing efficient evaluation of operator performance independent of the physical hardware environment, significantly improving the evaluation speed while ensuring the accuracy of the evaluation, and effectively reducing the dependence on and occupation of hardware resources. By using a deep learning model to fit each data type corresponding to each DMA operation between each level of cache, there is a corresponding model to predict the corresponding performance for each DMA operation in the operator, which is more accurate than using only one model to fit all data types on the entire DMA hardware; using a deep learning model to fit the performance of the computing kernel can quickly calculate the kernel performance of the corresponding tensor shape and data type without consuming hardware resources. In addition, for different kernels of different data types, each uses a deep learning model to fit its performance, which is more accurate than using a single model to predict the result. Furthermore, when training the bandwidth model and the operator computing kernel performance model, various software and hardware parameters are used in combination to make the evaluation more accurate.

[0132] Figure 4 This is a flowchart of another method for evaluating operator performance in an AI chip, provided by an embodiment of the present invention. This embodiment is optimized based on the above embodiments. Specifically, when the target AI chip has a three-level cache architecture, the operation of "obtaining multiple partitioning strategies for the operator to be evaluated across cache levels of the target AI chip based on at least one of the operator characteristics of the operator to be evaluated in the target AI chip, the storage characteristics of each cache level of the target AI chip, and the computing core characteristics" is concretized.

[0133] Correspondingly, such as Figure 4 As shown, the method may specifically include:

[0134] S410 : Determine, based on the operator calculation transfer type of the operator to be evaluated, the data calculation transfer form on the three-level cache architecture when the operator to be evaluated performs calculations.

[0135] In this embodiment, the operators to be evaluated are data handling and computation operators, and the target AI chip is implemented with a three-level cache architecture. Specifically, in descending order of proximity to the compute cores in the target AI chip, the three-level cache architecture consists of a level 1 cache (also known as L1 or L1 cache), a level 2 cache (also known as L2 or L2 cache), and a level 3 cache (also known as L3 or L3 cache).

[0136] In an optional implementation of this embodiment, the operator calculation transfer type of the operator to be evaluated includes: a first-type L3 operator and a second-type L3 operator.

[0137] Among them, the L3 operator can be understood as an operator that moves the data required for calculation from the third-level cache and finally moves it to the first-level cache to perform the calculation. At the same time, the calculation results will eventually be moved back to the third-level cache.

[0138] Correspondingly, the first data calculation and transfer form of the first type of L3 operator is to first transfer the data from the third-level cache to the second-level cache, then transfer it from the second-level cache to the first-level cache for calculation, and then transfer the calculation result back to the third-level cache;

[0139] The second data calculation and transfer form of the second type of L3 operator is to transfer data directly from the third-level cache to the first-level cache for calculation, and then transfer the calculation results back to the third-level cache.

[0140] It's understandable that the data splitting strategy for the first type of L3 operators, including data splitting across all cache levels, needs to include data splitting on the L3 cache (generally based on the tensor shape of the input data), data splitting on the L2 cache, and data splitting on the L1 cache. However, the data splitting strategy for the second type of L3 operators, including data splitting across all cache levels, only includes data splitting on the L3 cache and data splitting on the L1 cache.

[0141] In another optional implementation of this embodiment, the operator calculation transfer type of the operator to be evaluated may also include: L1 fusion operator (ie, fusion operator) and L2 fusion operator.

[0142] The L1 fusion operator can be understood as implementing operator fusion calculations only in the L1 cache. The other levels of cache are only used to split the data into multiple levels and pass it to the L1 cache for calculation. The L2 fusion operator can be understood as implementing operator fusion calculations through the cooperation of the L1 cache and the L2 cache.

[0143] Correspondingly, the third data calculation and transport form of the L1 fusion operator is the first data calculation and transport form or the second data calculation and transport form;

[0144] The fourth data calculation and transfer form of the L2 fusion operator is to first transfer the data from the third-level cache to the second-level cache, and then perform multiple loop operations between the second-level cache and the first-level cache, transferring the data from the second-level cache to the first-level cache for calculation and then transferring it back to the second-level cache until the final calculation result is obtained, and then the final calculation result is transferred back to the third-level cache.

[0145] It is understood that the data segmentation among various cache levels included in the segmentation strategy constructed for the L1 fusion operator may include data segmentation on the L3 cache, data segmentation on the L2 cache, and data segmentation on the L1 cache, or may only include data segmentation on the L3 cache and data segmentation on the L1 cache. The data segmentation among various cache levels included in the segmentation strategy constructed for the L2 fusion operator may include data segmentation on the L3 cache, data segmentation on the L2 cache, and data segmentation on the L1 cache.

[0146] In addition, for the special data transfer form of the L2 fusion operator, when calculating the final total time consumption of the operator, it is also necessary to refer to this special data transfer form and customize the calculation process of the total time consumption of the operator.

[0147] It's important to reiterate that existing technologies struggle to accurately evaluate the performance of even simple L3 operators. Fusion operators, which involve the computation of multiple operator cores and the complex data movement between caches, are even more challenging to evaluate. The various embodiments of the present invention creatively propose a novel approach to operator performance in AI chips. This approach not only enables effective operator evaluation for various types of L3 operators, but also for various fusion operators, achieving technical benefits unattainable with existing technologies.

[0148] S420. Obtain multiple partitioning strategies for the operator to be evaluated between cache levels of the target AI chip based on the data computation and transfer format, operator characteristics of the operator to be evaluated, and at least one of the storage characteristics and computing core characteristics of cache levels of the target AI chip.

[0149] In an optional implementation of this embodiment, based on the data computation and transport form, the operator characteristics of the operator to be evaluated, and the storage characteristics and computing core characteristics of the caches at each level of the target AI chip, obtaining multiple partitioning strategies for the operator to be evaluated between the caches at each level of the target AI chip may include:

[0150] S4201. Based on the operator calculation transport type, tensor shape, data type and calculation core constraint of the operator to be evaluated, as well as the cache size and pipeline strategy of the first-level cache, multiple first-class splits of the operator to be evaluated on the first-level cache are calculated, and multiple original strategies are constructed based on the initialization of each first-class split.

[0151] Compute core constraints can be understood as alignment constraints on each input shape of the operator, or alignment requirements for compute instructions, such as 32B alignment or 64B alignment. A pipeline strategy on the L1 cache specifically refers to a strategy that parallelizes data transfers on the L1 cache to improve data transfer throughput, such as a ping-pong strategy.

[0152] The first type of split on the L2 cache can be understood as the data block size of the data to be calculated on the L2. For example, the data size of the left operand is [64B, 128B], and the data size of the right operand is [128B, 96B]. These values ​​are determined by combining the operator calculation handling type, tensor shape, data type, and compute kernel constraints of the operator to be evaluated, as well as the cache size and pipeline strategy of the L2 cache.

[0153] That is, all of the above information can be used as constraints to determine the first type of split. For example, the first type of split needs to be smaller than the cache size of the first-level cache, or, if the pipeline strategy is a ping-pong strategy, the first type of split needs to be smaller than half of the cache size of the first-level cache. In addition, the first type of split also needs to meet the computing core constraint. Generally speaking, each computing core in the target AI chip is configured with a one-to-one L1 cache. The computing core is used to use a computing core operator that matches the operator type to perform calculations on input data of a set tensor shape. Furthermore, the split on the first-level cache is generally an integer multiple of the tensor shape. Accordingly, the first type of split and the tensor shape input to the computing core operator need to meet a preset proportional relationship.

[0154] Based on the above constraints, we can traverse and obtain all feasible first-class splits on the L1 cache. Furthermore, we can use each first-class split as a primitive policy, i.e., n1 primitive policies on L1.

[0155] S4202. When it is determined that the second type of split is included based on the data calculation and transportation form, multiple second type of splits corresponding to each first type of split are calculated respectively according to the cache size of the secondary cache, the pipeline strategy and the number of computing cores used when executing the operator calculation, and multiple update strategies are constructed based on the second type of splits corresponding to each first type of split.

[0156] When the data to be calculated is determined to need to be moved from L2 to L1 based on the data calculation and transfer format, it is determined that the data to be calculated needs to be appropriately partitioned in the L2 cache. In other words, the resulting partitioning strategy must include the second type of partitioning. The second type of partitioning can be understood as the data block size of the data to be calculated in L3.

[0157] Similarly, after using each first-category split as the initial strategy, you can use the L2 cache size, pipeline strategy, and the number of cores used for operator calculations as constraints to traverse all second-category splits corresponding to each first-category split. The number of cores used for operator calculations also needs to be pre-programmed into each split strategy.

[0158] Specifically, considering that each second-type split on L2 needs to be transferred to L1 multiple times, and the data transferred once by L2 often needs to satisfy the requirement that the computing cores used to execute the operator calculation can be used for one cycle, then, when determining the second-type split on L2, in addition to the various factors considered when determining the first-type split, it is necessary to further consider the number of computing cores used to execute the operator calculation. In a specific example, the second-type split on L2 can be an integer multiple of the product of the first-type split size * the number of computing cores used to execute the operator calculation.

[0159] In a specific example, if the second-type splits corresponding to the first-type split C1 are C21, C22, and C23, then three update strategies can be constructed: (C1, C21), (C1, C22), and (C1, C23). For example, for the n1 original strategies mentioned above, a total of n2 L2_L1 update strategies are obtained.

[0160] S4203: When the data transfer form is calculated based on the data and it is determined that the second type of segmentation is not included, the original strategy is directly determined as the update strategy.

[0161] Furthermore, if, based on the data calculation and transfer form, it is determined that the data to be calculated is directly transferred from L3 to L1 to implement the calculation of the corresponding operator type, then it means that the complete segmentation strategy does not include the second type of segmentation on L2. At this time, the original strategy on L1 can be directly determined as the update strategy.

[0162] S4204. Obtain the third type of segmentation according to the tensor shape of the input data.

[0163] Generally speaking, data segmentation is not performed in the three-level cache, and the tensor shape of the input data can be directly determined as the third type of segmentation. Specifically, the tensor shape of the input data can be one or more, and this embodiment does not limit this. Among them, the tensor shape of the input data can be understood as the actual data size of the large-scale input data required to be calculated by the operator to be evaluated, because the data size is much larger than the tensor shape that can be calculated by the computing core operator adapted to the operator to be evaluated. Therefore, it is necessary to perform multi-level segmentation of the tensor shape of the input data on multiple caches, and pass it to L1 step by step, and the computing core adapted to L1 uses the computing core operator to implement local data calculations.

[0164] S4205: Combine the third type of segmentation with each update strategy to obtain segmentation of the multiple data in the caches at each level, and form the multiple segmentation strategies based on the multiple segmentations obtained by the combination.

[0165] Specifically, by combining the first, second, and third types of splits, we can derive a data partitioning strategy across cache levels. By factoring in the number of cores used when executing operator calculations, the data movement between cache levels, and the mutual hiding strategies during calculations, we can form a partitioning strategy. In a specific example, if the input data tensor shape is unique, then the third type of split is unique. In this case, a strategy for updating n2 L2_L1s can yield a corresponding strategy for partitioning n2 L3_L2_L1s.

[0166] S430. Based on a pre-built bandwidth model adapted to the target AI chip, calculate the data transfer time consumed by each splitting strategy of the operator to be evaluated when performing data transfer between caches at various levels.

[0167] In an optional implementation of this embodiment, a specific calculation process of the data transfer time consumption is described in detail by taking the segmentation strategy including both the first type of segmentation and the second type of segmentation as an example. Specifically, a specific calculation process of the data transfer time consumption is described in detail by taking a target segmentation strategy as an example.

[0168] Accordingly, based on a pre-built bandwidth model adapted to the target AI chip, the data transfer time consumed by each splitting strategy of the operator to be evaluated when performing data transfer between cache levels can be calculated, which may include:

[0169] S4301. Obtain a target segmentation strategy from each segmentation strategy, and obtain a target first-category segmentation and a target second-category segmentation from the target segmentation strategy.

[0170] S4302: Based on the tensor shape, data type, and data transfer type in the target first-category segmentation, call the bandwidth model to obtain first bandwidth data when transferring data from the second-level cache to the first-level cache.

[0171] The first bandwidth data may be understood as the data transport volume per unit time when the data to be calculated is transported from L2 to L1.

[0172] Furthermore, based on the tensor shape, data type, and data transfer type in the target first-category split, calling the bandwidth model to obtain first bandwidth data when transferring data from the second-level cache to the first-level cache may include:

[0173] Obtain a target bandwidth sub-model in the bandwidth model based on the data type, data transfer type, and data transfer direction from the second-level cache to the first-level cache in the target first-category segmentation;

[0174] As mentioned above, since the bandwidth sub-model corresponds to the two buffers for setting the data transfer type, setting the data type, and setting the data transfer execution, after obtaining the corresponding tensor shape, data type, and data transfer type from the target first-category split, the target bandwidth sub-model can be obtained from each bandwidth sub-model included in the bandwidth model.

[0175] The tensor shape in the target first-category split and the hardware attribute information of the matching direct access storage hardware in the target AI chip are input into the target bandwidth sub-model to obtain the first bandwidth data output by the target bandwidth sub-model.

[0176] In this optional embodiment, when using the target bandwidth sub-model to predict the first bandwidth data, in addition to using the tensor shape in the target first-category split, information such as the bus bandwidth, bus frequency, and L1 and L2 cache capacity of the DMA hardware in the target AI chip is also used. This allows the prediction results to be closely associated with the data handling hardware in the target AI chip, resulting in higher accuracy of the prediction results.

[0177] S4303: Calculate the first data transfer time between the second-level cache and the first-level cache according to the first bandwidth data and the tensor shape in the target first-type segmentation.

[0178] When determining the amount of data required to be moved from L2 to L1 per unit time, combined with the tensor shape in the target first-category split, the data movement time required to move the data blocks on L2 to L1 according to the first bandwidth data, i.e., the first data movement time, can be calculated accordingly.

[0179] Specifically, the total amount of data to be transported in a single time can be determined based on the tensor shape in the target first type of segmentation, and then the total amount of data is divided by the first bandwidth data to obtain the first data transport time.

[0180] S4304: Based on the tensor shape, data type, and data transfer type in the target second-category split, call the bandwidth model to obtain second bandwidth data when transferring data from the third-level cache to the second-level cache.

[0181] Similarly, the second bandwidth data can be understood as the data transport volume per unit time when the data to be calculated is transported from L3 to L2.

[0182] Furthermore, based on the tensor shape, data type, and data transfer type in the target second-type split, the bandwidth model is called to obtain the second bandwidth data when transferring data from the third-level cache to the second-level cache, which may specifically include:

[0183] According to the data type, data transfer type, and data transfer direction from the third-level cache to the second-level cache in the target second-category split, the target bandwidth sub-model is obtained in the bandwidth model; the tensor shape in the target second-category split and the hardware attribute information of the matching direct access storage hardware in the target AI chip are input into the target bandwidth sub-model to obtain the second bandwidth data output by the target bandwidth sub-model.

[0184] S4305: Calculate the second data transfer time between the L3 cache and the L2 cache according to the second bandwidth data and the tensor shape in the target second-type segmentation.

[0185] Similarly, the total amount of data that needs to be moved from L3 to L2 can be determined based on the tensor shape in the target second-type split. Then, the total amount of data is divided by the second bandwidth data to obtain the second data transfer time between the third-level cache and the second-level cache.

[0186] S440. Calculate the computational time of the operator to be evaluated when executing calculations for each splitting strategy on the first-level cache based on a pre-built operator computing core performance model adapted to the target AI chip.

[0187] In an optional implementation of this embodiment, based on a pre-built operator computing core performance model adapted to the target AI chip, calculating the computation time consumed by the operator to be evaluated when executing the computation for each slicing strategy on the L1 cache may include:

[0188] S4401. In the target segmentation strategy, obtain the first type of target segmentation.

[0189] S4402: Obtain a target operator calculation core model from the operator calculation core performance model according to the data type in the target first-category segmentation and the operator type of the operator to be evaluated.

[0190] As previously mentioned, the operator calculation core performance model includes multiple operator calculation core sub-models, each of which corresponds to a specified operator type and data type. Furthermore, the target operator calculation core sub-model can be uniquely obtained from all operator calculation core performance models based on the data type in the target first-category split and the operator type of the operator to be evaluated.

[0191] S4403. Input the tensor shape in the first category of target segmentation and the hardware attribute information of the computing hardware in the target AI chip into the target performance sub-model to obtain the target calculation time corresponding to the target segmentation strategy output by the target operator calculation core model.

[0192] In this embodiment, in the process of using the target performance sub-model to predict the target calculation time of the target segmentation strategy, in addition to using the tensor shape in the target first-category segmentation, the hardware attribute information of the computing hardware in the target AI chip is also used, that is, the chip hardware information such as the number of registers and register width of each computing core in the target AI chip is also used. This can make the prediction result closely related to the computing hardware in the target AI chip, and the prediction result is more accurate.

[0193] S450 , based on each calculation time consumption and each data transfer time consumption, combined with the second hidden strategy included in the segmentation strategy, calculate the total operator time consumption of the operator to be evaluated for each segmentation strategy, and perform operator performance evaluation on the operator to be evaluated based on the total operator time consumption.

[0194] In an optional implementation of this embodiment, based on the respective computational time consumptions and the respective data handling time consumptions, combined with the second hidden strategy included in the segmentation strategy, calculating the total operator time consumption of the operator to be evaluated for each segmentation strategy may include:

[0195] S4501. For the target segmentation strategy, calculate the number of first-loop transfers in the first-level cache and the number of second-loop transfers in the second-level cache based on the tensor size of the input data, the tensor shape in the target first-category segmentation, the tensor shape in the target second-category segmentation, and the number of computing cores contained in the target AI chip.

[0196] The first-cycle transfer count in the L1 cache can be understood as the total number of transfers required to transfer all the data required for calculation from L2 to L1. The second-cycle transfer count in the L2 cache can be understood as the total number of transfers required to transfer all the data required for calculation from L3 to L2.

[0197] Specifically, after obtaining the tensor size of the input data, the total amount of data required for calculation (or data required for data transfer) can be determined. Combined with the target second-type split, the number of second-loop transfers in the L2 cache can be calculated.

[0198] Furthermore, as previously mentioned, when moving data from L2 to L1, each move must be sufficient to be used by all cores used to execute the operator calculation for at least one cycle. Therefore, the number of first-cycle moves in the first cache can be calculated based on the tensor shape in the target first-class split and the number of cores used to execute the operator calculation.

[0199] S4502. Based on the tensor shape in the first type of target segmentation, the tensor shape in the second type of target segmentation, the pipeline strategy of the first-level cache, the pipeline strategy in the second-level cache, the first data transfer time, the second data transfer time, the first loop transfer times, the second loop transfer times and the target calculation time, combined with the second hidden strategy included in the segmentation strategy, calculate the total operator time of the operator to be evaluated for the target segmentation strategy.

[0200] In this embodiment, after obtaining the pipeline strategy of the first-level cache and the pipeline strategy in the second-level cache, for example, both use the ping-pong strategy, and then combine the specific data included in the splitting strategy to move between different levels of cache and the mutual hiding strategy when performing calculations, by using the above-mentioned data movement time, calculation time and number of loop movement times, the total operator time of the operator to be evaluated for the target splitting strategy can be calculated accordingly.

[0201] More generally, if different operations on the target AI chip can be executed in parallel by different hardware units, the time consumed by these two operations can be hidden from each other. Furthermore, when the above-mentioned operations performed by different hardware units are data transfer and data calculation between multiple levels of cache, the time it takes for the target AI chip to perform data transfer between different levels of cache can be hidden from each other, and the time it takes for the target AI chip to perform data transfer and the time it takes to perform data calculation can also be hidden from each other. In a specific example, if the operator calculation transfer type of the operator to be evaluated is a first-class L3 operator, the mutual hiding strategy for data transfer between different levels of cache and the execution of calculations is determined by the two hardware features of the bidirectionality of data transfer by DMA hardware and the fact that the data transfer and calculation operations of the target AI chip can be performed in parallel by different hardware units. Accordingly, combined with the second hiding strategy, the operation of calculating the total operator time consumed by the operator to be evaluated for the target segmentation strategy can be concretized as follows:

[0202] If the pipeline strategy of the first-level cache is the ping-pong strategy, then if the target calculation time is greater than the first data transfer time, the target calculation time is determined as the first-level cache sub-time; otherwise, the first data transfer time is determined as the first-level cache sub-time;

[0203] Calculate the rounded value of the tensor shape in the target second-class split divided by the tensor shape in the target first-class split as the number of L1 cache sub-cycles, and calculate the L1 cache time obtained by multiplying the L1 cache sub-time by the number of L1 cache sub-cycles;

[0204] If the pipeline strategy of the second-level cache is the ping-pong strategy, then when the first-level cache time is greater than or equal to the second data transfer time, the cumulative sum of the preset data transfer pre-data, the first data transfer time, the second data transfer time, the time value obtained by multiplying the first-level cache time by the number of first cycle transfers, and the preset last storage time is calculated as the total operator time of the first type L3 operator for the target splitting strategy;

[0205] If the pipeline strategy of the second-level cache is the ping-pong strategy, then when the first-level cache time is less than the second data transfer time, the cumulative sum of the preset data transfer pre-data, the second loop transfer times multiplied by the second data transfer time, the first-level cache time and the preset last storage time is calculated as the total operator time of the first-type L3 operator for the target splitting strategy.

[0206] In another specific example, if the operator calculation transfer type of the operator to be evaluated is a second-class L3 operator, then the target splitting strategy only includes the target first-class split on the first-level cache and the target second-class split on the third-level cache. Furthermore, the data transfer time required to transfer all the data required for calculation from L3 to L1, the number of cyclic transfers of data on L1, and the calculation time required for calculation of data on L1 can be calculated accordingly. Combined with the above information, the pipeline strategy of the first-level cache and the mutual hiding strategy of data transfer between different levels of cache and when performing calculations, the total operator time of the second-class L3 operator for the target splitting strategy can also be calculated.

[0207] Similarly, when the operator calculation and transfer type of the operator to be evaluated is the L1 fusion operator or the L2 fusion operator, the total operator time of the L1 fusion operator or the L2 fusion operator for the target segmentation strategy can also be calculated based on the specific segmentation strategy and data calculation and transfer form.

[0208] Finally, the main technical points and innovations of each embodiment of the present invention are summarized as follows:

[0209] First, each embodiment of the present invention divides the operator time consumption into two parts: the DMA data transfer time consumption and the kernel data calculation time consumption. The time consumption of these two parts is difficult to quickly and accurately evaluate the performance using traditional modeling methods in the case of large-scale data.

[0210] The time it takes to move the DAM portion of data for the same input tensor shape (also called shape) is affected by hardware parameters such as the AI ​​chip's cache hierarchy, bus bandwidth, and DMA controller design. These hardware parameters vary significantly between different AI chips, making it difficult to establish a unified model for the DMA of different chips using traditional methods. This requires remodeling for each generation of chips, which is inefficient. Using machine learning models or deep learning models to establish a bandwidth evaluation model for DMA between different caches can effectively solve this problem. Furthermore, the computational time of the kernel portion of the same input tensor shape is affected by hardware parameters such as the AI ​​chip's computational instruction efficiency, instruction alignment, register width, number of registers, and instruction packing. The influencing conditions are complex, and these hardware parameters vary significantly between different AI chips. It's difficult to establish a unified traditional model to evaluate kernel performance across different chips, requiring remodeling for each generation of chips based on the chip architecture. Using machine learning models or deep learning models to establish computing power evaluation models for different kernels can effectively solve this problem.

[0211] Based on this, the various embodiments of the present invention combine the hardware architecture (including the chip's cache level, DMA characteristics, and the characteristics of the chip's computing instructions) to generate a split space executed on the chip corresponding to the operator input shape. DMA models at different levels are used in combination with DMA hardware parameters to calculate the bandwidth and time consumption of transfers between different caches in the split shape. The kernel model is used in combination with chip instructions and register parameters to calculate the kernel time consumption corresponding to the L1 shape in the split. Finally, the time consumption corresponding to each split in the operator split space is forward calculated by combining the flow between data transfer and calculation. The results are accurate, reliable, and well-founded, and the operator performance evaluation is accurate.

[0212] Furthermore, the embodiments of the present invention do not need to implement operator templates. They only need to simply establish an operator calculation model based on the chip architecture and operator calculation process. They can use a deep learning model or a machine learning model on the CPU (without using the computing power resources of the actual AI chip) to test the performance corresponding to all splitting strategies, thereby calculating the operator performance. This can quickly and comprehensively evaluate the operator performance, solving the above-mentioned problem.

[0213] Accordingly, the main innovations of the embodiments of the present invention include:

[0214] 1. By building a deep learning model to fit and learn hardware characteristics, efficient evaluation of operator performance can be achieved without the physical hardware environment. This significantly improves the evaluation speed while ensuring evaluation accuracy and effectively reduces dependence on and occupation of hardware resources.

[0215] 2. A deep learning model is used to fit each data type corresponding to each DMA operation between each level of cache. Each DMA operation in the operator has a corresponding model to predict the corresponding performance, which is more accurate than using only one model to fit all data types on the entire DMA hardware.

[0216] 3. When training a DMA deep learning model, the model input includes the following: input tensor shape, data type (dtype), chip bus bandwidth, bus frequency, cache capacity, and cache level. The model input includes software input and corresponding hardware parameters. The model output is the predicted bandwidth. Combining software and hardware parameters makes the evaluation more accurate.

[0217] 4. Use deep learning model fitting to calculate kernel performance, which can quickly calculate the kernel performance of the corresponding tensor shape and data type without consuming hardware resources.

[0218] 5. When training a kernel model, the model input includes the following: input shape, data dtype, number of chip registers, and chip register width. The model input includes software input and corresponding hardware parameters. The model output is the actual kernel computing power corresponding to the shape and dtype. Combining software and hardware parameters makes the evaluation more accurate.

[0219] 6. For different computing kernels of different data types, using a deep learning model to fit their performance is more accurate than using a single model to predict the results.

[0220] 7. By splitting the operator performance into the DMA part and the kernel part and evaluating them using different models, it is more accurate than using one model to directly calculate the performance of the entire operator.

[0221] 8. Currently, testing operator performance requires implementing an operator template for the corresponding data stream and then running the operator template on a specific computing device (i.e., the target AI chip mentioned above) to obtain the corresponding performance. Different data streams require different operator templates, which requires a significant amount of manpower. The embodiments of the present invention only require the corresponding DMA tensor size and kernel tensor size, using a DMA model to predict DMA performance, and a kernel model to predict kernel performance. This eliminates the limitations of existing technologies that require implementing operator templates and running on the computing device, significantly saving human and equipment resources.

[0222] 9. The embodiments of the present invention can calculate individual L3 operators or fusion operators at all levels. The performance evaluation of fusion operators is difficult because it involves the movement of data of different shapes between different caches among multiple kernels. The methods of the embodiments of the present invention can quickly evaluate the optimal performance of the fusion operator.

[0223] 10. The embodiments of the present invention get rid of the limitation of the prior art that operator templates must be used to run on the computing power device side, and the operator performance corresponding to the splitting strategy can be calculated on the CPU or GPU.

[0224] 11. In the process of generating the segmentation space, each embodiment of the present invention will consider the influence of the operator implementation logic and chip hardware parameters. The chip hardware parameters include: the size of cache capacity at each level, the alignment requirements of computing instructions, the pipeline arrangement during the operator execution process, and the ping-pong strategy, etc.

[0225] 12. The logic for calculating the partitioning space in each embodiment of the present invention is as follows:

[0226] Legal splits on L1 are calculated based on the size of L1, the ping-pong strategy, the alignment requirements for computational instructions, and the operator input dtype. Legal splits on L2 are calculated based on the size of L2, legal splits on L1, and different pipeline layout strategies. This calculation method ensures that the entire generated operator split strategy is legal and valid, effectively eliminating illegal splits and saving evaluation time.

[0227] 13. Current hardware tuners (also known as tunners) calculate the splitting strategies in the splitting space. This requires specific operator templates to be implemented and run on real AI chips to test the performance of the corresponding splitting strategies and select the optimal splitting strategy. This prerequisite means that if you want to test all splitting strategies in the splitting space, you must implement all operator templates and run all the splitting strategies in a large environment for a long time. In practice, this is almost impossible to achieve. One reason is that it is impossible to implement all splitting strategies, which requires a huge amount of manpower. Another reason is that even if all splitting strategies are implemented, running all splitting strategies on AI chips for all tensor shapes in all models will consume a huge amount of environmental resources. Therefore, the hardware tuner will trim the splitting space and only select the splitting strategies with implemented templates. This inevitably trims the best-performing splitting strategies for certain tensor shapes, resulting in suboptimal performance results. Since this patent breaks away from the limitation of having to use operator templates to run on the AI ​​chip side, it can use CPU or GPU multi-threaded calculations to calculate all the segmentation strategies in the segmentation space without any trimming. This way, no segmentation strategies will be missed, and the optimal performance of the operator can be calculated more comprehensively. It also saves a lot of manpower to develop operator templates and run tests in a large number of environments.

[0228] 14. Each embodiment of the present invention calculates the time consumption of each part of the operation in the generated split, uses the DMA model to calculate the time consumption of all DMA transfer parts in the split, uses the kernel calculation model to calculate the kernel time consumption, combines the pipeline arrangement strategy and the ping-pong strategy to forwardly calculate the operator time consumption, adopts the deterministic calculation flow, and derives the time consumption composition of each part of the split in the operator split strategy step by step to ensure that the results are traceable and credible.

[0229] 15. The embodiments of the present invention support the root cause analysis of performance bottlenecks (such as insufficient bus bandwidth, cache size or unfriendly instruction alignment) through the explicit mapping relationship between hardware parameters and model inputs, providing a quantitative basis for chip design iteration.

[0230] 16. Each embodiment of the present invention will calculate the performance of all splitting strategies of the operator, can perform calculations on typical tensor shapes, obtain the performance of all splitting strategies, summarize the rules of splitting strategies with good performance, extract typical data streams, and provide guidance for subsequent operator implementation.

[0231] 17. In addition to outputting the time-consuming data of the tensor shape corresponding to the operator, each embodiment of the present invention will also output the optimal division of the operator on the chip and the optimal number of computing cores used when executing the operator calculation, providing a solution for optimizing the operator to the corresponding performance.

[0232] 18. In the embodiments of the present invention, before the chip is in mass production, the DMA performance and the corresponding kernel performance can be tested and obtained, and the performance of the operator on the newly developed chip and the corresponding segmentation strategy can be obtained through calculation, which can guide the implementation of the operator and save the trial and error cost in the process of implementing the operator on the new chip.

[0233] Figure 5 This is a schematic diagram of the structure of an operator performance evaluation device in an AI chip provided by an embodiment of the present invention. Figure 5 As shown, the device includes: a segmentation strategy determination module 510, a transport time calculation module 520 and a first performance evaluation module 530, wherein:

[0234] A partitioning strategy determination module 510 is configured to obtain multiple partitioning strategies for the operator to be evaluated between cache levels of the target AI chip based on at least one of the operator characteristics of the operator to be evaluated in the target AI chip, the storage characteristics of the cache levels of the target AI chip, and the computing core characteristics;

[0235] The data transfer time calculation module 520 is used to calculate the data transfer time when each splitting strategy of the operator to be evaluated performs data transfer between cache levels based on a pre-built bandwidth model adapted to the target AI chip;

[0236] The bandwidth model is used to predict bandwidth data when transferring data of a set data transfer type, a set data type, and a set tensor shape between two set caches in the target AI chip;

[0237] The first performance evaluation module 530 is used to calculate the total operator time of the operator to be evaluated for each splitting strategy based on the time consumption of each data transfer and the first hidden strategy included in the splitting strategy if the operator to be evaluated is a pure data handling operator, and perform operator performance evaluation on the operator to be evaluated based on the total time consumption of each operator.

[0238] The technical solution of the embodiment of the present invention realizes efficient evaluation of operator performance by constructing a bandwidth model and an operator computing core performance model to effectively fit and learn various bandwidth data and computing time data, significantly improves the evaluation speed while ensuring the accuracy of the evaluation, and can effectively reduce the dependence on and occupation of hardware resources in the AI ​​chip. In addition, the technical solution of the embodiment of the present invention does not require manpower to implement the operator template, and the calculation part performed based on the bandwidth model and the operator computing core performance model does not need to run on the AI ​​chip, which can save a lot of human resources and equipment resources and effectively save evaluation time. At the same time, the solution of the embodiment of the present invention can provide optimization guidance for the operator implementation of the operator in the AI ​​chip when the AI ​​chip has not yet been mass-produced, provide an effective solution for optimizing the operator to the optimal performance, and save the trial and error cost of implementing the operator on the new chip.

[0239] On the basis of the above embodiments, the following may also be included:

[0240] A computational time calculation module is used to calculate the data transfer time of each splitting strategy of the operator to be evaluated when performing data transfer between the various cache levels. If the operator to be evaluated is a data transfer and computation operator, the computational time of the operator to be evaluated when executing the computation for each splitting strategy in the first-level cache is calculated based on a pre-built operator computing core performance model adapted for the target AI chip. The first-level cache is set close to the computing core in the target AI chip.

[0241] The operator computing core performance model is used to predict the computational time required to execute a set type of operator calculation on data of a set data type and a set tensor shape in the target AI chip.

[0242] The second performance evaluation module is used to calculate the total operator time of the operator to be evaluated for each splitting strategy based on the calculation time and the data transfer time, combined with the second hiding strategy included in the splitting strategy, and perform operator performance evaluation on the operator to be evaluated based on the total operator time. Based on the above embodiments, when the operator to be evaluated is a pure data transfer operator, the splitting strategy specifically includes: data segmentation in caches at all levels, and a first hiding strategy for hiding data from each other when transferring between caches at different levels;

[0243] When the operator to be evaluated is a data handling and computing operator, the splitting strategy specifically includes: data splitting in caches at all levels, the number of computing cores used when executing operator calculations, and a second hiding strategy in which data is hidden from each other when being moved between caches at different levels and when performing calculations.

[0244] On the basis of the above embodiments, the following may also be included:

[0245] The measured bandwidth data measurement module is used to run a bandwidth test program adapted to the target AI chip on the target AI chip before calculating the data transfer time for each splitting strategy of the operator to be evaluated when performing data transfer between cache levels based on a pre-built bandwidth model adapted to the target AI chip. The module measures the measured bandwidth data when performing data transfer between cache levels of the target AI chip under multiple data transfer types, multiple data types, and multiple tensor shape combinations.

[0246] The bandwidth model construction module is used to construct a bandwidth model of the target AI chip based on the measured bandwidth data.

[0247] Based on the above embodiments, the bandwidth model construction module can be specifically used to:

[0248] Obtaining respectively the measured bandwidth data of data of the same data transfer type and the same data type when performing data transfer between the same two caches, and constructing a plurality of measured bandwidth data groups;

[0249] In each measured bandwidth data of each measured data group, hardware attribute information of the matching direct access storage hardware in the target AI chip is added;

[0250] Using each measured bandwidth data group to train a preset machine learning model to obtain multiple bandwidth sub-models, and organizing each bandwidth sub-model to obtain the bandwidth model;

[0251] The bandwidth sub-model corresponds to two buffers for setting a data transfer type, setting a data type, and performing data transfer.

[0252] On the basis of the above embodiments, the following may also be included:

[0253] A computing core test program acquisition module is used to obtain the computing core test program corresponding to each operator on the target AI chip before calculating the computational time of the operator to be evaluated when executing the computation for each slicing strategy on the first-level cache based on a pre-built operator computing core performance model adapted for the target AI chip;

[0254] A measured computation time measurement module, configured to run a test program for each computing core on the target AI chip and measure the measured computation time when the computing core of the target AI chip performs various types of operator calculations on data of multiple data types and multiple tensor shapes on the first cache;

[0255] The operator computing core performance model construction module is used to construct the operator computing core performance model of the target AI chip based on the measured calculation time.

[0256] Based on the above embodiments, the operator calculation core performance model construction module can be specifically used to:

[0257] Obtain the measured computation time of the same operator type under the same data type, and construct multiple groups of measured computation time.

[0258] The hardware attribute information of the computing hardware in the target AI chip is added to each measured computing time of each measured computing time group;

[0259] The preset machine learning model is trained using each measured computing time group to obtain multiple performance sub-models, and each performance sub-model is organized to obtain the operator computing core performance model;

[0260] The performance sub-model corresponds to the set operator type and the set data type.

[0261] Based on the above embodiments, the target AI chip has a three-level cache architecture, which is composed of a first-level cache, a second-level cache, and a third-level cache in the order from closest to the computing core in the target AI chip.

[0262] Accordingly, the segmentation strategy determination module 510 may specifically include:

[0263] a data calculation and transport form determination unit, configured to determine, based on the operator calculation and transport type of the operator to be evaluated, the data calculation and transport form of the data on the three-level cache architecture when the operator to be evaluated performs the calculation;

[0264] A partitioning strategy acquisition unit is configured to acquire multiple partitioning strategies for the operator to be evaluated across cache levels of the target AI chip based on the data computation and transport format, the operator characteristics of the operator to be evaluated, and at least one of the storage characteristics and computing core characteristics of the cache levels of the target AI chip.

[0265] On the basis of the above embodiments, the operator calculation transport type of the operator to be evaluated may include: a first type L3 operator and a second type L3 operator;

[0266] The first data calculation and transfer form of the first type of L3 operator is to first transfer the data from the third level cache to the second level cache, then transfer it from the second level cache to the first level cache for calculation, and then transfer the calculation results back to the third level cache;

[0267] The second data calculation and transfer form of the second type of L3 operator is to transfer data directly from the third-level cache to the first-level cache for calculation, and then transfer the calculation results back to the third-level cache.

[0268] On the basis of the above embodiments, the operator calculation transport type of the operator to be evaluated may further include: L1 fusion operator and L2 fusion operator;

[0269] The third data calculation and transport form of the L1 fusion operator is the first data calculation and transport form or the second data calculation and transport form;

[0270] The fourth data calculation and transfer form of the L2 fusion operator is to first transfer the data from the third-level cache to the second-level cache, and then perform multiple loop operations between the second-level cache and the first-level cache, transferring the data from the second-level cache to the first-level cache for calculation and then transferring it back to the second-level cache until the final calculation result is obtained, and then the final calculation result is transferred back to the third-level cache.

[0271] Based on the above embodiments, the segmentation strategy acquisition unit can be specifically used to:

[0272] Based on the operator computation handling type, tensor shape, data type, and computation core constraints of the operator to be evaluated, as well as the cache size and pipeline strategy of the first-level cache, multiple first-class splits of the operator to be evaluated on the first-level cache are calculated, and multiple primitive strategies are initialized based on each first-class split.

[0273] When the data calculation and transfer form determines that the second type of split is included, multiple second type splits corresponding to each first type of split are calculated based on the cache size of the secondary cache, the pipeline strategy, and the number of computing cores used when executing the operator calculation. Multiple update strategies are then constructed based on each second type split corresponding to each first type of split.

[0274] When the data is calculated based on the transport form and it is determined that the second type of segmentation is not involved, the original strategy is directly determined as the update strategy;

[0275] Get the third type of split based on the tensor shape of the input data;

[0276] The third type of segmentation is combined with each update strategy to obtain segmentation of multiple data in caches at various levels, and the multiple segmentation strategies are formed based on the multiple segmentations obtained by the combination.

[0277] Based on the above embodiments, the transport time calculation module 520 can be specifically used to:

[0278] Obtain a target segmentation strategy from each segmentation strategy, and obtain a target first-category segmentation and a target second-category segmentation from the target segmentation strategy;

[0279] Based on the tensor shape, data type, and data transfer type in the target first-category split, the bandwidth model is called to obtain the first bandwidth data when transferring data from the second-level cache to the first-level cache.

[0280] Calculate the first data transfer time between the second-level cache and the first-level cache according to the first bandwidth data and the tensor shape in the target first-category split;

[0281] Based on the tensor shape, data type, and data transfer type in the target second-category split, the bandwidth model is called to obtain the second bandwidth data when transferring data from the third-level cache to the second-level cache.

[0282] A second data transfer time between the L3 cache and the L2 cache is calculated according to the second bandwidth data and the shape of the tensor in the target second-type segmentation.

[0283] Based on the above embodiments, the transport time calculation module 520 can be specifically used to:

[0284] In the target segmentation strategy, obtain the first type of target segmentation;

[0285] According to the data type in the target first-category segmentation and the operator type of the operator to be evaluated, the target operator calculation core model is obtained in the operator calculation core performance model;

[0286] The tensor shape in the target first-category split and the hardware attribute information of the computing hardware in the target AI chip are input into the target performance sub-model to obtain the target computation time corresponding to the target split strategy output by the target operator computation core model.

[0287] Based on the above embodiments, the transport time calculation module can be further used to:

[0288] Obtain a target bandwidth sub-model in the bandwidth model based on the data type, data transfer type, and data transfer direction from the second-level cache to the first-level cache in the target first-category segmentation;

[0289] The tensor shape in the target first-category split and the hardware attribute information of the matching direct access storage hardware in the target AI chip are input into the target bandwidth sub-model to obtain the first bandwidth data output by the target bandwidth sub-model.

[0290] Based on the above embodiments, the second performance evaluation module can be specifically used to:

[0291] For the target partitioning strategy, the first-loop transfer count in the L1 cache and the second-loop transfer count in the L2 cache are calculated based on the input data tensor size, the tensor shape in the target first-type partitioning, the tensor shape in the target second-type partitioning, and the number of compute cores in the target AI chip.

[0292] According to the tensor shape in the first type of target segmentation, the tensor shape in the second type of target segmentation, the pipeline strategy of the first-level cache, the pipeline strategy in the second-level cache, the first data handling time, the second data handling time, the first loop handling times, the second loop handling times and the target calculation time, combined with the second hidden strategy included in the segmentation strategy, the total operator time of the operator to be evaluated for the target segmentation strategy is calculated.

[0293] Based on the above embodiments, the first performance evaluation module 530 can be specifically used to:

[0294] Obtain the optimal splitting strategy corresponding to the minimum total operator time;

[0295] According to the various strategy information contained in the optimal segmentation strategy, the operator to be evaluated is optimized and guided so that the operator to be evaluated achieves the best performance.

[0296] The operator performance evaluation device in the AI ​​chip provided by the embodiment of the present invention can execute the operator performance evaluation method in the AI ​​chip provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0297] In the technical solutions of the embodiments of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0298] Figure 6A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0299] like Figure 6 As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0300] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0301] Processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 executes the various methods and processes described above, such as the operator performance evaluation method in an AI chip as described in any of the embodiments of the present invention.

[0302] That is, based on the operator characteristics of the operator to be evaluated in the target AI chip, the storage characteristics of the caches at each level of the target AI chip, and at least one of the computing core characteristics, multiple partitioning strategies for the operator to be evaluated between the caches at each level of the target AI chip are obtained;

[0303] Based on a pre-built bandwidth model adapted to the target AI chip, the data transfer time between cache levels is calculated for each splitting strategy of the operator to be evaluated.

[0304] The bandwidth model is used to predict bandwidth data when transferring data of a set data transfer type, a set data type, and a set tensor shape between two set caches in the target AI chip;

[0305] If the operator to be evaluated is a pure data handling operator, then the total operator time of the operator to be evaluated for each splitting strategy is calculated based on the data handling time and the first hidden strategy included in the splitting strategy, and the operator performance of the operator to be evaluated is evaluated based on the total operator time.

[0306] In some embodiments, the method for evaluating the performance of an operator in an AI chip as described in any of the embodiments of the present invention may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for evaluating the performance of an operator in an AI chip as described in any of the embodiments of the present invention described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for evaluating the performance of an operator in an AI chip as described in any of the embodiments of the present invention by any other appropriate means (for example, by means of firmware).

[0307] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0308] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0309] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0310] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0311] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0312] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0313] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0314] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for evaluating operator performance in an AI chip, characterized in that: include: Obtain multiple partitioning strategies for the operator to be evaluated across cache levels of the target AI chip based on at least one of the operator characteristics of the operator to be evaluated in the target AI chip, the storage characteristics of cache levels of the target AI chip, and the computing core characteristics; Based on a pre-built bandwidth model adapted to the target AI chip, the data transfer time between cache levels is calculated for each splitting strategy of the operator to be evaluated. The bandwidth model is used to predict bandwidth data when transferring data of a set data transfer type, a set data type, and a set tensor shape between two set caches in the target AI chip; If the operator to be evaluated is a pure data handling operator, then the total operator time consumed by the operator to be evaluated for each splitting strategy is calculated based on the data handling time consumed by each operator and the first hidden strategy included in the splitting strategy. The operator performance of the operator to be evaluated is then evaluated based on the total operator time consumed by each operator. After calculating the data transfer time between cache levels for each splitting strategy of the operator to be evaluated, the following is also included: If the operator to be evaluated is a data handling and computing operator, the computational time of the operator to be evaluated when executing calculations for each sharding strategy on the L1 cache is calculated based on a pre-built operator computing core performance model adapted for the target AI chip. The L1 cache is set close to the computing core in the target AI chip. The operator computing core performance model is used to predict the computational time required to execute a set type of operator calculation on data of a set data type and a set tensor shape in the target AI chip. According to each calculation time consumption and each data transfer time consumption, combined with the second hidden strategy included in the segmentation strategy, the total operator time consumption of the operator to be evaluated for each segmentation strategy is calculated, and the operator performance of the operator to be evaluated is evaluated based on the total operator time consumption.

2. The method according to claim 1, characterized in that When the operator to be evaluated is a pure data transfer operator, the partitioning strategy specifically includes: partitioning the data in caches at all levels, and a first hiding strategy in which data is hidden from each other when transferred between caches at different levels; When the operator to be evaluated is a data handling and computing operator, the splitting strategy specifically includes: data splitting in caches at all levels, the number of computing cores used when executing operator calculations, and a second hiding strategy in which data is hidden from each other when being moved between caches at different levels and when performing calculations.

3. The method according to claim 1, characterized in that Before calculating the data transfer time between cache levels for each splitting strategy of the operator to be evaluated based on a pre-built bandwidth model adapted to the target AI chip, the following is also included: Run a bandwidth test program adapted for the target AI chip on the target AI chip to measure the actual bandwidth of data transfers between cache levels on the target AI chip, across multiple data transfer types, data types, and tensor shape combinations. Based on the measured bandwidth data, a bandwidth model of the target AI chip is constructed.

4. The method according to claim 3, characterized in that Based on the measured bandwidth data, a bandwidth model of the target AI chip is constructed, including: Obtaining respectively the measured bandwidth data of data of the same data transfer type and the same data type when performing data transfer between the same two caches, and constructing a plurality of measured bandwidth data groups; In each measured bandwidth data of each measured data group, hardware attribute information of the matching direct access storage hardware in the target AI chip is added; Using each measured bandwidth data group to train a preset machine learning model to obtain multiple bandwidth sub-models, and organizing each bandwidth sub-model to obtain the bandwidth model; The bandwidth sub-model corresponds to two buffers for setting a data transfer type, setting a data type, and performing data transfer.

5. The method according to claim 1, wherein Before calculating the computational time of the operator to be evaluated for each splitting strategy on the L1 cache based on the pre-built operator computing core performance model adapted to the target AI chip, the following is also included: Obtain the computing core test program corresponding to each operator on the target AI chip; Run the test program for each computing core on the target AI chip to measure the actual computation time of the computing core of the target AI chip when executing various types of operator calculations on data of multiple data types and multiple tensor shapes in the first cache. Based on the measured calculation time, the operator computing core performance model of the target AI chip is constructed.

6. The method according to claim 5, characterized in that Based on the measured computation time, a performance model of the operator computing core of the target AI chip is constructed, specifically including: Obtain the measured computation time of the same operator type under the same data type, and construct multiple groups of measured computation time. The hardware attribute information of the computing hardware in the target AI chip is added to each measured computing time of each measured computing time group; The preset machine learning model is trained using each measured computing time group to obtain multiple performance sub-models, and each performance sub-model is organized to obtain the operator computing core performance model; The performance sub-model corresponds to the set operator type and the set data type.

7. The method according to claim 1, characterized in that The target AI chip has a three-level cache architecture, which consists of a first-level cache, a second-level cache, and a third-level cache, in descending order of distance from the computing cores in the target AI chip. Accordingly, based on the operator characteristics of the operator to be evaluated in the target AI chip, the storage characteristics of the cache at each level of the target AI chip, and the computing core characteristics, multiple partitioning strategies for the operator to be evaluated between the caches at each level of the target AI chip are obtained, including: According to the operator calculation and transport type of the operator to be evaluated, determine the data calculation and transport form on the three-level cache architecture when the operator to be evaluated performs the calculation; Based on the data calculation and transportation form, the operator characteristics of the operator to be evaluated, the storage characteristics of the cache at each level of the target AI chip, and at least one of the computing core characteristics, multiple segmentation strategies for the operator to be evaluated between the caches at each level of the target AI chip are obtained.

8. The method according to claim 7, characterized in that The operator calculation and transport types of the operator to be evaluated include: first-class L3 operators and second-class L3 operators; The first data calculation and transfer form of the first type of L3 operator is to first transfer the data from the third level cache to the second level cache, then transfer it from the second level cache to the first level cache for calculation, and then transfer the calculation results back to the third level cache; The second data calculation and transfer form of the second type of L3 operator is to transfer data directly from the third-level cache to the first-level cache for calculation, and then transfer the calculation results back to the third-level cache.

9. The method according to claim 8, characterized in that The operator calculation transfer types of the operator to be evaluated also include: L1 fusion operator and L2 fusion operator; The third data calculation and transport form of the L1 fusion operator is the first data calculation and transport form or the second data calculation and transport form; The fourth data calculation and transfer form of the L2 fusion operator is to first transfer the data from the third-level cache to the second-level cache, and then perform multiple loop operations between the second-level cache and the first-level cache, transferring the data from the second-level cache to the first-level cache for calculation and then transferring it back to the second-level cache until the final calculation result is obtained, and then the final calculation result is transferred back to the third-level cache.

10. The method according to claim 7, characterized in that Based on the data computation and transport format, the operator characteristics of the operator to be evaluated, and the storage characteristics and computing core characteristics of the target AI chip's cache at each level, multiple partitioning strategies are obtained for the operator to be evaluated across the target AI chip's cache at each level, including: Based on the operator computation handling type, tensor shape, data type, and computation core constraints of the operator to be evaluated, as well as the cache size and pipeline strategy of the first-level cache, multiple first-class splits of the operator to be evaluated on the first-level cache are calculated, and multiple primitive strategies are initialized based on each first-class split. When the data calculation and transfer form determines that the second type of split is included, multiple second type splits corresponding to each first type of split are calculated based on the cache size of the secondary cache, the pipeline strategy, and the number of computing cores used when executing the operator calculation. Multiple update strategies are then constructed based on each second type split corresponding to each first type of split. When the data is calculated based on the transport form and it is determined that the second type of segmentation is not involved, the original strategy is directly determined as the update strategy; Get the third type of split based on the tensor shape of the input data; The third type of segmentation is combined with each update strategy to obtain segmentation of multiple data in caches at various levels, and the multiple segmentation strategies are formed based on the multiple segmentations obtained by the combination.

11. The method according to claim 10, characterized in that Based on a pre-built bandwidth model adapted to the target AI chip, the data transfer time between cache levels for each splitting strategy of the operator to be evaluated is calculated. Specifically, the data transfer time includes: Obtain a target segmentation strategy from each segmentation strategy, and obtain a target first-category segmentation and a target second-category segmentation from the target segmentation strategy; Based on the tensor shape, data type, and data transfer type in the target first-category split, the bandwidth model is called to obtain the first bandwidth data when transferring data from the second-level cache to the first-level cache. Calculate the first data transfer time between the second-level cache and the first-level cache according to the first bandwidth data and the tensor shape in the target first-category split; Based on the tensor shape, data type, and data transfer type in the target second-category split, the bandwidth model is called to obtain the second bandwidth data when transferring data from the third-level cache to the second-level cache. A second data transfer time between the L3 cache and the L2 cache is calculated according to the second bandwidth data and the shape of the tensor in the target second-type segmentation.

12. The method according to claim 11, characterized in that Based on the pre-built operator computing core performance model adapted to the target AI chip, the computational time of the operator to be evaluated when executing calculations on the L1 cache for each slicing strategy is calculated, including: In the target segmentation strategy, obtain the first type of target segmentation; According to the data type in the target first-category segmentation and the operator type of the operator to be evaluated, the target operator calculation core model is obtained in the operator calculation core performance model; The tensor shape in the target first-category split and the hardware attribute information of the computing hardware in the target AI chip are input into the target performance sub-model to obtain the target computation time corresponding to the target split strategy output by the target operator computation core model.

13. The method according to claim 11, characterized in that Based on the tensor shape, data type, and data transfer type in the target first-category split, the bandwidth model is called to obtain the first bandwidth data when transferring data from the second-level cache to the first-level cache, including: Obtain a target bandwidth sub-model in the bandwidth model based on the data type, data transfer type, and data transfer direction from the second-level cache to the first-level cache in the target first-category segmentation; The tensor shape in the target first-category split and the hardware attribute information of the matching direct access storage hardware in the target AI chip are input into the target bandwidth sub-model to obtain the first bandwidth data output by the target bandwidth sub-model.

14. The method according to claim 12, characterized in that Based on the computational time and data handling time, combined with the second hidden strategy included in the splitting strategy, the total operator time of the operator to be evaluated for each splitting strategy is calculated, including: For the target split strategy, the number of first-loop transfers in the L1 cache and the number of second-loop transfers in the L2 cache are calculated based on the input data tensor size, the tensor shape in the target first-type split, the tensor shape in the target second-type split, and the number of compute cores used when executing the operator calculation. According to the tensor shape in the first type of target segmentation, the tensor shape in the second type of target segmentation, the pipeline strategy of the first-level cache, the pipeline strategy in the second-level cache, the first data handling time, the second data handling time, the first loop handling times, the second loop handling times and the target calculation time, combined with the second hidden strategy included in the segmentation strategy, the total operator time of the operator to be evaluated for the target segmentation strategy is calculated.

15. The method according to any one of claims 1 to 14, characterized in that Operator performance evaluation is performed on the operators to be evaluated based on the total time consumed by each operator, including: Obtain the optimal splitting strategy corresponding to the minimum total operator time; According to the various strategy information contained in the optimal segmentation strategy, the operator to be evaluated is optimized and guided so that the operator to be evaluated achieves the best performance.

16. An operator performance evaluation device in an AI chip, characterized in that: include: a partitioning strategy determination module, configured to obtain multiple partitioning strategies for the operator to be evaluated across cache levels of the target AI chip based on at least one of the operator characteristics of the operator to be evaluated in the target AI chip, the storage characteristics of the cache levels of the target AI chip, and the computing core characteristics; The data transfer time calculation module is used to calculate the data transfer time when each splitting strategy of the operator to be evaluated executes data transfer between cache levels based on a pre-built bandwidth model adapted to the target AI chip; The bandwidth model is used to predict bandwidth data when transferring data of a set data transfer type, a set data type, and a set tensor shape between two set caches in the target AI chip; A first performance evaluation module is configured to calculate, if the operator to be evaluated is a pure data handling operator, the total operator time consumed by the operator to be evaluated for each splitting strategy based on the data handling time consumed and the first hidden strategy included in the splitting strategy, and perform operator performance evaluation on the operator to be evaluated based on the total operator time consumed; Wherein, the device further comprises: A computational time calculation module is used to calculate the data transfer time of each splitting strategy of the operator to be evaluated when performing data transfer between the various cache levels. If the operator to be evaluated is a data transfer and computation operator, the computational time of the operator to be evaluated when executing the computation for each splitting strategy in the first-level cache is calculated based on a pre-built operator computing core performance model adapted for the target AI chip. The first-level cache is set close to the computing core in the target AI chip. The operator computing core performance model is used to predict the computational time required to execute a set type of operator calculation on data of a set data type and a set tensor shape in the target AI chip. The second performance evaluation module is used to calculate the total operator time of the operator to be evaluated for each splitting strategy based on the calculation time and the data processing time, combined with the second hidden strategy included in the splitting strategy, and perform operator performance evaluation on the operator to be evaluated based on the total operator time.

17. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so as to enable the at least one processor to execute the operator performance evaluation method in the AI ​​chip according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a processor to implement the operator performance evaluation method in the AI ​​chip according to any one of claims 1 to 15 when executed.

19. A computer program product, characterized in that The computer program product includes a computer program, which, when executed by a processor, implements the operator performance evaluation method in the AI ​​chip according to any one of claims 1 to 15.

Images