A method and device for optimizing value-based operators on NPU

By optimizing the data transfer path of value-based operators on the NPU, and determining the target storage location based on the input and output data size, the problem of inefficient value is solved and more efficient network reasoning performance is achieved.

CN118520920BActive Publication Date: 2024-11-12STREAM COMPUTING INC
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410968677.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-12
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

The inefficient value-taking operators on NPUs lead to network inference performance problems, especially when the input data is large or small, and the third-level cache migration is required, which increases the time.

Method used

By obtaining the size of the input data and output data, the target storage location is determined to minimize the number of transfer cycles, thereby optimizing the data transfer path of the value-based operator, and using cyclic value acquisition or matrix multiplication to achieve data transfer.

Benefits of technology

It reduces the time to move when calculating input data to output data, improves the value efficiency of value-based operators on the NPU, reduces hardware dependence and R&D costs, and improves the inference efficiency of accelerated inference models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118520920B_ABST
    Figure CN118520920B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses a method and device for optimizing value-taking operators on an NPU. The embodiment of the present invention obtains the information of the input data and the size of the output data of any value-taking operator, wherein the information of the input data includes the size and storage location of the input data; the target storage location of the output data is determined according to the information of the input data and the size of the output data, and the target storage location is the storage location of the output data when the number of movement cycles of the input data is minimized when the value-taking operator completes the calculation of the input data to obtain the output data. Through the above method, the value-taking efficiency of the value-taking operators on the NPU can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for optimizing a value-based operator on an NPU. Background Art

[0002] With the evolution of neural network algorithms, the computing power requirements for artificial intelligence (AI) chips are getting higher and higher. Different AI compilation frameworks need to support various types of operator implementations. However, due to different hardware architectures, the implementation methods of different operators vary greatly, especially for network processing units (NPUs). NPUs lack support for value-taking operators, which makes the value-taking efficiency of value-taking operators low, which in turn causes performance problems in network reasoning in applications.

[0003] In the prior art, for the NPU design structure of three-level cache, the three-level cache is respectively double-rate synchronous dynamic random access memory DDR, the last level buffer (LLB) last level cache and the local memory space L1 of the computing core. When the value-taking operator takes values, there are two situations, as follows: Situation 1: If the input data is large, the input data and index information cannot be completely placed on L1. If the input data and index information are on DDR, it is necessary to complete the three-level movement from DDR to LLB and then to L1 before the value can be taken, and the time consumption of the value-taking operator will be greatly increased; Situation 2: If the input data is small, the input data and index information can be completely placed on L1. At this time, the table lookup instruction is a linear instruction, and the time consumption of the value-taking operator will also be greatly increased; the above two situations will cause the problem of low value-taking efficiency.

[0004] In summary, how to improve the value-taking efficiency of value-taking operators on the NPU is a problem that needs to be solved at present. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for optimizing value-taking operators on an NPU, which can reduce the time consumed by value-taking operators in completing the calculation from input data to output data and the time consumed in moving input data to output data, thereby improving the value-taking efficiency of value-taking operators on the NPU.

[0006] In a first aspect, an embodiment of the present invention provides a method for optimizing a value-based operator on an NPU, the method comprising:

[0007] Obtaining the information of input data and the size of output data of any value-taking operator, wherein the information of input data includes the size and storage location of the input data;

[0008] The target storage location of the output data is determined based on the information of the input data and the size of the output data. The target storage location is the storage location of the output data when the number of movement cycles of the input data is minimized when the value-taking operator completes the calculation of the input data to obtain the output data.

[0009] Optionally, determining the target storage location of the output data according to the information of the input data and the size of the output data specifically includes:

[0010] Determine at least one candidate storage correspondence mode according to the storage location of the input data, wherein the candidate storage correspondence mode is a correspondence relationship between the storage location of the input data and the storage location of the output data;

[0011] Determining a target storage correspondence mode from the at least one candidate storage correspondence mode according to the size of the input data and the size of the output data;

[0012] The target storage location of the output data is determined according to the target storage corresponding mode.

[0013] Optionally, determining a target storage corresponding mode from the at least one candidate storage corresponding mode according to the size of the input data and the size of the output data specifically includes:

[0014] According to the size of the input data and the size of the output data, querying the number of transfer cycles of any value-taking operator when completing the calculation of the input data to obtain the output data under each candidate storage corresponding mode;

[0015] The candidate storage corresponding mode corresponding to the minimum transfer cycle number among the transfer cycle numbers is determined as the target storage corresponding mode.

[0016] Optionally, the storage location includes a double data rate synchronous dynamic random access memory DDR, a last level cache LLB, and a local memory space L1 of a computing core.

[0017] Optionally, the candidate storage correspondence modes include DDR to DDR, DDR to LLB, DDR to L1, LLB to LLB, LLB to L1, LLB to DDR, L1 to L1, L1 to LLB and L1 to DDR.

[0018] Optionally, after determining the target storage location of the output data according to the information of the input data and the size of the output data, the method further includes:

[0019] The transfer from the storage location of the input data to the target storage location of the output data is achieved in a pre-set manner.

[0020] Optionally, the pre-set method includes cyclic value acquisition or matrix multiplication.

[0021] In a second aspect, an embodiment of the present invention provides a device for optimizing a value-based operator on an NPU, the device comprising:

[0022] An acquisition unit, used to acquire information about input data and the size of output data of any value-taking operator, wherein the information about the input data includes the size and storage location of the input data;

[0023] A determination unit is used to determine the target storage location of the output data based on the information of the input data and the size of the output data, wherein the target storage location is the storage location of the output data when the number of movement cycles of the input data is minimized when the value-taking operator completes the calculation of the input data to obtain the output data.

[0024] Optionally, the determining unit is specifically configured to:

[0025] Determine at least one candidate storage correspondence mode according to the storage location of the input data, wherein the candidate storage correspondence mode is a correspondence relationship between the storage location of the input data and the storage location of the output data;

[0026] Determining a target storage correspondence mode from the at least one candidate storage correspondence mode according to the size of the input data and the size of the output data;

[0027] The target storage location of the output data is determined according to the target storage corresponding mode.

[0028] Optionally, the determining unit is specifically configured to:

[0029] According to the size of the input data and the size of the output data, querying the number of transfer cycles of any value-taking operator when completing the calculation of the input data to obtain the output data under each candidate storage corresponding mode;

[0030] The candidate storage corresponding mode corresponding to the minimum transfer cycle number among the transfer cycle numbers is determined as the target storage corresponding mode.

[0031] Optionally, the storage location includes a double data rate synchronous dynamic random access memory DDR, a last level cache LLB, and a local memory space L1 of a computing core.

[0032] Optionally, the candidate storage correspondence modes include DDR to DDR, DDR to LLB, DDR to L1, LLB to LLB, LLB to L1, LLB to DDR, L1 to L1, L1 to LLB and L1 to DDR.

[0033] Optionally, the device further comprises:

[0034] The moving unit is used to realize the moving from the storage location of the input data to the target storage location of the output data in a preset manner.

[0035] Optionally, the pre-set method includes cyclic value acquisition or matrix multiplication.

[0036] In a third aspect, an embodiment of the present invention provides a computer program instruction, which, when executed by a processor, implements a method as described in the first aspect or any possible embodiment of the first aspect.

[0037] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement a method as described in the first aspect or any possible embodiment of the first aspect.

[0038] In a fifth aspect, an embodiment of the present invention provides a chip comprising a memory and a processing core, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processing core to implement a method as described in the first aspect or any possible embodiment of the first aspect.

[0039] In a sixth aspect, an embodiment of the present invention provides a board card, which includes the chip according to the fifth aspect.

[0040] In a seventh aspect, an embodiment of the present invention provides a server, wherein the server includes the board card according to the sixth aspect.

[0041] The embodiment of the present invention obtains the information of the input data and the size of the output data of any value-taking operator, wherein the information of the input data includes the size and storage location of the input data; the target storage location of the output data is determined according to the information of the input data and the size of the output data, wherein the target storage location is the storage location of the output data when the number of cycles of moving the input data is the smallest when the value-taking operator completes the calculation of the input data to obtain the output data. The above method can reduce the time spent on moving the input data to the output data when the value-taking operator completes the calculation from the input data to the output data, thereby improving the value-taking efficiency of the value-taking operator on the NPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0043] Figure 1 This is a schematic diagram of three-level data transfer in the prior art;

[0044] Figure 2 It is a flow chart of a method for optimizing a value-based operator on an NPU in an embodiment of the present invention;

[0045] Figure 3 is a flow chart of a method for determining a target storage location in an embodiment of the present invention;

[0046] Figure 4 It is a flow chart of another method for optimizing value-based operators on NPU in an embodiment of the present invention;

[0047] Figure 5 is a flowchart for implementing the matrix multiplication method in an embodiment of the present invention;

[0048] Figure 6 is a flowchart for implementing the cyclic value acquisition in an embodiment of the present invention;

[0049] Figure 7 It is a schematic diagram of a device for optimizing a value-based operator on an NPU in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention is described below based on the embodiments, but the present invention is not limited to these embodiments. In the detailed description of the present invention below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present invention without the description of these details. In order to avoid confusing the essence of the present invention, known methods, processes, procedures, components and circuits are not described in detail.

[0051] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0052] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0053] In the description of the present invention, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" is two or more.

[0054] In the prior art, for the NPU design structure of the three-level cache (DDR-LLB-L1), there are two situations when the value-taking operator takes a value, as follows:

[0055] Case 1: If the input data is large, the input data and index information cannot be completely placed on L1. If the input data and index information are on DDR, they need to be moved from DDR to LLB and then to L1 in three levels before they can be retrieved. For example: Figure 1 As shown, the input data of DDR is moved to LLB. In addition to the necessary input data, the waste data other than the input data needs to be moved during the moving process. When the input data is large, the input data needs to be cut first, and then moved after the cutting. That is to say, the larger the input data, the more waste data needs to be moved, and the time consumption will also increase. After the block input data is moved to LLB, the valid input data is moved to L1. Since the three-level move must be completed, the time consumption of the value operator will increase significantly, as shown Figure 1 Only one block of data is taken as an example to illustrate the process of moving valid input data and waste data.

[0056] Case 2: If the input data is small, the input data and index information can be completely placed on L1. At this time, the table lookup instruction is a linear instruction. The larger the shape of the input data, the more time-consuming the linear instruction is. The time consumption increases with the increase of the shape, and the time consumption of the value-taking operator will also increase significantly. The above two cases will cause the problem of low value-taking efficiency.

[0057] In summary, how to improve the value-taking efficiency of value-taking operators on the NPU is a problem that needs to be solved at present.

[0058] In order to solve the above problems, an optimization method for value-based operators on NPU is proposed in the embodiment of the present invention. Figure 2 As shown, Figure 2 This is a flow chart of a method for optimizing value-based operators on an NPU according to an embodiment of the present invention. Specifically, it includes:

[0059] Step S200: Obtain information about input data and the size of output data of any value-taking operator, wherein the information about the input data includes the size and storage location of the input data.

[0060] Specifically, the input data and output data of any value-taking operator are stored in bytes, and different input data sizes and output data sizes can be set according to different hardware limitations.

[0061] In a possible implementation, taking a gather operator as an example, the storage location of the gather operator may be: a double data rate synchronous dynamic random access memory (DDR SDRAM, referred to as DDR in the present invention), a last level buffer (LLB), or a local memory space L1 of a computing core.

[0062] Assume that the relationship between the input data size and the output data size is as shown in Table 1 below:

[0063] Table 1

[0064] Input data size Output data size Larger than 8MB Less than or equal to 1MB Larger than 8MB Greater than 1MB and less than or equal to 8MB Larger than 8MB The output data is greater than or equal to the input data Less than or equal to 8MB, greater than 1MB Less than or equal to 1MB Less than or equal to 8MB, greater than 1MB Greater than or equal to 1MB, less than or equal to 8MB and the output data is less than or equal to the input data Less than or equal to 8MB, greater than 1MB The output data is greater than or equal to the input data Less than or equal to 1MB Less than or equal to 1MB Less than or equal to 1MB Larger than 1MB

[0065] In the above Table 1, the input data size and the output data size are determined according to the hardware, which is only an example description here.

[0066] Step S201, determine the target storage location of the output data according to the information of the input data and the size of the output data, the target storage location being the storage location of the output data when the number of movement cycles of the input data is minimized when the value class operator completes the calculation of the input data to obtain the output data.

[0067] Specifically, the target storage location of the output data is determined according to the information of the input data and the size of the output data. The specific process is as follows: Figure 3 As shown, the following steps are included:

[0068] Step S300: Determine at least one candidate storage corresponding mode according to the storage location of the input data.

[0069] Specifically, the candidate storage correspondence modes include DDR to DDR, DDR to LLB, DDR to L1, LLB to LLB, LLB to L1, LLB to DDR, L1 to L1, L1 to LLB and L1 to DDR, which are 9 in total.

[0070] In the embodiment of the present invention, the DDR to DDR indicates that the storage location of the input data is DDR, and the storage location of the output data is also DDR; the DDR to LLB indicates that the storage location of the input data is DDR, and the storage location of the output data is LLB; the DDR to L1 indicates that the storage location of the input data is DDR, and the storage location of the output data is L1; the LLB to LLB indicates that the storage location of the input data is LLB, and the storage location of the output data is also LLB; the LLB to L1 indicates that the storage location of the input data is LLB, and the storage location of the output data is L1; the LLB to DDR indicates that the storage location of the input data is LLB, and the storage location of the output data is DDR; the L1 to L1 indicates that the storage location of the input data is L1, and the storage location of the output data is also L1; the L1 to LLB indicates that the storage location of the input data is L1, and the storage location of the output data is LLB; the L1 to DDR indicates that the storage location of the input data is L1, and the storage location of the output data is DDR.

[0071] In an embodiment of the present invention, if the storage location of the input data is DDR, the candidate storage corresponding modes are DDR to DDR, DDR to LLB, and DDR to L1; if the storage location of the input data is LLB, the candidate storage corresponding modes are LLB to LLB, LLB to L1, and LLB to DDR; if the storage location of the input data is L1, the candidate storage corresponding modes are L1 to L1, L1 to LLB, and L1 to DDR.

[0072] Step S301: Determine a target storage correspondence mode from the at least one candidate storage correspondence mode according to the size of the input data and the size of the output data.

[0073] Specifically, according to the size of the input data and the size of the output data, the number of transfer cycles of any value-taking operator when completing the calculation from input data to output data under each candidate storage corresponding mode is queried; and the candidate storage corresponding mode corresponding to the minimum number of transfer cycles among at least one of the transfer cycle numbers is determined as the target storage corresponding mode.

[0074] In the embodiment of the present invention, when any of the value-taking operators completes the calculation of the input data to obtain the output data, the current hardware will provide at least one hardware-available instruction, and each hardware-available instruction corresponds to a corresponding number of transfer cycles, which can also be called a theoretical number of cycles. When the same value-taking operator is in different candidate storage corresponding modes, it involves different calculation schemes and uses different instructions, resulting in different corresponding numbers of transfer cycles for the same value-taking operator when selecting different candidate storage corresponding modes. Specifically, while the input data is being moved from its storage location to the target storage location of the output data, the calculation of the input data is completed to obtain the output data. The calculation can be a hardware-available instruction or a mathematical formula calculation scheme. Because the instructions are different, the number of transfer cycles is different.

[0075] In a possible implementation, the size of the input data and the size of the output data are required, and the corresponding number of transfer cycles for different candidate storage corresponding modes is pre-calculated. Assume that the value class operator is a gather operator, the input data is greater than 8MB, the output data is less than 1MB, and the storage location of the input data is DDR. The corresponding number of transfer cycles for the output data on DDR, LLB and L1 is pre-calculated. If the number of transfer cycles for the output data on DDR is the smallest, then in the subsequent processing process, when the input data is greater than 8MB and the output data is less than 1MB, the storage location of the output data is selected in DDR. This is only an exemplary description, and it will be determined according to the actual situation.

[0076] In the embodiment of the present invention, although the number of shift cycles (ie, the theoretical number of cycles) is somewhat different from the actual number of cycles, it can be the reference closest to the actual value, so the embodiment of the present invention only needs to consider the theoretical number of cycles.

[0077] Step S302: Determine the target storage location of the output data according to the target storage corresponding mode.

[0078] In the embodiment of the present invention, since the target storage correspondence is a correspondence between an input data storage location and an output data storage location, when the target storage correspondence is determined, the target storage location of the output data is determined.

[0079] For example, assuming that the target storage correspondence mode is DDR to L1, the target storage location of the output data is L1, and the valid input data is directly moved from DDR to the L1.

[0080] In the embodiment of the present invention, when the output data is at the target storage location, the number of transfer cycles of any value-taking operator is minimized when completing the calculation from input data to output data, which reduces the time consumption during the transfer, thereby improving the value-taking efficiency of the value-taking operators on the NPU.

[0081] In a possible implementation, after step S201, the following steps are further included: Figure 4 As shown, Figure 4 This is a flow chart of a method for optimizing value-based operators on an NPU according to an embodiment of the present invention. Specifically, it includes:

[0082] Step S202: Using a preset method to achieve the migration from the storage location of the input data to the target storage location of the output data.

[0083] Specifically, the pre-set method includes cyclic value acquisition or matrix multiplication.

[0084] The following nine specific embodiments are used to describe data migration in detail, as follows: Specific embodiment 1:

[0086] When the input data is greater than 8MB, the output data is greater than 8MB, and the input data is 2 times or more than the output data, the storage location of the input data is DDR, and the target storage location of the output data is DDR, the number of migration cycles is the smallest, and the DDR to DDR data migration can be achieved by using a cyclic value taking method. Specific embodiment 2:

[0088] When the input data is greater than 8MB, the output data is less than or equal to 8MB and greater than 1MB, the storage location of the input data is DDR, and the target storage location of the output data is LLB, the number of migration cycles is minimal, and the data migration from DDR to LLB can be achieved by cyclic value taking. Specific embodiment three:

[0090] When the input data is 2 times or more than the output data, the output data is less than or equal to 8MB, the storage location of the input data is DDR, and the target storage location of the output data is L1, the number of migration cycles is minimal, and the data migration from DDR to L1 can be achieved by cyclic value taking. Specific embodiment 4:

[0092] When the input data is less than or equal to 8MB, the input data is smaller than the output data, the storage location of the input data is LLB, and the target storage location of the output data is DDR, the number of migration cycles is the smallest, and the data migration from LLB to DDR can be achieved by matrix multiplication or cyclic value taking.

[0093] Specifically, it is necessary to calculate the transfer cycle data corresponding to the matrix multiplication method and the cyclic value acquisition method respectively for further comparison. Specific embodiment 5:

[0095] When the input data is less than or equal to 8MB and greater than 1MB, the output data is also less than or equal to 8MB and greater than 1MB, the storage location of the input data is LLB, and the target storage location of the output data is LLB, the number of migration cycles is the smallest, and data migration from LLB to LLB can be achieved by matrix multiplication or cyclic value taking.

[0096] Specifically, it is necessary to calculate the transfer cycle data corresponding to the matrix multiplication method and the cyclic value acquisition method respectively for further comparison. Specific embodiment six:

[0098] When the input data is less than or equal to 8MB, the output data is less than or equal to 1MB, the storage location of the input data is LLB, and the target storage location of the output data is L1, the number of migration cycles is the smallest, and the data migration from LLB to L1 can be achieved by matrix multiplication or cyclic value taking.

[0099] Specifically, it is necessary to calculate the transfer cycle data corresponding to the matrix multiplication method and the cyclic value acquisition method respectively for further comparison. Specific embodiment seven,

[0101] When the input data is less than or equal to 1MB, the output data is less than or equal to 1MB, the storage location of the input data is L1, and the target storage location of the output data is L1, the number of migration cycles is minimal, and matrix multiplication can be used to achieve data migration from L1 to L1. Specific embodiment eight,

[0103] When the input data is less than or equal to 1MB, the output data is greater than 1MB and less than or equal to 8MB, the storage location of the input data is L1, and the target storage location of the output data is LLB, the number of migration cycles is the smallest, and the reverse data migration from L1 to LLB can be achieved by matrix multiplication or cyclic value taking.

[0104] Specifically, it is necessary to calculate the transfer cycle data corresponding to the matrix multiplication method and the cyclic value acquisition method respectively for further comparison. Specific embodiment nine,

[0106] When the input data is less than or equal to 1MB, the output data is greater than 8MB, the storage location of the input data is L1, and the target storage location of the output data is DDR, the number of migration cycles is the smallest, and the reverse data migration from L1 to DDR can be achieved by matrix multiplication or cyclic value taking.

[0107] Specifically, it is necessary to calculate the transfer cycle data corresponding to the matrix multiplication method and the cyclic value acquisition method respectively for further comparison.

[0108] In the embodiment of the present invention, the matrix multiplication method and the loop value acquisition method are respectively described in detail.

[0109] Specifically, the matrix multiplication method is implemented in a specific process, such as Figure 5 As shown, the details are as follows:

[0110] Step S500: convert the index matrix into a global index matrix using the index axis information.

[0111] Step S501: Convert the global index matrix into a 01 matrix.

[0112] Specifically, the 01 matrix contains only two numbers, 0 and 1. The 1 in the 01 matrix indicates that a value needs to be taken at the current index position, and the 0 in the 01 matrix indicates that a value does not need to be taken.

[0113] Step S502: transform the input data to generate a transformation matrix.

[0114] Specifically, the transformation matrix meets the matrix multiplication requirements.

[0115] Step S503: Calculate the 01 matrix and the transformation matrix using a matrix multiplication instruction to obtain a value result.

[0116] Step S504: The value-taking result is transformed into a shape to obtain a result matrix.

[0117] Specifically, the loop value acquisition method implements the process as follows: Figure 6 As shown, the details are as follows:

[0118] Step S600: confirm the index position where the value needs to be taken according to the value axis information and the index information, and record the number of times the value needs to be taken.

[0119] Step S601: construct a value acquisition loop according to the number of times the value is required.

[0120] Step S602: Each time a loop is executed, the input corresponding position data is retrieved according to the axis information and index information used in the current loop.

[0121] Step S603: Repeat step S602 until all required values ​​in all indexes are obtained.

[0122] The present invention is implemented in the above manner. In hardware that lacks value-taking hardware instructions or has low value-taking instructions, the value-taking efficiency of value-taking operators can be improved by adopting the above method in the compilation software, thereby reducing the dependence of value-taking operators on NPU hardware, reducing hardware R&D costs, and improving the reasoning efficiency of the accelerated reasoning model.

[0123] Figure 7 FIG. 1 is a schematic diagram of a device for optimizing a value-based operator on an NPU according to an embodiment of the present invention. Figure 7 As shown, the apparatus of this embodiment includes an acquiring unit 701 and a determining unit 702 .

[0124] Among them, the acquisition unit 701 is used to obtain the information of input data and the size of output data of any value-taking operator, wherein the information of input data includes the size and storage location of input data; the determination unit 702 is used to determine the target storage location of the output data according to the information of input data and the size of output data, wherein the target storage location is the storage location of the output data when the number of movement cycles of the input data is minimized when the value-taking operator completes the calculation of the input data to obtain the output data.

[0125] Furthermore, the determining unit is specifically configured to:

[0126] Determine at least one candidate storage correspondence mode according to the storage location of the input data, wherein the candidate storage correspondence mode is a correspondence relationship between the storage location of the input data and the storage location of the output data;

[0127] Determining a target storage correspondence mode from the at least one candidate storage correspondence mode according to the size of the input data and the size of the output data;

[0128] The target storage location of the output data is determined according to the target storage corresponding mode.

[0129] Furthermore, the determining unit is specifically configured to:

[0130] According to the size of the input data and the size of the output data, query the number of transfer cycles of any value-taking operator when completing the calculation from input data to output data under each candidate storage corresponding mode; determine the candidate storage corresponding mode corresponding to the minimum number of transfer cycles among at least one of the transfer cycle numbers as the target storage corresponding mode.

[0131] Furthermore, the storage location includes a double data rate synchronous dynamic random access memory DDR, a last level cache LLB and a local memory space L1 of a computing core.

[0132] Furthermore, the candidate storage correspondence modes include DDR to DDR, DDR to LLB, DDR to L1, LLB to LLB, LLB to L1, LLB to DDR, L1 to L1, L1 to LLB and L1 to DDR.

[0133] Furthermore, the device also includes:

[0134] The moving unit is used to realize the moving from the storage location of the input data to the target storage location of the output data in a preset manner.

[0135] Furthermore, the pre-set method includes cyclic value acquisition or matrix multiplication.

[0136] In an embodiment of the present invention, a computer program instruction is further provided. When the computer program instruction is executed by a processor, the method described in any one of the above embodiments is implemented.

[0137] In an embodiment of the present invention, a computer-readable storage medium is further provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in any one of the above embodiments is implemented.

[0138] An embodiment of the present invention provides a chip, including a memory and a processing core, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processing core to implement any of the methods described in the above embodiments.

[0139] An embodiment of the present invention provides a board, which includes the chip.

[0140] An embodiment of the present invention provides a server, and the server includes the board.

[0141] As will be appreciated by those skilled in the art, various aspects of the embodiments of the present invention may be implemented as a system, method or computer program product. Therefore, various aspects of the embodiments of the present invention may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation that combines software aspects with hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." In addition, various aspects of the embodiments of the present invention may take the form of a computer program product implemented in one or more computer-readable media having a computer-readable program code implemented thereon.

[0142] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, (but not limited to) electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples (non-exhaustive enumeration) of computer-readable storage media will include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of an embodiment of the present invention, a computer-readable storage medium can be any tangible medium that can contain or store a program used by an instruction execution system, device or apparatus or a program used in conjunction with an instruction execution system, device or apparatus.

[0143] A computer readable signal medium may include a propagated data signal having a computer readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any of the following computer readable media: not a computer readable storage medium, and may communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0144] Program code embodied on a computer readable medium may be transmitted using any appropriate medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0145] The computer program code for performing operations for various aspects of the embodiments of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional process programming languages ​​such as "C" programming language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer as a stand-alone software package; partially on the user's computer and partially on a remote computer; or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by using the Internet of an Internet service provider).

[0146] The flowchart legend and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention described above describe various aspects of the embodiment of the present invention.It will be understood that each block of the flowchart legend and / or block diagram and the combination of the blocks in the flowchart legend and / or block diagram can be implemented by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that the instructions (executed by the processor of the computer or other programmable data processing device) create a device for implementing the function / action specified in the flowchart and / or block diagram block or block.

[0147] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing device, or other apparatus to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0148] The computer program instructions may also be loaded onto a computer, other programmable data processing device or other apparatus so that a series of operable steps are performed on the computer, other programmable device or other apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in the flowchart and / or block diagram block or blocks.

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for optimizing a value-based operator on an NPU, characterized in that: The method includes: Obtaining the information of input data and the size of output data of any value-taking operator, wherein the information of input data includes the size and storage location of the input data; Determine a target storage location of the output data according to the information of the input data and the size of the output data, wherein the target storage location is the storage location of the output data when the number of transfer cycles of the input data is the smallest when the value operator completes the calculation of the input data to obtain the output data; Wherein, determining the target storage location of the output data according to the information of the input data and the size of the output data specifically includes: Determine at least one candidate storage correspondence mode according to the storage location of the input data, wherein the candidate storage correspondence mode is a correspondence relationship between the storage location of the input data and the storage location of the output data; Determining a target storage correspondence mode from the at least one candidate storage correspondence mode according to the size of the input data and the size of the output data; The target storage location of the output data is determined according to the target storage corresponding mode.

2. The method according to claim 1, characterized in that Determining a target storage corresponding mode from the at least one candidate storage corresponding mode according to the size of the input data and the size of the output data specifically includes: According to the size of the input data and the size of the output data, querying the number of transfer cycles of any value-taking operator when completing the calculation of the input data to obtain the output data under each candidate storage corresponding mode; The candidate storage corresponding mode corresponding to the minimum transfer cycle number among the transfer cycle numbers is determined as the target storage corresponding mode.

3. The method according to claim 1, characterized in that The storage locations include a double data rate synchronous dynamic random access memory DDR, a last level cache LLB and a local memory space L1 of a computing core.

4. The method according to claim 3, characterized in that The candidate storage correspondences include DDR to DDR, DDR to LLB, DDR to L1, LLB to LLB, LLB to L1, LLB to DDR, L1 to L1, L1 to LLB and L1 to DDR.

5. The method according to claim 1, characterized in that After determining the target storage location of the output data according to the information of the input data and the size of the output data, the method further comprises: The transfer from the storage location of the input data to the target storage location of the output data is achieved in a pre-set manner.

6. The method according to claim 5, characterized in that The pre-set method includes cyclic value acquisition or matrix multiplication.

7. A device for optimizing value-based operators on an NPU, characterized in that: The device includes: An acquisition unit, used to acquire information about input data and the size of output data of any value-taking operator, wherein the information about the input data includes the size and storage location of the input data; a determination unit, configured to determine a target storage location of the output data according to the information of the input data and the size of the output data, wherein the target storage location is the storage location of the output data when the number of transfer cycles of the input data is minimized when the value class operator completes the calculation of the input data to obtain the output data; Wherein, the determination unit is specifically used to: determine at least one candidate storage correspondence method according to the storage location of the input data, wherein the candidate storage correspondence method is the correspondence between the storage location of the input data and the storage location of the output data; determine a target storage correspondence method from the at least one candidate storage correspondence method according to the size of the input data and the size of the output data; and determine the target storage location of the output data according to the target storage correspondence method.

8. A computer-readable storage medium storing computer program instructions, characterized in that: The computer program instructions implement the method according to any one of claims 1 to 6 when executed by a processor.

Citation Information

Patent Citations

  • Data processing method, network processor and computer storage medium

    CN109413122A

  • Algorithm processing method and device, storage medium and terminal equipment

    CN109656719A

  • Operator optimization method and related equipment

    CN116012633A

  • Data processing method, device and equipment and readable storage medium

    CN116432718A

  • Input / Output method and device thereof

    JP1999345153A