Optimization Method for Computing Power of Neural Network Module, Chip, Electronic Device, and Medium

Abstract

Provided are a method for optimizing the computing power of a neural network by a chip, the chip, an electronic device, and a medium. The method includes the steps of: the chip obtaining a computational graph of a neural network having operators, where the neural network receives application data and obtains a processing output result for the application data; the chip performing at least once at least one of the following adjustments on the computational graph based on the specific operations of each operator: an adjustment of exchanging the positions of a first operator and a consequent operator or an antecedent operator in the computational graph with the consequent operator or the antecedent operator in the computational graph; an adjustment of dividing the first operator into two or more similar operators; an adjustment of inserting a plurality of first operators that can cancel each other out, where the first operator does not change the numerical values in the input tensor; the chip determining a second operator as described below adjacent to the adjusted first operator, where the operation of the adjusted first operator and the operation of the second operator can be combined or canceled without affecting the operation result; and combining or canceling the adjusted first operator and the second operator and executing the operation of the combined or canceled computational graph.

Description

Technical Field

[0001] This application claims the priority of Chinese Patent Application No. 202210835887.0, filed on July 15, 2022, and hereby incorporates by reference in its entirety all the content disclosed in the above Chinese patent application as part of this application.

[0002] This application relates to the field of neural networks, and more specifically, to a method for optimizing the computing power of a neural network module, a chip, an electronic device, and a medium.

Background Art

[0003] An artificial intelligence (AI) chip is a chip dedicated to neural network operations, and is mainly a chip specially designed to accelerate the execution of neural networks.

[0004] A neural network can be represented by pure mathematical formulas. Based on these formulas, the neural network can be represented by a computational graph model. A computational graph is a visual representation of these formulas. The computational graph model can divide one composite operation into multiple sub-operations, and each sub-operation is called an operator (abbreviated as Operator, Op). A computational graph consists of nodes and edges. Nodes represent operators, and edges represent the respective dependency relationships of the calculation objects. Solid lines represent the dependency of data transfer, and the data transferred is a tensor. Dashed lines generally represent control dependencies, that is, the execution priorities.

[0005] In a computational graph, an operator has its own input data. Since the computational logic of operators is different, each operator outputs output data calculated using the input data based on its own computational logic.

[0006] It is necessary to design a chip that accelerates the execution speed of the computational graph of a neural network and optimizes the computing power of the neural network module.

Summary of the Invention

Means for Solving the Problems

[0007] According to one aspect of the present application, a method for optimizing the computing power of a neural network module by a chip, comprising: obtaining, by the chip, a computational graph of a neural network module having each operator, wherein the neural network module receives application data and obtains a processing output result for the application data; based on the specific operations of each operator, the chip performs at least once at least one of the following adjustments on the first operator in the computational graph: exchanging the position of the first operator with a subsequent operator or a preceding operator in the computational graph, splitting the first operator into two or more similar operators, and inserting a plurality of first operators that can cancel each other out, wherein the first operator is a tensor operation that does not change the numerical values in the input tensor; the chip determines a second operator adjacent to the adjusted first operator in the computational graph based on the specific operations of each operator, wherein the operations of the adjusted first operator and the second operator can be combined or canceled without affecting the operation results; and the chip combines or cancels the adjusted first operator and the second operator and executes the operations of each operator in the combined or canceled computational graph. A method for optimizing the computing power of a neural network module by a chip is provided.

[0008] According to one aspect of the present application, there is provided an artificial intelligence chip, comprising: a memory configured to store received application data and input the application data into a computational graph of a neural network module having respective operators to obtain a processing output result for the application data; and a controller configured to perform at least one of the following adjustments on the computational graph for a first operator based on specific operations of the respective operators: exchanging the position of the first operator with a consequent operator or an antecedent operator in the computational graph, splitting the first operator into two or more similar operators, and inserting a plurality of first operators that can cancel each other out, wherein the first operator is a tensor operation that does not change the numerical values in the input tensor, determining a second operator adjacent to the adjusted first operator in the computational graph based on specific operations of the respective operators, combining or canceling out the operation of the adjusted first operator and the operation of the second operator without affecting the operation result, and causing the chip to combine or cancel out the adjusted first operator and the second operator and execute the operations of the respective operators in the combined or canceled computational graph.

[0009] According to one aspect of the present application, there is provided an electronic device configured to optimize the computing power of a neural network module by a chip, comprising: a memory configured to store instructions; and a processor configured to read the instructions stored in the memory and execute the method described in the present disclosure.

[0010] According to one aspect of the present application, there is provided a non-transitory storage medium storing instructions, wherein when the instructions are read by a processor, the method described in the present disclosure is executed by the processor.

Advantages of the Invention

[0011] In this way, for example, these operators can be optimized according to the specific operation rules of the operators in the computational graph of a neural network that processes image data, reducing the number of calculated operators or the amount of calculation performed by the operators, and shortening the delay for a neural network that processes image data to operate at a later time on a chip.

Brief Description of the Drawings

[0012] To more clearly explain the embodiments of the present disclosure or the technical solutions in the prior art, the drawings that need to be used in the following description of the embodiments or the prior art will be briefly described below. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and those skilled in the art can obtain other drawings based on these drawings without creative labor.

[0013]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 3C

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8A

Figure 8B

Figure 9A

Figure 9B

Figure 10

Figure 11

Figure 12A

Figure 12B

Figure 13A

Figure 13B

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0014] Specific examples of the present application are referred to in detail, and examples of the present application are illustrated in the drawings. Although the specific examples are combined to describe the present application, it should be understood that the present application is not intended to be limited to the described examples. On the contrary, it is intended to overwrite changes, modifications and equivalents included within the spirit and scope of the present application defined by the appended claims. It should be noted that the method steps described here can be realized by any functional block or functional configuration, and any functional block or functional configuration can be realized as a physical entity or a logical entity, or a combination of both.

[0015] Before using the technical solutions disclosed in each embodiment of the present disclosure, it should be understood that the types, scope of use, usage scenarios, etc. of personal information related to the present disclosure should be notified to the user in an appropriate manner based on relevant laws and regulations, and the permission of the user must be obtained.

[0016] For example, in response to receiving a user's spontaneous request, present information is sent to the user, explicitly presenting to the user that the operation of requesting execution needs to obtain and use the user's personal information. Therefore, based on the presented information, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application, a server, or a storage medium that executes the technical solution of the present disclosure.

[0017] As a selectable and non-limiting embodiment, the form of sending present information to the user in response to receiving a user's spontaneous request may be, for example, in the form of a pop-up that can present present information in text. Also, the pop-up window can be equipped with a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0018] It should be noted that the process of obtaining the above notification and the user's permission is merely schematic and does not limit the embodiments of the present disclosure. Other forms that meet the relevant laws and regulations can also be applied to the embodiments of the present disclosure.

[0019] It should be noted that the data related to this technical solution (including the data itself, the acquisition or use of data, but not limited thereto) should comply with the requirements of the corresponding laws, regulations and related provisions.

[0020] Currently, neural networks and machine learning systems use tensors as the basic data structure. A tensor is a data container, and since most of the data it contains is always numerical data, it is a digital container. The specific numerical values in a tensor may be application data, including, for example, image data, natural language data, etc.

[0021] For example, a scalar is a tensor of rank 0 and can be, for example, 2, 3, or 5. In a specific application scenario, it can be, for example, image data. Here, 2 represents, for example, the grayscale value of one pixel in the image data, 3 represents the grayscale value of one pixel in the image data, and 5 represents, for example, the grayscale value of one pixel in the image data. For example, a vector is a tensor of rank 1 and can be, for example, [0, 3, 20]. A matrix is a tensor of rank 2 and can be, for example,

Number

[0022] The identification process of the above application scenarios receives various application data input by the neural network as tensors and is realized through the calculations of the neural network. As described above, the calculations of the neural network can be composed of a series of tensor operations, and these tensor operations can also be complex geometric transformations of input data of tensors of several ranks. These tensor operations are also called operators and can transform the calculations of the neural network into a computational graph. The computational graph has multiple operators, and between the multiple operators, they may be connected by lines to represent the dependency relationships between the calculations of each operator.

[0023] The types of operators include activation - type operators (e.g., Relu operator, sigmoid operator, Tan operator), two - tensor calculation - type operators (e.g., BiasAdd operator (adding bias operation), MatMul operator (matrix multiplication), Axpy operator (vector addition)), multiple - Tensor calculation - type operators (e.g., EltwiseOp operator (performing one operation of multiplication, addition, or taking the maximum value on the corresponding elements of multiple tensors at corresponding positions)), normalization - type operators (e.g., BatchNorm operator (accelerating the training convergence speed of neural networks), LRN operator (Local Response Normalization, i.e., local response normalization layer), SsdNormalize operator (limiting the range of primary normalization)), feature extraction - type operators (Convolution operator (extracting feature values by region), FullyConnected operator (extracts features using the entire input as a feature map), Correlation operator, DeConvolution operator (maps a single low-dimensional space to a high-dimensional space while maintaining their connection relationships / patterns), DepConvolution operator (uses a method calculated in multiple steps to complete the calculation of a normal 3D core)), overfitting prevention operators (e.g., Pooling operator (divides the input image into multiple rectangular regions and outputs the maximum or average value for each sub-region), Mean operator (a sliding window operator with only the function of taking the average value)), dimension transformation operators (e.g., Flatten operator (combines the start_axis dimension to end_axis dimension input to the tensor into one dimension), Reshape operator (converts the description input to the Tensor into a new shape), FreespaceExtract operator (changes the dimension of h to 1 without changing other dimensions, thereby completing sampling for h, where h is a positive integer), Pack operator (converts a tensor array with dimension R to a tensor with dimension R + 1), Pad operator (performs data augmentation), Permute operator (adjusts the input dimension order of the Tensor), ShuffleChannel (adjusts the arrangement order of the C dimension), Title operator (generates output data by copying the input data the specified number of times in each dimension), transpose operator (transposes and changes the arrangement order of the values in the tensor), reshape operator (converts the tensor to a tensor of another shape), broadcast operator (extends the input tensor to another tensor), gather operator, reverse operator, concat operator (realizes the combination of multiple operators), cast operator, etc. may be included. Also, the user can customize some operators. Since there are too many operator names and specific types of operations, all types, names, and corresponding operations of the operators are not listed one by one here.

[0024] In the prior art, after obtaining the computational graph of a neural network, generally, these operators are sequentially calculated according to the lines between the operators shown in the computational graph to obtain the final calculation result. However, it takes a certain amount of time to perform each calculation for each operator, and in the prior art, these operators have not been optimized to increase the calculation speed of the operators, and the calculation time of the operators has not been reduced.

[0025] For example, when studying the operators in the computational graph of a neural network for processing image data, for some operators, depending on their specific calculation content, they can be moved up and down and then combined or canceled (cancelled out, mutually cancelled) with the calculations of other operators, optimizing these operators and reducing the number of calculated operators or the amount of calculation performed by the operators. For example, it reduces the latency of a neural network for processing image data operating at a late stage on a chip. Also, by combining some specific hardware loading functions of the artificial intelligence chip further developed by the applicant, it is ensured that the calculation result is consistent with the calculation result of these operators before optimization, and it can be realized that the number of calculated operators or the amount of calculation performed by the operators can be reduced. This optimization is performed based on the high-level semantic representation of the computational graph, avoiding the complex and troublesome optimization operations at the instruction level, and can optimize more general scenarios, making it highly versatile.

[0026] FIG. 1 shows an example of a computational graph in a neural network for processing or identifying image data according to an embodiment of the present application.

[0027] For example, a tensor with image data (e.g., the chromaticity values of pixels) is input into the computational graph shown in an example of FIG. 1. The computational graph shows only some of the operators according to an embodiment of the present application to facilitate the reader's viewing. During the calculation of the computational graph, first, calculations are performed on the tensor using a Transpose operator, then calculations are performed on one branch via a Reshape operator, and calculations are performed on the other branch via a Fully connected operator.

[0028] Suppose the tensor is first input into the Transpose operator. The Transpose operator is a tensor operation that does not change the numerical values in the input tensor. The effect of the Transpose operator is to change the order of the dimensions (axes) of the array. For example, for a two-dimensional array, swapping the order of the two dimensions results in a matrix transpose. The Transpose operator can be applied to cases with more dimensions. The input parameter of the Transpose operator is the order of the dimensions of the output array, and the numbers are counted from 0. Taking the tensor input into the Transpose operator as a two-dimensional matrix [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]] or,

Number

Number

[0029] As can be seen from this, the Transpose operator changes the order of the dimensions, that is, changes the shape of the tensor, but does not change the numerical values in the tensor. For example, it is still 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. Of course, the Transpose operator may not change the shape of the tensor. For example, a 3*3 matrix is still a 3*3 matrix after transposition, and the numerical values in the tensor have not changed, but the order of the numerical values in the transposed matrix is different.

[0030] And the tensor calculated by the above Transpose operator

Number

[0031] The specific operation of the Reshape operator is to change the shape attribute of the tensor, and an m*n matrix a can be arranged into a matrix b of size i*j. For example, the Reshape operator (Reshape(A, 2, 6), where A is the input tensor) changes the shape of the above tensor

Number

Number

[0032] Note that the Reshape operator changes the shape of the tensor but does not change the numerical values in the tensor. For example, they are still 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12.

[0033] The Fully connected operator (also called the Full Connection operator) can be regarded as a special convolutional layer, or can be regarded as the product of tensors. It is an operation that takes the entire tensor input as a feature map and extracts features. That is, it performs a linear transformation from one feature space to another feature space, and the output tensor is the weighted sum of the input tensor. For example, the Fully connected operator takes the input tensor (the tensor output by the Transpose operator)

Number

Number

[0034] So far, the specific operation process of the computational graph in the neural network for processing or identifying the image data shown in FIG. 1 has been introduced.

[0035] As can be seen from the example of FIG. 1, the Transpose operator changes the dimension ordering, that is, changes the shape of the tensor, but does not change the numerical values in the tensor. For example, it is still 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. In the computational graph, there are two subsequent operators, the Reshape operator and the Fully connected operator. The numerical values in the tensors output by the Transpose operator to the Reshape operator and the Fully connected operator are the same as the numerical values in the tensor input to the Transpose operator. For operators such as the Transpose operator, in order to optimize the execution speed and execution time of these operators, it is possible to consider whether to make a certain change in the computational graph.

[0036] FIG. 2 shows an optimization method for the computing power of the neural network module by the chip according to the embodiment of the present application.

[0037] As shown in FIG. 2, the method 200 for optimizing the computing power of a neural network module by a chip includes steps 201 to 204. In step 201, the chip obtains the computation graph of the neural network module having each operator. The neural network module receives application data and obtains a processing output result for the application data. In step 202, based on the specific operation of each operator, in the computation graph, for the first operator, the chip performs at least once at least one of the following adjustments: exchanging the position of the first operator with a subsequent operator or a preceding operator in the computation graph, splitting the first operator into two or more similar operators, and inserting a plurality of first operators that can cancel each other out. The first operator is a tensor operation that does not change the numerical values in the input tensor. In step 203, based on the specific operation of each operator, the chip determines the second operator adjacent to the adjusted first operator in the computation graph. The operation of the adjusted first operator and the operation of the second operator can be combined or canceled without affecting the operation result. In step 204, the chip combines or cancels the adjusted first operator and the second operator, and executes the operation of each operator in the combined or canceled computation graph.

[0038] Based on the example of FIG. 1, the specific process of the above method 200 will be described in conjunction with FIGS. 3A to 3C. FIG. 3A shows a schematic diagram of the original computation graph including a plurality of operators according to an embodiment of the present application. FIG. 3A shows a schematic diagram of the original computation graph including a plurality of operators according to an embodiment of the present application. FIG. 3B shows a schematic diagram of splitting the first operator into two or more similar operators in the computation graph according to an embodiment of the present application. FIG. 3C shows a schematic diagram of optimizing the computation graph after adjusting the first operator according to an embodiment of the present application.

[0039] First, in step 201, the chip obtains the computational graph of the neural network module having each operator. The neural network module receives application data and obtains a processing output result for the application data. For example, as shown in FIG. 3A, the chip obtains the computational graph of the neural network module having each operator such as, for example, a Transpose operator, a Reshape operator, and a Fully connected operator. The neural network module receives image data and obtains a processing output result for the image data.

[0040] In step 202, based on the specific operations of each operator, the chip performs at least once at least one of the following adjustments on the first operator in the computational graph: swapping the position of the first operator with a consequent operator or an antecedent operator in the computational graph, splitting the first operator into two or more similar operators, and inserting a plurality of first operators that can cancel each other out. The first operator is a tensor operation that does not change the numerical values in the input tensor. For example, the tensor operation that does not change the numerical values in the input tensor may be a tensor operation that changes the shape or arrangement order of the tensor. As shown in FIG. 3A, the Transpose operator is determined as the first operator of the tensor operation that does not change the numerical values in the input tensor.

[0041] In one embodiment, when the first operator has two or more consequent or antecedent operators that are branched, the first operator is split into two or more similar operators so that each first operator and the consequent operator or the antecedent operator are in one-to-one correspondence.

[0042] Since the Transpose operator has two consequent operators, the branched Reshape operator and the Fully connected operator, as shown in Fig. 3B, each Transpose operator and the consequent operator are in one-to-one correspondence, that is, one Transpose operator corresponds to the consequent Reshape operator, and the other Transpose operator corresponds to the consequent Fully connected operator. The Transpose operator is split into two similar Transpose operators. That is, originally, the operation of the Transpose operator is performed once on the input tensor, and then the operations of the Reshape operator and the Fully connected operator are respectively performed on the tensor output by the Transpose operator to obtain two output tensors and continue to be used for the operations of the consequent operators. After adjustment, one Transpose operator is performed on the input tensor, the operation of the Reshape operator is performed, and the other Transpose operator is performed on the input tensor, and then the operation of the Fully connected operator is performed.

[0043] Of course, in this case, further adjustment may be performed. For example, in Fig. 3C, by exchanging the positions of the Transpose operator and the consequent Reshape operator in the left branch, the operation of the Reshape operator is performed on the input tensor, and then the operation of the Transpose operator is performed. Here, since the operation of the Transpose operator is a tensor operation that does not change the numerical values in the input tensor, after moving it down, the operations of each operator will not change the final operation result. By exchanging the positions of the Transpose operator and the consequent Reshape operator in this way, it is possible to consider whether the (here, moved down) Transpose operator with the exchanged position can be combined or offset with its further antecedent operator or consequent operator (here, the consequent operator).

[0044] In the examples of FIGS. 3A to 3C, the Transpose operator is split and then moved downward in the left branch. Actually, the Transpose operator can move up and down in each branch. In order to split one Transpose operator into the number of Transpose operators according to the number of branches, the Transpose operator can move up and down in each branch, thereby finding opportunities to combine or cancel with other operators.

[0045] In step 203, the chip determines the following second operator adjacent to the adjusted first operator in the computational graph based on the specific operations of each operator. The operation of the adjusted first operator and the operation of the second operator can be combined or canceled without affecting the operation result.

[0046] The chip determines the Fully connected operator adjacent to the split Transpose operator in the computational graph based on the specific operations of each operator. The operation of the split Transpose operator and the Fully connected operator can be combined without affecting the operation result.

[0047] In one embodiment, if the operation of the second operator is the operation of the second operator on the numerical value in the output tensor of the adjusted first operator before being combined or canceled, after being combined or canceled, the adjusted first operator and the second operator are combined as the fourth operator. The operation of the fourth operator is to replace the tensor operation of the adjusted first operator itself by reading the numerical value in the input tensor of the adjusted first operator adjusted in a predetermined order by the chip. The predetermined order is determined based on the tensor operation of the adjusted first operator itself and includes performing the operation of the second operator on the numerical value read by the chip based on the operation of the second operator.

[0048] As shown in the right branch in FIG. 3C, the split Transpose operator and the consequent Fully connected operator can be coalesced as a Fully connected operator with Transpose combined. That is, since the Transpose operator is actually a transposition of a tensor, the originally input tensor was transposed by the Transpose operator, and the Fully connected operator was applied to the transposed tensor. However, after coalescing, the chip reads each numerical value in the input tensor according to the transposed order and directly performs the operation of the Fully connected operator. This is because although the Transpose operator transposes the input tensor, each numerical value in the tensor remains unchanged, only changing its position in the row or column.

[0049] For example, the tensor originally input to the Transpose operator

Number

Number

Number

Number

[0050] The tensor input to the Fully connected operator with Transpose combined is [Number] , that is, the matrix before transposition. The Fully connected operator with Transpose combined reads the values from the first column of the directly input tensor to perform matrix multiplication, then reads the second column, and then reads the third column. That is, the product of 1 and 40 + the product of 4 and 50 + the product of 7 and 60 + the product of 10 and 70 is the first value of the result tensor of the Fully connected operator, the product of 2 and 40 + the product of 5 and 50 + the product of 8 and 60 + the product of 11 and 70 is the second value of the result tensor of the Fully connected operator, and the product of 3 and 40 + the product of 6 and 50 + the product of 9 and 60 + the product of 12 and 70 is the third value of the result tensor of the Fully connected operator.

[0051] Such an operation of selecting and loading (changing from loading the first, second, and third rows of the originally transposed matrix to loading the first, second, and third columns of the matrix before transposition) may be performed by chip loading. That is, in the chip, by loading the numerical values in the tensor input to the Fully connected operator combined with Transpose in a predetermined order, the tensor operation of the divided Transpose operator itself is replaced. In this example, in the hardware operation of the chip loading the numerical values in the tensor input in the predetermined order of loading the first, second, and third columns, the transposition operation of the Transpose operator itself can be replaced. The predetermined order is determined based on the tensor operation of the divided Transpose operator itself. That is, here it is transposition, and the predetermined order should achieve the transposition effect.

[0052] In one embodiment, the chip loading the numerical values in the input tensor of the first operator adjusted in a predetermined order is realized by setting at least one of the stride, offset amount, and loop count for chip loading by software. In this way, the user can flexibly set at least one of the stride, offset amount, and loop count for chip loading by software to change the loading order, thereby replacing some tensor operations of the operators themselves in the chip hardware loading.

[0053] Such a hardware operation for chip loading may specifically have at least one of the stride, offset amount, and loop count for reading / writing from the static random access memory (SRAM) set by software of some static random access memory (SRAM) controllers. For example, the input tensor of the residual in the SRAM

Number

[0054] In step 204, the chip combines or cancels the adjusted first operator and the second operator, and executes the operations of each operator in the combined or canceled computational graph.

[0055] Here, in the computational graph, after adjusting, combining, or canceling the operator, the chip can perform the operations of each operator in the optimized entire computational graph.

[0056] In this way, by optimizing these operators according to the specific operation rules of the operators in the computational graph of a neural network that processes, for example, image data, the number of calculated operators is reduced or the amount of calculation performed by the operators is reduced, and for example, the delay of a neural network that processes image data operating at the post-stage time on a chip is reduced.

[0057] In one embodiment, when the consequent operators of two or more similar first operators are the same operator, the two or more similar first operators are integrated as one operator.

[0058] FIG. 4 shows a schematic diagram of combining two or more similar first operators as one operator and moving them downward in the computational graph according to an embodiment of the present application.

[0059] When the consequent operators of two or more similar first operators are the same operator, that is, as shown in FIG. 4, the consequent operators of three Transpose operators are the same SomeOp operator (referring to an operator, not limited here), that is, when all of the multiple antecedent operators of the SomeOp operator are similar Transpose operators, the three antecedent Transpose operators of the SomeOp operator are combined as one Transpose operator. However, there may be three antecedent operators respectively in the three branches of the three Transpose operators, which are not shown in FIG. 4.

[0060] In order to find opportunities for combination or cancellation with other operators, the combined Transpose operator may be moved upward (that is, exchange positions with the antecedent operator) or downward (that is, exchange positions with the consequent operator). In the example of FIG. 4, the SomeOp operator can move the Transpose operator downward to the consequent operator (not shown) of the SomeOp operator, thereby reducing redundant Transpose operators by combining or canceling with the consequent operator, or creating possibilities for subsequent combination or cancellation operations.

[0061] In the above example, the Transpose operator is moved downward. However, in practice, in order to achieve the purpose of reducing the calculation amount, the Transpose operator may be moved upward to find opportunities for optimization, that is, opportunities to combine or cancel out with adjacent operators. For example, after moving the Transpose operator upward, it is discovered that it is the inverse operation of another adjacent Transpose operator, and thus both can be cancelled out.

[0062] In the above examples, the Transpose operator is moved downward once in each case. However, in order to achieve the purpose of reducing the calculation amount, the Transpose operator may be moved upward or downward multiple times to find opportunities for optimization, that is, opportunities to combine or cancel out with adjacent operators.

[0063] FIG. 5 shows a schematic diagram of continuously moving the Transpose operator downward multiple times and combining it with the Fully connected operator in the computational graph according to the embodiment of the present application.

[0064] As shown in FIG. 5, after continuously moving the Transpose operator downward twice (that is, exchanging positions with the subsequent SomeOp operator twice), it is combined with the Fully connected operator. As described above, this combination can achieve the same meaning of the Transpose operator, that is, the transpose effect, by setting at least one of the stride, offset amount, and number of loop iterations for reading / writing from the static random access memory (SRAM) by some hardware operations read by this chip, such as some static random access memory (SRAM) controllers. In this way, the Transpose operator that has been continuously moved downward twice can be combined with the adjacent subsequent Fully connected operator as a Fully connected operator with the Transpose operator combined.

[0065] In addition to the Fully connected operator that can be combined with the Transpose operator, such optimized operators that support combination with the Transpose operator include the reshape operator, the broadcast operator, the gather operator, the reverse operator, the concat operator, the flatten operator, the cast operator, the elementwise binary operator, the batch fully connected operator, the convolution operator, and. Other Transpose operators that cannot be canceled but can be combined may also be used.

[0066] In one embodiment, if the first operator and the second operator that are adjusted before being combined or canceled are inverse operations of each other, the first operator and the second operator are canceled.

[0067] FIG. 6 shows a schematic diagram of canceling the first operator and the second operator in the computational graph according to the embodiment of the present application.

[0068] As shown in FIG. 6, in the original computational graph on the left, the upper Transpose operator and the lower Transpose operator are inverse operations of each other. For example, the upper Transpose operator transposes a 3*4 matrix into a 4*3 matrix, and the lower Transpose operator transposes a 4*3 matrix into a 3*4 matrix. Therefore, the two Transpose operators are clearly inverse operations. In the original computational graph, since the two Transpose operators are not adjacent, they cannot be directly canceled. Therefore, by moving the upper Transpose operator down continuously twice (exchanging positions with the consequent operator twice), it can be adjacent to the lower Transpose operator, and these two Transpose operators can be canceled and deleted from each other.

[0069] In this way, the number of calculated operators is reduced or the amount of calculation performed by the operators is reduced, for example, reducing the delay of a neural network that processes image data from operating at a consequent time on the chip.

[0070] In one embodiment, if the second operator is a tensor operation that does not change the numerical values in the input tensor before being combined or canceled out, after being combined or canceled out, the adjusted first operator and the second operator are combined as a third operator, and the tensor operation of the third operator realizes the tensor operations of the adjusted first operator and the second operator together.

[0071] That is, the first operator is a tensor operation that does not change the numerical values in the input tensor, and the adjacent second operator is also a tensor operation that does not change the numerical values in the input tensor, but the two are not the same, not inverse operations to each other, but different forms or different types of tensor operations.

[0072] For example, the first operator may be at least one of a transpose operator, a reshape operator, a broadcast operator, a gather operator, a reverse operator, a concat operator, and a cast operator. The second operator may be at least one of a transpose operator, a reshape operator, a broadcast operator, a gather operator, a reverse operator, a concat operator, and a cast operator. For example, the first operator is a Transpose operator, and the second operator is also a Transpose operator. However, the two Transpose operators are neither exactly the same operation nor inverse operations.

[0073] FIG. 7 shows a schematic diagram of combining one Transpose operator after moving it downward in the computational graph according to the embodiment of the present application with an adjacent Transpose operator as one Transpose operator.

[0074] As shown in FIG. 7, in the computational graph, first, one Transpose operator can be moved downward, and then an adjacent Transpose operator can be found and combined. Therefore, the Transpose operator moved downward and the adjacent Transpose operator are combined as one combined Transpose operator. Since the operations of the Transpose operators all change the shape of the tensor, the combined Transpose operator realizes the tensor operation of the Transpose operator moved downward and the tensor operation of the adjacent Transpose operator together. That is, instead of performing two tensor operations of the Transpose operator in two steps to achieve the final result, by using the Transpose operator once, the shape of the tensor can be changed to achieve the final result.

[0075] As described above, the Transpose operator is taken as an example of the first operator that can be combined or canceled with other operators. Other operators will be introduced as the first operator below.

[0076] In one embodiment, the first operator may be a Reshape operator. The Reshape operator converts one tensor into a tensor of another shape. Note that the operation of the Reshape operator is a tensor operation that does not change the numerical values in the input tensor.

[0077] FIG. 8A shows a schematic diagram of canceling a Reshape operator that is an inverse operation of the Reshape operator in the computational graph according to an embodiment of the present application.

[0078] As shown in FIG. 8A, in the original computational graph on the left side, the upper Reshape operator and the lower Reshape operator are inverse operations to each other. For example, the operation of the upper Reshape operator changes a matrix with a shape of 3*4 into a matrix with a shape of 2*6, and the operation of the lower Reshape operator changes a matrix with a shape of 2*6 into a matrix with a shape of 3*4. The two Reshape operators are inverse operations. In the original computational graph, since the two Reshape operators are not adjacent to each other, they cannot be directly canceled out. Therefore, if the upper Reshape operator is continuously moved down twice (the positions are exchanged with the subsequent operator twice), it can be adjacent to the lower Reshape operator, and then these two Reshape operators can be mutually canceled out and deleted.

[0079] In this way, the number of calculated operators is reduced or the amount of calculation performed by the operators is reduced. For example, the delay of a neural network that processes image data operating at a subsequent time in the chip is reduced.

[0080] FIG. 8B shows a schematic diagram of combining one Reshape operator that has been moved down with an adjacent Reshape operator as one Reshape operator in the computational graph according to an embodiment of the present application.

[0081] As shown in FIG. 8B, in the computational graph, first, one Reshape operator can be moved down and then an adjacent Reshape operator can be found and combined. Therefore, the Reshape operator that has been moved down and the adjacent Transpose operator are combined as one combined Reshape operator. Since the operations of the Reshape operators all change the shape of the tensor, the combined Reshape operator realizes the tensor operation of the Reshape operator that has been moved down and the tensor operation of the adjacent Reshape operator together. That is, instead of performing the two tensor operations of the Reshape operator in two steps to achieve the final result, if the Reshape operator is used once, the shape of the tensor can be changed to achieve the final result.

[0082] For example, the operation of the upper Reshape operator changes a matrix with a shape of 3*4 into a matrix with a shape of 2*6, and the operation of the lower Reshape operator changes a matrix with a shape of 2*6 into a matrix with a shape of 6*2. Then, the operation of the combined Reshape operator may change a matrix with a shape of 3*4 into a matrix with a shape of 6*2.

[0083] Of course, taking the Reshape operator as an example above is just some optimization forms and not a limitation. Actually, it is also possible to combine the Reshape operator with other operators (the second operator). Since it is a tensor operation that does not change the numerical values in the input tensor, by reading the numerical values in the tensor input to the Reshape operator adjusted in a predetermined order in the chip, the tensor operation of the adjusted Reshape operator itself can be replaced.

[0084] In one embodiment, the first operator may be a Broadcast operator. The Broadcast operator expands the input tensor into another tensor. For example, it expands a tensor with a shape of A into a tensor with a shape of B, where A and B are tensor shapes represented by dimensions. Note that the operation of the Broadcast operator is a tensor operation that does not change the numerical values in the input tensor.

[0085] FIG. 9A shows a schematic diagram of a computational graph that optimizes after splitting a Broadcast operator according to an embodiment of the present application into two Broadcast operators.

[0086] The Broadcast operator has two consequent operators, namely a branched Reshape operator and a Binary operator. As shown in FIG. 9A, each Broadcast operator corresponds one-to-one with the consequent operators, that is, one Broadcast operator corresponds to the consequent Reshape operator, and the other Broadcast operator corresponds to the consequent Binary operator. The Broadcast operator is split into two similar Broadcast operators. That is, originally, the operation of the Broadcast operator is performed once on the input tensor, and then the operations of the Reshape operator and the Binary operator are respectively performed on the tensor output by the Broadcast operator to obtain two output tensors and continue to be used in the operations of the consequent operators. After adjustment, one Broadcast operator is performed on the input tensor, and then the operation of the Reshape operator is performed, and the other Broadcast operator is performed on the input tensor, and then the operation of the Binary operator is performed. The Binary operator, also called the Elementwise Binary operator, binarizes the numerical values in the input tensor.

[0087] Of course, in this case, further adjustment may be performed. For example, in FIG. 9A, by exchanging the positions of the Broadcast operator and the consequent Reshape operator in the left branch, the operation of the Reshape operator is performed on the input tensor first, and then the operation of the Broadcast operator is performed. Here, since the operation of the Broadcast operator is a tensor operation that does not change the numerical values in the input tensor, after moving it down, the operations of each operator will not change the final operation result. By exchanging the positions of the Broadcast operator and the consequent Reshape operator in this way, it is possible to consider whether the (here, moved down) Broadcast operator after the position exchange and its further antecedent operator or consequent operator (here, the consequent operator) can be combined or canceled out.

[0088] As shown in the right branch in FIG. 9A, the split Broadcast operator and the consequent Binary operator are combined as a Binary operator with Broadcast combined. That is, since the Broadcast operator is actually an expansion of the tensor, the originally input tensor is expanded by the Broadcast operator, and then the Binary operator operation is performed on the expanded tensor. After being combined, the chip reads each numerical value in the tensor input to the Broadcast operator according to the order of the expanded tensor. For example, it reads more numerical values that are expanded (i.e., copied), and directly performs the Binary operator operation on the read numerical values. This is because the Broadcast operator expands the input tensor but does not change each numerical value in the tensor, only changing the number of times each numerical value appears in the tensor.

[0089] If the Broadcast operator changes [1, 2, 4] to [[1, 1], [2, 2], [4, 4]], and the binary operator takes 1 when it is greater than 3 and 0 when it is less than 3, then the Binary operator combined with Broadcast can be realized by the chip reading in a predetermined order. For example, the chip reads the position where 1 is stored twice, performs the Binary operator operation each time it reads, and determines that it is 0, 0. The chip reads the position where 2 is stored twice, performs the Binary operator operation each time it reads, and determines that it is 0, 0. The chip reads the position where 4 is stored twice, performs the Binary operator operation each time it reads, and determines that it is 1, 1. Finally, the tensor output by the Binary operator combined with Broadcast is [[0, 0], [0, 0], [1, 1]].

[0090] FIG. 9B shows a schematic diagram of moving the Broadcast operator downward and combining it with the Binary operator in the computational graph according to the embodiment of the present application.

[0091] As shown in FIG. 9B, after moving the Broadcast operator downward (i.e., exchanging positions with the consequent Reshape operator), it is combined with the Binary operator. As described above, this combination may achieve the same meaning of the Broadcast operator, i.e., the expansion effect, by setting at least one of the stride, offset amount, and number of loop iterations for the hardware operation read by this chip, for example, the stride for reading / writing by some static random access memory (SRAM) controllers from the static random access memory (SRAM). In this way, the downward-moved Broadcast operator can be combined with the adjacent consequent Binary operator as a Binary operator combined with the Broadcast operator.

[0092] In addition to the Binary operator that can be combined with the Broadcast operator, such an optimized operator that supports combination with the Broadcast operator may be, for example, a batch fully connected operator (also called a batch matmul operator).

[0093] In one embodiment, the first operator may be a Gather operator. The operation of the Gather operator selects a tensor that outputs some numerical values from some numerical values in the input tensor. Note that the operation of the Gather operator is a tensor operation that does not change the numerical values in the input tensor.

[0094] FIG. 10 shows a schematic diagram of combining two Gather operators into one Gather operator and moving it downward according to an embodiment of the present application.

[0095] If the consequent operators of two or more similar first operators are the same operator, i.e., as shown in FIG. 10, the consequent operators of the two Gather operators are the same SomeOp operator, then the two antecedent Gather operators of the SomeOp operator can be combined into one Gather operator.

[0096] In order to find opportunities to combine or cancel with other operators, the combined Gather operator can be continuously moved downward (i.e., exchange positions with the consequent operator). In the example of FIG. 10, the SomeOp operator can move the Gather operator downward to the consequent operator (not shown) of the SomeOp operator, thereby reducing redundant Gather operators by combining or canceling with the consequent operator, or creating possibilities for subsequent combining or canceling operations.

[0097] In one embodiment, the first operator may be a Reverse operator. The function of the Reverse operator is to reverse the order of data in a certain dimension of the input tensor.

[0098] FIG. 11 shows a schematic diagram of moving the Reverse operator downward and combining it with the Convolution operator according to an embodiment of the present application.

[0099] The operation of the Convolution operator is convolution to obtain weights. The Reverse operator can be combined with the Convolution operator. If the tensor input to the Convolution operator is operated by the Reverse operator in a certain direction and then the layout of the weights of the Convolution operator is changed according to the operation rules of the Reverse operator, it can conform to the same meaning of the original Convolution operator and the output will not change. Therefore, the purpose of optimizing the redundant Reverse operator is achieved by moving the Reverse operator downward and combining it with the Convolution operator.

[0100] Above, embodiments of combining the Reshape operator and the Transpose operator with operators of the same type have been introduced. Furthermore, other operators can be combined with operators of the same type.

[0101] FIGS. 12A and 12B respectively show schematic diagrams of combining the Concat operator and the Cast operator according to embodiments of the present application.

[0102] The operation of the Concat operator combines two or more tensors. For example, it combines a 3*4 tensor and a 3*6 tensor into a 3*10 tensor. Note that the Concat operator is a tensor operation that does not change the numerical values in the input tensors.

[0103] As shown in FIG. 12A, the Concat operator can be combined with another adjacent Concat operator to form a Concat operator. For example, the operation of the Concat operator combines a 3*4 tensor and a 3*6 tensor (two input tensors, refer to the two lower input arrows above the left Concat operator in FIG. 12A) into a 3*10 tensor, and the operation of another Concat operator combines the tensor output by the left Concat operator (3*10 tensor) and also two input tensors, for example, a 3*5 tensor and a 3*7 tensor (also two input tensors, refer to the two lower input arrows in the upper right of the middle Concat operator in FIG. 12A) into a 3*22 tensor. The operation of the Concat operator formed by combining these two Concat operators may also be to input four tensors, namely a 3*4 tensor, a 3*6 tensor, a 3*5 tensor, and a 3*7 tensor, and combine them all into a 3*22 tensor.

[0104] In this way, the number of calculated operators is reduced or the amount of calculation performed by the operators is reduced, for example, reducing the delay of a neural network for processing image data that operates at a later time on the chip.

[0105] The operation of the Cast operator performs type conversion on numerical values. For example, it converts the integer 1 to the floating-point number 1.0, or converts a 32-bit integer type to a 64-bit integer type. Note that the Cast operator only changes the type of the numerical value and does not change the numerical values in the input tensors.

[0106] As shown in FIG. 12B, the conversion from Cast A at the upper left to B represents the conversion of the tensor from type A to type B, and the conversion from Cast B at the lower left to C represents the conversion of the tensor from type B to type C. The operation of the Cast operator combining these two Cast operators may be from Cast A to C, that is, the tensor of type A may be directly converted to type C.

[0107] In this way, the number of calculated operators is reduced or the calculation amount calculated by the operators is reduced. For example, the delay of a neural network that processes image data and operates at a later time on a chip is reduced.

[0108] In one embodiment, in the adjustment process of the first operator, since these multiple first operators can perform operations such as combining or canceling with other adjacent operators, the insertion of multiple first operators that can cancel each other can be considered. Here, mutual cancellation may be an inverse operation to each other.

[0109] FIG. 13A shows a schematic diagram of an embodiment in which, by inserting multiple first operators that can cancel each other according to an embodiment of the present application, these multiple first operators can perform operations of combining or canceling with other adjacent operators.

[0110] First, there are two formats for the image data channel, namely the NCHW layout, also called "channels_first", and the NHWC, also called "channels_last". In the NCHW layout, since C is on the outside, within each channel, the image pixels are adjacent, that is, "RRRGGGBBB", but in the NHWC layout, since C is on the innermost side, within each channel, the image pixels are spaced apart, that is, "RGBRGBRGB".

[0111] The formats of some neural network models support the Convolution operator in NCHW layout by default. However, when performing operations on a chip with NHWC layout, in the computational graph, it is necessary to convert the NHWC layout to the NCHW layout, perform NCHW convolution, and then convert the convolution result to the NHWC layout. In the computational graph, as shown on the left side of Figure 13A, it passes through the Transpose NHWC2NCHW operator, then through the NCHW_Convolution operator, and then through the Transpose NCHW2NHWC operator.

[0112] To convert this layout by inserting multiple Transposes that can cancel each other out in the original computational graph, then move down, optimize the entire computational graph in a form of combining and / or canceling out, the cost of the converted layout can be minimized, and in the optimal case, the cost can be eliminated, that is, all inserted Transposes can be optimized.

[0113] For example, as shown in Figure 13A, in the first step, two mutually cancelable Transpose operators of the Transpose NCHW2NHWC operator and the Transpose NHWC2NCHW operator (which are inverse operations to each other) are inserted below the original Transpose NHWC2NCHW operator and above the Transpose NCHW2NHWC operator respectively. In the second step, since the inserted Transpose NCHW2NHWC operator and the Transpose NHWC2NCHW operator in the original computational graph are inverse operations, they can be canceled out, and since the inserted Transpose NHWC2NCHW operator and the Transpose NCHW2NHWC operator in the original computational graph are inverse operations, they can be canceled out. In this way, the original computational graph can be optimized so that only the NHWC_Convolution operator remains.

[0114] In this way, the number of calculated operators is reduced or the amount of calculation performed by the operators is reduced, for example, reducing the latency of a neural network that processes image data from operating at a late stage on the chip.

[0115] FIG. 13B shows a schematic diagram of another embodiment in which a plurality of mutually cancellable first operators according to an embodiment of the present application are inserted, and these plurality of first operators can perform a combining or cancelling operation with other adjacent operators.

[0116] First, the input of the operator of the neural network model can be represented by a left-hand-side (LHS) input tensor and a right-hand-side (RHS) input tensor in the case of two inputs.

[0117] Some neural network model formats support a fully connected operator (i.e., a matmul operator) without transposing the RHS (the second input) by default, but some chips require a transpose to be attached to the rhs of the fully connected operator (i.e., the matmul operator). When performing the operation of the fully connected operator on two tensors, attention must be paid to the problem of dimension matching. For example, in order to succeed, it is necessary to multiply a 3*4 tensor by a 4*5 tensor, that is, only when the number of columns 4 of the first tensor is the same as the number of rows 4 of the second tensor, can we multiply and add one by one, that is, perform matrix multiplication. Therefore, in order to perform the operation of another input tensor and the subsequent fully connected operator smoothly, there is an NC_Transpose operator in the original computational graph to convert the dimension of one input tensor.

[0118] Therefore, two mutually cancellable Transposes are inserted into the original computational graph to transform the layout, and then moved down, combined, and cancelled to optimize the entire computational graph in a way that minimizes the cost of the transformed layout, and in the optimal case, the cost can be eliminated, that is, all inserted Transposes can be optimized. Similarly, when inserting a Transpose to transform the rhs, there is a weight for the Convolution, and it supports converting the weight from NCHW or HWCN to NHWC.

[0119] As shown in Figure 13B, in the first step, two Transpose operators that are inverse operations of each other and mutually cancellable are inserted into the original computational graph on the left side of Figure 13B. The NC_Transpose operator (shown shaded in the figure and needs to be set as the inverse operation of the original NC_Transpose operator), and the NC_Transpose operator that is an inverse operation of each other are inserted below the original NC_Transpose operator and above the Fully connected operator respectively. In the second step, the NC_Transpose operator inserted above the Fully connected operator and the Fully connected operator in the original computational graph are combined to form a Fully connected operator with RHS Transpose (for example, the Fully connected rhs transposed shown in Figure 13B). In the third step, since the inserted NC_Transpose operator (shown shaded in the figure) and the NC_Transpose operator in the original computational graph are inverse operations, they can be cancelled. As a result, the original computational graph is optimized to only the Fully connected operator with RHS Transpose.

[0120] In this way, the number of calculated operators is reduced or the amount of calculation performed by the operators is reduced, for example, reducing the latency of a neural network that processes image data from operating at a late stage on the chip.

[0121] The combination of one adjusted first operator and one adjacent second operator has been introduced above. The combined operator can further move up and down to perform an optimization operation of combining or canceling with other adjacent operators, that is, continuous combination and possible cancellation can be performed.

[0122] Thus, for some operators, depending on their specific operation content, they can be inserted in the computational graph or moved up and down and then combined or canceled with the operations of other operators, thereby optimizing these operators, reducing the number of calculated operators or the amount of calculation performed by the operators, for example, reducing the delay of a neural network for processing image data when operating at a later time on a chip. Also, by combining some specific hardware loading functions of the artificial intelligence chip further developed by the applicant, it is ensured that the calculation result is consistent with the calculation result of these operators before optimization, and it is realized that the number of calculated operators or the amount of calculation performed by the operators can be reduced. This optimization is performed based on the high-level semantic representation of the computational graph, avoiding the complex and cumbersome optimization operations at the instruction level, and can optimize more general scenarios, making it highly versatile.

[0123] Summarizing the above, each embodiment of the present application can achieve at least the following effects.

[0124] 1. Split the Transpose operator in the computational graph, move it down, combine or cancel it to perform combined optimization, change the scheduling order of the Transpose operator, and finally realize the layout transformation of the computational graph, reduce the calculation amount of redundant operators, and optimize the delay during the execution of the computational graph.

[0125] 2. Move the Reshape operator in the computational graph down, combine or cancel it to perform combined optimization, change the scheduling order of the Reshape operator, and finally reduce the redundant calculation by the Reshape operator in the computational graph and optimize the delay during the execution of the computational graph.

[0126] 3. The Broadcast operator can expand the shape of the tensor, split the Broadcast operator in the computational graph, move it downward, combine and perform combination optimization, reduce the calculation amount of the Broadcast consequent operator in the original computational graph, and finally optimize the latency during the execution of the computational graph.

[0127] 4. Since the Gather operator generally selects some data in the input as the output, the amount of data output from the gather operator is generally smaller than the amount of data input. However, in some chips, when the difference between the amount of input data and the amount of output data is smaller than a certain threshold, calculating the gather operator twice and making the cost of the consequent node of the gather operator for calculating a small amount of data much larger than the amount of data calculated by the consequent node, but slightly larger than the cost of calculating the gather operator only once. Therefore, the technique of combining and moving down two similar gather operators achieves the optimization effect.

[0128] 5. By changing the layout of the weights of the Convolution operator, the antecedent Reverse operator input to the Convolution operator can be combined. Combining this feature, moving it downward and combining it can optimize redundant Reverse operators and achieve the optimization goal of reducing the calculation amount.

[0129] 6. Based on the mathematical meaning, operators can be continuously combined. Specifically, in a chip, based on chip parameters, selectively combine to reduce redundant operators, thereby achieving the optimization goal of reducing the calculation amount.

[0130] 7. Depending on different specific scenarios, in different chips, this optimization technique can move downward, combine and cancel each other to convert the order, and flexibly combine while meeting the requirement that the mathematical meaning remains unchanged, thereby achieving the optimization effect.

[0131] Using an 8.1GHz simulator to simulate the above process, when the above optimization operation is not performed, the inference delay of 4pe 4batch is 4562102 ns. After the optimization operation, the inference delay becomes 4450052 ns. That is, in this model, the optimization percentage of the delay of the chip simulator executing the neural network at the post-condition runtime is 2.46%, and the absolute value is 112.05 us.

[0132] Figure 14 shows a schematic block diagram of an artificial intelligence chip according to an embodiment of the present application.

[0133] As shown in Figure 14, the artificial intelligence chip 1400 stores the received application data, inputs it into the computational graph of the neural network module having each operator, and obtains a processing output result for the application data, and based on the specific operations of each operator, in the computational graph, for the first operator, at least one of the adjustments of exchanging the position of the first operator with the subsequent operator or the previous operator in the computational graph, splitting the first operator into two or more similar operators, and inserting a plurality of first operators that can cancel each other out is performed at least once. The first operator is a tensor operation that does not change the numerical values in the input tensor, and based on the specific operations of each operator, the following second operator adjacent to the adjusted first operator in the computational graph is determined, and the operation of the adjusted first operator and the operation of the second operator can be combined or canceled without affecting the operation result, and the chip combines or cancels the adjusted first operator and the second operator and executes the operations of each operator in the combined or canceled computational graph, including a controller 1402.

[0134] In one embodiment, the application data includes at least one of image data and natural language data.

[0135] In one embodiment, when the chip is a computational graph, for the first operator, the step of performing at least once at least one of the following adjustments: exchanging the position of the first operator with the consequent operator or the antecedent operator in the computational graph, splitting the first operator into two or more similar operators, and inserting a plurality of first operators that can cancel each other out; when the first operator has two or more consequent operators or antecedent operators that are branched, the step of splitting the first operator into two or more similar operators so that each first operator corresponds one-to-one with the consequent operator or the antecedent operator; and when the consequent operators of two or more similar first operators are the same operator, the step of integrating the two or more similar first operators into one operator.

[0136] In one embodiment, the step of the chip combining or canceling the adjusted first operator and the second operator and performing the operations of each operator in the combined or canceled computational graph: when the operation of the second operator is an operation of the second operator on the numerical value in the output tensor of the adjusted first operator before being combined or canceled, after being combined or canceled, the step of combining the adjusted first operator and the second operator as the fourth operator is included, and the operation of the fourth operator is a step of replacing the tensor operation of the adjusted first operator itself by reading the numerical value in the input tensor of the adjusted first operator in a predetermined order by the chip, where the predetermined order is determined based on the tensor operation of the adjusted first operator itself, and the step of performing the operation of the second operator on the numerical value read by the chip based on the operation of the second operator.

[0137] In one embodiment, the chip reading the numerical value in the input tensor of the adjusted first operator in a predetermined order is realized by software setting at least one of the read stride, offset amount, and number of loop iterations by the chip.

[0138] In one embodiment, the step of combining or canceling the adjusted first operator and the second operator by the chip and executing the operations of each operator in the combined or canceled computational graph includes, when the second operator is a tensor operation that does not change the numerical values in the input tensor before being combined or canceled, after being combined or canceled, combining the adjusted first operator and the second operator as a third operator, and the tensor operation of the third operator realizes the tensor operations of the adjusted first operator and the second operator together.

[0139] In one embodiment, the step of combining or canceling the adjusted first operator and the second operator by the chip and executing the operations of each operator in the combined or canceled computational graph includes, when the adjusted first operator and the second operator are inverse operations to each other before being combined or canceled, the step of canceling the first operator and the second operator.

[0140] In one embodiment, the first operator is at least one of a transpose operator, a reshape operator, a broadcast operator, a gather operator, a reverse operator, a concat operator, and a cast operator.

[0141] In one embodiment, the second operator is at least one of a transpose operator, a reshape operator, a broadcast operator, a gather operator, a reverse operator, a concat operator, a flatten operator, a cast operator, an elementwise binary operator, a batch fully connected operator, and a convolution operator.

[0142] In this way, for some operators, depending on their specific operation content, they can be inserted in the computational graph or moved up and down and then combined or canceled with the operations of other operators, thereby optimizing these operators, reducing the number of calculated operators or the amount of calculation performed by the operators, for example, reducing the latency of a neural network that processes image data and operates at a later time on a chip. Further, by combining some specific hardware loading functions of the artificial intelligence chip developed by the applicant, it is ensured that the operation results are consistent with those of these operators before optimization, and it is realized that the number of calculated operators or the amount of calculation performed by the operators can be reduced. This optimization is performed based on a high-level semantic representation called a computational graph, avoiding the complex and cumbersome optimization operations at the instruction level, and can optimize more general scenarios, making it highly versatile.

[0143] FIG. 15 shows a block diagram of an exemplary electronic device suitable for implementing the embodiments of the present application according to the embodiments of the present application.

[0144] The electronic device may include a processor (H1) and a storage medium (H2) coupled to the processor (H1) and storing computer-executable instructions that, when executed by the processor, perform the steps of each method of the embodiments of the present application.

[0145] The processor (H1) may include, for example, but is not limited to, one or more processors or microprocessors.

[0146] The storage medium (H2) may include, for example, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (such as hard disks, floppy disks, solid-state hard disks, removable disks, CD-ROMs, DVD-ROMs, Blu-ray disks, etc.).

[0147] In addition to this, the electronic device may further include a data bus (H3), an input / output (I / O) bus (H4), a display (H5), and input / output devices (H6) (for example, a keyboard, a mouse, a speaker, etc.).

[0148] The processor (H1) can communicate with external devices (such as H5, H6, etc.) via a wired or wireless network (not shown) by means of the I / O bus (H4).

[0149] The storage medium (H2) can further store at least one computer-executable instruction that, when executed by the processor (H1), performs each function and / or method step in the embodiments described in this technology.

[0150] In one embodiment, the at least one computer-executable instruction may be compiled or configured as a software product, and when one or more computer-executable instructions are executed by the processor, they perform each function and / or method step in the embodiments described in this technology.

[0151] FIG. 16 shows a schematic diagram of a non-transitory computer-readable storage medium according to an embodiment of the present application.

[0152] As shown in FIG. 16, instructions are stored in the computer-readable storage medium 1620, and the instructions are, for example, computer-readable instructions 1610. When the computer-readable instructions 1610 are executed by a processor, the methods described above can be executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or high-speed cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash, etc. For example, the computer-readable storage medium 1620 can be connected to a computing device such as a computer, and when the computing device executes the computer-readable instructions 1610 stored in the computer-readable storage medium 1620, various methods as described above can be executed.

[0153] This application may include the following items.

[0154] Item 1. A method for optimizing the computing power of a neural network module by a chip, comprising: obtaining, by a chip, a computational graph of a neural network module having each operator, the neural network module receiving application data and obtaining a processing output result for the application data; based on the specific operations of each operator, the chip performing at least once at least one of the following adjustments on the computational graph for the first operator: exchanging the position of the first operator with a subsequent operator or a preceding operator in the computational graph, splitting the first operator into two or more similar operators, and inserting a plurality of first operators that can cancel each other out, the first operator being a tensor operation that does not change the numerical value in the input tensor; The step in which the chip determines the following second operator adjacent to the adjusted first operator in the computational graph based on the specific operations of each operator, where the operation of the adjusted first operator and the operation of the second operator can be combined or canceled without affecting the operation result. Including the step in which the chip combines or cancels the adjusted first operator and the second operator, and executes the operations of each operator in the combined or canceled computational graph.

[0155] Item 2. The method according to item 1, wherein the application data includes at least one of image data and natural language data.

[0156] Item 3. The step in which the chip performs at least one of the following adjustments on the first operator in the computational graph at least once: an adjustment to exchange the position of the first operator with a subsequent operator or a preceding operator in the computational graph, an adjustment to divide the first operator into two or more similar operators, and an adjustment to insert a plurality of first operators that can cancel each other out. When the first operator has two or more subsequent operators or preceding operators that are branched, the step of dividing the first operator into two or more similar operators so that each first operator corresponds one-to-one with a subsequent operator or a preceding operator. The method according to item 1, including the step of integrating two or more similar first operators into one operator when the subsequent operators of the two or more similar first operators are the same operator.

[0157] Item 4. The step in which the chip combines or cancels the adjusted first operator and the second operator, and executes the operations of each operator in the combined or canceled computational graph. Before combining or canceling, when the operation of the second operator is an operation of the second operator on the numerical values in the output tensor of the adjusted first operator, after combining or canceling, the method includes the step of combining the adjusted first operator and the second operator as a fourth operator, and the operation of the fourth operator is to replace the tensor operation of the adjusted first operator itself by reading the numerical values in the input tensor of the adjusted first operator in a predetermined order by the chip, where the predetermined order is determined based on the tensor operation of the adjusted first operator itself, and the step of performing the operation of the second operator on the numerical values read by the chip based on the operation of the second operator. The method according to item 1.

[0158] Item 5. Reading the numerical values in the input tensor of the adjusted first operator in a predetermined order by the chip is realized by software setting at least one of the stride, offset amount, and number of loop iterations for reading by the chip. The method according to item 4.

[0159] Item 6. The step of combining or canceling the adjusted first operator and the second operator by the chip and executing the operation of each operator in the combined or canceled computational graph is Before combining or canceling, when the second operator is a tensor operation that does not change the numerical values in the input tensor, after combining or canceling, the method includes the step of combining the adjusted first operator and the second operator as a third operator, and the tensor operation of the third operator realizes the tensor operation of the adjusted first operator and the tensor operation of the second operator together. The method according to item 1.

[0160] Item 7. The step of combining or canceling the adjusted first operator and the second operator by the chip and executing the operation of each operator in the combined or canceled computational graph is Before combining or canceling, when the adjusted first operator and the second operator are inverse operations to each other, the method according to item 1 includes the step of canceling the first operator and the second operator.

[0161] Item 8. The method according to Item 1, wherein the first operator is at least one of a transpose operator, a reshape operator, a broadcast operator, a gather operator, a reverse operator, a concat operator, and a cast operator.

[0162] Item 9. The method according to Item 1, wherein the second operator is at least one of a transpose operator, a reshape operator, a broadcast operator, a gather operator, a reverse operator, a concat operator, a flatten operator, a cast operator, an elementwise binary operator, a batch fully connected operator, and a convolution operator.

[0163] Item 10. An artificial intelligence chip, a memory that stores received application data and inputs it into a computational graph of a neural network module having each operator to obtain a processing output result for the application data; based on the specific operations of each operator, in the computational graph, for the first operator, at least one of the following adjustments is performed at least once: an adjustment to exchange the position of the first operator with a subsequent operator or a preceding operator in the computational graph, an adjustment to divide the first operator into two or more similar operators, and an adjustment to insert a plurality of first operators that can cancel each other out, and the first operator is a tensor operation that does not change the numerical values in the input tensor; based on the specific operations of each operator, determine the following second operator adjacent to the adjusted first operator in the computational graph, and the operations of the adjusted first operator and the second operator can be combined or canceled without affecting the operation results; including a controller that combines or cancels the adjusted first operator and the second operator by the chip and executes the operations of each operator in the combined or canceled computational graph.

[0164] Item 11. The chip according to Item 10, wherein the application data includes at least one of image data and natural language data.

[0165] Item 12. The chip according to Item 10, wherein the step of the chip performing at least once at least one of the following adjustments on the first operator in the computational graph: exchanging the position of the first operator with a consequent operator or an antecedent operator in the computational graph; splitting the first operator into two or more similar operators; and inserting a plurality of first operators that can cancel each other out, includes: when the first operator has two or more consequent operators or antecedent operators that are branched, splitting the first operator into two or more similar operators so that each first operator corresponds one-to-one with a consequent operator or an antecedent operator; when the consequent operators of two or more similar first operators are the same operator, integrating the two or more similar first operators as one operator.

[0166] Item 13. The chip according to Item 10, wherein the step of the chip combining or canceling the adjusted first operator and the second operator and performing the operation of each operator in the combined or canceled computational graph includes: before combining or canceling, when the operation of the second operator is an operation of the second operator on the numerical value in the output tensor of the adjusted first operator, after combining or canceling, combining the adjusted first operator and the second operator as a fourth operator, and the operation of the fourth operator is a step of replacing the tensor operation of the adjusted first operator itself by reading the numerical value in the input tensor of the adjusted first operator by the chip in a predetermined order, where the predetermined order is determined based on the tensor operation of the adjusted first operator itself, and performing the operation of the second operator on the numerical value read by the chip based on the operation of the second operator.

[0167] Item 14. Reading the numerical values in the input tensor of the first operator adjusted in a predetermined order by the chip is realized by setting at least one of the stride, offset amount, and number of loop iterations for reading by the chip by software, the chip according to Item 13.

[0168] Item 15. The step of combining or canceling the first operator adjusted by the chip and the second operator and executing the operations of each operator in the combined or canceled computational graph is If, before being combined or canceled, the second operator is a tensor operation that does not change the numerical values in the input tensor, after being combined or canceled, it includes the step of combining the adjusted first operator and the second operator as a third operator, and the tensor operation of the third operator realizes the tensor operation of the adjusted first operator and the tensor operation of the second operator together, the chip according to Item 10.

[0169] Item 16. The step of combining or canceling the first operator adjusted by the chip and the second operator and executing the operations of each operator in the combined or canceled computational graph is If the adjusted first operator and the second operator are inverse operations to each other before being combined or canceled, it includes the step of canceling the first operator and the second operator, the chip according to Item 10.

[0170] Item 17. The first operator is at least one of a transpose operator, a reshape operator, a broadcast operator, a gather operator, a reverse operator, a concat operator, and a cast operator, the chip according to Item 10.

[0171] Item 18. The chip according to Item 10, wherein the second operator is at least one of a transpose operator, a reshape operator, a broadcast operator, a gather operator, a reverse operator, a concat operator, a flatten operator, a cast operator, an elementwise binary operator, a batch fully connected operator, and a convolution operator.

[0172] Item 19. An electronic device that optimizes the computing power of a neural network module by a chip, a memory that stores instructions, and a processor that reads the instructions in the memory and executes the method according to any one of Items 1 to 9.

[0173] Item 20. A non-transitory storage medium in which instructions are stored, wherein when the instructions are read by a processor, the method according to any one of Items 1 to 9 is executed by the processor.

[0174] Of course, the specific embodiments described above are examples and not limitations. Those skilled in the art can combine some steps and devices from the various embodiments described separately above based on the concept of the present application to achieve the effects of the present application. Such combined embodiments are also included in the present application, and the combinations and combinations thereof will not be described one by one here.

[0175] It should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are illustrative and not limiting. It is not considered that each embodiment of the present application must have these advantages, benefits, effects, etc. Also, the specific details of the above disclosure are not limitations, but only for the purpose of illustration and for ease of understanding. The above details do not limit that the present application must be implemented using the above specific details.

[0176] The block diagrams of the devices, apparatuses, equipment, and systems according to the present disclosure are used only as exemplary examples and are not intended to require or imply that the connections, arrangements, and configurations should be made as shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any way. Terms such as "including," "comprising," and "having" are open terms meaning "including but not limited to" and can be used interchangeably therewith. The terms "or" and "and" used herein mean "and / or" and can be used interchangeably therewith unless the context clearly indicates otherwise. The term "for example" used herein means "for example, without limitation" and can be used interchangeably therewith.

[0177] The step flowcharts and the above-described methods in the present disclosure are described only as exemplary examples and are not intended to require or imply that the steps of each embodiment must be performed in the given order. As those skilled in the art will recognize, the order of the steps in the above embodiments can be performed in any order. Words such as "subsequently," "and," and "next" are not intended to limit the order of the steps. These words are used only to guide the reader in reading the description of these methods. Also, for example, any reference to a single element using the articles "one," "a," or "the" is not to be construed as limiting that element to being singular.

[0178] In addition, the steps and apparatuses in each embodiment of this specification are not limited to a certain embodiment. Actually, based on the concept of this specification, a new embodiment can be conceived by combining a part of the related steps and a part of the apparatuses in each embodiment of this specification, and these new embodiments are also included within the scope of this specification.

[0179] Each operation of the method described above can be performed by any suitable means capable of performing the corresponding function. Such means can include, but are not limited to, hardware circuits, application specific integrated circuits (ASICs), or processors, and can include various hardware and / or software components and / or modules.

[0180] Various exemplary logic blocks, modules, and circuits implemented or described by a general purpose processor, digital signal processor (DSP), ASIC, field programmable gate array signal (FPGA), or other programmable logic device (PLD), discrete gates or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein can be utilized. The general purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may further be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a microprocessor cooperating with a DSP core, or any other such configuration.

[0181] The steps of a method or algorithm described in connection with the present disclosure can be directly incorporated into hardware, software modules executed by a processor, or a combination of the two. The software modules can be present in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, and the like. The storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium can be integral to the processor. The software modules can be a single instruction, or many instructions, and can be distributed over several different code segments, among different programs, and across multiple storage media.

[0182] The methods disclosed herein include operations that implement the described methods. The methods and / or operations can be interchanged with each other without departing from the scope of the claims. In other words, unless a specific order of operations is specified, the order and / or use of specific operations can be modified without departing from the scope of the claims.

[0183] The above functions can be realized by hardware, software, firmware, or any combination thereof. If realized by software, the functions can be stored as instructions in a suitable computer-readable medium. The storage medium may be any available suitable medium accessible by a computer. By way of example and not limitation, such computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible medium that can be used to carry or store the desired program code in the form of instructions or data structures and that can be accessed by a computer. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disk typically magnetically reproduces data and disc optically reproduces data using a laser.

[0184] Accordingly, a computer program product can perform the operations provided herein. For example, such a computer program product may be a computer-readable tangible medium having tangible (and / or encoded) instructions thereon that can be executed by a processor to perform the operations described herein. The computer program product can include packaging materials.

[0185] Software or instructions may be transmitted by a transmission medium. For example, the software may be transmitted by a website, server, or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, RF, or microwave.

[0186] Furthermore, modules and / or other suitable means for implementing the methods and techniques described herein may be downloaded by a user terminal and / or a base station when appropriate, and / or obtained in other ways. For example, such devices may be coupled to a server to facilitate the transmission of means for implementing the methods described herein. Alternatively, the various methods described herein may be provided via a storage member (e.g., a physical storage medium such as RAM, ROM, CD, or floppy disk) so that the user terminal and / or the base station can obtain the various methods while being connected to the device or providing a storage member to the device. Furthermore, the methods and techniques described herein can be utilized in other suitable techniques of the device.

[0187] Other examples and embodiments are within the scope and spirit of the present disclosure and the appended claims. For example, the above-described functions can be realized using software executed by a processor, hardware, firmware, hardwire, or any combination thereof due to the nature of software. The features for realizing the functions can also be physically located at various positions, including being distributed such that some parts of the functions are realized at different physical locations. Furthermore, as used in this specification and the claims, for example, the enumeration of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C), so the "or" used in the enumeration of terms starting with "at least one" means a disjunctive enumeration. Furthermore, the term "exemplary" does not mean that the examples described are preferred or superior to other examples.

[0188] Various changes, substitutions, and modifications to the technology described herein can be made without departing from the teachings of the disclosed technology as defined by the appended claims. Also, the scope of the appended claims of the present disclosure is not limited to the specific aspects of the above processes, machines, manufactures, configurations of events, means, methods, and acts. Using the corresponding aspects described herein, processes, machines, manufactures, configurations of events, means, methods, or acts that are substantially the same in function or that achieve substantially the same result, whether currently existing or later developed, can be realized. Accordingly, the appended claims include such processes, machines, manufactures, configurations of events, means, methods, or acts within their scope.

[0189] The foregoing description of the disclosed aspects is provided so that those skilled in the art can make or use the present application. Various changes to these aspects are obvious to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present application. Accordingly, the present application is not intended to be limited to the aspects shown herein, but rather to follow the broadest scope consistent with the principles and novel features disclosed herein.

[0190] The foregoing description has been provided for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the form of the present disclosure. Although several exemplary aspects and embodiments have been described above, those skilled in the art will recognize some variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for optimizing the computing power of a neural network module by a chip, comprising: a step in which the chip obtains a computational graph of a neural network module having each operator, the neural network module receiving application data and obtaining a processing output result for the application data; a step in which the chip, based on the specific operations of each operator, in the computational graph, for a first operator, performs at least once at least one of an adjustment of exchanging the position of the first operator with a subsequent operator or a preceding operator in the computational graph, an adjustment of dividing the first operator into two or more similar operators, and an adjustment of inserting a plurality of first operators that can cancel each other out, the first operator being a tensor operation that does not change the numerical value in the input tensor; a step in which the chip, based on the specific operations of each operator, determines a second operator adjacent to the adjusted first operator in the computational graph, the operation of the adjusted first operator and the operation of the second operator being combinable or cancellable without affecting the operation result; a method for optimizing the computing power of a neural network module by a chip, comprising combining or cancelling the adjusted first operator and the second operator by the chip and executing the operations of each operator in the combined or cancelled computational graph.

2. The method for optimizing the computing power of a neural network module by a chip according to claim 1, wherein the application data includes at least one of image data and natural language data.

3. The step in which the chip, in the computational graph, for a first operator, performs at least once at least one of an adjustment of exchanging the position of the first operator with a subsequent operator or a preceding operator in the computational graph, an adjustment of dividing the first operator into two or more similar operators, and an adjustment of inserting a plurality of first operators that can cancel each other out is: when the first operator has two or more branched subsequent operators or preceding operators, dividing the first operator into two or more similar operators so that each first operator corresponds one-to-one with a subsequent operator or a preceding operator. When the consequential operators of two or more similar first operators are the same operator, integrating the two or more similar first operators as one operator, the method for optimizing the computing power of the neural network module by the chip according to claim 1.

4. The step of combining or canceling the adjusted first operator and the second operator by the chip and executing the operations of each operator in the combined or canceled computational graph is Before combining or canceling, when the operation of the second operator is an operation of the second operator on the numerical value in the output tensor of the adjusted first operator, after combining or canceling, combining the adjusted first operator and the second operator as a fourth operator, the operation of the fourth operator is to replace the tensor operation of the adjusted first operator itself by reading the numerical values in the input tensor of the adjusted first operator in a predetermined order by the chip, the predetermined order is determined based on the tensor operation of the adjusted first operator itself, and the step of performing the operation of the second operator on the numerical values read by the chip based on the operation of the second operator, the method for optimizing the computing power of the neural network module by the chip according to claim 1.

5. The step of reading the numerical values in the input tensor of the adjusted first operator in a predetermined order by the chip is realized by setting at least one of the read stride, offset amount and number of loop iterations by software by the chip, the method for optimizing the computing power of the neural network module by the chip according to claim 4.

6. The step of combining or canceling the adjusted first operator and the second operator by the chip and executing the operations of each operator in the combined or canceled computational graph is Before combining or canceling, when the second operator is a tensor operation that does not change the numerical values in the input tensor, after combining or canceling, combining the adjusted first operator and the second operator as a third operator, the tensor operation of the third operator realizes the tensor operation of the adjusted first operator and the tensor operation of the second operator together, the method for optimizing the computing power of the neural network module by the chip according to claim 1.

7. The step of combining or canceling out the adjusted first operator and the second operator by the chip and executing the operations of each operator in the combined or canceled computational graph is The method for optimizing the computing power of a neural network module by a chip according to claim 1, including the step of canceling out the first operator and the second operator when the adjusted first operator and the second operator are inverse operations to each other before being combined or canceled out.

8. The method for optimizing the computing power of a neural network module by a chip according to claim 1, wherein the first operator is at least one of a transpose operator, a reshape operator, a broadcast operator, a gather operator, a reverse operator, a concat operator, and a cast operator.

9. The method for optimizing the computing power of a neural network module by a chip according to claim 1, wherein the second operator is at least one of a transpose operator, a reshape operator, a broadcast operator, a gather operator, a reverse operator, a concat operator, a flatten operator, a cast operator, an elementwise binary operator, a batch fully connected operator, and a convolution operator.

10. A memory that stores the received application data, inputs it into the computational graph of the neural network module having each operator, and obtains a processing output result for the application data, and a controller, wherein the controller Based on the specific operations of each operator, in the computational graph, for the first operator, at least one of the adjustments of exchanging the position of the first operator with the subsequent operator or the preceding operator in the computational graph, dividing the first operator into two or more similar operators, and inserting a plurality of first operators that can cancel each other out is performed at least once. The first operator is a tensor operation that does not change the numerical value in the input tensor. Based on the specific operations of each operator, determine the second operator adjacent to the adjusted first operator in the computational graph, and the operations of the adjusted first operator and the second operator can be combined or canceled out without affecting the operation results. An artificial intelligence chip that combines or cancels out the adjusted first operator and the second operator by a chip and executes the operations of each operator in the combined or canceled computational graph.

11. The artificial intelligence chip according to claim 10, wherein the application data includes at least one of image data and natural language data.

12. The step of, by a chip, performing at least once at least one of an adjustment of exchanging the position of the first operator with a consequent operator or an antecedent operator in the computational graph with respect to the first operator, an adjustment of dividing the first operator into two or more similar operators, and an adjustment of inserting a plurality of first operators that can cancel each other out with respect to the first operator in the computational graph is When the first operator has two or more consequent operators or antecedent operators that are branched, dividing the first operator into two or more similar operators so that each first operator has a one-to-one correspondence with a consequent operator or an antecedent operator; The artificial intelligence chip according to claim 10, further comprising: when consequent operators of two or more similar first operators are the same operator, integrating the two or more similar first operators into one operator.

13. The step of combining or canceling out the adjusted first operator and the second operator by a chip and executing the operations of each operator in the combined or canceled computational graph is Before combining or canceling out, when the operation of the second operator is an operation of the second operator with respect to the numerical value in the output tensor of the adjusted first operator, after combining or canceling out, combining the adjusted first operator and the second operator as a fourth operator, including the step of, The operation of the fourth operator is to replace the tensor operation of the adjusted first operator itself by reading the numerical values in the input tensor of the adjusted first operator in a predetermined order by a chip, and the predetermined order is determined based on the tensor operation of the adjusted first operator itself; The artificial intelligence chip according to claim 10, further comprising performing the operation of the second operator on the numerical values read by the chip based on the operation of the second operator.

14. The artificial intelligence chip according to claim 13, wherein reading the numerical values in the input tensor of the adjusted first operator in a predetermined order by a chip is realized by software setting at least one of a read stride, an offset amount, and the number of loop iterations by the chip.

15. The step of combining or canceling out the adjusted first operator and the second operator by the chip and executing the operations of each operator in the combined or canceled computational graph is: Before combining or canceling out, if the second operator is a tensor operation that does not change the numerical values in the input tensor, the step of combining the adjusted first operator and the second operator as a third operator after combining or canceling out is included, and the tensor operation of the third operator realizes the tensor operation of the adjusted first operator and the tensor operation of the second operator together. The artificial intelligence chip according to claim 10.

16. The step of combining or canceling out the adjusted first operator and the second operator by the chip and executing the operations of each operator in the combined or canceled computational graph is: Before combining or canceling out, if the adjusted first operator and the second operator are inverse operations to each other, the step of canceling out the first operator and the second operator is included. The artificial intelligence chip according to claim 10.

17. The first operator is at least one of a transpose operator, a reshape operator, a broadcast operator, a gather operator, a reverse operator, a concat operator, and a cast operator. The artificial intelligence chip according to claim 10.

18. The second operator is at least one of a transpose operator, a reshape operator, a broadcast operator, a gather operator, a reverse operator, a concat operator, a flatten operator, a cast operator, an elementwise binary operator, a batch fully connected operator, and a convolution operator. The artificial intelligence chip according to claim 10.

19. An electronic device that optimizes the computing power of a neural network module by a chip, A memory for storing instructions, An electronic device including a processor that reads the instructions in the memory and executes a method for optimizing the computing power of a neural network module by the chip according to any one of claims 1 to 9.

20. A non-transitory storage medium in which instructions are stored, When the instructions are read by a processor, the method for optimizing the computing power of a neural network module by the chip according to any one of claims 1 to 9 is executed by the processor. A non-transitory storage medium.

Images