Method for optimizing the computing power of a neural network module, chip, electronic device, and medium.

Abstract

Provided are a method for optimizing the computing power of a neural network by a chip, the chip, an electronic device, and a medium. The method includes the steps of: the chip obtaining a computational graph of a neural network having operators, where the neural network receives application data and obtains a processing output result for the application data; the chip performing at least once at least one of the following adjustments on the computational graph based on the specific operations of each operator: an adjustment of exchanging the positions of a first operator and a consequent operator or an antecedent operator in the computational graph with the consequent operator or the antecedent operator in the computational graph; an adjustment of dividing the first operator into two or more similar operators; an adjustment of inserting a plurality of first operators that can cancel each other out, where the first operator does not change the numerical values in the input tensor; the chip determining a second operator as described below adjacent to the adjusted first operator, where the operation of the adjusted first operator and the operation of the second operator can be combined or canceled without affecting the operation result; and combining or canceling the adjusted first operator and the second operator and executing the operation of the combined or canceled computational graph.

Description

Technical Field

[0001] This application claims the priority of Chinese Patent Application No. 202210835887.0 filed on July 15, 2022, and hereby incorporates by reference in its entirety all the content disclosed in the above Chinese patent application as part of this application.

[0002] This application relates to the field of neural networks, and more specifically, to a method for optimizing the computing power of neural network modules, chips, electronic devices, and media.

Background Art

[0003] An artificial intelligence (AI) chip is a chip dedicated to neural network operations and is mainly a chip specifically designed to accelerate the execution of neural networks.

[0004] A neural network can be represented by pure mathematical formulas. Based on these formulas, the neural network can be represented by a computational graph model. A computational graph is a visual representation of these formulas. The computational graph model can divide one composite operation into multiple sub-operations, and each sub-operation is called an operator (abbreviated as Operator, Op). A computational graph consists of nodes and edges. Nodes represent operators, and edges represent the respective dependency relationships of the calculation targets. Solid lines indicate the existence of data transfer dependencies, and the data to be transferred is a tensor. Dashed lines generally represent control dependencies, that is, the execution priorities.

[0005] In a computational graph, an operator has its own input data. Since the operation logics of operators are different, each operator outputs output data calculated using the input data based on its own operation logic.

[0006] It is necessary to design a chip that accelerates the execution speed of the computational graph of a neural network and optimizes the computing power of the neural network module. [Overview of the Initiative] [Means for solving the problem]

[0007] According to one aspect of this application, a method for optimizing the computing power of a neural network module by a chip, comprising the steps of: obtaining a computation diagram of a neural network module having operators by the chip; the neural network module receiving application data and obtaining a processing output result for the application data; and, based on the specific calculation of each operator, the chip making adjustments in the computation diagram, such as swapping the position of a first operator with a consequent or antecedent operator in the computation diagram; dividing the first operator into two or more similar operators; and inserting a plurality of first operators that can cancel each other out. A chip provides a method for optimizing the computing power of a neural network module, comprising the steps of: performing at least one of the following at least once, wherein the first operator is a tensor operation that does not change the numerical value in the input tensor; determining a second operator adjacent to the adjusted first operator in the computation diagram based on the specific operation of each operator, wherein the operation of the adjusted first operator and the operation of the second operator can be combined or canceled out without affecting the result of the operation; and combining or canceling the adjusted first operator and the second operator by the chip and performing the operation of each operator in the combined or canceled computation diagram.

[0008] According to one aspect of this application, an artificial intelligence chip is provided, comprising: a memory that stores received application data and inputs it into a computation diagram of a neural network module having operators to obtain processing output results for the application data; and a controller that, based on the specific operations of each operator, performs at least one of the following adjustments at least once on the computation diagram: an adjustment that swaps the position of the first operator with a consequent or antecedent operator in the computation diagram; an adjustment that divides the first operator into two or more similar operators; and an adjustment that inserts a plurality of mutually cancelable first operators, wherein the first operator is a tensor operation that does not change the numerical value in the input tensor; a second operator adjacent to the adjusted first operator in the computation diagram is determined based on the specific operations of each operator, the operations of the adjusted first operator and the operations of the second operators can be combined or canceled out without affecting the operation results; the chip combines or cancels out the adjusted first operator and the second operators, and performs operations on each operator in the combined or canceled computation diagram.

[0009] According to one aspect of this application, an electronic device that optimizes the computing power of a neural network module by means of a chip, Memory for storing instructions, The present invention provides an electronic device including a processor that reads instructions from the aforementioned memory and performs the method described herein.

[0010] According to one aspect of this application, a non-temporary storage medium is provided which stores instructions, wherein when the instructions are read by a processor, the processor performs a method according to the Disclosure. [Effects of the Invention]

[0011] In this way, for example, by optimizing the specific operation rules of operators in the computational diagram of a neural network that processes image data, the number of operators calculated or the amount of computation performed by the operators can be reduced, and for example, the delay in the subsequent time it takes for the neural network that processes image data to operate on the chip can be reduced. [Brief explanation of the drawing]

[0012] To more clearly illustrate the embodiments of this disclosure or the technical concepts in the prior art, the drawings that may be used in the description of the embodiments or the prior art are briefly described below. Obviously, the drawings in the following description are only a few embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these without any creative effort.

[0013] [Figure 1] This document shows one example of a computational diagram in a neural network for processing or identifying image data according to an embodiment of the present application. [Figure 2] This invention describes a method for optimizing the computing power of a chip-based neural network module according to an embodiment of this application. [Figure 3A] A schematic diagram of the original calculation diagram including multiple operators according to the embodiment of this application is shown. [Figure 3B] The computational diagram according to the embodiment of this application shows a schematic diagram in which the first operator is divided into two or more similar operators. [Figure 3C] A schematic diagram is shown showing how to optimize the calculation diagram after adjusting the first operator according to the embodiment of this application. [Figure 4] The computational diagram according to the embodiment of this application shows a schematic diagram in which two or more similar first operators are combined as a single operator and moved downwards. [Figure 5] The computational diagram according to the embodiment of this application shows a schematic diagram in which the Transpose operator is moved downward multiple times in succession and combined with the Fully connected operator. [Figure 6]A schematic diagram showing the cancellation of the first operator and the second operator according to the embodiment of this application is provided. [Figure 7] The schematic diagram of the computational diagram according to the embodiment of this application shows how one transpose operator is moved downwards and then combined with an adjacent transpose operator to form a single transpose operator. [Figure 8A] The schematic diagram of the calculation according to the embodiment of this application shows how the Reshape operator and the Reshape operator that is the inverse operation of each other cancel each other out. [Figure 8B] The schematic diagram of the embodiment of this application shows how one Reshape operator is moved downwards and then combined with an adjacent Reshape operator to form a single Reshape operator. [Figure 9A] A schematic diagram of the calculation method for splitting the Broadcast operator into two Broadcast operators and then optimizing them according to the embodiment of this application is shown. [Figure 9B] The schematic diagram of the calculation according to the embodiment of this application shows the Broadcast operator being moved downwards and combined with the Binary operator. [Figure 10] A schematic diagram is shown below illustrating how two Gather operators according to the embodiment of this application are combined into a single Gather operator and moved downwards. [Figure 11] A schematic diagram is shown illustrating the Reverse operator being moved downwards and combined with the Convolution operator according to the embodiment of this application. [Figure 12A] A schematic diagram showing the concatenation of the Concat operator according to the embodiment of this application is shown. [Figure 12B] A schematic diagram showing the combination of Cast operators according to the embodiment of this application is shown. [Figure 13A] This diagram shows a schematic representation of one embodiment in which, by inserting a plurality of mutually cancelable first operators according to the embodiment of this application, these plurality of first operators can perform operations of joining or canceling with other adjacent operators. [Figure 13B]A schematic diagram of another example is shown in which a plurality of mutually cancellable first operators according to an embodiment of the present application are inserted, and these plurality of first operators can perform an operation of combining or cancelling with other adjacent operators. [Figure 14] A schematic block diagram of an artificial intelligence chip according to an embodiment of the present application is shown. [Figure 15] A block diagram of an exemplary electronic device suitable for implementing an embodiment of the present application according to an embodiment of the present application is shown. [[ID=​​​​​​​​​​​​​​​​For example, upon receiving a voluntary request from a user, the system sends the user information to explicitly inform the user that the requested operation requires the acquisition and use of the user's personal information. Thus, the user can autonomously choose, based on the information provided, whether to provide personal information to software or hardware such as electronic devices, applications, servers, or storage media that perform the operation of the proposed technology of this disclosure.

[0017] As an optional and non-limiting embodiment, the form in which information is sent to the user in response to a voluntary request from the user may be, for example, a pop-up that can present the information as text. The pop-up window may also include a choice control that allows the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0018] The above notice and the process for obtaining user permission are merely illustrative and do not limit the embodiments of this disclosure. Other forms that comply with applicable laws and regulations may also be applied to the embodiments of this disclosure.

[0019] Furthermore, the data relating to this proposed technology (including, but not limited to, the data itself, or the acquisition or use of the data) must comply with the requirements of the corresponding laws, regulations, and related provisions.

[0020] Currently, neural networks and machine learning systems use tensors as their fundamental data structure. A tensor is a data container, and since most of the data it contains is always numerical, it is a digital container. The specific numerical values ​​in a tensor can be application data, such as image data or natural language data.

[0021] For example, a scalar is a 0th-order tensor, for example, 2, 3, 5. In a specific application scenario, for example, image data, where 2 represents, for example, the grayscale value of a single pixel in the image data, 3 represents, for example, the grayscale value of a single pixel in the image data, 5 represents, for example, the grayscale value of a single pixel in the image data, and so on. For example, a vector is a 1st-order tensor, for example, [0, 3, 20]. A matrix is ​​a 2nd-order tensor, for example,

number

[0022] The identification process for the above application scenario is realized by receiving various application data input to the neural network as tensors and performing computations within the neural network. As mentioned above, the computation of the neural network can consist of a series of tensor operations, which may be complex geometric transformations of multi-order tensor input data. These tensor operations are also called operators, and the computation of the neural network can be converted into a computation diagram, which may have multiple operators and be connected by lines to represent the dependencies between each operator computation.

[0023] The types of operators are: activation operators (e.g., Relu, sigmoid, Tan operators), two-tensor computation operators (e.g., BiasAdd (adds a bias operation), MatMul (matrix multiplication), Axpy (vector addition)), multiple tensor computation operators (e.g., EltwiseOp (performs one operation: multiplication, addition, or maximization on elements at corresponding positions of multiple tensors)), normalization operators (e.g., BatchNorm (speeds up the training convergence speed of a neural network), LRN (Local Response Normalization, i.e., local response normalization layer), SsdNormalize (limits the range of first-order normalization)), and feature extraction operators (Convolution (extracts features region by region), FullyThe operators include: connected operator (extracts features from the entire input as a feature diagram), Correlation operator, DeConvolution operator (maps one low-dimensional space to a higher dimension while maintaining their connection relationships / patterns), DepConvolution operator (completes the calculation of a normal 3D core using a method that calculates in multiple steps), anti-overfitting operators (e.g., Pooling operator (divides the input image into multiple rectangular regions and outputs the maximum or average value for each sub-region), Mean operator (a sliding window operator that only takes the average value)), dimensional transformation operators (e.g., Flatten operator (combines the start_axis dimension ~ end_axis dimension input to the tensor into a single dimension), Reshape operator (converts the description input to the Tensor into a new shape), Freespace This may include operators such as the Extract operator (changes the dimension of h to 1, leaving the other dimensions unchanged, thereby completing sampling for h, where h is a positive integer), the Pack operator (converts a tensor array of dimension R to a tensor of dimension R+1), the Pad operator (performs data augmentation), the Permute operator (adjusts the order of the input dimensions of a tensor), the ShuffleChannel operator (adjusts the order of the C dimensions), the Title operator (generates output data by copying the input data a specified number of times in each dimension), the transpose operator (transposes and changes the order of values ​​in the tensor), the reshape operator (converts a tensor to a tensor of another shape), the broadcast operator (extends the input tensor to another tensor), the gather operator, the reverse operator, the concat operator (achieves the combination of multiple operators), and the cast operator. Users can also customize some operators. Because there are too many names and specific types of operators, it is not possible to list all operator types, names, and their operations here.

[0024] Conventional techniques involve obtaining a computational diagram of a neural network, and then sequentially performing calculations on the operators shown in the diagram, following the lines connecting them, to obtain the final calculation result. However, performing calculations for each operator takes a certain amount of time, and conventional techniques do not optimize these operators to speed up their calculations, thus failing to reduce the calculation time for each operator.

[0025] For example, when studying operators in a neural network computation diagram for processing image data, some operators can be moved up or down depending on their specific operation and then combined with or canceled out (cancel out) the operations of other operators, thereby optimizing these operators and reducing the number of operators calculated or the computational load they perform. For example, this reduces the delay required for a neural network processing image data to operate in real time on a chip. Furthermore, by combining this with several specific hardware read functions of the artificial intelligence chip developed by the applicant, it is possible to ensure that the calculation results match the calculation results of these operators before optimization, while also reducing the number of operators calculated or the computational load they perform. This optimization is performed based on a high-level semantic representation called a computation diagram, avoiding complex and cumbersome optimization operations at the instruction level, enabling optimization of more general scenarios, and increasing versatility.

[0026] Figure 1 shows an example of a computational diagram in a neural network for processing or identifying image data according to an embodiment of the present application.

[0027] For example, a tensor containing image data (e.g., pixel chromaticity values) is input into an example calculation diagram shown in Figure 1. This calculation diagram shows only a portion of the operators according to the embodiment of this application for the purpose of ease of viewing by the reader. During the calculation of this calculation diagram, the tensor is first calculated using the Transpose operator, then one branch is calculated using the Reshape operator, and the other branch is calculated using the Fully connected operator.

[0028] If the tensor in question is first input to the Transpose operator, the Transpose operator is a tensor operation that does not change the numerical values ​​in the input tensor. The action of the Transpose operator is to change the order of the dimensions (axis) of the array. For example, swapping the order of two dimensions in a two-dimensional array is matrix transposition. The Transpose operator can be applied to more dimensions. The input parameter of the Transpose operator is the order of the dimensions of the output array, and the indices are counted from 0. The tensor input to the Transpose operator is, for example, a two-dimensional matrix [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]] or

number

number

[0029] As can be seen, the Transpose operator changes the order of dimensions, that is, changes the shape of the tensor, but does not change the numerical values ​​in the tensor; for example, they are still 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. Of course, the Transpose operator may not change the shape of the tensor; for example, a 3x3 matrix remains a 3x3 matrix after transposition, and the numerical values ​​in the tensor remain unchanged, but the order of the numerical values ​​in the transposed matrix is ​​different.

[0030] Then, the tensor calculated using the Transpose operator described above.

number

[0031] The specific operation of the Reshape operator is to change the shape attribute of a tensor, so that an m*n matrix a can be arranged into an i*j matrix b. For example, the Reshape operator (Reshape(A,2, 6), where A is the input tensor) can be transformed into the above tensor

number

number

[0032] Note that the Reshape operator changes the shape of the tensor but does not change the numerical values ​​within the tensor; for example, they remain 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12.

[0033] The Fully Connected operator (also called the Full Connection operator) can be considered a special convolutional layer or a tensor product, and is an operation that extracts features from the entire tensor input as a feature diagram. In other words, it performs a linear transformation from one feature space to another, and the output tensor is a weighted sum of the input tensors. For example, the Fully Connected operator takes the input tensor (the tensor output by the Transpose operator) as...

number

number

[0034] Up to this point, we have introduced the specific computational process of the neural network used to process or identify the image data shown in Figure 1.

[0035] As can be seen from the example in Figure 1, the Transpose operator changes the dimensional order, i.e., the shape of the tensor, but does not change the numerical values ​​in the tensor; for example, they remain 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. In the computation diagram, there are two subsequent operators, the Reshape operator and the Fully connected operator, but the numerical values ​​in the tensor that the Transpose operator outputs to the Reshape operator and the Fully connected operator are the same as the numerical values ​​in the tensor input to the Transpose operator. For operators such as the Transpose operator, one can consider whether to make certain changes in the computation diagram in order to optimize the execution speed and execution time of these operators.

[0036] Figure 2 shows a method for optimizing the computing power of a neural network module using a chip according to an embodiment of this application.

[0037] As shown in Figure 2, the chip-based method 200 for optimizing the computational power of a neural network module includes steps 201 to 204. In step 201, the chip obtains a computational diagram of the neural network module having each operator. The neural network module receives application data and obtains the processing output result for the application data. In step 202, based on the specific operation of each operator, the chip performs at least one of the following adjustments on the computational diagram at least once: an adjustment to swap the position of the first operator with a consequent or antecedent operator in the computational diagram; an adjustment to divide the first operator into two or more similar operators; and an adjustment to insert a plurality of mutually cancelable first operators. The first operator is a tensor operation that does not change the numerical value in the input tensor. In step 203, based on the specific operation of each operator, the chip determines the second operator adjacent to the adjusted first operator in the computational diagram. The operations of the adjusted first operators and the operations of the second operators can be combined or canceled out without affecting the operation results. In step 204, the chip combines or cancels the adjusted first and second operators and performs the operations on each operator in the combined or canceled calculation diagram.

[0038] Based on the example in Figure 1, the specific process of the above method 200 will be explained in conjunction with Figures 3A to 3C. Figure 3A shows a schematic diagram of the original calculation diagram including multiple operators according to the embodiment of this application. Figure 3B shows a schematic diagram of the calculation diagram according to the embodiment of this application in which the first operator is divided into two or more similar operators. Figure 3C shows a schematic diagram of the calculation diagram after adjusting the first operator according to the embodiment of this application.

[0039] First, in step 201, the chip obtains a computational diagram of the neural network module having each operator. The neural network module receives application data and obtains the processing output result for the application data. For example, as shown in Figure 3A, the chip obtains a computational diagram of the neural network module having each operator, such as the Transpose operator, Reshape operator, and Fully connected operator. The neural network module receives image data and obtains the processing output result for the image data.

[0040] In step 202, based on the specific operation of each operator, the chip performs at least one of the following adjustments at least once in the calculation diagram: an adjustment to swap the position of the first operator with a consequent or antecedent operator in the calculation diagram; an adjustment to divide the first operator into two or more similar operators; and an adjustment to insert a plurality of mutually cancelable first operators. The first operator is a tensor operation that does not change the numerical value in the input tensor. For example, a tensor operation that does not change the numerical value in the input tensor may be a tensor operation that changes the shape or order of the tensor. As shown in Figure 3A, the Transpose operator is determined to be the first tensor operation that does not change the numerical value in the input tensor.

[0041] In one embodiment, if the first operator has two or more consequent or antecedent operators from which it has branched, the first operator is divided into two or more similar operators such that each first operator corresponds one-to-one with the consequent or antecedent operator.

[0042] The Transpose operator has two consequent operators, the Reshape operator and the Fully connected operator, which are branched. Therefore, as shown in Figure 3B, the Transpose operator is split into two similar Transpose operators so that there is a one-to-one correspondence between each Transpose operator and its consequent operators; that is, one Transpose operator corresponds to the consequent Reshape operator, and the other Transpose operator corresponds to the consequent Fully connected operator. In other words, the Transpose operator is first performed on the input tensor, and then the Reshape operator and the Fully connected operator are performed on the tensor output by the Transpose operator to obtain two output tensors, which are then used in the subsequent operators. After adjustment, one Transpose operator is performed on the input tensor, then the Reshape operator is performed, then the other Transpose operator is performed on the input tensor, and then the Fully connected operator is performed.

[0043] Of course, in this case, further adjustments may be made. For example, in Figure 3C, by swapping the positions of the Transpose operator and the consequent Reshape operator in the left branch, the Reshape operator operation is performed on the input tensor first, and then the Transpose operator operation is performed. Here, since the Transpose operator operation is a tensor operation that does not change the numerical value in the input tensor, moving it down and then performing the operations of each operator does not change the final result. By swapping the positions of the Transpose operator and the consequent Reshape operator in this way, it is possible to consider whether the swapped (in this case, moved down) Transpose operator and its further antecedent or consequent operator (in this case, the consequent operator) can be combined or canceled out.

[0044] In the examples in Figures 3A and 3C, the Transpose operator is split and then moved down in the left branch, but in practice, the Transpose operator can move up and down in each branch. By splitting one Transpose operator into the number of Transpose operators according to the number of branches, the Transpose operator can move up and down in each branch, thereby finding opportunities to combine or cancel out with other operators.

[0045] In step 203, the chip determines the following second operator adjacent to the adjusted first operator in the calculation diagram, based on the specific operations of each operator. The operations of the adjusted first operator and the second operator can be combined or canceled out without affecting the result of the operations.

[0046] The chip determines the fully connected operators adjacent to the partitioned transpose operators in the computation diagram, based on the specific operations of each operator. The operations of the partitioned transpose operators and the fully connected operators can be combined without affecting the results of the operations.

[0047] In one embodiment, if the operation of the second operator is an operation of the second operator on a numerical value in the output tensor of the first operator that has been adjusted before being combined or canceled out, the adjusted first and second operators are combined as a fourth operator after being combined or canceled out. The operation of the fourth operator involves replacing the tensor operation of the adjusted first operator itself by reading the numerical value in the input tensor of the first operator that has been adjusted in a predetermined order by the chip, the predetermined order being determined based on the tensor operation of the adjusted first operator itself, and performing the operation of the second operator on the numerical value read by the chip based on the operation of the second operator.

[0048] As shown in the right branch in Figure 3C, the separated Transpose operator and its consequent Fully connected operator can be combined (coalescing) as a Fully connected operator with Transpose. In other words, since the Transpose operator is essentially the transpose of a tensor, the original input tensor is transposed by the Transpose operator, and then the Fully connected operator is performed on the transposed tensor. However, after the combination, the chip reads each value in the input tensor in the transposed order and directly performs the Fully connected operator operation. This is because the Transpose operator transposes the input tensor, but the individual values ​​in the tensor do not change; only their position in the row or column changes.

[0049] The tensor originally input to the Transpose operator is, for example

number

number

number

number

[0050] The tensor input to the fully connected operator with transpose is:

number

[0051] Such selective reading (changing from reading the first, second, and third rows of the originally transposed matrix to reading the first, second, and third columns of the matrix before transposition) may be performed by a chip. That is, the chip reads the numerical values ​​in the tensor input to the Fully connected operator to which Transpose is coupled in a predetermined order, thereby replacing the tensor operation of the partitioned Transpose operator itself. In this example, the chip can replace the transpose operation of the Transpose operator itself through a hardware operation in which it reads the numerical values ​​in the input tensor in a predetermined order of reading the first, second, and third columns. This predetermined order is determined based on the tensor operation of the partitioned Transpose operator itself, i.e., transpose, and the predetermined order should achieve the transpose effect.

[0052] In one embodiment, reading numerical values ​​in the input tensor of a first operator, which has been adjusted in a predetermined order by the chip, is achieved by setting at least one of the reading stride, offset amount, and loop count by the chip in software. In this way, the user can change the reading order by flexibly setting at least one of the reading stride, offset amount, and loop count by the chip in software, thereby replacing some of the tensor operations of the operators themselves in the chip's hardware reading.

[0053] The hardware operations read by such a chip may be configured by software, specifically by several static random access memory (SRAM) controllers, which set at least one of the stride, offset, and loop count for reading / writing to / from static random access memory (SRAM). For example, the input tensor of residuals in SRAM.

number

[0054] In step 204, the chip combines or cancels the adjusted first and second operators and performs the operations on each operator in the combined or canceled calculation diagram.

[0055] Here, the chip can perform calculations on each operator in the optimized calculation diagram after the operators have been adjusted, combined, or canceled out.

[0056] In this way, for example, by optimizing these operators by specific operation rules in the computational diagram of a neural network that processes image data, the number of operators calculated or the computational amount performed by the operators can be reduced, thereby reducing the delay in which the neural network that processes image data operates in real time on the chip.

[0057] In one embodiment, if the consequent operators of two or more similar first operators are the same operator, the two or more similar first operators are merged into a single operator.

[0058] Figure 4 shows a schematic diagram of a calculation diagram according to an embodiment of the present application, in which two or more similar first operators are combined as a single operator and moved downward.

[0059] If two or more similar first operators have the same consequent operator, that is, if the consequent operators of three transpose operators are the same SomeOp operator (referring to any operator, not limited thereto), as shown in Figure 4, that is, if the multiple antecedent operators of a SomeOp operator are all similar transpose operators, then the three antecedent transpose operators of the SomeOp operator are combined into a single transpose operator. However, there is a possibility that each of the three branches of the three transpose operators has three antecedent operators, but this is not shown in Figure 4.

[0060] To find opportunities to combine or cancel with other operators, a combined Transpose operator may be moved up (i.e., swapped positions with the antecedent operator) or down (i.e., swapped positions with the consequent operator). In the example in Figure 4, the SomeOp operator can move the Transpose operator down to its consequent operator (not shown), thereby reducing the redundant Transpose operator by combining or canceling with the consequent operator, or creating possibilities for subsequent operations such as combining or canceling.

[0061] In the above example, the Transpose operator is moved downwards, but in practice, to achieve the objective of reducing computation time, the Transpose operator may be moved upwards to find opportunities for optimization, namely, opportunities to determine whether it can combine or cancel out with adjacent operators. For example, one might move the Transpose operator upwards and then discover that it is the inverse operation of another adjacent Transpose operator, thereby allowing both to cancel out.

[0062] In the examples above, the Transpose operator is moved down once, but to achieve the goal of reducing computation time, the Transpose operator may be moved up or down multiple times before finding opportunities for optimization, i.e., opportunities to determine whether it can be combined or canceled out with adjacent operators.

[0063] Figure 5 shows a schematic diagram of a computational diagram according to an embodiment of this application, in which the Transpose operator is moved downward multiple times in succession and combined with the Fully connected operator.

[0064] As shown in Figure 5, the Transpose operator is moved downwards twice in a row (i.e., its position is swapped twice with the consequent SomeOp operator) and then coupled with the Fully connected operator. As mentioned above, this coupling may achieve the same meaning as the Transpose operator, i.e., the transpose effect, by having the hardware operations read by this chip, for example, by having some static random access memory (SRAM) controllers set at least one of the stride, offset amount, and loop count for reading / writing to and from static random access memory (SRAM). In this way, the Transpose operator, moved downwards twice in a row, can be coupled with the adjacent consequent Fully connected operator as a Fully connected operator with the Transpose operator coupled.

[0065] In addition to fully connected operators that can be combined with the Transpose operator, such optimized operators that support combination with the Transpose operator include the reshape operator, broadcast operator, gather operator, reverse operator, concat operator, flatten operator, cast operator, elementwise binary operator, batch fully connected operator, convolution operator, and other Transpose operators that can be combined but cannot be canceled out.

[0066] In one embodiment, if the first and second operators, after being adjusted before being combined or canceled out, are inverse operations of each other, the first and second operators are canceled out.

[0067] Figure 6 shows a schematic diagram of the calculation according to the embodiment of this application, in which the first operator and the second operator cancel each other out.

[0068] As shown in Figure 6, in the original calculation diagram on the left, the upper and lower Transpose operators are inverse operations of each other. For example, the upper Transpose operator transposes a 3*4 matrix to a 4*3 matrix, and the lower Transpose operator transposes a 4*3 matrix to a 3*4 matrix. Therefore, the two Transpose operators are clearly inverse operations. In the original calculation diagram, the two Transpose operators are not adjacent and cannot directly cancel each other out. Therefore, by moving the upper Transpose operator downwards twice in succession (swapping positions with the consequent operator twice), it becomes adjacent to the lower Transpose operator, and these two Transpose operators can then cancel each other out and be deleted.

[0069] In this way, the number of operators calculated is reduced, or the computational amount performed by the operators is reduced, thereby reducing the delay in the subsequent time it takes for a neural network processing image data to operate on the chip.

[0070] In one embodiment, if the second operator is a tensor operation that does not change the numerical value in the input tensor before being combined or canceled out, then the adjusted first and second operators are combined as a third operator after being combined or canceled out, and the tensor operation of the third operator realizes the tensor operation of the adjusted first operator and the tensor operation of the second operator together.

[0071] In other words, the first operator is a tensor operation that does not change the numerical value in the input tensor, and the adjacent second operator is also a tensor operation that does not change the numerical value in the input tensor. However, the two are not the same, and they are not inverse operations of each other, but rather different forms or different types of tensor operations.

[0072] For example, the first operator may be at least one of the transpose, reshape, broadcast, gather, reverse, concat, and cast operators. The second operator may be at least one of the transpose, reshape, broadcast, gather, reverse, concat, and cast operators. For example, the first operator may be the transpose operator, and the second operator may also be the transpose operator. However, the two transpose operators are neither exactly the same nor their inverse.

[0073] Figure 7 shows a schematic diagram of a computational diagram according to an embodiment of the present application, in which one transpose operator is moved downward and then combined with an adjacent transpose operator as a single transpose operator.

[0074] As shown in Figure 7, the computation diagram allows us to first move one Transpose operator downwards, then find an adjacent Transpose operator and combine them. Therefore, the Transpose operator that was moved downwards and the adjacent Transpose operators are combined into a single combined Transpose operator. Since all operations of the Transpose operator change the shape of the tensor, this combined Transpose operator realizes the tensor operation of the Transpose operator that was moved downwards and the tensor operation of the adjacent Transpose operator together. In other words, instead of performing two tensor operations of the Transpose operator in two separate steps to achieve the final result, the final result can be achieved by changing the shape of the tensor using the Transpose operator only once.

[0075] Above, the Transpose operator was used as an example of a first operator that can be combined or canceled out with other operators. Below, other operators that can be used as first operators will be introduced.

[0076] In one embodiment, the first operator may be the Reshape operator. The Reshape operator transforms one tensor into another tensor of a different shape. Note that the operation of the Reshape operator is a tensor operation that does not change the numerical values ​​in the input tensor.

[0077] Figure 8A shows a schematic diagram of a calculation according to an embodiment of this application, illustrating how the Reshape operator and its inverse Reshape operator cancel each other out.

[0078] As shown in Figure 8A, in the original calculation diagram on the left, the upper Reshape operator and the lower Reshape operator are inverse operations of each other. For example, the operation of the upper Reshape operator changes a 3*4 matrix to a 2*6 matrix, and the operation of the lower Reshape operator changes a 2*6 matrix to a 3*4 matrix. The two Reshape operators are inverse operations. In the original calculation diagram, the two Reshape operators are not adjacent and therefore cannot directly cancel each other out. Thus, by moving the upper Reshape operator downwards twice in succession (swapping its position with the consequent operator twice), it can become adjacent to the lower Reshape operator, and these two Reshape operators can then cancel each other out and be deleted.

[0079] In this way, the number of operators calculated is reduced, or the computational amount performed by the operators is reduced, thereby reducing the delay in the subsequent time it takes for a neural network processing image data to operate on the chip.

[0080] Figure 8B shows a schematic diagram of a computational diagram according to an embodiment of the present application, in which one Reshape operator is moved downward and then combined with an adjacent Reshape operator as a single Reshape operator.

[0081] As shown in Figure 8B, the computational diagram allows us to first move one Reshape operator downwards, then find an adjacent Reshape operator and combine them. Therefore, the Reshape operator that was moved downwards and the adjacent Transpose operator are combined into a single combined Reshape operator. Since all Reshape operations change the shape of a tensor, this combined Reshape operator performs the tensor operation of the Reshape operator that was moved downwards and the tensor operation of the adjacent Reshape operator together. In other words, instead of performing two tensor operations on the Reshape operator in two separate steps to achieve the final result, the shape of the tensor can be changed and the final result achieved by using the Reshape operator only once.

[0082] For example, the operation of the upper Reshape operator could change a 3*4 matrix to a 2*6 matrix, and the operation of the lower Reshape operator could change a 2*6 matrix to a 6*2 matrix, and then the combined operation of the Reshape operators could change a 3*4 matrix to a 6*2 matrix.

[0083] Of course, the above example using the Reshape operator is only one of several optimization forms, and is not limited to them. In practice, it is also possible to combine the Reshape operator with other operators (second operators). Since this is a tensor operation that does not change the numerical values ​​in the input tensor, by reading the numerical values ​​in the input tensor into the Reshape operator which has been adjusted in a predetermined order by the chip, the tensor operation of the adjusted Reshape operator itself can be replaced.

[0084] In one embodiment, the first operator may be the Broadcast operator. The Broadcast operator extends an input tensor into another tensor, for example, extending a tensor of shape A into a tensor of shape B, where A and B are tensor shapes represented by dimensions. Note that the Broadcast operator is a tensor operation that does not change the numerical values ​​in the input tensor.

[0085] Figure 9A shows a schematic diagram of a calculation in which a Broadcast operator according to an embodiment of this application is divided into two Broadcast operators and then optimized.

[0086] The Broadcast operator has two consequent operators, a Reshape operator and a Binary operator, which branch off. As shown in Figure 9A, each Broadcast operator and its consequent operator have a one-to-one correspondence; that is, one Broadcast operator corresponds to the consequent Reshape operator, and the other Broadcast operator corresponds to the consequent Binary operator. The Broadcast operator is thus split into two similar Broadcast operators. In other words, the Broadcast operator is first performed on the input tensor, and then the Reshape operator and the Binary operator are performed on the tensor output by the Broadcast operator to obtain two output tensors, which are then used in the subsequent operators. After adjustment, one Broadcast operator is performed on the input tensor, then the Reshape operator is performed, then the other Broadcast operator is performed on the input tensor, and then the Binary operator is performed. The Binary operator, also called the Elementwise Binary operator, binarizes the numerical values ​​in the input tensor.

[0087] Of course, in this case, further adjustments may be made. For example, in Figure 9A, by swapping the positions of the Broadcast operator and the consequent Reshape operator in the left branch, the Reshape operator operation is performed on the input tensor first, and then the Broadcast operator operation is performed. Here, since the Broadcast operator operation is a tensor operation that does not change the numerical value in the input tensor, moving it down and then performing the operations of each operator does not change the final result. By swapping the positions of the Broadcast operator and the consequent Reshape operator in this way, it is possible to consider whether the swapped (in this case, moved down) Broadcast operator and its further antecedent or consequent operator (in this case, consequent operator) can be combined or canceled out.

[0088] As shown in the right branch in Figure 9A, the divided Broadcast operator and the subsequent Binary operator are combined as a Binary operator to which the Broadcast operator is combined. In other words, since the Broadcast operator is in practice an extension of a tensor, the original input tensor is extended by the Broadcast operator, and then the Binary operator operation is performed on the extended tensor and combined. After that, the chip reads each value in the input tensor to the Broadcast operator in the order of the extended tensor, for example, reading many of the values ​​that are extended (i.e., copied), and directly performing the Binary operator operation on the read values. This is because the Broadcast operator extends the input tensor but does not change each value in the tensor, only the number of times it appears in that tensor.

[0089] If the Broadcast operator changes [1,2,4] to [[1,1],[2,2],[4,4]], and the binary operator takes 1 if it is greater than 3 and 0 if it is less than 3, then the Binary operator with Broadcast attached can be realized by the chip reading in a predetermined order. For example, the chip reads the position where 1 is stored twice, performs the Binary operator operation each time it reads and determines that it is 0,0, then the chip reads the position where 2 is stored twice, performs the Binary operator operation each time it reads and determines that it is 0,0, then the chip reads the position where 4 is stored twice, performs the Binary operator operation each time it reads and determines that it is 1,1, and finally the tensor output by the Binary operator with Broadcast attached is [[0,0],[0,0],[1,1]].

[0090] Figure 9B shows a schematic diagram of a calculation according to an embodiment of this application, in which the Broadcast operator is moved downward and combined with the Binary operator.

[0091] As shown in Figure 9B, the Broadcast operator is moved down (i.e., swapped positions with its consequent Reshape operator) and then coupled with the Binary operator. As mentioned above, this coupling may achieve the same meaning, or extended effect, of the Broadcast operator by setting at least one of the hardware operations read by the chip, such as the stride, offset amount, and loop count for reading / writing to / from static random access memory (SRAM) by some static random access memory (SRAM) controllers. In this way, the Broadcast operator moved down can be coupled with its adjacent consequent Binary operator as a Binary operator into which the Broadcast operator is coupled.

[0092] In addition to the Binary operator which can be combined with the Broadcast operator, such optimized operators that support combination with the Broadcast operator may also include the batch fully connected operator (also known as the batch matmul operator).

[0093] In one embodiment, the first operator may be the Gather operator. The Gather operator selects a number of values ​​from a number of values ​​in the input tensor to be output as a tensor. Note that the Gather operator is a tensor operation that does not change the numbers in the input tensor.

[0094] Figure 10 shows a schematic diagram of an embodiment of the present application in which two Gather operators are combined into a single Gather operator and moved downwards.

[0095] If the consequent operators of two or more similar first operators are the same operator, that is, if the consequent operators of two Gather operators are the same SomeOp operator as shown in Figure 10, then the two antecedent Gather operators of the SomeOp operator can be combined into a single Gather operator.

[0096] To find opportunities to combine or cancel with other operators, combined Gather operators can be continuously moved down (i.e., swapped positions with their consequent operators). In the example in Figure 10, the SomeOp operator can move the Gather operator down to its consequent operator (not shown), thereby reducing the redundant Gather operator by combining or canceling with that consequent operator, or creating possibilities for subsequent operations such as combining or canceling.

[0097] In one embodiment, the first operator may be the Reverse operator. The action of the Reverse operator is to reverse the order of data of a certain dimension in the input tensor.

[0098] Figure 11 shows a schematic diagram of the Reverse operator being moved downward and combined with the Convolution operator according to an embodiment of this application.

[0099] The operation of the Convolution operator is a convolution that calculates weights. The Reverse operator can be combined with the Convolution operator. After the tensor input to the Convolution operator is processed by the Reverse operator in a certain direction, the layout of the weights in the Convolution operator can be changed according to the rules of the Reverse operator, so that it matches the same meaning as the original Convolution operator and the output remains unchanged. Therefore, the operation of moving the Reverse operator down and combining it with the Convolution operator achieves the objective of optimizing the redundant Reverse operator.

[0100] The above describes embodiments for combining the Reshape and Transpose operators with other operators of the same type, and furthermore, other operators can be combined with operators of the same type.

[0101] Figures 12A and 12B show schematic diagrams of the combination of the Concat and Cast operators according to the embodiments of this application, respectively.

[0102] The Concat operator combines two or more tensors; for example, it combines a 3*4 tensor and a 3*6 tensor into a 3*10 tensor. It's important to note that the Concat operator is a tensor operation that does not change the numerical values ​​of the input tensors.

[0103] As shown in Figure 12A, a Concat operator can be formed by combining one adjacent Concat operator with another. For example, the operation of one Concat operator combines a 3*4 tensor and a 3*6 tensor (two input tensors, see the two downward input arrows above the left Concat operator in Figure 12A) into a 3*10 tensor, and the operation of another Concat operator combines the tensor output by the left Concat operator (a 3*10 tensor) with two more input tensors, for example, a 3*5 tensor and a 3*7 tensor (again two input tensors, see the two downward input arrows to the upper right of the intermediate Concat operator in Figure 12A) into a 3*22 tensor. The operation of this combined Concat operator may take four tensors as input, namely a 3*4 tensor, a 3*6 tensor, and a 3*5 tensor and a 3*7 tensor, and combine them all at once into a 3*22 tensor.

[0104] In this way, the number of operators calculated is reduced, or the computational amount performed by the operators is reduced, thereby reducing the delay in the subsequent time it takes for a neural network processing image data to operate on the chip.

[0105] The Cast operator performs a type conversion on a number. For example, it converts the integer type 1 to the floating-point number 1.0, or a 32-bit integer type to a 64-bit integer type. Note that the Cast operator only changes the type of the number; it does not change the number in the input tensor.

[0106] As shown in Figure 12B, the Cast A to B operation in the upper left represents a conversion from a tensor of type A to type B, and the Cast B to C operation in the lower left represents a conversion from a tensor of type B to type C. The Cast operator operation, which combines these two Cast operators, may also be Cast A to C, that is, it may directly convert a tensor of type A to type C.

[0107] In this way, the number of operators calculated is reduced, or the computational amount performed by the operators is reduced, thereby reducing the delay in the subsequent time it takes for a neural network processing image data to operate on the chip.

[0108] In one embodiment, during the adjustment process of the first operator, the insertion of multiple first operators that can mutually cancel each other can be considered so that these multiple first operators can perform operations such as joining or canceling with other adjacent operators. Here, mutual cancellation may be mutually inverse operations.

[0109] Figure 13A shows a schematic diagram of one embodiment in which, by inserting a plurality of mutually cancelable first operators according to the embodiment of the present application, these plurality of first operators can perform operations of combining or canceling with other adjacent operators.

[0110] First, there are two image data channel formats: the NCHW layout, also known as "channels_first," and the NHWC layout, also known as "channels_last." In the NCHW layout, C is on the outside, so within each channel, the image pixels are adjacent, i.e., "RRRGGGBBB." In the NHWC layout, C is on the inside, so within each channel, the image pixels are spaced apart, i.e., "RGBRGBRGB."

[0111] While some neural network model formats support the NCHW layout Convolution operator by default, when performing calculations on an NHWC layout chip, the computation diagram requires converting the NHWC layout to the NCHW layout, performing the NCHW convolution, and then converting the convolution result back to the NHWC layout. In the computation diagram, as shown on the left side of Figure 13A, the process involves passing through the Transpose NHWC2NCHW operator, then the NCHW_Convolution operator, and finally the Transpose NCHW2NHWC operator.

[0112] To transform this layout by inserting multiple mutually cancelable transposes into the original calculation diagram, the entire calculation diagram can then be optimized by moving them down, joining them, and / or canceling them out, thereby minimizing the cost of the transformed layout, and in optimal cases eliminating the cost altogether, i.e., optimizing all inserted transposes.

[0113] For example, as shown in Figure 13A, in the first step, two mutually cancelable Transpose operators, the Transpose NCHW2NHWC operator and the Transpose NHWC2NCHW operator (which are mutually inverse operations), are inserted below the original Transpose NHWC2NCHW operator and above the Transpose NCHW2NHWC operator, respectively. In the second step, the inserted Transpose NCHW2NHWC operator and the Transpose NHWC2NCHW operator in the original calculation diagram can cancel each other out because they are inverse operations, and the inserted Transpose NHWC2NCHW operator and the Transpose NCHW2NHWC operator in the original calculation diagram can cancel each other out because they are inverse operations. In this way, the original calculation diagram can be optimized so that only the NHWC_Convolution operator remains.

[0114] In this way, the number of operators calculated is reduced, or the computational amount performed by the operators is reduced, thereby reducing the delay in the subsequent time it takes for a neural network processing image data to operate on the chip.

[0115] Figure 13B shows a schematic diagram of another embodiment in which a plurality of mutually cancelable first operators are inserted according to the embodiment of the present application, thereby enabling these multiple first operators to perform operations of joining or canceling with other adjacent operators.

[0116] First, the inputs to the operators of a neural network model can be represented by a left-hand-side (LHS) input tensor and a right-hand-side (RHS) input tensor, in the case of two inputs.

[0117] Some neural network model formats support the fully connected operator (i.e., the matmul operator) without transpose on the RHS (second input) by default, but some chips require transpose on the RHS of the fully connected operator (i.e., the matmul operator). When performing the fully connected operator operation on two tensors, one must be careful of the dimension matching problem. For example, to succeed, one needs to multiply a 3*4 tensor by a 4*5 tensor, that is, one can multiply and add them one by one, i.e., perform matrix multiplication, only if the number of columns (4) of the first tensor is the same as the number of rows (4) of the second tensor. Therefore, in order to successfully perform the consequent fully connected operator operation with another input tensor, the original computation diagram has an NC_Transpose operator that transforms the dimension of one of the input tensors.

[0118] Therefore, by inserting two mutually cancelable transposes into the original computational diagram to transform the layout, and then optimizing the entire computational diagram by moving them down, joining them, and canceling them out, the cost of the transformed layout can be minimized, and in optimal cases eliminated, meaning that all inserted transposes can be optimized. Similarly, when inserting transposes to transform rhs, there are convolution weights, and the weights support the conversion from NCHW or HWCN to NHWC.

[0119] As shown in Figure 13B, in the first step, two transpose operators that are mutually inverse operations and mutually cancel each other out are inserted into the original calculation diagram on the left side of Figure 13B: the NC_Transpose operator (shown in shaded in the figure and which must be set as the inverse operation of the original NC_Transpose operator) and the mutually inverse NC_Transpose operator are inserted below the original NC_Transpose operator and above the Fully connected operator, respectively. In the second step, the NC_Transpose operator inserted above the Fully connected operator and the Fully connected operator in the original calculation diagram are combined to form a Fully connected operator with RHS transpose (for example, the Fully connected rhs transposed shown in Figure 13B). In the third step, the inserted NC_Transpose operator (shown in shaded in the figure) and the NC_Transpose operator in the original calculation diagram are mutually inverse operations and can cancel each other out. As a result, the original calculation diagram is optimized to only the Fully connected operator with RHS transpose.

[0120] In this way, the number of operators calculated is reduced, or the computational amount performed by the operators is reduced, thereby reducing the delay in the subsequent time it takes for a neural network processing image data to operate on the chip.

[0121] The above describes the combination of one (adjusted) first operator and one adjacent second operator. The combined operators can then be moved up or down to perform optimization operations that combine or cancel out other adjacent operators, that is, continuous combinations and possible cancellations are possible.

[0122] Thus, for some operators, depending on their specific calculation content, they can be inserted or moved up or down in the calculation diagram before being combined or canceled out with the calculations of other operators. This optimizes these operators, reducing the number of operators calculated or the computational load they perform, thereby reducing the latency required for a neural network processing image data to operate in real time on the chip. Furthermore, by combining this with several specific hardware read functions of the artificial intelligence chip developed by the applicant, it is possible to ensure that the calculation results match those of these operators before optimization, while also reducing the number of operators calculated or the computational load they perform. This optimization is performed based on a high-level semantic representation called a calculation diagram, avoiding the complex and cumbersome optimization operations at the instruction level, enabling optimization of more general scenarios, and increasing versatility.

[0123] In summary, each embodiment of this application can achieve at least the following effects.

[0124] 1. In the computation diagram, the Transpose operator is split, moved downwards, combined, or canceled out to perform combinatorial optimization, change the scheduling order of the Transpose operator, and ultimately achieve a layout transformation of the computation diagram, reduce the computation amount of redundant operators, and optimize the runtime delay of the computation diagram.

[0125] 2. Move the Reshape operators in the calculation diagram downwards, combine or cancel them out, perform combinatorial optimization, change the scheduling order of the Reshape operators, and ultimately reduce redundant calculations by the Reshape operators in the calculation diagram, optimizing the runtime delay of the calculation diagram.

[0126] 3. The Broadcast operator can extend the shape of the tensor, split the Broadcast operator in the computation diagram, move it down, combine it, and perform combinatorial optimization to reduce the computational complexity of the Broadcast consequent operator in the original computation diagram, ultimately optimizing the runtime delay of the computation diagram.

[0127] 4. The Gather operator generally selects a portion of the input data as the output, so the amount of data output from the Gather operator is generally smaller than the amount of input data. However, in some chips, if the difference between the input and output data amounts is smaller than a certain threshold, calculating the Gather operator twice and making the consequent node calculate a small amount of data is far greater than the amount of data the consequent node calculates, but slightly greater than the cost of calculating the Gather operator only once. Therefore, the technique of combining two similar Gather operators and moving them down achieves an optimization effect.

[0128] 5. By changing the layout of the weights of the Convolution operator, it is possible to combine the antecedent Reverse operator input to the Convolution operator. Combining this feature, by moving it downwards and combining it, the optimization objective of optimizing redundant Reverse operators and reducing computation time can be achieved.

[0129] 6. Based on mathematical meaning, operators can be sequentially combined, and specifically in a chip, they can be selectively combined based on chip parameters to reduce redundant operators, thereby achieving the optimization objective of reducing computational complexity.

[0130] 7. Depending on the specific scenario, the optimization technique can be used to change the order of elements by moving them down, combining and canceling them out on different chips, allowing for flexible combinations while satisfying the requirement that the mathematical meaning remains unchanged, and achieving the effect of optimization.

[0131] When the above process is simulated using an 8.1GHz simulator and the optimization operation is not performed, the estimated delay for 4pe 4batch is 4,562,102 ns. After the optimization operation, the estimated delay becomes 4,450,052 ns. In other words, in this model, the optimization percentage of the delay at which the chip simulator executes the neural network in subsequent time is 2.46%, and its absolute value is 112.05 us.

[0132] Figure 14 shows a schematic block diagram of an artificial intelligence chip according to an embodiment of this application.

[0133] As shown in Figure 14, the artificial intelligence chip 1400 includes a memory 1401 that stores received application data and inputs it into a computation diagram of a neural network module having each operator to obtain the processing output result for the application data, and a controller 1402 that, based on the specific operation of each operator, performs at least one of the following at least once in the computation diagram: an adjustment in which the first operator is swapped in position with a consequent or antecedent operator in the computation diagram; an adjustment in which the first operator is divided into two or more similar operators; and an adjustment in which a plurality of mutually cancelable first operators are inserted. The first operator is a tensor operation that does not change the numerical value in the input tensor, and based on the specific operation of each operator, determines the following second operator adjacent to the adjusted first operator in the computation diagram, and the operation of the adjusted first operator and the operation of the second operator can be combined or canceled out without affecting the operation result, and the chip combines or cancels out the adjusted first operator and the second operator and performs the operation of each operator in the combined or canceled computation diagram.

[0134] In one embodiment, the application data includes at least one of image data and natural language data.

[0135] In one embodiment, the step of performing at least one of the following adjustments at least once with respect to a first operator in a calculation diagram: adjusting the position of the first operator with respect to a consequent or antecedent operator in the calculation diagram; adjusting the first operator to divide it into two or more similar operators; and adjusting the insertion of a plurality of mutually cancelable first operators, includes, if the first operator has two or more branched consequent or antecedent operators, the step of dividing the first operator into two or more similar operators so that each first operator has a one-to-one correspondence with a consequent or antecedent operator; and if the consequent operators of two or more similar first operators are the same operator, the step of integrating two or more similar first operators into a single operator.

[0136] In one embodiment, the steps of the chip combining or canceling out a adjusted first operator and a second operator, and performing operations on each operator in the combined or canceled calculation diagram, include, if the operation of the second operator before combining or canceling out is an operation of the second operator on a numerical value in the output tensor of the adjusted first operator, then combining the adjusted first operator and the second operator as a fourth operator, wherein the operation of the fourth operator is a step of replacing the tensor operation of the adjusted first operator itself by reading numerical values ​​in the input tensor of the first operator adjusted by the chip in a predetermined order, the predetermined order being determined based on the tensor operation of the adjusted first operator itself, and the steps of performing the operation of the second operator on the numerical value read by the chip based on the operation of the second operator.

[0137] In one embodiment, the reading of numerical values ​​in the input tensor of the adjusted first operator by the chip in a predetermined order is achieved by setting at least one of the reading stride, offset amount, and loop count by the chip in software.

[0138] In one embodiment, the step of the chip combining or canceling out the adjusted first and second operators and performing the operation of each operator in the combined or canceled calculation diagram includes, if the second operator is a tensor operation that does not change the numerical value in the input tensor before combining or canceling, the step of combining the adjusted first and second operators as a third operator after combining or canceling, wherein the tensor operation of the third operator realizes the tensor operation of the adjusted first operator and the tensor operation of the second operator together.

[0139] In one embodiment, the steps of a chip combining or canceling out a first operator and a second operator, and performing operations on each operator in the combined or canceled calculation diagram, include the step of canceling out the first operator and the second operator if the first and second operators, which were combined or canceled out before, were inverse operations on each other.

[0140] In one embodiment, the first operator is at least one of the transpose operator, reshape operator, broadcast operator, gather operator, reverse operator, concat operator, and cast operator.

[0141] In one embodiment, the second operator is at least one of the following: transpose operator, reshape operator, broadcast operator, gather operator, reverse operator, concat operator, flatten operator, cast operator, elementwise binary operator, batch fully connected operator, and convolution operator.

[0142] Thus, for some operators, depending on their specific calculation content, they can be inserted or moved up or down in the calculation diagram before being combined or canceled out with the calculations of other operators. This optimizes these operators, reducing the number of operators calculated or the computational load they perform, thereby reducing the latency required for a neural network processing image data to operate in real time on the chip. Furthermore, by combining this with several specific hardware read functions of the artificial intelligence chip developed by the applicant, it is possible to ensure that the calculation results match those of these operators before optimization, while also reducing the number of operators calculated or the computational load they perform. This optimization is performed based on a high-level semantic representation called a calculation diagram, avoiding the complex and cumbersome optimization operations at the instruction level, enabling optimization of more general scenarios, and increasing versatility.

[0143] Figure 15 shows a block diagram of an exemplary electronic device suitable for realizing the embodiment of this application according to the embodiment of this application.

[0144] The electronic device may include a processor (H1) and a storage medium (H2) coupled to the processor (H1) and storing computer-executable instructions that, when executed by the processor, perform the steps of each embodiment of this application.

[0145] The processor (H1) may include, but is not limited to, one or more processors or / or microprocessors.

[0146] The storage medium (H2) may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, and computer storage media (such as hard disks, floppy disks, solid-state hard disks, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.).

[0147] In addition, the electronic device may further include a data bus (H3), an input / output (I / O) bus (H4), a display (H5), and input / output devices (H6) (e.g., a keyboard, mouse, speaker, etc.).

[0148] The processor (H1) can communicate with external devices (H5, H6, etc.) via a wired or wireless network (not shown) through an I / O bus (H4).

[0149] The storage medium (H2) can further store at least one computer executable instruction that, when executed by the processor (H1), performs a step of each function and / or method in the embodiments described herein.

[0150] In one embodiment, the at least one computer executable instruction may be compiled or configured as a software product, and one or more computer executable instructions, when executed by a processor, perform the steps of each function and / or method in the embodiments described herein.

[0151] Figure 16 shows a schematic diagram of a non-temporary computer-readable storage medium according to an embodiment of the present application.

[0152] As shown in Figure 16, the computer-readable storage medium 1620 stores instructions, which are, for example, computer-readable instructions 1610. When a computer-readable instruction 1610 is executed by a processor, it can be executed by referring to the methods described above. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or high-speed cache memory (cache). Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, the computer-readable storage medium 1620 can be connected to a computing device such as a computer, and when the computing device executes a computer-readable instruction 1610 stored in the computer-readable storage medium 1620, it can execute various methods as described above.

[0153] This application may include the following items:

[0154] Item 1. A method for optimizing the computing power of a neural network module using a chip, A step of obtaining a computation diagram of a neural network module having each operator using a chip, the step of the neural network module receiving application data and obtaining the processing output result for the application data, The chip performs, at least once, the following adjustments on the calculation diagram based on the specific operations of each operator: an adjustment to swap the position of the first operator with a consequent or antecedent operator in the calculation diagram; an adjustment to divide the first operator into two or more similar operators; and an adjustment to insert a plurality of mutually cancelable first operators, wherein the first operator is a tensor operation that does not change the numerical value in the input tensor. The chip includes the step of determining a second operator adjacent to a coordinated first operator in a calculation diagram based on the specific calculation of each operator, wherein the calculation of the coordinated first operator and the calculation of the second operator can be combined or canceled out without affecting the calculation result. The process includes the steps of combining or canceling the adjusted first operator and the second operator using a chip, and performing operations on each operator in the combined or canceled calculation diagram.

[0155] Item 2, the method according to Item 1, wherein the application data includes at least one of image data and natural language data.

[0156] Item 3, tip, in the calculation diagram, the step of performing at least one of the following adjustments at least once with respect to the first operator: an adjustment that swaps the position of the first operator with a consequent or antecedent operator in the calculation diagram; an adjustment that divides the first operator into two or more similar operators; and an adjustment that inserts a plurality of mutually cancelable first operators. If the first operator has two or more consequent or antecedent operators from which it has been branched, the first operator is divided into two or more similar operators so that each first operator has a one-to-one correspondence with a consequent or antecedent operator. The method according to Item 1, further comprising the step of integrating two or more similar first operators into a single operator if the consequent operators of those two or more similar first operators are the same operator.

[0157] The step of combining or canceling the first and second operators adjusted by item 4, the chip, and performing the operation of each operator in the combined or canceled calculation diagram is: The method according to Item 1, wherein, if the operation of the second operator is an operation of the second operator on a numerical value in the output tensor of the adjusted first operator before being combined or canceled, the adjusted first operator and the second operator are combined as a fourth operator, the operation of the fourth operator includes the step of replacing the tensor operation of the adjusted first operator itself by reading numerical values ​​in the input tensor of the adjusted first operator in a predetermined order by a chip, the predetermined order being determined based on the tensor operation of the adjusted first operator itself, and the step of performing the operation of the second operator on numerical values ​​read by the chip based on the operation of the second operator.

[0158] Item 5, the method according to item 4, wherein reading numerical values ​​in the input tensor of the adjusted first operator in a predetermined order by a chip is achieved by software setting at least one of the stride, offset amount, and loop count for reading by the chip.

[0159] The step of combining or canceling the first and second operators adjusted by item 6, chip, and performing the operation of each operator in the combined or canceled calculation diagram is: The method according to Item 1, wherein if the second operator is a tensor operation that does not change the numerical value of the input tensor before being joined or canceled, the method includes the step of joining the adjusted first operator and the second operator as a third operator after being joined or canceled, the tensor operation of the third operator implements the tensor operation of the adjusted first operator and the tensor operation of the second operator together.

[0160] The step of combining or canceling the first and second operators adjusted by item 7, chip, and performing the operation of each operator in the combined or canceled calculation diagram is: The method according to item 1, further comprising the step of canceling out the first and second operators if the adjusted first and second operators are inverse operations of each other before they are combined or canceled out.

[0161] Item 8, the method according to Item 1, wherein the first operator is at least one of the transpose operator, reshape operator, broadcast operator, gather operator, reverse operator, concat operator, and cast operator.

[0162] Item 9, the method according to Item 1, wherein the second operator is at least one of the transpose operator, reshape operator, broadcast operator, gather operator, reverse operator, concat operator, flatten operator, cast operator, elementwise binary operator, batch fully connected operator, and convolution operator.

[0163] Item 10. An artificial intelligence chip, A memory that stores received application data, inputs it into the computation diagram of a neural network module having each operator, and obtains the processing output result for the application data, Based on the specific operations of each operator, the calculation diagram performs at least one of the following adjustments at least once on the first operator: an adjustment to swap the position of the first operator with the consequent or antecedent operator in the calculation diagram; an adjustment to divide the first operator into two or more similar operators; and an adjustment to insert a plurality of mutually cancelable first operators, wherein the first operator is a tensor operation that does not change the numerical value in the input tensor. Based on the specific operations of each operator, the following second operators adjacent to the adjusted first operator in the calculation diagram are determined, and the operations of the adjusted first operator and the operations of the second operator can be combined or canceled out without affecting the calculation result. The chip includes a controller that combines or cancels the adjusted first operator and the second operator, and performs operations on each operator in the combined or canceled calculation diagram.

[0164] Item 11, the chip according to Item 10, wherein the application data includes at least one of image data and natural language data.

[0165] Item 12, tip, in the calculation diagram, the step of performing at least one of the following adjustments at least once with respect to a first operator: an adjustment that swaps the position of the first operator with a consequent or antecedent operator in the calculation diagram; an adjustment that divides the first operator into two or more similar operators; and an adjustment that inserts a plurality of mutually cancelable first operators. If the first operator has two or more consequent or antecedent operators from which it has been branched, the first operator is divided into two or more similar operators so that each first operator has a one-to-one correspondence with a consequent or antecedent operator. The tip according to item 10, which includes the step of integrating two or more similar first operators into a single operator if the consequent operators of those two or more similar first operators are the same operator.

[0166] The step of combining or canceling the first and second operators adjusted by item 13, chip, and performing the operation of each operator in the combined or canceled calculation diagram is: The chip according to item 10, wherein, if the operation of the second operator is an operation of the second operator on a numerical value in the output tensor of the adjusted first operator before it is combined or canceled, the chip is combined or canceled and then the adjusted first operator and the second operator are combined as a fourth operator, the operation of the fourth operator being a step of replacing the tensor operation of the adjusted first operator itself by reading numerical values ​​in the input tensor of the adjusted first operator in a predetermined order by the chip, the predetermined order being determined based on the tensor operation of the adjusted first operator itself, and the chip being used to perform the operation of the second operator on numerical values ​​read by the chip based on the operation of the second operator.

[0167] Item 14, the chip described in Item 13, wherein reading numerical values ​​in the input tensor of the adjusted first operator in a predetermined order by the chip is achieved by setting at least one of the reading stride, offset amount and loop count by the chip in software.

[0168] The step of combining or canceling the first and second operators adjusted by item 15, chip, and performing the operation of each operator in the combined or canceled calculation diagram is: The tip according to item 10, wherein if the second operator is a tensor operation that does not change the numerical value of the input tensor before being combined or canceled, the tip includes the step of combining the adjusted first operator and the second operator as a third operator after being combined or canceled, the tensor operation of the third operator implements the tensor operation of the adjusted first operator and the tensor operation of the second operator together.

[0169] The step of combining or canceling the first and second operators adjusted by item 16, chip, and performing the operation of each operator in the combined or canceled calculation diagram is: The tip according to item 10, which includes the step of canceling out the first and second operators if the adjusted first and second operators are inverse operations of each other before they are combined or canceled out.

[0170] Item 17, the tip according to item 10, wherein the first operator is at least one of the transpose operator, reshape operator, broadcast operator, gather operator, reverse operator, concat operator, and cast operator.

[0171] Item 18, the tip according to item 10, wherein the second operator is at least one of the transpose operator, reshape operator, broadcast operator, gather operator, reverse operator, concat operator, flatten operator, cast operator, elementwise binary operator, batch fully connected operator, and convolution operator.

[0172] Item 19. An electronic device that optimizes the computing power of a neural network module using a chip, Memory for storing instructions, Electronic device including a processor that reads instructions from the aforementioned memory and performs the method described in any of items 1 to 9.

[0173] Item 20. A non-temporary storage medium on which instructions are stored, A non-temporary storage medium in which the instruction is stored, wherein, when the instruction is read by the processor, the processor performs the method described in any of items 1 to 9.

[0174] Of course, the specific embodiments described above are examples and not limitations, and those skilled in the art can combine and integrate several steps and apparatus from the various embodiments described separately above based on the concept of the present application to achieve the effects of the present application. These combined embodiments are also included in the present application, and such combinations will not be described individually here.

[0175] Furthermore, the advantages, merits, and effects mentioned in this disclosure are illustrative and not limiting, and it is not considered necessary for each embodiment of this application to possess these advantages, merits, and effects. In addition, the specific details of the above disclosure are for illustrative and easy-to-understand purposes only, and are not limiting, and the above details do not limit the fact that this application must be realized using the above specific details.

[0176] The block diagrams of the devices, apparatus, equipment, and systems relating to this disclosure are provided for illustrative purposes only and are not intended to require or imply that connections, arrangements, or configurations must be made as shown in the block diagrams. As a person skilled in the art will recognize, these devices, apparatus, equipment, and systems can be connected, arranged, or configured in any way. The words “including,” “equipped,” and “possess” are open terms meaning “including, but not limited to,” and can be used interchangeably with them. The terms “or” and “and” as used herein mean the terms “and / or” unless the context explicitly indicates otherwise, and can be used interchangeably with them. The term “for example” as used herein means the phrase “for example, not limited to,” and can be used interchangeably with it.

[0177] The step flowcharts and methods described herein are illustrative examples only and are not intended to require or imply that the steps of each embodiment must be performed in a given order. As those skilled in the art will recognize, the steps in the above embodiments can be performed in any order. Words such as “then,” “and,” and “next” are not intended to limit the order of the steps. These words are used only to guide the reader in reading the description of these methods. Also, any reference to a singular element using, for example, the article “one,” “1,” or “it” is not intended to limit that element to singular.

[0178] Furthermore, the steps and apparatus in each embodiment of this specification are not limited to any particular embodiment. In practice, new embodiments can be conceived based on the concepts of this specification by combining some of the relevant steps and some of the apparatus in each embodiment of this specification, and these new embodiments are also included within the scope of this specification.

[0179] Each operation of the methods described above can be performed by any suitable means capable of performing the corresponding function. This means may include, but is not limited to, hardware circuits, application-specific integrated circuits (ASICs), or processors, and may include a variety of hardware and / or software components and / or modules.

[0180] Various exemplary logic blocks, modules, and circuits can be utilized, implemented or described by general-purpose processors, digital signal processors (DSPs), ASICs, field-programmable gate array signal (FPGA) or other programmable logic devices (PLDs), discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller or state machine. The processor may further be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, a microprocessor working with a DSP core, or any other such configuration.

[0181] Steps of the methods or algorithms described in connection with this disclosure can be directly incorporated into hardware, software modules executed by a processor, or a combination of the two. The software modules can reside in any form of tangible storage medium. Some examples of usable storage media include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, and CD-ROMs. The storage medium can be coupled to the processor so that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor. The software module may be a single instruction or a number of instructions, and may be distributed across several different code segments, different programs, and multiple storage media.

[0182] The methods disclosed herein include actions to implement the described methods. The methods and / or actions are interchangeable without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of any particular actions can be modified without departing from the scope of the claims.

[0183] The above functions can be implemented by hardware, software, firmware, or any combination thereof. If implemented by software, the functions can be stored as instructions on a suitable computer-readable medium. The storage medium may be any available suitable medium accessible by a computer. By example, but not limited to, such computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other reliable medium accessible by a computer that can be used to carry or store desired program code in the form of instructions or data structures. As used herein, disk and disc include compact discs (CDs), laserdiscs, optical discs, digital general-purpose discs (DVDs), floppy disks, and Blu-ray discs, where a disk typically reproduces data magnetically and a disc optically reproduces data using a laser.

[0184] Therefore, a computer program product can perform the operations given herein. For example, such a computer program product may be a computer-readable tangible medium having tangible stored (and / or encoded) instructions, which can be executed by a processor to perform the operations described herein. The computer program product may include packaging materials.

[0185] Software or instructions may be transmitted by a transmission medium. For example, software may be transmitted by a website, server, or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, RF, or microwave.

[0186] Furthermore, modules and / or other suitable means for performing the methods and techniques described herein may be downloaded and / or otherwise obtained by user terminals and / or base stations at appropriate times. For example, such equipment may be coupled to a server to facilitate the transmission of means for performing the methods described herein. Alternatively, the various methods described herein may be provided via storage materials (e.g., physical storage media such as RAM, ROM, CD, or floppy disk) so that user terminals and / or base stations can obtain the various methods while they are connected to the equipment or providing the equipment with storage materials. Furthermore, the methods and techniques described herein can be utilized in other suitable techniques of the equipment.

[0187] Other examples and embodiments are within the scope and spirit of the claims of this disclosure and the appended patent. For example, the functions described above can be implemented using software that runs by a processor, hardware, firmware, hardwire, or any combination thereof, due to the nature of the software. The features that implement the functions can also be physically located in various locations, including being distributed so that some of the functions are implemented in different physical locations. Furthermore, as used herein and in the claims, the "or" used in an enumeration of a term beginning with "at least one" means a separate enumeration, for example, the enumeration of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A, B, and C). Furthermore, the term "exemplary" does not mean that the examples described are preferred or superior to other examples.

[0188] Various changes, substitutions, and modifications of the techniques described herein can be implemented without departing from the taught techniques as defined by the appended claims. Furthermore, the scope of the claims in this disclosure is not limited to specific embodiments of the configurations, means, methods, and operations of the processes, machines, manufactures, and events described herein. By utilizing the corresponding embodiments described herein, existing or later developed configurations, means, methods, or operations of processes, machines, manufactures, and events that are substantially the same in function or produce substantially the same results can be realized. Accordingly, the appended claims include such configurations, means, methods, or operations of processes, machines, manufactures, and events within their scope.

[0189] The above description of the disclosed embodiments is provided so that a person skilled in the art can prepare or use the present application. Various modifications to these embodiments will be very obvious to a person skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the scope of the present application. Accordingly, the present application is not intended to be limited to the embodiments shown herein, but rather to adhere to the broadest scope that is consistent with the principles and novel features disclosed herein.

[0190] The above description is given for illustrative and explanatory purposes. Furthermore, this description is not intended to limit the embodiments of the present application to the forms of the present disclosure. Although several exemplary embodiments and examples have been described above, those skilled in the art will be able to recognize several variations, modifications, changes, additions, and subcombinations thereof.

Claims

1. A method for optimizing the computing power of a neural network module using a chip, A step in which the chip obtains a computation diagram of a neural network module having each operator, wherein the neural network module receives application data and obtains the processing output result for the application data, The chip performs, at least once, an adjustment on the calculation diagram, based on the specific operation of each operator, which is a tensor operation that does not change the numerical value in the input tensor. A step in which the chip determines a second operator adjacent to a coordinated first operator in a calculation diagram based on the specific calculations of each operator, wherein the calculations of the coordinated first operator and the calculations of the second operator can be combined or canceled out without affecting the calculation results, The chip includes the steps of combining or canceling the adjusted first operator and the second operator, and performing the operations on each operator in the combined or canceled calculation diagram, The step of combining or canceling the adjusted first operator and the second operator by the chip, and performing the calculation of each operator in the combined or canceled calculation diagram, A method for optimizing the computing power of a neural network module by a chip, comprising the steps of: if the operation of the second operator is an operation of the second operator on a numerical value in the output tensor of the adjusted first operator before it is combined or canceled out, then combining the adjusted first operator and the second operator as a fourth operator, wherein the operation of the fourth operator is a step of replacing the tensor operation of the adjusted first operator itself by reading numerical values ​​in the input tensor of the adjusted first operator from static random access memory SRAM in a predetermined order by a read operation of the chip hardware, the predetermined order being determined based on the tensor operation of the adjusted first operator itself, and performing the operation of the second operator on numerical values ​​read by the chip based on the operation of the second operator.

2. The method for optimizing the computing power of a neural network module using a chip according to claim 1, wherein the application data includes at least one of image data and natural language data.

3. The step of the chip performing at least one of the following adjustments at least once with respect to the first operator in the calculation diagram: an adjustment to swap the position of the first operator with a consequent or antecedent operator in the calculation diagram; an adjustment to divide the first operator into two or more similar operators; and an adjustment to insert a plurality of mutually cancelable first operators. If the first operator has two or more consequent or antecedent operators from which it has been branched, the first operator is divided into two or more similar operators so that each first operator has a one-to-one correspondence with a consequent or antecedent operator. A method for optimizing the computing power of a neural network module using a chip according to claim 1, comprising the step of integrating two or more similar first operators into a single operator if the consequent operators of two or more similar first operators are the same operator.

4. The method for optimizing the computing power of a neural network module using a chip according to claim 1, wherein the chip reads numerical values ​​in the input tensor of the first operator adjusted in a predetermined order, which is achieved by setting at least one of the reading stride, offset amount, and loop count by the chip using software.

5. The steps of combining or canceling the adjusted first operator and the second operator by the chip, and performing the operations on each operator in the combined or canceled calculation diagram, A method for optimizing the computing power of a neural network module using a chip according to claim 1, comprising the step of combining the adjusted first operator and the second operator as a third operator after the second operator is a tensor operation that does not change the numerical value of the input tensor before it is combined or canceled out, wherein the tensor operation of the third operator realizes the tensor operation of the adjusted first operator and the tensor operation of the second operator together.

6. The steps of combining or canceling the adjusted first operator and the second operator by the chip, and performing the operations on each operator in the combined or canceled calculation diagram, A method for optimizing the computing power of a neural network module using a chip according to claim 1, comprising the step of canceling out the first operator and the second operator if the adjusted first operator and the second operator are inverse operations of each other before they are combined or canceled out.

7. A method for optimizing the computing power of a neural network module using a chip according to claim 1, wherein the first operator is at least one of the following: the transfer operator, the reshape operator, the broadcast operator, the gather operator, the reverse operator, the concat operator, and the cast operator.

8. A method for optimizing the computing power of a neural network module using a chip according to claim 1, wherein the second operator is at least one of the following: the transfer operator, the reshape operator, the broadcast operator, the gather operator, the reverse operator, the concat operator, the flatten operator, the cast operator, the elementwise binary operator, the batch fully connected operator, and the convolution operator.

9. A memory that stores received application data, inputs it into the computation diagram of a neural network module having each operator, and obtains the processing output result for the application data, Including the controller, The controller in question is, Based on the specific operations of each operator, the calculation diagram performs at least one of the following adjustments at least once on the first operator: an adjustment to swap the position of the first operator with the consequent or antecedent operator in the calculation diagram; an adjustment to divide the first operator into two or more similar operators; and an adjustment to insert a plurality of mutually cancelable first operators, wherein the first operator is a tensor operation that does not change the numerical value in the input tensor. Based on the specific operations of each operator, a second operator adjacent to the adjusted first operator in the calculation diagram is determined, and the operations of the adjusted first operator and the operations of the second operator can be combined or canceled out without affecting the calculation result. The chip combines or cancels the adjusted first operator and the second operator, and performs the calculations of each operator in the combined or canceled calculation diagram. The chip combines or cancels the adjusted first operator and the second operator, and performs the operations on each operator in the combined or canceled calculation diagram, An artificial intelligence chip comprising the steps of: if the operation of the second operator is an operation of the second operator on a numerical value in the output tensor of the adjusted first operator before it is combined or canceled out, then combining the adjusted first operator and the second operator as a fourth operator, wherein the operation of the fourth operator is a step of replacing the tensor operation of the adjusted first operator itself by reading numerical values ​​in the input tensor of the adjusted first operator from static random access memory SRAM in a predetermined order by a read operation of the chip hardware, the predetermined order being determined based on the tensor operation of the adjusted first operator itself, and performing the operation of the second operator on numerical values ​​read by the chip based on the operation of the second operator.

10. The artificial intelligence chip according to claim 9, wherein the application data includes at least one of image data and natural language data.

11. The chip performs at least one of the following adjustments at least once with respect to the first operator in the calculation diagram: an adjustment to swap the position of the first operator with a consequent or antecedent operator in the calculation diagram; an adjustment to divide the first operator into two or more similar operators; and an adjustment to insert a plurality of mutually cancelable first operators. If the first operator has two or more consequent or antecedent operators from which it has been branched, the first operator is divided into two or more similar operators so that each first operator has a one-to-one correspondence with a consequent or antecedent operator. The artificial intelligence chip according to claim 9, further comprising the step of integrating two or more similar first operators into a single operator if the consequent operators of two or more similar first operators are the same operator.

12. The artificial intelligence chip according to claim 9, wherein the chip reads numerical values ​​from the input tensor of the first operator that has been adjusted in a predetermined order, by software setting at least one of the reading stride, offset amount, and loop count for the chip.

13. The steps of combining or canceling the adjusted first operator and the second operator by the chip, and performing the operations on each operator in the combined or canceled calculation diagram, The artificial intelligence chip according to claim 9, wherein, if the second operator is a tensor operation that does not change the numerical value of the input tensor before being combined or canceled, the steps include combining the adjusted first operator and the second operator as a third operator after being combined or canceled, the tensor operation of the third operator implements the tensor operation of the adjusted first operator and the tensor operation of the second operator together.

14. The steps of combining or canceling the adjusted first operator and the second operator by the chip, and performing the operations on each operator in the combined or canceled calculation diagram, The artificial intelligence chip according to claim 9, further comprising the step of canceling out the first operator and the second operator if the adjusted first operator and the second operator are inverse operations of each other before they are combined or canceled out.

15. The artificial intelligence chip according to claim 9, wherein the first operator is at least one of the following: the transfer operator, the reshape operator, the broadcast operator, the gather operator, the reverse operator, the concat operator, and the cast operator.

16. The artificial intelligence chip according to claim 9, wherein the second operator is at least one of the following: the transfer operator, the reshape operator, the broadcast operator, the gather operator, the reverse operator, the concat operator, the flatten operator, the cast operator, the elementwise binary operator, the batch fully connected operator, and the convolution operator.

17. An electronic device that optimizes the computing power of a neural network module using a chip, Memory for storing instructions, An electronic device comprising a processor that reads instructions from the memory and executes a method for optimizing the computing power of a neural network module using a chip according to any one of claims 1 to 8.

18. A non-temporary storage medium in which instructions are stored, A non-temporary storage medium in which, when the instruction is read by the processor, the processor executes the method for optimizing the computing power of a neural network module using a chip according to any one of claims 1 to 8.

Images