Method and apparatus for inserting a conversion operator

Abstract

The application provides a method for inserting a conversion operator, which comprises: determining an initial data format of a first operator in a neural network; determining a current data format of the first operator; when the initial data format of the first operator is different from the current data format, inserting at least one first conversion operator and at least one second conversion operator around the first operator, the first conversion operator being used for converting input data of the first operator from the initial data format to the current data format, and the second conversion operator being used for converting output data of the first operator from the current data format to the initial data format. In the method, the initial data format is a data format beneficial to hardware performance, and the current data format is a data format beneficial to improving processing efficiency of the first operator; therefore, the conversion operator is inserted according to the initial data format and the current data format of the first operator, so that the requirements of the operator and the requirements of hardware can be met simultaneously.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to a method and apparatus for inserting a transformation operator. Background Technology

[0002] Neural networks are tools for realizing artificial intelligence (AI). They are characterized by large computational demands, large-scale and frequent memory accesses when processing input data. Neural network chips often have very little high-speed memory directly accessed, while the amount of input data used by neural networks is very large, requiring caching on relatively low-speed, large memory. Therefore, data throughput has become the current computational bottleneck for neural networks.

[0003] One method to improve the processing speed of neural networks is to rationally plan the data slicing strategy, copying the sliced ​​data to high-speed memory to improve memory access efficiency. The data slicing strategy depends on the hardware-supported data layout; different hardware may be suitable for different data layouts, and different operators may also be suitable for different data layouts. Therefore, there may be a problem where the data layout that improves memory access efficiency differs from the data layout that improves operator performance. Transformation operators need to be inserted into the neural network to solve this problem. Furthermore, different operators in a neural network may have different requirements for data types, which also necessitates the insertion of transformation operators into the neural network.

[0004] Since using conversion operators to transform data arrangement and data type consumes chip computing power, it is necessary to plan the insertion strategy of conversion operators reasonably in order to improve the acceleration effect of inserting conversion operators on the chip. Summary of the Invention

[0005] This application provides a method and apparatus for inserting a transformation operator, which can meet the requirements of the operator and the hardware, thereby improving the acceleration effect of the insertion transformation operator on the chip.

[0006] In a first aspect, a method for inserting transformation operators is provided, comprising: determining an initial data format of a first operator in a neural network; determining a current data format of the first operator; and when the initial data format of the first operator differs from the current data format, inserting at least one first transformation operator and at least one second transformation operator around the first operator, wherein the first transformation operator is used to transform the input data of the first operator from the initial data format to the current data format, and the second transformation operator is used to transform the output data of the first operator from the current data format to the initial data format.

[0007] In the above method, the initial data format is one that benefits hardware performance (improves memory access efficiency), and the current data format is one that improves the processing efficiency of the first operator. When executing the step of inserting conversion operators, each operator has an initial data format and a current data format. For operators with input data, a format conversion operator can be inserted before the operator to convert the input data read from memory from the initial data format to the current data format, thereby improving the processing efficiency of format-sensitive operators. For operators with output data, a format conversion operator can be inserted after the operator to convert the output data from the current data format to the initial data format, thereby improving the storage efficiency of storing the output data in memory. Therefore, inserting conversion operators based on the initial and current data formats of the first operator can simultaneously satisfy both the operator's requirements and the hardware's requirements.

[0008] Optionally, when the first operator is a non-format-sensitive operator, determining the current data format of the first operator in the neural network includes: determining the current data format of the first operator based on the current data format of the second operator, wherein the current data format of the first operator is the same as the current data format of the second operator, the first operator is used to process the output data of the second operator, or the second operator is used to process the output data of the first operator, and the second operator is a format-sensitive operator.

[0009] Since format-sensitive operators only benefit the AI ​​system when using data in a specific format (improving the processing efficiency of format-sensitive operators), while the processing efficiency of non-format-sensitive operators is almost the same when using data in different formats, the current data format of non-format-sensitive operators can be determined based on the current data format of format-sensitive operators, thereby improving the benefits of insertion transformation operators to the AI ​​system.

[0010] Optionally, when the first operator is a non-format-sensitive operator, determining the initial data format of the first operator in the neural network includes: determining the initial data format of the first operator based on the initial data format of the second operator, wherein the initial data format of the first operator is the same as the initial data format of the second operator, the first operator is used to process the output data of the second operator, or the second operator is used to process the output data of the first operator, and the second operator is a format-sensitive operator.

[0011] When the initial data format of a format-sensitive operator is the same as that of a non-format-sensitive operator, and when the current data format of a format-sensitive operator is the same as that of a non-format-sensitive operator, adjacent transformation operators can be eliminated. Therefore, the above-described optional embodiments can reduce the number of transformation operators inserted in the neural network and reduce the computational power consumption of the transformation operators on the hardware.

[0012] Optionally, the neural network includes multiple format-sensitive operators, and the second operator is the format-sensitive operator located at the center of the neural network among the multiple format-sensitive operators.

[0013] Typically, format-sensitive operators located at the center of a neural network connect to a large number of non-format-sensitive operators, placing them at the linchpin of the entire network. Therefore, using the format-sensitive operator at the center of the neural network as the baseline operator (i.e., the second operator mentioned above) helps improve the consistency of the initial and current data formats of each operator. That is, it makes as many operators as possible have the same initial data format, and / or makes as many operators as possible have the same current data format. This is beneficial for subsequent elimination of inserted format conversion operators.

[0014] Optionally, the method further includes: when the at least one second transformation operator comprises multiple second transformation operators with the same function connected together, merging the multiple second transformation operators with the same function.

[0015] The above-mentioned optional embodiments can reduce the number of transformation operators inserted in the neural network and reduce the computational power consumption of the transformation operators on the hardware.

[0016] Optionally, the method further includes: when the functions of the at least one second transformation operator are the same, merging the first operator and the at least one second transformation operator, wherein the first operator is a memory operator.

[0017] The above-mentioned optional embodiments can reduce the number of transformation operators inserted in the neural network and reduce the computational power consumption of the transformation operators on the hardware.

[0018] Optionally, the method further includes: when there are two conversion operators with opposite conversion functions between the first operator and the second operator, eliminating the two conversion operators with opposite conversion functions.

[0019] The above-mentioned optional embodiments can reduce the number of transformation operators inserted in the neural network and reduce the computational power consumption of the transformation operators on the hardware.

[0020] Optionally, the data processed by each operator in the neural network includes tensor data.

[0021] Optionally, the tensor data is image data, and the image data format includes four dimensions: the number of images, the number of channels in the image, the height of the image, and the width of the image.

[0022] Optionally, the tensor data is text data, and the data format of the text data includes at least one dimension, wherein the at least one dimension is text content.

[0023] Secondly, a method for inserting a conversion operator is provided, comprising: determining the current data type of a first operator and the current data type of a second operator, wherein the first operator and the second operator are two adjacent operators in a neural network; when the current data type of the first operator is different from the current data type of the second operator, inserting a conversion operator between the first operator and the second operator, wherein the conversion operator is used to convert the current data type of the first operator and the current data type of the second operator.

[0024] If the current data types of two operators are the same, it means that the output data of one operator can be directly used by the other operator, and a data type conversion operator does not need to be inserted between the two operators. If the current data types of two operators are different, it means that the output data of one operator cannot be directly used by the other operator, and a data type conversion operator needs to be inserted between the two operators. This can reduce the number of conversion operators inserted in the neural network and reduce the computational power consumption of the conversion operators on the hardware.

[0025] Optionally, determining the current data type of the first operator and the current data type of the second operator includes: when the first operator and the second operator have data type identifiers, determining the current data type of the first operator based on the data type identifier of the first operator, and determining the current data type of the second operator based on the data type identifier of the second operator; or,

[0026] When neither the first operator nor the second operator has a data type identifier, the current data type of the first operator is determined based on the initial data type of the first operator, and the current data type of the second operator is determined based on the initial data type of the second operator.

[0027] Generally, when an operator has a data type identifier, the data type indicated by the data type identifier is beneficial to improving the performance of the operator. The current data type of the operator (i.e., the data type actually used by the operator) can be determined based on the data type identifier, thereby improving the performance of the operator.

[0028] Optionally, the method further includes: determining the initial data type of the first operator and the initial data type of the second operator based on the type of input data of the neural network.

[0029] The initial data type of each operator is determined based on the type of input data, so that the initial data types of each operator are as similar as possible. In this way, when the current data type is determined based on the initial data type, the current data types of adjacent operators are also the same. Therefore, there is no need to insert conversion operators between adjacent conversion operators, thereby reducing the number of conversion operators inserted in the neural network and reducing the computational power consumption of conversion operators on the hardware.

[0030] Optionally, the method further includes: when the output of the first operator is connected to multiple conversion operators with the same function, merging the multiple conversion operators with the same function.

[0031] The above-mentioned optional embodiments can reduce the number of transformation operators inserted in the neural network and reduce the computational power consumption of the transformation operators on the hardware.

[0032] Optionally, the method further includes: when the output of the first conversion operator is connected to at least one conversion operator, and the at least one conversion operator has the same function, merging the first operator and the at least one conversion operator, wherein the first operator is a memory operator.

[0033] The above-mentioned optional embodiments can reduce the number of transformation operators inserted in the neural network and reduce the computational power consumption of the transformation operators on the hardware.

[0034] Optionally, the method further includes: when there are two conversion operators with opposite conversion functions between the first operator and the second operator, eliminating the two conversion operators with opposite conversion functions.

[0035] The above-mentioned optional embodiments can reduce the number of transformation operators inserted in the neural network and reduce the computational power consumption of the transformation operators on the hardware.

[0036] Optionally, the data processed by each operator in the neural network includes tensor data.

[0037] Optionally, the tensor data is image data, and the image data format includes four dimensions: the number of images, the number of channels in the image, the height of the image, and the width of the image.

[0038] Optionally, the tensor data is text data, and the data format of the text data includes at least one dimension, wherein the at least one dimension is text content.

[0039] Thirdly, this application provides an apparatus for inserting a conversion operator, including units for performing the method described in the first aspect. This apparatus may be a terminal device or a server, or a chip within a terminal device or server. The apparatus may include a processing unit.

[0040] When the device is a terminal device or a server, the processing unit may be a processor; the terminal device may also include a storage unit, which may be a memory; the storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the terminal device to perform the method described in the first aspect.

[0041] When the device is a chip within a terminal device or server, the processing unit can be a processing module. The processing unit can be connected to the storage unit through input / output interfaces, pins, or circuits, and execute the instructions stored in the storage unit to cause the chip to perform the method described in the first aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip (e.g., a read-only memory, random access memory, etc.).

[0042] Fourthly, this application provides an apparatus for inserting a conversion operator, including a unit for performing the method described in the second aspect. The apparatus may be a terminal device or a server, or a chip within a terminal device or server. The apparatus may include a processing unit.

[0043] When the device is a terminal device or a server, the processing unit may be a processor; the terminal device may also include a storage unit, which may be a memory; the storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the terminal device to perform the method described in the second aspect.

[0044] When the device is a chip within a terminal device or server, the processing unit can be a processing module. The processing unit can be connected to the storage unit through input / output interfaces, pins, or circuits, and execute the instructions stored in the storage unit to cause the chip to perform the method described in the second aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip (e.g., a read-only memory, random access memory, etc.).

[0045] Fifthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the method described in the first aspect.

[0046] In a sixth aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the method described in the second aspect.

[0047] In a seventh aspect, this application provides a computer program product comprising: computer program code, which, when executed by a processor, causes the processor to perform the method described in the first aspect.

[0048] Eighthly, this application provides a computer program product comprising: computer program code, which, when executed by a processor, causes the processor to perform the method described in the second aspect. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of an AI system applicable to this application;

[0050] Figure 2 This is a schematic diagram of a method for inserting a format conversion operator provided in this application;

[0051] Figure 3 This is a schematic diagram of a method for determining the initial data format of an operator provided in this application;

[0052] Figure 4 This is a schematic diagram of another method for determining the initial data format of an operator provided in this application;

[0053] Figure 5 This is a schematic diagram of a method for determining the current data format of an operator provided in this application;

[0054] Figure 6 This is a schematic diagram of another method for determining the current data format of an operator provided in this application;

[0055] Figure 7 This is a schematic diagram of another method for inserting format conversion operators provided in this application;

[0056] Figure 8 This is a schematic diagram of a method for an insertion type conversion operator provided in this application;

[0057] Figures 9 to 10 This is a schematic diagram of the method for determining the initial data type of an operator provided in this application;

[0058] Figure 11 This is a schematic diagram of the method for determining the current data type of an operator provided in this application;

[0059] Figure 12 This is a schematic diagram of a method for an insertion type conversion operator provided in this application;

[0060] Figure 13 This is a schematic diagram of a method for eliminating type conversion operators provided in this application;

[0061] Figure 14 This is a schematic diagram of a horizontal fusion of conversion operators provided in this application;

[0062] Figure 15 This is a schematic diagram of a vertical fusion of conversion operators provided in this application;

[0063] Figure 16 This is a schematic diagram illustrating the fusion of a conversion operator and a memory operator provided in this application;

[0064] Figure 17 This is a schematic diagram of an apparatus for an insertion format conversion operator provided in this application;

[0065] Figure 18 This is a schematic diagram of an apparatus for an insertion type conversion operator provided in this application;

[0066] Figure 19 This is a schematic diagram of a device for inserting a conversion operator provided in this application. Detailed Implementation

[0067] Figure 1 This is a schematic diagram of an AI system applicable to this application, which is used to develop AI applications (i.e., neural networks) and includes four modules: chip, chip enablement, framework, and application enablement. These four modules are briefly introduced below.

[0068] Chips are used to provide hardware computing resources for AI systems. Chips with a unified architecture can be used in multiple scenarios with only one operator development, reducing the difficulty of cross-scenario development.

[0069] Different chips are suited to different data formats. For example, when a central processing unit (CPU) is used to provide hardware computing resources for an AI system, data in the number heightwidth channel (NHWC) format is processed faster; when a graphics processing unit (GPU) is used to provide hardware computing resources for an AI system, data in the number channel heightwidth (NCHW) format is processed faster.

[0070] In the example above, when the data is an RGB image, N represents the number of RGB images, C represents the RGB channels of the RGB image, H represents the length of the RGB image, and W represents the width of the RGB image. Different data formats correspond to different data arrangements. Data arrangement refers to the order in which multidimensional vectors are placed in memory space, which can be column-major or row-major. Different data arrangements have different memory access efficiencies. Therefore, the data arrangement can be changed by changing the data format, thereby changing the memory access efficiency. For the goal of changing memory access efficiency, data arrangement and data format can be considered equivalent.

[0071] Chip enablement provides chip operator libraries and operator development toolkits for AI systems, such as compute architecture for neural networks (CANN). CANN provides a unified domain-specific language (DSL) interface, along with a pre-built set of high-level template encapsulation and automated performance tuning tools, allowing users to easily develop custom operators on the chip.

[0072] Operators are the building blocks of neural networks. Each operator represents a segment of computational logic, and the ordered combination of multiple operators constitutes a neural network. Neural networks, also known as artificial neural networks (ANNs), are mathematical or computational models that mimic the structure and function of biological neural networks (the central nervous system of animals, especially the brain), used to estimate or approximate functions. Typical neural networks include convolutional neural networks (CNNs), deep neural networks (DNNs), and multilayer perceptrons (MLPs).

[0073] A framework is used to provide a unified training / inference framework for AI systems. For example, the MindSpore framework can be used for neural network design. The MindSpore framework includes core subsystems such as model libraries, graph computation, and tuning toolkits, and features a unified distributed architecture and a flexible programming interface, adaptable to various development environments such as machine learning, deep learning, and reinforcement learning. The framework applicable to this application is not limited to this.

[0074] Application enablement provides AI systems with one-stop services such as end-to-end services, hierarchical application programming interfaces (APIs), and pre-integrated solutions, including data acquisition, model training, and adaptation to changes.

[0075] To improve memory access efficiency, different chips may use different data formats. For example, the data format used by CPUs is NHWC, while the data format used by GPUs is NCHW. Similarly, different operators may use different data formats. For instance, the convolution (conv) operator is suitable for data in a specific format, while the rectified linear unit (ReLU) operator is suitable for data in various formats. A specific format refers to the data obtained by processing one or more dimensions of tensor data. For example, splitting or padding one or more dimensions of tensor data in NCHW format results in data in a specific format. Since there are various ways to process tensor data, we use specific format data to describe the results of tensor data processing. Furthermore, different operators may have different requirements for data types. Therefore, when designing neural networks using the aforementioned AI system, different transformation operators need to be inserted to meet the data format requirements of both the chip and the neural network's operators.

[0076] The following describes the method for the insertion transformation operator provided in this application.

[0077] The insertion transformation operator method provided in this application can be applied to the framework of AI systems, such as... Figure 2 As shown, method 200 includes:

[0078] S210, determine the initial data format of the first operator in the neural network.

[0079] The initial data format of the first operator refers to the initial data format of the output data and / or output data of the first operator.

[0080] The initial data format of the first operator can be preset or derived from the initial data formats of other operators.

[0081] For example, for format-sensitive operators, such as the conv operator, the output data and / or the data format of the output data cannot be changed arbitrarily. The initial data format can be determined according to the identifier in the operator information of the format-sensitive operator. That is, the initial data format of the format-sensitive operator can be preset.

[0082] For example, for non-format-sensitive operators, such as the elementWise operator, each element in the input data is operated on individually, and the format of the input data has no effect on its operation. In order to reduce the number of format conversion operators inserted into the neural network, the initial data format of the non-format-sensitive operator can be determined to be the initial data format of the nearby format-sensitive operators. The beneficial effects of this method of determining the initial data format will be described in detail below.

[0083] In this application, adjectives such as "first" and "second" are used to describe different individuals within the same class of objects. For example, "first operator" and "second operator" as described below are two different operators, and there are no other limitations.

[0084] S220, determine the current data format of the first operator.

[0085] The current data format of the first operator is the data format of its actual input and / or output during operation, which can also be called the target data format. It is usually a data format that is beneficial to improving the performance of the first operator.

[0086] For example, if the first operator is a format-sensitive operator, such as the conv operator, then the current data format of the first operator is a specific format; if the first operator is a non-format-sensitive operator, such as the ReLU operator, then the current data format of the first operator can be the same as the initial data format.

[0087] When the first operator is a non-format-sensitive operator, the current data format of the first operator can also be determined through format derivation.

[0088] For example, the second operator is a format-sensitive operator adjacent to the first operator, where the first operator is adjacent to the second operator in the following ways: the first operator is used to process the output data of the second operator, or the second operator is used to process the output data of the first operator; the execution device of method 200 can deduce the current data format of the first operator based on the current data format of the second operator, and if the current data format of the second operator is a specific format, then the current data format of the first operator is also a specific format.

[0089] Below, in conjunction with Figures 3 to 6 Explain how to determine the initial and current data formats of an operator.

[0090] Figure 3 This application illustrates a neural network. In this neural network, most operators are format-insensitive operators, meaning that most operators do not require specific formats for the input and / or output data. Directly inserting format conversion operators would result in randomness in the inserted format conversion operators, making it difficult to eliminate adjacent format conversion operators.

[0091] The initial data format of format-sensitive operators is preset. The initial data format of non-format-sensitive operators can be determined based on the initial data format of format-sensitive operators. Furthermore, the current data format of non-format-sensitive operators can be determined based on the current data format of format-sensitive operators. In this way, the initial data format and current data format of most operators are consistent. When inserting format-converted data based on the initial data format and the current data format, two adjacent format-converting operators can be made to be format-converting operators with opposite functions, so as to eliminate two adjacent format-converting operators.

[0092] Figure 3 In the neural network shown, the format-sensitive operators are the conv operator and the pooling operator. When there are multiple format-sensitive operators in a neural network, one format-sensitive operator can be selected as the base operator, and the initial data format of the base operator is determined as the initial data format of all non-format-sensitive operators.

[0093] Optionally, the format-sensitive operator located at the center of the neural network can be designated as the baseline operator. Typically, the format-sensitive operator at the center of the neural network connects to a large number of non-format-sensitive operators, placing it at the pivotal position of the entire neural network. Therefore, using the format-sensitive operator at the center of the neural network as the baseline operator helps improve the consistency of the initial data format of each operator, that is, ensuring that as many operators as possible have the same initial data format. This facilitates the subsequent elimination of inserted format transformation operators.

[0094] Alternatively, the format-sensitive operator that connects the most non-format-sensitive operators can be determined as the baseline operator, with the same beneficial effects as those described in the previous paragraph.

[0095] Based on the above principles, one can choose... Figure 3 The `conv` operator is used as the baseline operator because it is located at the center of the neural network, and it has the largest number of non-format-sensitive operators that are directly or indirectly connected to it. A direct connection means that there are no other operators between the format-sensitive and non-format-sensitive operators, such as the connection between the `conv` operator and the ReLU operator. An indirect connection means that there is no other format-sensitive operator between the format-sensitive and non-format-sensitive operators, such as the connection between the `conv` operator and the addition (addition) operator.

[0096] After determining the conv operator as the baseline operator, the initial data format for all non-format-sensitive operators can be determined, such as... Figure 4 As shown, the initial data format for all non-format sensitive operators is NCHW.

[0097] Since format-sensitive operators only benefit the AI ​​system when using data in a specific format (improving the processing efficiency of format-sensitive operators), while the processing efficiency of non-format-sensitive operators is almost the same when using data in different formats, the current data format of non-format-sensitive operators can be determined based on the current data format of format-sensitive operators.

[0098] Similar to the method for determining the initial data format, the current data format of a format-sensitive operator can be determined based on the format identifier in the operator information, such as... Figure 5 As shown, the current data format of the conv operator is NCHW, and the current data format of the pooling operator is NHWC.

[0099] Subsequently, the current data formats of all non-format-sensitive operators are determined based on the current data format of the baseline operator (conv operator), as shown in the following figure. Figure 6 As shown.

[0100] After determining the current data format of each operator in the neural network, an identifier can be added to the format information of each operator. This identifier is used to indicate the current data format of each operator. Some operators do not have format information, but they express their format requirements through axis attribute settings. These operators are also format-sensitive operators, and the axis attribute is the identifier of the current data format.

[0101] Subsequently, the execution device of method 200 can perform the following steps according to the identifier.

[0102] S230, when the initial data format of the first operator is different from the current data format, at least one first conversion operator and at least one second conversion operator are inserted around the first operator. The first conversion operator is used to convert the input data of the first operator from the initial data format to the current data format, and the second conversion operator is used to convert the output data of the first operator from the current data format to the initial data format.

[0103] Optionally, when the initial data format of the first operator is the same as the current data format, it is determined not to insert a transformation operator around the first operator. "Around the first operator" refers to: the input terminal of the first operator and / or the input terminal of the first operator.

[0104] As shown above, in method 200, the initial data format is one that benefits hardware performance (improves memory access efficiency), and the current data format is one that improves the processing efficiency of the first operator. When executing the step of inserting conversion operators, each operator has an initial data format and a current data format. For operators with input data, a format conversion operator can be inserted before the operator to convert the input data read from memory from the initial data format to the current data format, thereby improving the processing efficiency of format-sensitive operators. For operators with output data, a format conversion operator can be inserted after the operator to convert the output data of the operator from the current data format to the initial data format, thereby improving the storage efficiency of storing the output data in memory. Therefore, inserting conversion operators based on the initial and current data formats of the first operator can simultaneously satisfy both the requirements of the operator and the hardware.

[0105] When executing S230, each operator has an initial data format and a current data format. For operators with input data, a format conversion operator can be inserted before the operator to convert the input data read from memory from the initial data format to the current data format, thereby improving the processing efficiency of format-sensitive operators. For operators with output data, a format conversion operator can be inserted after the operator to convert the output data of the operator from the current data format to the initial data format, thereby improving the storage efficiency of the output data stored in memory.

[0106] Since most operators have the same initial data format as the current data format, most adjacent format conversion operators have opposite functions, and therefore, the functions of most adjacent format conversion operators can be eliminated.

[0107] Figure 7 This application illustrates a method for eliminating format conversion operators.

[0108] Figure 7 The neural network includes a conv operator and a ReLU operator. The initial data format of the conv operator is NCHW, and the initial data format of the ReLU operator is NCHW. The current data format of the conv operator is a specific format, and the initial data format of the ReLU operator is a specific format.

[0109] Because the initial data format of the conv operator is different from the current data format, a transformation operator needs to be inserted before and after the conv operator. Figure 3Only one transformation operator (Trans1) inserted after the conv operator is shown. This transformation operator converts the output data of the conv operator from a specific format to NCHW. Since the initial data format of the ReLU operator is different from the current data format, a transformation operator needs to be inserted before and after the ReLU operator. The transformation operator (Trans2) inserted before the ReLU operator is used to convert the output data of the conv operator to a specific format.

[0110] Since the two format conversion operators between the conv and ReLU operators have opposite functions and can be eliminated, determining the initial data format of non-format-sensitive operators based on the initial data format of format-sensitive operators is beneficial for eliminating conversion operators between format-sensitive and non-format-sensitive operators.

[0111] Figure 7 This shows the case where the conv operator is connected to one operator. If the conv operator is connected to multiple operators, multiple transformation operators can be inserted before and after the conv operator.

[0112] The above describes the method of the insertion format conversion operator provided in this application. The following describes the method of the insertion type conversion operator provided in this application.

[0113] like Figure 8 As shown, method 800 includes:

[0114] S810, determine the current data type of the first operator and the current data type of the second operator, where the first operator and the second operator are two adjacent operators in the neural network.

[0115] In method 800, the first operator can be either before or after the second operator.

[0116] The current data type of an operator can be determined based on its data type identifier.

[0117] For example, when the first operator has a data type identifier, the current data type of the first operator is determined to be the data type indicated by the data type identifier; when the first operator does not have a data type identifier, the current data type of the first operator is determined to be the initial data type.

[0118] The method for determining the current data type of the second operator is the same as in the example above.

[0119] Optionally, the initial data type of the first operator and the initial data type of the second operator can be determined by the type of input data to the neural network.

[0120] Figures 9 to 11 This illustrates a method for determining the initial data type of operators. The first and second operators can be... Figures 9 to 11 Any two adjacent operators in the array.

[0121] After obtaining the architecture of the neural network, the execution device of method 800 can obtain the type of the operator at the starting position of the architecture, that is, obtain the type of the input data of the neural network. For example... Figure 9 As shown, the type of input data is, for example, floating-point (float).

[0122] Subsequently, the execution device determines the initial data type of each operator based on the type of input data, such as... Figure 10 As shown, the initial data type of each operator is floating point.

[0123] Next, the execution device determines the current data type of each operator based on the data type identifier in the operator information. For example... Figure 11 As shown, the current data type of the conv operator, batch normalization (bn) operator, ReLU operator, fully connected (fc) operator, and softmax operator is 16-bit floating-point (fp16), while the current data type of the constant operator and network output operator is floating-point.

[0124] In method 800, the data type of an operator refers to the data type of the input data and / or output data of the operator. For example, the current data type of the conv operator is fp16, which means that the current input data and the current output data of the conv operator are both of type fp16. Since the const operator has no input data, the current data type of the const operator is floating-point, which means that the current output data of the const operator is of type floating-point.

[0125] After determining the initial data type and current data type of each operator, the execution device can perform the following steps.

[0126] S420, when the current data type of the first operator is different from the current data type of the second operator, a conversion operator is inserted between the first operator and the second operator, the conversion operator being used to convert the current data type of the first operator and the current data type of the second operator.

[0127] Optionally, when the current data type of the first operator is the same as the current data type of the second operator, it is determined not to insert a data type conversion operator between the first operator and the second operator.

[0128] If the current data types of the two operators are the same, it means that the output data of one operator can be directly used by the other operator, and a data type conversion operator does not need to be inserted between the two operators. If the current data types of the two operators are different, it means that the output data of one operator cannot be directly used by the other operator, and a data type conversion operator needs to be inserted between the two operators.

[0129] Figure 12 An example of an insertion data type conversion operator is shown.

[0130] The const operator and the conv operator have different current data types. The output data of the const operator cannot be directly used by the conv operator. Therefore, a data type conversion operator needs to be inserted between the const operator and the conv operator. Similarly, data type conversion operators also need to be inserted between the const operator and the bn operator, and between the softmax operator and the network output operator.

[0131] Since the current data types of the conv and bn operators are the same, there is no need to insert a data type conversion operator between them; similarly, there is no need to insert a conversion operator between the bn operator and the ReLU operator, or between the fc operator and the softmax operator.

[0132] Alternatively, the neural network can be optimized after the transformation operator is inserted.

[0133] like Figure 13 As shown, since the const operator is a memory operator, it is equivalent to a variable without computational logic. The data type of the const operator can be modified directly before running the neural network. That is, the conversion operator "floating-point → fp16" can be merged into the const operator. This can avoid executing the conversion operator during the running of the neural network, thereby reducing the number of conversion operators inserted in the neural network.

[0134] As can be seen from the above, Method 800 first determines the current data type of two adjacent operators, and then determines whether to insert a type conversion operator between the two operators based on the current data type of these two operators, which can reduce the amount of type conversion operators used.

[0135] The method for inserting transformation operators provided in this application has been described in detail above. Since the inserted transformation operators consume the computing power of the hardware running the neural network, it is necessary to eliminate the transformation operators as much as possible in order to reduce the computing power consumed by the transformation operators. The method for eliminating transformation operators provided in this application is described below.

[0136] There are three methods for eliminating transformation operators: horizontal fusion of transformation operators, vertical fusion of transformation operators, and fusion of transformation operators and non-transformation operators. The following section combines... Figures 14 to 16 To explain, Figures 14 to 16 middle.

[0137] Figure 14 A method for horizontal fusion of transformation operators is shown.

[0138] The output data of operator 0 is in 4D format, the data format required by operators 1 and 2 is 5D, and the data format required by operators 3 and 4 is FZ. Therefore, a format conversion operator needs to be inserted before operators 1 and 2 respectively, and a format conversion operator needs to be inserted before operators 3 and 4 respectively.

[0139] Since the format transformation operators preceding operator 1 and operator 2 are the same, the format transformation operators preceding operator 1 and operator 2 can be merged; similarly, since the format transformation operators preceding operator 3 and operator 4 are the same, the transformation operators between operator 3 and operator 4 can be merged. This reduces the number of transformation operators inserted into the neural network.

[0140] Horizontal fusion through transformation operators Figure 14 The neural network on the right side has two fewer transformation operators than the neural network on the left side.

[0141] Figure 15 A method for vertical fusion of transformation operators is shown.

[0142] There are four conversion operators between operator 0 and operator 5. Cast-1 and Cast-2 are two type conversion operators with opposite functions, and Transp-1 and Transp-2 are two format conversion operators with opposite functions. For example, Cast-1 can convert type 1 to type 2, and Cast-2 can convert type 1 to type 1; Transp-1 can convert format 1 to format 2, and Transp-2 can convert format 2 to format 1. Therefore, Cast-1 and Cast-2 can be eliminated, and Transp-1 and Transp-2 can be eliminated. Operator 0 can be directly connected to operator 5. The functions of these conversion operators can be determined based on the operator's attribute information.

[0143] There are two conversion operators between operator 0 and operator 7. Cast-1 and Cast-3 are two format conversion operators with opposite functions. Cast-1 and Cast-3 can be eliminated, and operator 0 can be directly connected to operator 7.

[0144] Vertical fusion through transformation operators Figure 15 The neural network on the right side has three fewer transformation operators than the neural network on the left side.

[0145] Figure 16 A method for fusing transformation operators and non-transformation operators is shown.

[0146] The data type of the output data of operator 0 is 4D, while the data type required by operators 8 to 10 is 5D. Therefore, a transformation operator needs to be inserted before each of operators 8 to 10. Since all transformation operators after operator 0 are the same, all transformation operators after operator 0 can be merged into operator 0.

[0147] By fusing transformation operators and non-transformation operators Figure 16 The neural network on the right side has three fewer transformation operators than the neural network on the left side.

[0148] The three operator elimination schemes described above can be used individually or in combination without conflict. For example, Figure 14 The lateral fusion method shown is similar to Figure 15 The vertical fusion methods shown can be used in combination. Figure 16 The fusion method of transformation operators and non-transformation operators shown is as follows: Figure 15 The vertical fusion methods shown can also be used in combination.

[0149] The foregoing has detailed examples of the methods for the insertion transformation operator provided in this application. It is understood that the apparatus for the insertion transformation operator, in order to achieve the above-described functions, includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0150] For example, different functions can be divided into different functional units, or two or more functions can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used.

[0151] The following are schematic diagrams of the apparatus for an insertion format conversion operator and an insertion type conversion operator provided in this application.

[0152] Figure 17 This is a schematic diagram of an apparatus for insert format conversion operator provided in this application. The apparatus 1700 includes a processing unit 1710 and an optional storage unit 1720, which can store the input data and / or output data of each operator in the neural network.

[0153] The processing unit 1710 is configured to perform: determining the initial data format of a first operator in a neural network; determining the current data format of the first operator; and when the initial data format of the first operator differs from the current data format, inserting at least one first conversion operator and at least one second conversion operator around the first operator, wherein the first conversion operator is used to convert the input data of the first operator from the initial data format to the current data format, and the second conversion operator is used to convert the output data of the first operator from the current data format to the initial data format.

[0154] Optionally, when the first operator is a non-format-sensitive operator, the processing unit 1710 is specifically used to: determine the current data format of the first operator according to the current data format of the second operator, wherein the current data format of the first operator is the same as the current data format of the second operator, the second operator is a format-sensitive operator, the first operator is used to process the output data of the second operator, or the second operator is used to process the output data of the first operator.

[0155] Optionally, when the first operator is a non-format-sensitive operator, the processing unit 1710 is specifically used to: determine the initial data format of the first operator according to the initial data format of the second operator, wherein the initial data format of the first operator is the same as the initial data format of the second operator, the second operator is a format-sensitive operator, the first operator is used to process the output data of the second operator, or the second operator is used to process the output data of the first operator.

[0156] Optionally, the neural network includes multiple format-sensitive operators, and the second operator is the format-sensitive operator located at the center of the neural network among the multiple format-sensitive operators.

[0157] Optionally, the processing unit 1710 is further configured to: merge the multiple functionally identical second conversion operators when the at least one second conversion operator comprises multiple second conversion operators connected together.

[0158] Optionally, the processing unit 1710 is further configured to: merge the first operator and the at least one second conversion operator when the functions of the at least one second conversion operator are the same, wherein the first operator is a memory operator.

[0159] Optionally, the processing unit 1710 is further configured to: eliminate the two conversion operators with opposite conversion functions when there are two conversion operators with opposite conversion functions between the first operator and the second operator.

[0160] Optionally, the data processed by each operator in the neural network includes tensor data.

[0161] Optionally, the tensor data is image data, and the image data format includes four dimensions: the number of images, the number of channels in the image, the height of the image, and the width of the image.

[0162] Optionally, the tensor data is text data, and the data format of the text data includes at least one dimension, wherein the at least one dimension is text content.

[0163] The specific method by which the device 1700 executes the insertion format conversion operator and the resulting beneficial effects can be found in the relevant description in the method embodiments.

[0164] Figure 18 This is a schematic diagram of the structure of an insertion type conversion operator provided in this application. The device 1800 includes a processing unit 1810 and an optional storage unit 1820, which can store the input data and / or output data of each operator in the neural network.

[0165] The processing unit 1810 is configured to perform the following: determine the current data type of the first operator and the current data type of the second operator, wherein the first operator and the second operator are two adjacent operators in the neural network; when the current data type of the first operator and the current data type of the second operator are different, insert a conversion operator between the first operator and the second operator, wherein the conversion operator is used to convert the current data type of the first operator and the current data type of the second operator.

[0166] Optionally, the processing unit 1810 is specifically configured to: when the first operator and the second operator have data type identifiers, determine the current data type of the first operator based on the data type identifier of the first operator, and determine the current data type of the second operator based on the data type identifier of the second operator; or,

[0167] When neither the first operator nor the second operator has a data type identifier, the current data type of the first operator is determined based on the initial data type of the first operator, and the current data type of the second operator is determined based on the initial data type of the second operator.

[0168] Optionally, the processing unit 1810 is further configured to: determine the initial data type of the first operator and the initial data type of the second operator based on the type of input data of the neural network.

[0169] Optionally, the processing unit 1810 is further configured to: merge the multiple conversion operators with the same function when the output of the first operator is connected to multiple conversion operators with the same function.

[0170] Optionally, the processing unit 1810 is further configured to: merge the first operator and the at least one conversion operator when the output of the first conversion operator is connected to at least one conversion operator, and when the at least one conversion operator has the same function, wherein the first operator is a memory operator.

[0171] Optionally, the processing unit 1810 is further configured to: eliminate the two conversion operators with opposite conversion functions when there are two conversion operators with opposite conversion functions between the first operator and the second operator.

[0172] Optionally, the data processed by each operator in the neural network includes tensor data.

[0173] Optionally, the tensor data is image data, and the image data format includes four dimensions: the number of images, the number of channels in the image, the height of the image, and the width of the image.

[0174] Optionally, the tensor data is text data, and the data format of the text data includes at least one dimension, wherein the at least one dimension is text content.

[0175] The specific method by which the device 1800 executes the insertion type conversion operator and the resulting beneficial effects can be found in the relevant description in the method embodiments.

[0176] Figure 19 A schematic diagram of the structure of a device for an insertion conversion operator provided in this application is shown. Figure 19 The dashed lines indicate that the unit or module is optional. Device 1900 can be used to implement the methods described in the above method embodiments. Device 1900 can be a terminal device, a server, or a chip.

[0177] Device 1900 includes one or more processors 1901, which can support the implementation of the methods in the method embodiments. Processor 1901 can be a general-purpose processor or a special-purpose processor. For example, processor 1901 can be a central processing unit (CPU). The CPU can be used to control device 1900, execute software programs, and process data from the software programs. Device 1900 may also include a communication unit 1905 for implementing signal input (reception) and output (transmission).

[0178] For example, device 1900 may be a chip, communication unit 1905 may be the input and / or output circuit of the chip, or communication unit 1905 may be the communication interface of the chip, which may be a component of terminal equipment, network equipment or other electronic equipment.

[0179] For example, device 1900 may be a terminal device or a server, and communication unit 1905 may be a transceiver of the terminal device or the server, or communication unit 1905 may be a transceiver circuit of the terminal device or the server.

[0180] The device 1900 may include one or more memories 1902 storing a program 1904. The program 1904 can be executed by a processor 1901 to generate instructions 1903, causing the processor 1901 to perform the method described in the above method embodiments according to the instructions 1903. Optionally, the memory 1902 may also store data (such as a neural network). Optionally, the processor 1901 may also read data stored in the memory 1902, which may be stored at the same memory address as the program 1904, or it may be stored at a different memory address.

[0181] The processor 1901 and memory 1902 can be configured separately or integrated together, for example, integrated on the system-on-chip (SOC) of the terminal device.

[0182] For details on how processor 1901 executes the insertion transformation operator, please refer to the relevant description in the method embodiment.

[0183] It should be understood that the steps of the above method embodiments can be implemented by hardware logic circuits or software instructions in the processor 1901. The processor 1901 may be a CPU, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, such as discrete gate, transistor logic devices, or discrete hardware components.

[0184] This application also provides a computer program product that, when executed by processor 1901, implements the methods described in any of the method embodiments of this application.

[0185] The computer program product can be stored in memory 1902, for example, program 1904. Program 1904 is finally converted into an executable object file that can be executed by processor 1901 after processing such as preprocessing, compilation, assembly and linking.

[0186] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the methods described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0187] The computer-readable storage medium is, for example, memory 1902. Memory 1902 can be volatile memory or non-volatile memory, or memory 1902 can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0188] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and technical effects of the above-described apparatus and equipment can be referred to the corresponding processes and technical effects in the foregoing method embodiments, and will not be repeated here.

[0189] In the several embodiments provided in this application, the systems, apparatuses, and methods disclosed can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The apparatus embodiments described above are merely illustrative; the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components can be combined or integrated into another system. Furthermore, the coupling between units or components can be direct coupling or indirect coupling, including electrical, mechanical, or other forms of connection.

[0190] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0191] Furthermore, the terms "system" and "network" are often used interchangeably in this paper. The term "and / or" in this paper merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this paper generally indicates that the preceding and following related objects have an "or" relationship.

[0192] In summary, the above description is merely a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for inserting a transformation operator, characterized in that, include: Determine the initial data format for the first operator in the neural network; Determine the current data format of the first operator; When the initial data format of the first operator is different from the current data format, at least one first conversion operator and at least one second conversion operator are inserted around the first operator. The first conversion operator is used to convert the input data of the first operator from the initial data format to the current data format, and the second conversion operator is used to convert the output data of the first operator from the current data format to the initial data format. When the at least one second transformation operator comprises multiple second transformation operators with the same function connected together, the multiple second transformation operators with the same function are merged; When there are two conversion operators with opposite conversion functions between the first operator and an adjacent operator, eliminate the two conversion operators with opposite conversion functions; Wherein, when the first operator is a non-format-sensitive operator, determining the current data format of the first operator in the neural network includes: The current data format of the first operator is determined based on the current data format of the second operator, wherein the current data format of the first operator is the same as the current data format of the second operator, the first operator is used to process the output data of the second operator, or the second operator is used to process the output data of the first operator, and the second operator is a format-sensitive operator.

2. The method according to claim 1, characterized in that, When the first operator is a non-format-sensitive operator, determining the initial data format of the first operator in the neural network includes: The initial data format of the first operator is determined based on the initial data format of the second operator, wherein the initial data format of the first operator is the same as the initial data format of the second operator, the first operator is used to process the output data of the second operator, or the second operator is used to process the output data of the first operator, and the second operator is a format-sensitive operator.

3. The method according to claim 1 or 2, characterized in that, The neural network includes multiple format-sensitive operators, and the second operator is the format-sensitive operator located at the center of the neural network among the multiple format-sensitive operators.

4. The method according to claim 1 or 2, characterized in that, The method further includes: When the functions of the at least one second transformation operator are the same, the first operator and the at least one second transformation operator are merged, wherein the first operator is a memory operator.

5. The method according to claim 1 or 2, characterized in that, The data processed by each operator in the neural network includes tensor data.

6. The method according to claim 5, characterized in that, The tensor data is image data, and the image data format includes four dimensions: the number of images, the number of channels in the image, the height of the image, and the width of the image.

7. The method according to claim 5, characterized in that, The tensor data is text data, and the data format of the text data includes at least one dimension, which is the text content.

8. An apparatus for inserting a transformation operator, characterized in that, Includes a processing unit for: Determine the initial data format for the first operator in the neural network; Determine the current data format of the first operator; When the initial data format of the first operator is different from the current data format, at least one first conversion operator and at least one second conversion operator are inserted around the first operator. The first conversion operator is used to convert the input data of the first operator from the initial data format to the current data format, and the second conversion operator is used to convert the output data of the first operator from the current data format to the initial data format. When the at least one second transformation operator comprises multiple second transformation operators with the same function connected together, the multiple second transformation operators with the same function are merged; When there are two conversion operators with opposite conversion functions between the first operator and an adjacent operator, eliminate the two conversion operators with opposite conversion functions; Wherein, when the first operator is a non-format-sensitive operator, the processing unit is specifically used for: The current data format of the first operator is determined based on the current data format of the second operator, wherein the current data format of the first operator is the same as the current data format of the second operator, the first operator is used to process the output data of the second operator, or the second operator is used to process the output data of the first operator, and the second operator is a format-sensitive operator.

9. The apparatus according to claim 8, characterized in that, When the first operator is a non-format-sensitive operator, the processing unit is specifically used for: The initial data format of the first operator is determined based on the initial data format of the second operator, wherein the initial data format of the first operator is the same as the initial data format of the second operator, the first operator is used to process the output data of the second operator, or the second operator is used to process the output data of the first operator, and the second operator is a format-sensitive operator.

10. The apparatus according to claim 8 or 9, characterized in that, The neural network includes multiple format-sensitive operators, and the second operator is the format-sensitive operator located at the center of the neural network among the multiple format-sensitive operators.

11. The apparatus according to claim 8 or 9, characterized in that, The processing unit is also used for: When the functions of the at least one second transformation operator are the same, the first operator and the at least one second transformation operator are merged, wherein the first operator is a memory operator.

12. The apparatus according to claim 8 or 9, characterized in that, The data processed by each operator in the neural network includes tensor data.

13. The apparatus according to claim 12, characterized in that, The tensor data is image data, and the image data format includes four dimensions: the number of images, the number of channels in the image, the height of the image, and the width of the image.

14. The apparatus according to claim 12, characterized in that, The tensor data is text data, and the data format of the text data includes at least one dimension, which is the text content.

15. An apparatus for inserting a transformation operator, characterized in that, The device includes a processor and a memory for storing a computer program, and the processor for calling and running the computer program from the memory, causing the device to perform the method of any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 7.

Images