Convolutional neural network pipeline parallel implementation method and system based on computational accuracy

Abstract

The present invention discloses a method and system for implementing convolutional neural networks in parallel based on computational precision. This method belongs to the field of convolutional neural network technology and includes the following steps: S1: finding substructures; S2: operator conversion; S3: input and output precision conversion; and S4: low-precision calculation. The present invention divides and fuses the computational logic of a convolutional neural network into multiple independent modules for pipeline parallelization. It also quantizes modules suitable for quantization, thereby reducing throughput. The modules are then placed on high-precision computing cards or, after quantization, on low-precision cards for calculation. This method fully utilizes computing resources of different precisions to jointly complete convolutional neural network calculations, making it worthy of widespread use.

Description

Technical Field

[0001] The present invention relates to the technical field of convolutional neural networks, and in particular to a method and system for implementing convolutional neural networks in parallel based on computational accuracy. Background Art

[0002] Convolutional neural networks are the most popular neural network models in computer vision. They require a significant amount of computation and are typically run on expensive GPUs (Graphics Processing Units). As image resolution increases, the amount of video memory (GPU memory) required to run convolutional neural network operations is also increasing, and a single GPU is increasingly unable to accommodate the entire model.

[0003] There are currently many methods proposed to solve this problem, and the present invention involves two methods:

[0004] The first is pipeline parallelism, which splits a large model into multiple independent modules and deploys them across multiple GPUs or even multiple machines, completing the entire computational process in a pipelined manner. This approach enables lossless execution of large models, but its disadvantage is that its speed is limited by the bandwidth between GPUs or machines. When the module granularity is too small, frequent communication between multiple machines occurs, significantly increasing latency.

[0005] The second method is model quantization. Quantization refers to the mapping of high-precision values ​​to low-precision values. In neural networks, quantization is primarily used to reduce model size at the expense of a certain degree of precision. For example, replacing 32-bit calculations with 8-bit calculations can reduce the model size by a quarter. This method has the advantage of significantly reducing model size, but the disadvantage is a loss of computational precision. This is especially true in neural networks, where there are many operators, and different operators are affected by computational precision in different ways. These issues urgently need to be addressed. To this end, a pipelined parallel implementation method and system for convolutional neural networks based on computational precision are proposed. Summary of the Invention

[0006] The technical problem to be solved by the present invention is: how to make full use of computing resources of different precisions to jointly complete the calculation of convolutional neural networks. A pipeline parallel implementation method of convolutional neural networks based on computing precision is provided. By identifying the corresponding operators in the convolutional neural network, sub-computing modules are divided, and after fusion, they are sent to the low-precision computing card for calculation. After the calculation is completed, the results are sent back to the high-precision computing card, while the remaining operators are still calculated on the high-precision computing card.

[0007] The present invention solves the above technical problems through the following technical solutions, which include the following steps:

[0008] S1: Finding substructures

[0009] Read the computational structure diagram of the convolutional neural network computational model and find the substructure to be quantized that meets the quantization conditions;

[0010] S2: Operator Transformation

[0011] Quantize the high-precision substructure to be quantized found in step S1 into a low-precision operator;

[0012] S3: Input and output precision conversion

[0013] Inserting precision conversion operators before and after the substructure to be quantized that is quantized to low precision, thereby converting the input of the substructure to be quantized from high precision to low precision and the output from low precision to high precision;

[0014] S4: Low-precision calculations

[0015] Insert calculation sending nodes and result receiving nodes into the convolutional neural network calculation model, send the substructure to be quantized that has been converted into low-precision to the low-precision calculation card for calculation, and retrieve the calculation results from the low-precision calculation card.

[0016] Furthermore, in step S1, the specific process is as follows:

[0017] S101: topologically sorting the computational structure graph of the convolutional neural network computational model, and converting the computational structure graph into an operator sequence through topological sorting, where each node in the computational structure graph is an operator;

[0018] S102: Traverse the operator sequence. When encountering a convolutional layer operator, continue traversing backward to find the batch normalization operator and the subsequent linear rectification operator, that is, to obtain an operator group. If other operators do not depend on the intermediate output results of the operators in this group, merge the found convolutional layer operator, batch normalization operator, and linear rectification operator into one operator and mark it as the operator to be quantized. If the convolutional layer operator is not followed by a batch normalization operator or a linear rectification operator, or if other operators depend on the intermediate output of the operators in this group, only mark the current convolutional layer operator as the operator to be quantized.

[0019] S103: Traverse the operator sequence again and merge adjacent operators to be quantized into a quantization group, that is, a substructure to be quantized.

[0020] Furthermore, in step S102, the process of merging operators is as follows:

[0021] S1021: Delete the convolution layer operator, batch normalization operator, and linear rectification operator to be merged from the calculation structure diagram and replace them with a new convolution layer operator;

[0022] S1022: Integrate the batch normalization operator into the new convolutional layer operator. The weight and bias of the new convolutional layer operator are calculated by the weight and bias of the original convolutional layer operator and the weight and bias of the batch normalization operator, as shown in the following formula:

[0023] Weight_conv_new=Weight_bn·Weight_conv_old

[0024] Bias_conv_new=Weight_bn·Bias_conv_old+Bias_bn

[0025] Among them, Weight_conv_new represents the weight of the new convolutional layer operator, Bias_conv_new represents the bias of the new convolutional layer operator, Weight_conv_old represents the weight of the original convolutional layer operator, Bias_conv_old represents the bias of the original convolutional layer operator, Weight_bn represents the weight of the batch normalization operator, and Bias_bn represents the bias of the batch normalization operator;

[0026] That is, the weight of the new convolutional layer operator is obtained by multiplying the weight of the original convolutional layer operator by the weight of the batch normalization operator; the bias is obtained by multiplying the weight of the batch normalization operator by the bias of the original convolutional layer plus the bias of the batch normalization operator.

[0027] S1023: Integrate the linear rectification operator into the new convolutional layer operator. The linear rectification operator does not change the weights and biases of the new convolutional layer operator. Quantize the new convolutional layer operator directly using the scale and zeropoint after the linear rectification operator. Merge the linear rectification operator into the new convolutional layer operator.

[0028] Among them, Scale represents the proportional relationship between real numbers and integers, and Zeropoint represents the integer corresponding to zero in real numbers after quantization;

[0029] If there is no batch normalization operator or no linear rectification operator, the corresponding merging step will be skipped and will not affect the process.

[0030] Furthermore, in step S3, the precision conversion operator includes a high-precision to low-precision operator and a low-precision to high-precision operator, wherein the high-precision to low-precision operator is used to convert the input of the substructure to be quantized from high precision to low precision, and the low-precision to high-precision operator is used to convert the output of the substructure to be quantized from low precision to high precision.

[0031] Furthermore, in step S4, the high-precision to low-precision operator, the calculation sending node, the result receiving node, and the low-precision to high-precision operator are set in sequence from front to back.

[0032] The present invention also provides a convolutional neural network pipeline parallel implementation system based on computational precision, which utilizes the above method to jointly complete the convolutional neural network calculation using computing resources of different precisions, including:

[0033] The search module is used to read the computational structure diagram of the convolutional neural network computational model and find the substructure to be quantized that meets the quantization conditions;

[0034] A first conversion module is used to quantize the found high-precision substructure to be quantized into a low-precision operator;

[0035] The second conversion module is used to insert a precision conversion operator before and after the substructure to be quantized that is quantized to a low-precision operator, thereby converting the input of the substructure to be quantized from high precision to low precision and the output from low precision to high precision;

[0036] The computing module is used to insert a computing sending node and a result receiving node into the convolutional neural network computing model, send the substructure to be quantized that has been converted into low-precision data to a low-precision computing card for computing, and retrieve the computing results from the low-precision computing card;

[0037] The control processing module is used to send instructions to other modules and complete corresponding actions;

[0038] The search module, the first conversion module, the first transformation module, and the calculation module are all communicatively connected to the control processing module.

[0039] Compared with the existing technology, the present invention has the following advantages: the computing logic of the convolutional neural network is divided and integrated into multiple independent modules for pipeline parallelization, and the modules suitable for quantization are quantized to reduce the throughput, and the modules are placed on high-precision computing cards or quantized and placed on low-precision cards for calculation, thereby making full use of computing resources of different precisions to jointly complete the calculation of the convolutional neural network, which is worthy of promotion and use. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of the method for implementing a convolutional neural network pipeline in parallel based on computational accuracy in the first embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. However, the protection scope of the present invention is not limited to the following embodiment.

[0042] Example 1

[0043] like Figure 1As shown, this embodiment provides a technical solution: a pipeline parallel implementation method of a convolutional neural network based on computing accuracy. The input of this method is a convolutional neural network computing model, and the output is a pipeline parallel implementation solution of the convolutional neural network computing model including computing cards of different precisions, including the following steps:

[0044] Step 1: Identify substructures

[0045] Read the computational structure diagram of the convolutional neural network computational model and identify the substructures to be quantized that meet the quantization conditions;

[0046] Step 2: Operator conversion

[0047] Quantize the high-precision substructure to be quantized identified in step 1 into a low-precision operator;

[0048] Step 3: Input and output precision conversion

[0049] Inserting precision conversion operators before and after the substructure to be quantized that is quantized to low precision, thereby converting the input of the substructure to be quantized from high precision to low precision and the output from low precision to high precision;

[0050] Step 4: Low-precision calculation

[0051] Insert calculation sending nodes and result receiving nodes into the convolutional neural network calculation model, send the substructure to be quantized that has been converted into low-precision to the low-precision calculation card for calculation, and retrieve the calculation results from the low-precision calculation card.

[0052] In step one, the following steps are specifically included:

[0053] 1) Perform topological sorting on the computational structure graph of the convolutional neural network computational model, and convert the computational structure graph into an operator sequence through topological sorting. Each node in the computational structure graph is an operator.

[0054] 2) Traverse the operator sequence. When encountering a convolutional layer operator, continue to traverse backward to find the possible batch normalization operator and the subsequent possible linear rectification operator, that is, to obtain the operator group; if other operators do not depend on the intermediate output results of the group of operators, then merge the found convolutional layer operator, (possible) batch normalization operator and (possible) linear rectification operator into one operator and mark it as the operator to be quantized; if the convolutional layer operator is not followed by a batch normalization operator or a linear rectification operator, or if other operators depend on the intermediate output of the group of operators, then only mark the current convolutional layer operator as the operator to be quantized;

[0055] 3) Traverse the operator sequence again and merge adjacent operators to be quantized into a quantization group (i.e., substructure to be quantized), thereby avoiding inserting repeated precision conversion nodes in the middle.

[0056] In step 2), the specific process of merging the convolution layer operator, linear rectification operator, and batch normalization operator into one operator is as follows:

[0057] Step 21) Delete the multiple operators to be merged from the calculation structure diagram and replace them with a new convolutional layer operator;

[0058] Step 22) If there is a batch normalization operator in the operator group, integrate it into the new convolutional layer operator. The weight (Weight_conv_new) and bias (Bias_conv_new) of the new convolutional layer operator are calculated by the weight (Weight_conv_old) and bias (Bias_conv_old) of the original convolutional layer operator and the weight (Weight_bn) and bias (Bias_bn) of the batch normalization operator, as shown below:

[0059] Weight_conv_new=Weight_bn·Weight_conv_old

[0060] Bias_conv_new=Weight_bn·Bias_conv_old+Bias_bn;

[0061] Step 23) Integrate the linear rectification operator in the operator group into the new convolutional layer operator. The linear rectification operator does not affect the weight and bias of the new convolutional layer operator, but after integration, the corresponding linear rectification logic will be inserted into the new convolutional layer operator.

[0062] It should be noted that the above merging process has three cases: 1. All three operators are merged; 2. Only the convolutional layer and the batch normalization layer are merged; 3. Only the convolutional layer and the linear rectifier operator are merged. (The order should be to find the convolutional layer operator first, then find the possible batch normalization operator, and then find the possible linear rectifier operator.)

[0063] In step 2, low-precision operators are provided by GPU hardware manufacturers (or software frameworks). That is, the hardware manufacturers will provide different precision implementations for the same computing logic (the software framework may perform secondary packaging on it). When using it, you only need to convert the input into the required precision and call the corresponding implementation to achieve the corresponding precision calculation.

[0064] In step 3, the precision conversion operator is usually also provided by the hardware manufacturer. The principle of high-precision to low-precision conversion is to discard some bits in the high-precision data to find an approximate value within the low-precision range (such as approximating 1.254 to 1.25). The principle of low-precision to high-precision conversion is to fill the missing bits of the low-precision data in some way (such as converting 1.25 to 1.2500).

[0065] In step three, the precision conversion operator includes a high-precision to low-precision operator and a low-precision to high-precision operator, wherein the high-precision to low-precision operator is used to convert the input of the substructure to be quantized from high precision to low precision, and the low-precision to high-precision operator is used to convert the output of the substructure to be quantized from low precision to high precision.

[0066] In step 4, the high-precision to low-precision operator, the calculation sending node, the result receiving node, and the low-precision to high-precision operator are set in sequence from front to back.

[0067] It should be noted that some operators in convolutional neural networks have a high tolerance for precision; and the convolution layer (Convolution) + Rectified Linear Unit (ReLU) + Batch Normalization (BN) that are widely used in convolutional neural networks are operators with a high tolerance for precision.

[0068] To sum up, the above-mentioned embodiment of the convolutional neural network pipeline parallel implementation method based on computing accuracy divides and fuses the computing logic of the convolutional neural network into multiple independent modules for pipeline parallelization, and at the same time quantizes the modules suitable for quantization, thereby reducing the throughput, and placing the modules on high-precision computing cards or quantizing them on low-precision cards for calculation, thereby making full use of computing resources of different precisions to jointly complete the calculation of the convolutional neural network, which is worthy of promotion and use.

[0069] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A pipeline parallel implementation method for convolutional neural networks based on computational accuracy, characterized in that: The following steps are involved: S1: Finding substructure Read the computational structure diagram of the convolutional neural network computational model and find the substructure to be quantized that meets the quantization conditions; In step S1, the specific process is as follows: S101: topologically sorting the computational structure graph of the convolutional neural network computational model, and converting the computational structure graph into an operator sequence through topological sorting, where each node in the computational structure graph is an operator; S102: Traverse the operator sequence. When encountering a convolutional layer operator, continue traversing backward to find the batch normalization operator and the subsequent linear rectification operator, that is, to obtain an operator group. If other operators do not depend on the intermediate output results of the operators in this group, merge the found convolutional layer operator, batch normalization operator, and linear rectification operator into one operator and mark it as the operator to be quantized. If the convolutional layer operator is not followed by a batch normalization operator or a linear rectification operator, or if other operators depend on the intermediate output of the operators in this group, only mark the current convolutional layer operator as the operator to be quantized. S103: traverse the operator sequence again and merge adjacent operators to be quantized into a quantization group, i.e., a substructure to be quantized; S2: Operator Transformation Quantize the high-precision substructure to be quantized found in step S1 into a low-precision operator; S3: Input and output precision conversion Inserting precision conversion operators before and after the substructure to be quantized that is quantized to low precision, thereby converting the input of the substructure to be quantized from high precision to low precision and the output from low precision to high precision; S4: Low-precision calculations Insert calculation sending nodes and result receiving nodes into the convolutional neural network calculation model, send the substructure to be quantized that has been converted into low-precision to the low-precision calculation card for calculation, and retrieve the calculation results from the low-precision calculation card.

2. The method for implementing convolutional neural network pipelines in parallel based on computational accuracy according to claim 1, characterized in that: In step S102, the process of merging operators is as follows: S1021: Delete the convolution layer operator, batch normalization operator, and linear rectification operator to be merged from the calculation structure diagram and replace them with a new convolution layer operator; S1022: Integrate the batch normalization operator into the new convolutional layer operator. The weight and bias of the new convolutional layer operator are calculated by the weight and bias of the original convolutional layer operator and the weight and bias of the batch normalization operator, as shown in the following formula: Weight_conv_new=Weight_bn·Weight_conv_old Bias_conv_new=Weight_bn·Bias_conv_old+Bias_bn Among them, Weight_conv_new represents the weight of the new convolutional layer operator, Bias_conv_new represents the bias of the new convolutional layer operator, Weight_conv_old represents the weight of the original convolutional layer operator, Bias_conv_old represents the bias of the original convolutional layer operator, Weight_bn represents the weight of the batch normalization operator, and Bias_bn represents the bias of the batch normalization operator; S1023: Integrate the linear rectification operator into the new convolutional layer operator. The linear rectification operator does not change the weights and biases of the new convolutional layer operator. Quantize the new convolutional layer operator directly using the scale and zeropoint after the linear rectification operator. Merge the linear rectification operator into the new convolutional layer operator. Among them, Scale represents the proportional relationship between real numbers and integers, and Zeropoint represents the integer corresponding to zero in real numbers after quantization; If there is no batch normalization operator or no linear rectification operator, the corresponding merging step is skipped.

3. The method for implementing convolutional neural network pipelines in parallel based on computational accuracy according to claim 1, characterized in that: In step S3, the precision conversion operator includes a high-precision to low-precision operator and a low-precision to high-precision operator, wherein the high-precision to low-precision operator is used to convert the input of the substructure to be quantized from high precision to low precision, and the low-precision to high-precision operator is used to convert the output of the substructure to be quantized from low precision to high precision.

4. The method for implementing convolutional neural network pipelines in parallel based on computational accuracy according to claim 3, characterized in that: In step S4, a high-precision to low-precision operator, a calculation sending node, a result receiving node, and a low-precision to high-precision operator are set in order from front to back.

5. A system for pipelined parallel implementation of a convolutional neural network based on computational precision, utilizing computing resources of different precisions to jointly complete convolutional neural network computations using the method described in any one of claims 1 to 4, comprising: The search module is used to read the computational structure diagram of the convolutional neural network computational model and find the substructure to be quantized that meets the quantization conditions; A first conversion module is used to quantize the found high-precision substructure to be quantized into a low-precision operator; The second conversion module is used to insert a precision conversion operator before and after the substructure to be quantized that is quantized to a low-precision operator, thereby converting the input of the substructure to be quantized from high precision to low precision and the output from low precision to high precision; The computing module is used to insert a computing sending node and a result receiving node into the convolutional neural network computing model, send the substructure to be quantized that has been converted into low-precision data to a low-precision computing card for computing, and retrieve the computing results from the low-precision computing card; The control processing module is used to send instructions to other modules and complete corresponding actions; The search module, the first conversion module, the first transformation module, and the calculation module are all communicatively connected to the control processing module.

Images