A two-dimensional convolution splitting method for embedded AI inference chips

Abstract

The application relates to a two-dimensional convolution splitting method for an embedded AI inference chip, and comprises the following steps: S1, judging whether the current two-dimensional convolution conv1 input is out of limit, if not, performing normal convolution processing, otherwise, entering S2; S2, if the last layer operator of the current convolution is a convolution operator and the splitting condition is met, splitting the last layer convolution operator conv0 into two parallel convolutions conv3 and conv4 according to the output channel number out_ch0, the input input of the conv0 will pass through the conv3 and the conv4 respectively, and then the step S4 is performed; if the splitting condition is not met or the last layer is not a convolution operator, the step S3 is performed; S3, adding two parallel convolutions conv5 and conv6 in front of the current convolution conv1; S4, splitting the current convolution conv1 into two parallel convolutions conv7 and conv8 according to the input channel number in_ch1, and the output of the last step is passed through the conv7 and the conv8 respectively; S5, performing matrix addition on the outputs of the conv7 and the conv8 to obtain the final result. The application effectively improves the generality and flexibility of the chip.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of embedded AI neural network deployment, and particularly relates to a two-dimensional convolution splitting method for an embedded AI inference chip. BACKGROUND

[0002] At present, convolutional neural networks are widely used in the field of artificial intelligence, especially in the field of computer vision, but there are still the following limitations restricting the deployment of large neural networks on embedded AI inference chips: when the convolution kernel height * input image width * input image channel number of the convolution layer > sram single bank capacity limit, the layer convolution will not be deployed due to exceeding the input limit of the chip AI acceleration core, and the current mainstream neural networks, such as YOLOv5, VGG, ResNet, etc., contain a large number of over-limit convolutions, which greatly affects the versatility of the chip. SUMMARY

[0003] The purpose of the present application is to provide a two-dimensional convolution splitting method for an embedded AI inference chip, which uses a large number of parallel convolutions instead of over-limit convolutions, solves the problem of not being able to deploy large convolution networks due to hardware limitations of the chip, and effectively improves the versatility and flexibility of the chip.

[0004] To solve the above technical problems, the technical scheme of the present application is as follows: a two-dimensional convolution splitting method for an embedded AI inference chip, comprising:

[0005] S1: obtaining the sram single bank capacity input_bank in the embedded AI inference chip, and judging whether the current two-dimensional convolution conv1 input is over-limit based on input_bank, if not, processing according to normal convolution; otherwise, entering S2 for convolution splitting;

[0006] S2: if the previous layer operator of the current convolution is a convolution operator and the splitting condition is met, then the previous layer convolution operator conv0 is split into two output channels out_ch3 and out_ch4 according to the output channel number out_ch0, and out_ch3 + out_ch4 = out_ch0, the input of conv0 will pass through conv3 and conv4 respectively, and then go to S4; if the splitting condition is not met or the previous layer is not a convolution operator, go to S3;

[0007] S3: adding two parallel convolutions conv5 and conv6 with 1*1 convolution kernel, 1 step, 0 padding, and half of the input channel number before the current convolution conv1;

[0008] S4: split the current convolution conv1 into two parallel convolutions conv7 and conv8 with input channel number in_ch / 2 according to input channel number in_ch1, and pass the output of the previous step through conv7 and conv8 respectively;

[0009] S5: perform matrix addition on the outputs of conv7 and conv8 to obtain the final result.

[0010] Preferably, the technical scheme of the present application includes a preset step S0 before step S1:

[0011] Let the current two-dimensional convolution be marked as conv1, and the parameters of conv1 be: batch number batch, input channel number in_ch1, output channel number out_ch1, input height height1 and width width1, convolution kernel height kh1 and width kw1, step size s1, padding p1, weight w1, and bias b1.

[0012] If the previous layer of the current convolution is a convolution operator and can be split, it is marked as conv0, and the parameters of conv0 be: batch number batch, input channel number in_ch0, output channel number out_ch0, input height height0 and width width0, convolution kernel height kh0 and width kw0, step size s0, padding p0, weight w0, and bias b0, in_ch1 = out_ch0.

[0013] Preferably, the splitting method specifically includes:

[0014] S1: judge whether the input of the current two-dimensional convolution conv1 is out of limit, if not, process it according to normal convolution; otherwise, go to S2 for convolution splitting, and the condition for conv1 being out of limit is kh1* width1* in_ch1>input_bank.

[0015] S2: If the current convolutional upper layer operator is also a convolutional conv0 and meets the splitting condition, then conv0 is split into two output channels of parallel convolutional conv3 and conv4 with output channel numbers out_ch3 and out_ch4, and out_ch3+out_ch4 = out_ch0, and conv3 and conv4 are represented as conv3(in_ch0, out_ch3, height0, width0, kh0, kw0, s0, p0) and conv4(in_ch0, out_ch4, height0, width0, kh0, kw0, s0, p0), and the input data of conv0 will pass through conv3 and conv4 respectively, and then go to S4, if the splitting condition is not met or the upper layer is not a convolutional operator, go to S3;

[0016] S3: Before conv1, two parallel convolutional conv5 and conv6 with a convolution kernel of 1*1, a step of 1, padding of 0, input data of conv1 input, and output channel numbers of out_ch5 and out_ch6 are added, and out_ch5+out_ch6 = in_ch1, and conv5(in_ch1, out_ch5, height1, width1, 1, 1, 1, 0) and conv6(in_ch1, out_ch6, height1, width1, 1, 1, 1, 0);

[0017] S4: The convolutional conv1 is split into two input channels of parallel convolutional conv7 and conv8 with input channel numbers in_ch7 and in_ch8, and in_ch7+in_ch8 = in_ch1, and conv7 and conv8 are represented as conv7(in_ch7, out_ch1, height1, width1, kh1, kw1, s1, p1) and conv8(in_ch8, out_ch1, height1, width1, kh1, kw1, s1, p1), and the output of the previous step passes through conv7 and conv8 respectively, and the output of the previous step passes through conv7 and conv8 respectively;

[0018] S5: The outputs of conv7 and conv8 are subjected to matrix addition to obtain the final result.

[0019] The preferred technical scheme of the application is as follows:

[0020] S2.1: Determine whether conv0 meets the splitting condition, and the judgment condition is whether the result of conv0 will be used in other network layers except conv1.

[0021] S2.2: keep other parameters unchanged, replace conv0 with two parallel convolutions conv3 and conv4 with output channels out_ch3 and out_ch4 respectively, and out_ch3+out_ch4 = out_ch0, and conv3 and conv4 are represented as conv3(in_ch0, out_ch3, height0, width0, kh0, kw0, s0, p0) and conv4(in_ch0, out_ch4, height0, width0, kh0, kw0, s0, p0);

[0022] S2.3: split the weights w0 of conv0 into w3 and w4; the data shape of w0 is a four-dimensional array with a size of [out_ch0, in_ch0, kh0, kw0]; w0 is divided into w3[out_ch3, in_ch0, kh0, kw0] and w4[out_ch4, in_ch0, kh0, kw0] according to the 0th dimension;

[0023] S2.4: split the weights b0 of conv0 into b3 and b4; the data shape of b0 is a one-dimensional array with a size of [out_ch0]; b0 is divided into b3[out_ch3] and b4[out_ch4];

[0024] S2.5: insert w3 and b3 into conv3, and insert w4 and b4 into conv4;

[0025] S2.6: the input data of conv0 will pass through conv3 and conv4 in parallel.

[0026] For the preferred technical solution of the present application, S3, the specific steps are as follows:

[0027] S3.1: if the upper layer network of conv1 is not a convolution or does not meet the splitting condition, add two parallel convolutions conv5 and conv6 with a convolution kernel of 1*1, a step of 1, a padding of 0, an input data of the input of conv1, and output channel numbers of out_ch5 and out_ch6 before conv1, and out_ch5+out_ch6 = in_ch1, and conv5(in_ch1, out_ch5, height1, width1, 1, 1, 1, 0) and conv6(in_ch1, out_ch6, height1, width1, 1, 1, 1, 0);

[0028] S3.2: set the weights of conv5 and conv6 as w5, w6; the shape of w5 is [out_ch5, in_ch1, 1, 1], and the shape of w6 is [out_ch6, in_ch1, 1, 1], that is, conv5 has out_ch5 convolution kernels, each convolution kernel has in_ch1 layers with a size of 1*1, and conv6 has out_ch6 convolution kernels, each convolution kernel has in_ch1 layers with a size of 1*1; the first out_ch5 layers of each convolution kernel of w5 are 1 in turn, and the other layers are 0, and the last out_ch6 layers of each convolution kernel of w6 are 1 in turn, and the other layers are 0;

[0029] S3.3: set the bias of conv5 and conv6 as 0;

[0030] S3.4: insert w5 and w6 into conv5 and conv6;

[0031] S3.5: pass the input data through conv5 and conv6 respectively.

[0032] The preferred technical scheme of the application is as follows:

[0033] S4.1: keep other parameters unchanged, replace conv1 with two input channels in_ch7 and in_ch8 row convolution conv7 and conv8, and in_ch7 + in_ch8 = inc_ch1, and conv7 and conv8 are represented as conv7 (in_ch7, out_ch1, height1, width1, kh1, kw1, s1, p1) and conv8 (in_ch8, out_ch1, height1, width1, kh1, kw1, s1, p1), and the output of the previous step is passed through conv7 and conv8 respectively;

[0034] S4.2: set the weights of conv7 and conv8 as w7, w8; w7 and w8 are four-dimensional arrays with sizes of [out_ch1, in_ch7, kh1, kw1] and [out_ch1, in_ch8, kh1, kw1];

[0035] S4.3: insert w7 and w8 into conv7 and conv8 respectively, and insert the bias b1 of conv1 into conv7;

[0036] S4.4: pass the output of the two groups of upper layers through conv7 and conv8.

[0037] The application has the following beneficial effects:

[0038] The method can fully exert the AI core hardware acceleration capability of the embedded AI inference chip, replace super limit convolution with a large number of parallel convolution, solve the problem that a large convolution network cannot be deployed due to the hardware limitation of the chip, effectively improve the versatility and flexibility of the chip, and enable large neural networks such as YOLOv5 to be deployed on the chip. Meanwhile, the parallel convolution can maximize the use of the multi-AI core characteristics of the embedded AI inference chip, and improve the overall network performance. The application is suitable for AI acceleration chips of different platforms, and enhances the versatility and portability of the cross-platform convolution algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A flowchart of the splitting method of the application is shown in the figure.

[0040] Figure 2 A flowchart of the splitting method of the application is shown in the figure.

[0041] Figure 3 A flowchart of the splitting method of the application is shown in the figure.

[0042] Figure 4 A flowchart of the splitting method of the application is shown in the figure.

[0043] Figure 5 A flowchart of the splitting method of the application is shown in the figure.

[0044] Figure 6 A flowchart of the splitting method of the application is shown in the figure.

[0045] Figure 7 A flowchart of the splitting method of the application is shown in the figure.

[0046] Figure 8 A schematic diagram of w5 and w6 is shown in the figure.

[0047] Figure 9 A flowchart of the splitting method of the application is shown in the figure.

[0048] Figure 10 A flowchart of the splitting method of the application is shown in the figure. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below with reference to the drawings and specific embodiments.

[0050] The method of the embodiment can be used to realize the splitting and deployment of two-dimensional convolution as long as the chip supports convolution neural network algorithm. In the application, the chip can be HC3080 chip, but is not limited to HC3080 chip.

[0051] Please refer toFigure 1 The application is a two-dimensional convolution splitting method for an embedded AI inference chip, which comprises:

[0052] S0: Let the current two-dimensional convolution be marked as conv1, and the parameters of conv1 be: batch number batch, input channel number in_ch1, output channel number out_ch1, input height height1 and width width1, convolution kernel height kh1 and width kw1, step s1, padding p1, weight w1, and bias b1.

[0053] If the previous layer of the current convolution is a convolution operator and can be split, it is marked as conv0, and the parameters of conv0 are: batch number batch, input channel number in_ch0, output channel number out_ch0, input height height0 and width width0, convolution kernel height kh0 and width kw0, step s0, padding p0, weight w0, and bias b0, in_ch1 = out_ch0.

[0054] S1: Obtain the sram single bank capacity input_bank in the embedded AI inference chip, and determine whether the input of the current two-dimensional convolution conv1 is out of limit based on input_bank. If not, it is processed according to the normal convolution; otherwise, it enters S2 for convolution splitting. The condition for conv1 being out of limit is kh1* width1* in_ch1>input_bank.

[0055] S2: If the previous layer operator of the current convolution is also a convolution conv0 and meets the splitting condition, the previous layer convolution operator conv0 is split into two parallel convolutions conv3 and conv4 with output channels out_ch3 and out_ch4 according to the output channel number out_ch0, and out_ch3+out_ch4 = out_ch0. Conv3 and conv4 are represented as conv3(in_ch0, out_ch3, height0, width0, kh0, kw0, s0, p0) and conv4(in_ch0, out_ch4, height0, width0, kh0, kw0, s0, p0). The input data of conv0 will pass through conv3 and conv4 respectively, and then go to S4. If the splitting condition is not met or the previous layer is not a convolution operator, go to S3.

[0056] S3: add two parallel convolutions conv5 and conv6 with 1*1 kernel, stride 1, padding 0, input data conv1 input, and output channel number out_ch5 and out_ch6 before the current convolution conv1, and out_ch5+out_ch6 = in_ch1, conv5(in_ch1, out_ch5, height1, width1, 1, 1, 1, 0) and conv6(in_ch1, out_ch6, height1, width1, 1, 1, 1, 0);

[0057] S4: split the convolution conv1 into two input channels with in_ch1, and the two input channels are in_ch1 / 2, and the input channels of the two convolutions conv7 and conv8 are in_ch7 and in_ch8, and in_ch7+in_ch8 = inc_ch1, and conv7 and conv8 are represented as conv7(in_ch7, out_ch1, height1, width1, kh1, kw1, s1, p1) and conv8(in_ch8, out_ch1, height1, width1, kh1, kw1, s1, p1), and the output of the previous step is input into conv7 and conv8 respectively.

[0058] S5: perform matrix addition on the outputs of conv7 and conv8 to obtain the final result.

[0059] Specifically, the specific steps of S2 are as follows:

[0060] S2.1: determine whether the conv0 meets the splitting condition, and the determination condition is whether the result of the conv0 will be used in other network layers except the conv1;

[0061] S2.2: keep other parameters unchanged, and replace the conv0 with two parallel convolutions conv3 and conv4 with output channels out_ch3 and out_ch4, and out_ch3+out_ch4 = out_ch0, and conv3 and conv4 are represented as conv3(in_ch0, out_ch3, height0, width0, kh0, kw0, s0, p0) and conv4(in_ch0, out_ch4, height0, width0, kh0, kw0, s0, p0);

[0062] S2.3: split the weight w0 of conv0 into w3, w4; the data shape of w0 is four-dimensional array with size [out ch0, in ch0, kh0, kw0]; split w0 by the 0th dimension into w3 [out ch3, in ch0, kh0, kw0] and w4 [out ch4, in ch0, kh0, kw0];

[0063] S2.4: split the weight b0 of conv0 into b3, b4; the data shape of b0 is one-dimensional array with size [out ch0]; split b0 into b3 [out ch3] and b4 [out ch4];

[0064] S2.5: insert w3, b3 into conv3, and insert w4, b4 into conv4;

[0065] S2.6: the input data of conv0 will pass through conv3 and conv4 in parallel.

[0066] Specifically, S3 includes the following specific steps:

[0067] S3.1: if the upper layer network of conv1 is not convolution or does not meet the splitting condition, add two parallel convolutions conv5 and conv6 with 1*1 convolution kernel, step size 1, padding 0, input data conv1 input, output channel number out ch5 and out ch6, and out ch5 + out ch6 = in ch1, before conv1, and conv5 (in ch1, out ch5, height1, width1, 1, 1, 1, 0) and conv6 (in ch1, out ch6, height1, width1, 1, 1, 1, 0);

[0068] S3.2: set the weights of conv5 and conv6 as w5, w6; the shape of w5 is [out ch5, in ch1, 1, 1], and the shape of w6 is [out ch6, in ch1, 1, 1], that is, conv5 has out ch5 convolution kernels, each of which has in ch1 layer of 1*1 size, and conv6 has out ch6 convolution kernels, each of which has in ch1 layer of 1*1 size; the first out ch5 layers of each convolution kernel of w5 are 1 in turn, and the other layers are 0, and the last out ch6 layers of each convolution kernel of w6 are 1 in turn, and the other layers are 0;

[0069] S3.3: set the bias of conv5 and conv6 as 0;

[0070] S3.4: insert w5, w6 into conv5, conv6;

[0071] S3.5: input data is respectively passed through conv5 and conv6.

[0072] Specifically, the specific steps of S4 are as follows:

[0073] S4.1: keep other parameters unchanged, replace conv1 with two input channel row convolutions conv7 and conv8, and in_ch7 + in_ch8 = inc_ch1, conv7 and conv8 are represented as conv7 (in_ch7, out_ch1, height1, width1, kh1, kw1, s1, p1) and conv8 (in_ch8, out_ch1, height1, width1, kh1, kw1, s1, p1), and the output of the previous step is respectively passed through conv7 and conv8.

[0074] S4.2: set the weights of conv7 and conv8 as w7, w8; w7 and w8 are four-dimensional arrays with sizes [out_ch1, in_ch7, kh1, kw1] and [out_ch1, in_ch8, kh1, kw1].

[0075] S4.3: insert w7 and w8 into conv7 and conv8 respectively, and insert the bias b1 of conv1 into conv7.

[0076] S4.4: pass the output of the two groups of the upper layer through conv7 and conv8.

[0077] Two embodiments are given below to specifically illustrate the method of the application. Embodiment 1

[0078] Taking the deployment of YOLOv5 6.0s on HC3080 chip as an example, the calculation method of the application is further illustrated:

[0079] Taking the second layer convolution of YOLOv5 6.0s as conv1 and the first layer convolution as conv0, as shown in Figure 2 The specific parameters obtained after AI fine-tuning of HC3080 are as follows:

[0080] conv0: batch = 1, in_ch0 = 3, out_ch0 = 32, height0 = 420, width0 = 420, kh0 = 6, kw0 = 6, s = 2, p = 2

[0081] conv1: batch = 1, in_ch1 = 32, out_ch1 = 64, height1 = 210, width1 = 210, kh1 = 3, kw1 = 3, s1 = 2, p1 = 1

[0082] First, it is judged whether the current two-dimensional convolution conv1 input is out of limit, kh1 * width1 * in_ch1 = 210 * 3 * 32 = 20160, which is greater than the single bank capacity of the chip sram 16 * 1024. The convolution exceeds the chip limit and needs to be split.

[0083] The last layer convolution conv0 meets the splitting condition, as shown in Figure 3 conv0 is split into two output channel parallel convolutions conv3 and conv4 with 16 output channels each according to the output channel number 32, and conv3 and conv4 are represented as conv3(3, 16, 420, 420, 6, 6, 2, 2) and conv4(3, 16, 420, 420, 6, 6, 2, 2). The data format of the weight w0 of conv0 is [32, 3, 6, 6], and w0 is split into w3[0:16, 3, 6, 6] and w4[16:32, 3, 6, 6] from the 0th dimension. The shape of w3 and w4 is [16, 3, 6, 6]. The bias b0 of conv0 is split into b3[0:16] and b4[16:32], and the shape of b3 and b4 is

[16] . The input data [1, 3, 420, 420] of conv0 is passed through conv3 and conv4 respectively.

[0084] As shown in Figure 4 convolution conv1 is split into two input channel parallel convolutions conv7(16, 64, 210, 210, 3, 3, 2, 1) and conv8(16, 64, 210, 210, 3, 3, 2, 1) with 16 input channels each according to the input channel number 32. The weight w1 of conv1 has a shape of [64, 32, 3, 3], which is split into w7[64, 0:16, 3, 3] and w8[64, 16:32, 3, 3] according to the second dimension, and the size is [64, 16, 3, 3]. w7 and w8 are inserted into conv7 and conv8 respectively, and the bias b1 of conv1 is inserted into conv7.

[0085] The two groups of outputs of the upper layer are passed through conv7 and conv8, and the two outputs are added in matrix, as shown in Figure 5 the final result is obtained. Example 2

[0086] Taking deployment of YOLOv5 6.0s on HC3080 as an example, the second case that the conv0 of the present application cannot be split is illustrated:

[0087] Taking the fifth part of convolution of YOLOv5 6.0s as conv1 and the last convolution of the fourth part as conv0, as Figure 6 , the specific parameters obtained after AI fine-tuning of HC3080 are as follows:

[0088] conv0: batch = 1, in_ch0 = 128, out_ch0 = 128, height0 = 52, width0 = 52, kh0 = 1, kw0 = 1, s = 1, p = 0

[0089] conv1: batch = 1, in_ch1 = 128, out_ch1 = 256, height1 = 52, width1 = 52, kh1 = 3, kw1 = 3, s1 = 2, p1 = 1

[0090] Note: The padding mechanism of HC3080 is to fill 0 after the convolution operation is completed.

[0091] First, it is judged whether the input of the current two-dimensional convolution conv1 is out of limit, kh1* width1* in_ch1 = 3*54*128 = 20736, which is greater than the single bank capacity of the chip sram 16*1024, so the convolution exceeds the chip limit and needs to be split.

[0092] The output of the previous layer convolution conv0 will be input to the 16th part of the whole network, so it cannot be split, and the process will enter S3.

[0093] Two parallel convolutions conv5 (128, 64, 52, 52, 1, 1, 1, 0) and conv6 (128, 64, 52, 52, 1, 1, 1, 0) are added before conv1, as Figure 7 , the convolution kernel is 1*1, the step is 1, the padding is 0, the input data is the input of conv1, and the output channel number out_ch5 and out_ch6 are both 64. The weights of conv5 and conv6 are set as w5 and w6, and the shape of w5 and w6 is [64, 128, 1, 1]. w5 and w6 both have 64 convolution kernels, and each convolution kernel has 128 layers of 1*1 size. As Figure 8, the first 64 layers of each convolution kernel of w5 are 1 in turn, and the other layers are 0, and the last 64 layers of each convolution kernel of w6 are 1 in turn, and the other layers are 0. At the same time, the bias of conv5 and conv6 is set to 0, and w5 and w6 are inserted into conv5 and conv6, and finally the output of conv0 is respectively passed through conv5 and conv6.

[0094] As Figure 9 , the convolution conv1 is split into two parallel convolutions conv7 (64, 256, 52, 52, 3, 3, 2, 1) and conv8 (64, 256, 52, 52, 3, 3, 2, 1) with input channel number 128 and input channel number 64 respectively; the weight w1 of conv1 has a shape of [256, 128, 3, 3], which is split into w7 [256, 0:64, 3, 3] and w8 [256, 64:128, 3, 3] according to the second dimension, and the size of both is [256, 64, 3, 3]. w7 and w8 are inserted into conv7 and conv8 respectively, and the bias b1 of conv1 is inserted into conv7. The outputs of the two groups of upper layers are passed through conv7 and conv8, and the two outputs are matrix added, as shown in the following formula: Figure 10 , the final result is obtained.

[0095] The part not involved in the present application is the same as the prior art or realized by using the prior art.

[0096] The above is a further detailed description of the present application in combination with specific embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, some simple deductions or substitutions can be made without departing from the concept of the present application, and all of them should be regarded as falling within the protection scope of the present application.

Claims

1. A two-dimensional convolution splitting method for embedded AI inference chips, characterized in that: Comprising S1: obtain the sram single bank capacity input_bank in the embedded AI inference chip, judge whether the current two-dimensional convolution conv1 input is out of limit based on the input_bank, if not, carry out normal convolution processing, otherwise, enter S2 for convolution splitting; S2: if the last layer operator of the current convolution is a convolution operator and meets the splitting condition, then the last layer convolution operator conv0 is split into two output channels parallel convolutions conv3 and conv4 with output channel numbers out_ch3 and out_ch4 respectively according to the output channel number out_ch0, and out_ch3+out_ch4=out_ch0, the input of conv0 will pass through conv3 and conv4 respectively, and then go to S4; if the splitting condition is not met or the last layer is not a convolution operator, go to S3; S3: add two convolution kernels with 1*1, step length 1, padding 0 and half of the input channel number to the current convolution conv1 before conv1, and then go to S4; S4: split the current convolution conv1 into two input channel parallel convolutions conv7 and conv8 with input channel number in_ch1 / 2, and pass the output of the last step through conv7 and conv8 respectively; S5: add the outputs of conv7 and conv8 to obtain the final result.

2. The method of claim 1, wherein: Before step S1, including preset step S0: Suppose: the current two-dimensional convolution is marked as conv1, and the parameters of conv1 are: batch number batch, input channel number in_ch1, output channel number out_ch1, input height height1 and width width1, convolution kernel height kh1 and width kw1, step length s1, padding p1, weight w1 and bias b1; If the last layer of the current convolution is a convolution operator and can be split, it is marked as conv0, and the parameters of conv0 are: batch number batch, input channel number in_ch0, output channel number out_ch0, input height height0 and width width0, convolution kernel height kh0 and width kw0, step length s0, padding p0, weight w0 and bias b0, in_ch1=out_ch0.

3. The method of claim 2, wherein: The splitting method specifically comprises: S1: judge whether the current two-dimensional convolution conv1 input is out of limit, if not, carry out normal convolution processing, otherwise, enter S2 for convolution splitting, the condition for conv1 out of limit is kh1* width1* in_ch1>input_bank; S2: If the current convolutional upper layer operator is also a convolutional conv0 and meets the splitting condition, then conv0 is split into two parallel convolutions conv3 and conv4 with output channels out_ch3 and out_ch4 respectively according to the output channel number out_ch0, and out_ch3+out_ch4 = out_ch0, the parameters of conv3 are: input channel number in_ch0, output channel number out_ch3, input height height0, input width width0, kernel height kh0, kernel width kw0, stride s0, and padding number p0; the parameters of conv4 are: input channel number in_ch0, output channel number out_ch4, input height height0, input width width0, kernel height kh0, kernel width kw0, stride s0, and padding number p0, the input data of conv0 will pass through conv3 and conv4 respectively, and then go to S4, if the splitting condition is not met or the upper layer is not a convolutional operator, go to S3; S3: Before conv1, add two parallel convolutions conv5 and conv6 with 1*1 kernel, stride 1, padding 0, input data conv1 input, and output channel numbers out_ch5 and out_ch6, and out_ch5+out_ch6 = in_ch1, the parameters of conv5 are: input channel number in_ch1, output channel number out_ch5, input height height1, input width width1, kernel height 1, kernel width 1, stride 1, and padding number 0; the parameters of conv6 are: input channel number in_ch1, output channel number out_ch6, input height height1, input width width1, kernel height 1, kernel width 1, stride 1, and padding number 0; S4: split the convolution conv1 into two input channels in_ch7 and in_ch8 row convolution conv7, conv8, and in_ch7+in_ch8=in_ch1, the parameters of conv7 are: input channel number in_ch7, output channel number out_ch1, input height height1, input width width1, kernel height kh1, kernel width kw1, stride s1, padding number p1; the parameters of conv8 are: input channel number in_ch8, output channel number out_ch1, input height height1, input width width1, kernel height kh1, kernel width kw1, stride s1, padding number p1; the output of the previous step is respectively through conv7 and conv8, S5: the outputs of conv7 and conv8 are added in matrix to obtain the final result.

4. The method of claim 3, wherein: The specific steps of S2 are as follows: S2.1: judge whether the conv0 meets the splitting condition, the judgment condition is whether the result of the conv0 will be used in other network layers except the conv1; S2.2: keep other parameters unchanged, replace the conv0 with two parallel convolutions conv3 and conv4 with output channels out_ch3 and out_ch4 respectively, and out_ch3+out_ch4=out_ch0, the parameters of conv3 are: input channel number in_ch0, output channel number out_ch3, input height height0, input width width0, kernel height kh0, kernel width kw0, stride s0, padding number p0; the parameters of conv4 are: input channel number in_ch0, output channel number out_ch4, input height height0, input width width0, kernel height kh0, kernel width kw0, stride s0, padding number p0); S2.3: split the weight w0 of the conv0 into w3 and w4; The data shape of w0 is a four-dimensional array with the size of: output channel number out_ch0, input channel number in_ch0, kernel height kh0, kernel width kw0; w0 is divided into w3 and w4 according to the 0th dimension, the size of w3 is: output channel number out_ch3, input channel number in_ch0, kernel height kh0, kernel width kw0; the size of w4 is: output channel number out_ch4, input channel number in_ch0, kernel height kh0, kernel width kw0; S2.4: split the bias b0 of the conv0 into b3 and b4; The data shape of b0 is one-dimensional array, and the size is: the output channel number out_ch0, b0 is divided into b3 and b4, the size of b3 is: the output channel number out_ch3, and the size of b4 is: the output channel number out_ch4; S2.5: insert w3 and b3 into conv3, and the parameters of conv3 are: the input channel number is in_ch0, the output channel number is out_ch3, the input height is height0, the input width is width0, the kernel height is kh0, the kernel width is kw0, the step is s0, and the padding number is p0; and insert w4 and b4 into conv4, and the parameters of conv4 are: the input channel number is in_ch0, the output channel number is out_ch4, the input height is height0, the input width is width0, the kernel height is kh0, the kernel width is kw0, the step is s0, and the padding number is p0; S2.6: the input data of conv0 will pass through conv3 and conv4 in parallel.

5. The method of claim 3, wherein: The specific steps are as follows: S3.1: if the upper layer network of conv1 is not convolution or does not meet the splitting condition, add two parallel convolutions conv5 and conv6 with the kernel size of 1*1, the step of 1, the padding of 0, the input data of conv1, and the output channel number of out_ch5 and out_ch6 before conv1, and out_ch5+out_ch6=in_ch1, the parameters of conv5 are: the input channel number is in_ch1, the output channel number is out_ch5, the input height is height1, the input width is width1, the kernel height is 1, the kernel width is 1, the step is 1, and the padding number is 0; and the parameters of conv6 are: the input channel number is in_ch1, the output channel number is out_ch6, the input height is height1, the input width is width1, the kernel height is 1, the kernel width is 1, the step is 1, and the padding number is 0; S3.2: set the weights of conv5 and conv6 as w5 and w6; The shape of w5 is: the output channel number is out_ch5, the input channel number is in_ch1, the kernel height is 1, and the kernel width is 1; and the shape of w6 is: the output channel number is out_ch6, the input channel number is in_ch1, the kernel height is 1, and the kernel width is 1; that is, conv5 has out_ch5 convolution kernels, each convolution kernel has in_ch1 layer 1*1 size, and conv6 has out_ch6 convolution kernels, each convolution kernel has in_ch1 layer 1*1 size; the first out_ch5 layers of each convolution kernel of w5 are 1 in turn, and the other layers are 0, and the last out_ch6 layers of each convolution kernel of w6 are 1 in turn, and the other layers are 0; S3.3: set the bias of conv5 and conv6 as 0; S3.4: insert w5, w6 into conv5, conv6, the parameters of conv5 are: the number of input channels is in ch1, the number of output channels is out ch5, the input height is height1, the input width is width1, the kernel height is 1, the kernel width is 1, the stride is 1, the padding number is 0; the parameters of conv6 are: the number of input channels is in ch1, the number of output channels is out ch6, the input height is height1, the input width is width1, the kernel height is 1, the kernel width is 1, the stride is 1, the padding number is 0; S3.5: input data is respectively passed through conv5 and conv6.

6. The method of claim 3, wherein: The specific steps of S4 are as follows: S4.1: keep other parameters unchanged, replace conv1 with two input channels in ch7 and in ch8 row convolution conv7, conv8, and in ch7 + in ch8 = inc ch1, the parameters of conv7 are: the number of input channels is in ch7, the number of output channels is out ch1, the input height is height1, the input width is width1, the kernel height is kh1, the kernel width is kw1, the stride is s1, the padding number is p1; the parameters of conv8 are: the number of input channels is in ch8, the number of output channels is out ch1, the input height is height1, the input width is width1, the kernel height is kh1, the kernel width is kw1, the stride is s1, the padding number is p1; the output of the last step is respectively passed through conv7 and conv8; S4.2: set the weights of conv7 and conv8 as w7, w8; w7 and w8 are four-dimensional arrays, the size of w7 is: the number of output channels is out ch1, the number of input channels is in ch7, the kernel height is kh1, the kernel width is kw1, the size of w8 is: the number of output channels is out ch1, the number of input channels is in ch8, the kernel height is kh1, the kernel width is kw1; S4.3: insert w7 and w8 into conv7 and conv8 respectively, and insert the bias b1 of conv1 into conv7; S4.4: pass the output of the two groups of the upper layer through conv7 and conv8.

Images