An optimization method for convolution and pooling combination
By merging convolution and pooling into a single process, and using a 2x2 max pooling kernel with a stride of 2, the quantization process is optimized, solving the time waste problem caused by the limited number of registers and improving processing efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-12
- Publication Date
- 2026-03-31
AI Technical Summary
With a limited number of registers, the results of convolution calculations in existing technologies need to be loaded and saved twice, resulting in wasted time.
By combining convolution and pooling into a single process, using a 2x2 max pooling kernel with a stride of 2, pooling is performed first, followed by quantization, thus optimizing the quantization process.
This reduces the number of quantization steps, saves computation time, and improves processing efficiency.
Smart Images
Figure CN115905783B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to an optimized method for merging convolution and pooling. Background Technology
[0002] Integrated circuit technology is increasingly becoming a focal point of technological development, with chip manufacturers developing their own chips. However, in chip applications, different application scenarios lead to specific requirements and challenges in chip design. For example, the Beijing Junzheng Integrated Circuit Co., Ltd. (hereinafter referred to as Beijing Junzheng) produces chips, specifically the T30 and T31 models, which have 128-bit registers with a limited number of 32 registers. Using existing technology, exceeding 32 registers would cause previously loaded register data to be stored in memory, requiring reloading for subsequent processing, resulting in low efficiency. Therefore, optimizing the design necessitates considering the number of registers.
[0003] Therefore, the shortcomings of existing technology are:
[0004] With a limited number of registers, after convolutional computation on a feature map, the result is saved, a new feature map is generated, and this new feature map is used as input for pooling. The result is then saved as a completely new feature map. This process involves loading and saving data twice, resulting in wasted time.
[0005] In addition, the following are commonly used terms in the prior art:
[0006] 1. SIMD instructions: Single Instruction Stream, Multiple Data Stream, meaning a single instruction can execute multiple data streams, thus improving program processing speed. In simpler terms, it's a vector-based computation. Different chips have different instruction sets.
[0007] 2. Feature Map: The result obtained after input data is processed by convolution is called a feature map (or output data). The result generated after data is processed by a fully connected layer is also called a feature map (or output data). The size of a feature map is generally expressed as length × width × depth, or 1 × depth. Depth is also called a channel.
[0008] 3. Convolution: The center of the convolution kernel is placed on the pixel to be calculated. The product of each element in the kernel and the image pixel value it covers is calculated one by one and then summed. The resulting structure is the new pixel value at that position. This process is called convolution.
[0009] 4. Pooling: After obtaining features through convolutional layers, the next step is to integrate and classify these features. Common pooling methods include max pooling and average pooling. Summary of the Invention
[0010] In order to solve the problems in the prior art, the purpose of this application is to optimize the two data loading and two data saving operations into one data loading and one data processing operation, and to optimize some intermediate algorithms to reduce the overall running time.
[0011] Specifically, the present invention provides an optimized method for merging convolution and pooling, the method comprising:
[0012] S1, Analysis of Quantization and Pooling Functions:
[0013] S1.1, let the initial result of the convolution calculation be a, the quantization multiplication coefficient be b, the first shift coefficient be shift_left, the second shift coefficient be shift_right, and the final result be res, as shown in the following formula:
[0014] (1)
[0015] (2)
[0016] (3)
[0017] (4)
[0018] S1.2, analyzing the above formulas, with the independent variable being 'a' and the other parameters 'shift_left' and 'shift_right' being given constants, we have formulas (1), (2), and (3) that are all increasing functions. It is also an increasing function, so formula (4) is an increasing function. Then, res obtained from formula (1) to (4) is an increasing function about a. That is, quantization is a process of processing an increasing function.
[0019] S2, Optimized processing method:
[0020] First, pooling is performed on the initial results of the convolution calculation, and then quantization is used to quantize the pooling results.
[0021] The pooling process in the method involves comparing several data points from the same channel and selecting the largest one.
[0022] The method uses max pooling with a 2x2 pooling kernel and a step size of 2.
[0023] Step S2 further includes:
[0024] S2.1, In the optimization process, the pooling process uses a 2x2 core with a step size of 2;
[0025] Let the quantization module be vrd = quantization(vrs, vrt), where vrs and vrt are the initial result registers of the convolution calculation. The data in vrs and vrt are the results before quantization and only the cumulative sum of the convolution calculation. Each of vrs and vrt stores 8 16-bit data. vrd is the generated result, which stores 16 8-bit data.
[0026] S2.2, Based on the width and height of the feature map, pooling and quantization are performed to integrate convolution and pooling operations:
[0027] The initial results of the convolution calculation are stored in sum_0, sum_1, sum_2, sum_3, sum_4, sum_5, sum_6, and sum_7. These parameters are all stored in eight 16-bit registers.
[0028] The feature map depth is a multiple of 16, and each processing step is performed in multiples of 16. sum_0 and sum_1, sum_2 and sum_3, sum_4 and sum_5, sum_6 and sum_7 represent the 16 data points being processed.
[0029] The initial result of the convolution calculation is considered as a feature map. Only with two strides in width and height will a pooled result with a width and a height be generated.
[0030] Step S2.2 further includes:
[0031] a) The portion of the feature map whose width and height are divisible by 2:
[0032] Let the feature map of the preliminary result of the convolution calculation be...
[0033] If we take values in width wi and height hi, then we have sum_0 and sum_1;
[0034] If we take values at width wi+1 and height hi, then we have sum_2 and sum_3;
[0035] If the width wi is taken as the value of the height hi+1, then sum_4 and sum_5 are obtained.
[0036] If we take values at width wi+1 and height hi+1, then we have sum_6 and sum_7;
[0037] Pooling process:
[0038] Compare sum_0 and sum_2, select the larger data and store it in sum_0;
[0039] Compare sum_0 and sum_4, select the larger data and store it in sum_0;
[0040] Compare sum_0 and sum_6, select the larger data and store it in sum_0;
[0041] Compare sum_1 and sum_3, select the larger data and store it in sum_1;
[0042] Compare sum_1 and sum_5, select the larger data and store it in sum_1;
[0043] Compare sum_1 and sum_7, select the larger data and store it in sum_1;
[0044] Quantification:
[0045] Using the quantization function module, sum_0 and sum_1 are quantized, and the result is stored in sum_0, as follows:
[0046] sum_0= quantization(sum_0, sum_1),
[0047] The inputs sum_0 and sum_1 are the initial result registers for the convolution calculation. The data in sum_0 and sum_1 are the results before quantization and are only the cumulative sum of the convolution calculation. Each of sum_0 and sum_1 stores 8 16-bit data. The output sum_0 is the generated result, which stores 16 8-bit data.
[0048] b) The remainder after dividing the width of the feature map by 2, and the remainder after dividing the height by 2:
[0049] At this point, the feature map of the preliminary results of the convolution calculation is...
[0050] If we take values in width wi and height hi, then we have sum_0 and sum_1;
[0051] If the width wi is taken as the value of the height hi+1, then sum_4 and sum_5 are obtained.
[0052] Compare sum_0 and sum_4, select the larger data and store it in sum_0;
[0053] Compare sum_1 and sum_5, select the larger data and store it in sum_1;
[0054] Quantification:
[0055] Using the quantization function module, sum_0 and sum_1 are quantized, and the result is stored in sum_0, as follows:
[0056] sum_0= quantization(sum_0, sum_1);
[0057] c) The width of the feature map is divisible by 2, and the height is the remainder when divided by 2:
[0058] Feature map of preliminary results of convolution calculation
[0059] If we take values in width wi and height hi, then we have sum_0 and sum_1;
[0060] If we take values at width wi+1 and height hi, then we have sum_2 and sum_3;
[0061] Pooling process:
[0062] Compare sum_0 and sum_2, select the larger data and store it in sum_0;
[0063] Compare sum_1 and sum_3, select the larger data and store it in sum_1;
[0064] Quantification:
[0065] Using the quantization function module, sum_0 and sum_1 are quantized, and the result is stored in sum_0, as follows:
[0066] sum_0= quantization(sum_0, sum_1);
[0067] d) The width of the feature map is the remainder when divided by 2, and the height is the remainder when divided by 2;
[0068] Feature map of preliminary results of convolution calculation
[0069] The width wi and height hi take only two values: sum_0 and sum_1.
[0070] Width wi+1 and height hi, width wi and height hi+1, and width wi+1 and height hi+1 have all overflowed outside the feature map and do not exist.
[0071] Using the quantization function module, sum_0 and sum_1 are quantized, and the result is stored in sum_0, as follows:
[0072] sum_0= quantization(sum_0, sum_1);
[0073] The convolution and pooling operations are integrated through a), b), c), and d).
[0074] Therefore, the advantage of this application lies in providing an optimization method that reduces the quantization process from multiple steps to a single step, saving considerable computation time. The quantization computation is substantial throughout the convolution process, so this approach significantly reduces time and greatly improves efficiency. Attached Figure Description
[0075] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.
[0076] Figure 1 This is a flowchart of the method of the present invention.
[0077] Figure 2 This is a schematic diagram of step 2 in the method of the present invention. Detailed Implementation
[0078] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.
[0079] This optimization method is based on the SIMD instruction set of T and X series chips such as the Beijing Junzheng T30 and T31 models. This algorithm is suitable for vector instruction operations.
[0080] like Figure 1 As shown, the method of the present invention relates to an optimization method for merging convolution and pooling, the method comprising:
[0081] S1, Analysis of Quantization and Pooling Functions:
[0082] S1.1, let the initial result of the convolution calculation be a, the quantization multiplication coefficient be b, the first shift coefficient be shift_left, the second shift coefficient be shift_right, and the final result be res, as shown in the following formula:
[0083] (1)
[0084] (2)
[0085] (3)
[0086] (4)
[0087] S1.2, analyzing the above formulas, with the independent variable being 'a' and the other parameters 'shift_left' and 'shift_right' being given constants, we have formulas (1), (2), and (3) that are all increasing functions. It is also an increasing function, so formula (4) is an increasing function. Then, res obtained from formula (1) to (4) is an increasing function about a. That is, quantization is a process of processing an increasing function.
[0088] S2, Optimized processing method:
[0089] First, pooling is performed on the initial results of the convolution calculation, and then quantization is used to quantize the pooling results.
[0090] The pooling process in the method involves comparing several data points from the same channel and selecting the largest one.
[0091] The method uses max pooling with a 2x2 pooling kernel and a step size of 2.
[0092] like Figure 2 As shown, step S2 further includes:
[0093] S2.1, In the optimization process, the pooling process uses a 2x2 core with a step size of 2;
[0094] Let the quantization module be vrd = quantization(vrs, vrt), where vrs and vrt are the initial result registers of the convolution calculation. The data in vrs and vrt are the results before quantization and only the cumulative sum of the convolution calculation. Each of vrs and vrt stores 8 16-bit data. vrd is the generated result, which stores 16 8-bit data.
[0095] S2.2, Based on the width and height of the feature map, pooling and quantization are performed to integrate convolution and pooling operations:
[0096] Here, the initial results of the convolution calculation are sum_0, sum_1, sum_2, sum_3, sum_4, sum_5, sum_6, and sum_7. These parameters are all stored in eight 16-bit registers.
[0097] The feature map depth is a multiple of 16, and each processing step is performed in multiples of 16. sum_0 and sum_1, sum_2 and sum_3, sum_4 and sum_5, sum_6 and sum_7 represent the 16 data points being processed.
[0098] The initial result of the convolution calculation is considered as a feature map. Only with two strides in width and height will a pooled result with one width and one height be generated.
[0099] And it is divided into the following situations:
[0100] a) The portion of the feature map whose width and height are divisible by 2;
[0101] b) The remainder after dividing the width of the feature map by 2, and the height is the remainder after dividing by 2;
[0102] c) The width of the feature map is divisible by 2, and the height is the remainder when divided by 2;
[0103] d) The width of the feature map is the remainder when divided by 2, and the height is the remainder when divided by 2.
[0104] Specifically, for ease of understanding, the specific implementation of the method of this application can also be described as follows:
[0105] simd instruction algorithm.
[0106] 1) Introduction to SIMD commands: The SIMD commands involved are as follows:
[0107] A) Maximum value instruction: vd = ingenic_max_h(vs, vt);
[0108] The input variables are vs and vt, and the output variable is vd. vd stores 8 int16_t data, while vs and vt store 8 int16_t data.
[0109] Equivalent operations: vd0 := (vs0>vt0)?vs0:vt0; vd1 := (vs1>vt1)?vs1:vt1; ... vd7 := (vs7>vt7)?vs7:vt7;
[0110] 1. Analysis of Quantization and Pooling Functions:
[0111] Let the initial result of the convolution calculation be 'a', the quantization multiplication coefficient be 'b', the first shift coefficient be 'shift_left', the second shift coefficient be 'shift_right', and the final result be 'res'. The calculation formula is as follows.
[0112] ………………………………(1)
[0113] ……………… (2)
[0114] ……………….. (3)
[0115] …………………………(4)
[0116] Analyzing the formulas above, with the independent variable 'a' and the other parameters 'shift_left' and 'shift_right' being given constants, we have formulas (1), (2), and (3) that are all increasing functions. It is also an increasing function, so formula (4) is an increasing function. Therefore, res obtained from formulas (1) to (4) is an increasing function with respect to a.
[0117] Pooling compares several data points from the same channel and selects the largest one. Since the quantization part uses an increasing function, pooling also selects the maximum value. Utilizing two key pieces of information, we can perform pooling on the initial convolution calculation result first, and then quantize the pooling result again, keeping the final result unchanged. This reduces the amount of quantization work that would normally be done multiple times to just one, saving considerable computation time. Furthermore, the quantization computation in the entire convolution process is substantial, so this approach significantly reduces time consumption.
[0118] Specific examples are provided:
[0119] There are four convolutional datasets: x_1, x_2, x_3, and x_4 (let's assume x_1>=x_2>=x_3>=x_4). These are calculated using four formulas and then four functions, resulting in y4_1, y4_2, y4_3, and y4_3. Since the first four functions are increasing functions, y4_1>=y4_2>=y4_3>=y4_3. Pooling selects the maximum value from y4_1, y4_2, y4_3, choosing y4_1. Because the first four functions are increasing functions, the order of y4_1, y4_2, y4_3 is exactly the same as the order of x_1, x_2, x_3, x_4. Therefore, we only need to select the largest value from x_1, x_2, x_3, and x_4, which is x_1, and then calculate it using the four function formulas. The other three, x_2, x_3, and x_4, do not need to be calculated using the four function formulas; this is called quantization. If pooling is performed after quantization, x_1, x_2, x_3, and x_4 all need to be quantized before pooling. However, since we can compare and select the largest value among x_1, x_2, x_3, and x_4 in advance, we only need to calculate the quantization of x_1, so only one quantization is performed. The other three, x_2, x_3, and x_4, do not need to be quantized.
[0120] This uses max pooling, with a 2x2 pooling kernel and a stride of 2, which is the most commonly used pooling method.
[0121] 2. Optimize processing methods
[0122] The current optimization uses a 2x2 core with a stride of 2 for pooling. The quantization module is defined as `vrd = quantization(vrs, vrt)`, where `vrs` and `vrt` are the initial convolution result registers, representing the accumulated sum of convolution calculations before quantization. Each of `vrs` and `vrt` stores eight 16-bit data segments. `vrd` stores the generated result, containing sixteen 8-bit data segments. Even with the simplest quantization process, the quantization part requires at least 12 computation instructions, including two multiplication instructions, and involves loading model data, resulting in significant time consumption.
[0123] a) The portion of the feature map whose width and height are divisible by 2.
[0124] Let the initial convolution calculation results be sum_0, sum_1, sum_2, sum_3, sum_4, sum_5, sum_6, and sum_7. These parameters are stored in 8 16-bit registers. The feature map depth is a multiple of 16, and each processing step is a multiple of 16. sum_0 and sum_1, sum_2 and sum_3, sum_4 and sum_5, sum_6 and sum_7 represent the 16 data points processed. The initial convolution calculation result is considered as a single feature map. Two strides are needed to generate a pooled result with a width and height. Let the feature map of the initial convolution calculation result have values at width wi and height hi, then sum_0 and sum_1; values at width wi+1 and height hi, then sum_2 and sum_3; values at width wi and height hi+1, then sum_4 and sum_5; and values at width wi+1 and height hi+1, then sum_6 and sum_7.
[0125] Compare sum_0 and sum_2, select the larger data point and store it in sum_0, represented as:
[0126] sum_0 = ingenic_max_h (sum_0, sum_2) ,
[0127] The input variables sum_0 and sum_2 are used, and the output variable is sum_0, which stores 8 int16_t data. The input variables sum_0 and sum_2 store 8 int16_t data.
[0128] Compare sum_0 and sum_4, select the larger data point, and store it in sum_0, represented as:
[0129] sum_0 = ingenic_max_h (sum_0, sum_4),
[0130] The input variables sum_0 and sum_4 are used, and the output variable is sum_0, which stores 8 int16_t data. The input variables sum_0 and sum_4 store 8 int16_t data.
[0131] Compare sum_0 and sum_6, select the larger data point, and store it in sum_0, represented as:
[0132] sum_0 = ingenic_max_h (sum_0, sum_6),
[0133] The input variables sum_0 and sum_6 are used, and the output variable is sum_0, which stores 8 int16_t data. The input variables sum_0 and sum_6 store 8 int16_t data.
[0134] Compare sum_1 and sum_3, select the larger data point, and store it in sum_1, represented as:
[0135] sum_1 = ingenic_max_h (sum_1, sum_3),
[0136] The input variables sum_1 and sum_3 are used, and the output variable is sum_1, which stores 8 int16_t data. The input variables sum_1 and sum_3 store 8 int16_t data.
[0137] Compare sum_1 and sum_5, select the larger data point, and store it in sum_1, represented as:
[0138] sum_1= ingenic_max_h (sum_1, sum_5),
[0139] The input variables sum_1 and sum_5 are used, and the output variable is sum_1, which stores 8 int16_t data. The input variables sum_1 and sum_5 store 8 int16_t data.
[0140] Compare sum_1 and sum_7, select the larger data point, and store it in sum_1, represented as:
[0141] sum_1= ingenic_max_h (sum_1, sum_7),
[0142] The input variables are sum_1 and sum_7, and the output variable is sum_1, which stores 8 int16_t data points. The input variables sum_1 and sum_7 store 8 int16_t data points. The quantization function module is used to quantize sum_0 and sum_1, and the result is stored in sum_0, as shown below:
[0143] sum_0= quantization(sum_0, sum_1),
[0144] The inputs sum_0 and sum_1 are the initial result registers for the convolution calculation. The data in sum_0 and sum_1 are the results before quantization and are only the cumulative sum of the convolution calculation. Each of sum_0 and sum_1 stores 8 16-bit data. The output sum_0 is the generated result, which stores 16 8-bit data.
[0145] b) The remainder after dividing the width of the feature map by 2, and the height is the remainder after dividing by 2. In this case, taking values at the width wi and height hi of the feature map in the preliminary convolution calculation, we have sum_0 and sum_1; taking values at the width wi and height hi+1, we have sum_4 and sum_5.
[0146] Compare sum_0 and sum_4, select the larger data point, and store it in sum_0, represented as:
[0147] sum_0 = ingenic_max_h (sum_0, sum_4),
[0148] The input variables sum_0 and sum_4 are used, and the output variable is sum_0, which stores 8 int16_t data. The input variables sum_0 and sum_4 store 8 int16_t data.
[0149] Compare sum_1 and sum_5, select the larger data point, and store it in sum_1, represented as:
[0150] sum_1= ingenic_max_h (sum_1, sum_5),
[0151] The input variables sum_1 and sum_5 are used, and the output variable is sum_1, which stores 8 int16_t data. The input variables sum_1 and sum_5 store 8 int16_t data.
[0152] Using the quantization function module, sum_0 and sum_1 are quantized, and the result is stored in sum_0, as follows:
[0153] sum_0= quantization(sum_0, sum_1),
[0154] The inputs sum_0 and sum_1 are the initial result registers for the convolution calculation. The data in sum_0 and sum_1 are the results before quantization and are only the cumulative sum of the convolution calculation. Each of sum_0 and sum_1 stores 8 16-bit data. The output sum_0 is the generated result, which stores 16 8-bit data.
[0155] c) The width of the feature map is divisible by 2, and the height is the remainder when divided by 2. The initial convolution calculation results in the feature map taking values at width wi and height hi, with sum_0 and sum_1; and values at width wi+1 and height hi, with sum_2 and sum_3.
[0156] Compare sum_0 and sum_2, select the larger data point and store it in sum_0, represented as:
[0157] sum_0 = ingenic_max_h (sum_0, sum_2);
[0158] The input variables sum_0 and sum_2 are used, and the output variable is sum_0, which stores 8 int16_t data. The input variables sum_0 and sum_2 store 8 int16_t data.
[0159] Compare sum_1 and sum_3, select the larger data point, and store it in sum_1, represented as:
[0160] sum_1= ingenic_max_h (sum_1, sum_3),
[0161] The input variables sum_1 and sum_3 are used as input variables, and the output variable is sum_1, which stores 8 int16_t data. The input variables sum_1 and sum_3 store 8 int16_t data.
[0162] Using the quantization function module, sum_0 and sum_1 are quantized, and the result is stored in sum_0, as follows:
[0163] sum_0= quantization(sum_0, sum_1),
[0164] The inputs sum_0 and sum_1 are the initial result registers for the convolution calculation. The data in sum_0 and sum_1 are the results before quantization and are only the cumulative sum of the convolution calculation. Each of sum_0 and sum_1 stores 8 16-bit data. The output sum_0 is the generated result, which stores 16 8-bit data.
[0165] d) The width of the feature map is the remainder when divided by 2, and the height is the remainder when divided by 2. In the preliminary convolution calculation, the feature map width *wi* and height *hi* take only the values *sum_0* and *sum_1*; width *wi+1* and height *hi*, width *wi* and height *hi+1*, and width *wi+1* and height *hi+1* have all overflowed outside the feature map and do not exist.
[0166] Using the quantization function module, sum_0 and sum_1 are quantized, and the result is stored in sum_0, as follows:
[0167] sum_0= quantization(sum_0, sum_1),
[0168] The inputs sum_0 and sum_1 are the initial result registers for the convolution calculation. The data in sum_0 and sum_1 are the results before quantization and are only the cumulative sum of the convolution calculation. Each of sum_0 and sum_1 stores 8 16-bit data. The output sum_0 is the generated result, which stores 16 8-bit data.
[0169] By using methods a), b), c), and d), convolution and pooling operations can be integrated, while also reducing the need for three quantization steps. In conventional methods, convolution is performed first, followed by quantization and result generation, then pooling (using 2x2 pooling, meaning four data points are compared to generate one). Each pooling result requires four quantization steps. Here, pooling is moved to the position after convolution and before quantization (since quantization is an increasing function and pooling is a comparison function, the reason for moving pooling before quantization has been explained in the above analysis; calculating a pooling result only requires one quantization step, thus reducing three quantization steps and significantly improving efficiency).
[0170] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An optimization method for convolution and pooling combined, characterized in that, The method takes an image as input, comprising: S1, quantization and pooling function analysis: S1.1, set the convolution calculation preliminary result as a, the quantization multiplication coefficient as b, the first shift coefficient as shift_left, the second shift coefficient as shift_right, and the final result as res, the formula is as follows: (1) (2) (3) (4) S1.2, analyzing the above formulas, the independent variable is a, and other parameters shift_left, shift_right are given constants, then formulas (1), (2), (3) are all increasing functions, while (4) is also an increasing function, so formula (4) is an increasing function, then res obtained from formulas (1) to (4) is an increasing function about a, that is, the quantization process is a process of processing an increasing function; S2, optimization processing method: First, the convolution calculation preliminary result is processed by pooling, and then the pooling result is processed by quantization; further comprising: S2.1, the pooling processing in the optimization processing adopts a 2X2 kernel and a step length of 2; Suppose the quantization processing module is vrd= quantization(vrs, vrt), where vrs and vrt are convolution calculation preliminary result registers, the data in vrs and vrt is the result before quantization and is only the convolution calculation cumulative sum, and vrs and vrt each store 8 16-bit data; vrd is the generated result, which stores 16 8-bit data; S2.2, according to the width and height of the feature map, the pooling and quantization processing are performed to realize the integration of convolution operation and pooling operation: Wherein, suppose the convolution calculation preliminary result is archived to sum_0, sum_1, sum_2, sum_3, sum_4, sum_5, sum_6, sum_7, these parameters are all 8 16-bit registers; The feature map depth is a multiple of 16, and each processing is a multiple of 16, sum_0 and sum_1, sum_2 and sum_3, sum_4 and sum_5, sum_6 and sum_7 represent 16 data processed respectively; The convolution calculation preliminary result is regarded as a feature map, and in width and height, two step lengths will generate a width and a height of a result after pooling. The pooling processing in the method is to compare several data in the same channel and select the largest one.
2. The optimization method of convolution and pooling combined according to claim 1, characterized in that, The method uses pooling processing, which adopts maximum pooling, and the pooling kernel is 2X2 and the step length is 2.
3. The method of claim 1, wherein, The step S2.2 further comprises: a) the part of the width and height of the feature map that can be divided by 2: Suppose the feature map of the convolution calculation preliminary result takes values on the width wi and the height hi, then sum_0 and sum_1; Suppose the feature map of the convolution calculation preliminary result takes values on the width wi+1 and the height hi, then sum_2 and sum_3; Suppose the feature map of the convolution calculation preliminary result takes values on the width wi and the height hi+1, then sum_4 and sum_5; Suppose the feature map of the convolution calculation preliminary result takes values on the width wi+1 and the height hi+1, then sum_6 and sum_7; Pooling processing: Compare sum_0 and sum_2, select one of the larger data and store it in sum_0; Compare sum_0 and sum_4, select one of the larger data and store it in sum_0; Compare sum_0 and sum_6, select one of the larger data and store it in sum_0; Compare sum_1 and sum_3, select one of the larger data and store it in sum_1; Comparing sum_1 and sum_5, one of the larger data is selected and stored in sum_1; Comparing sum_1 and sum_7, one of the larger data is selected and stored in sum_1; Quantization processing: The quantization function module is used to quantize sum_0 and sum_1, and the processing result is stored in sum_0, which is represented as: sum_0 = quantization(sum_0, sum_1), where the output sum_0 is the generated result, and 16 8-bit data are stored; b) The remainder part of the feature map width divided by 2, and the part divided by 2: At this time, the feature map of the convolution calculation preliminary result takes values on the width wi height hi, and sum_0 and sum_1 are obtained; The feature map of the convolution calculation preliminary result takes values on the width wi height hi+1, and sum_4 and sum_5 are obtained; Comparing sum_0 and sum_4, one of the larger data is selected and stored in sum_0; Comparing sum_1 and sum_5, one of the larger data is selected and stored in sum_1; Quantization processing: The quantization function module is used to quantize sum_0 and sum_1, and the processing result is stored in sum_0, which is represented as: sum_0 = quantization(sum_0, sum_1); c) The part of the feature map width divided by 2, and the remainder part divided by 2: The feature map of the convolution calculation preliminary result takes values on the width wi height hi, and sum_0 and sum_1 are obtained; The feature map of the convolution calculation preliminary result takes values on the width wi+1 height hi, and sum_2 and sum_3 are obtained; Pooling processing: Comparing sum_0 and sum_2, one of the larger data is selected and stored in sum_0; Comparing sum_1 and sum_3, one of the larger data is selected and stored in sum_1; Quantization processing: The quantization function module is used to quantize sum_0 and sum_1, and the processing result is stored in sum_0, which is represented as: sum_0 = quantization(sum_0, sum_1); d) The remainder part of the feature map width divided by 2, and the remainder part divided by 2: The feature map of the convolution calculation preliminary result takes values on the width wi height hi, and only sum_0 and sum_1 are obtained; The width wi+1 height hi, the width wi height hi+1, and the width wi+1 height hi+1 have all overflowed outside the feature map, and do not exist; The quantization function module is used to quantize sum_0 and sum_1, and the processing result is stored in sum_0, which is represented as: sum_0 = quantization(sum_0, sum_1); The integration of convolution calculation and pooling operation is realized through a), b), c), and d). 4. The method of claim 3, wherein, The method adopts a maximum instruction in a simd instruction set: input variables vs, vt, and an output variable vd; vd stores 8 int16_t data, vs and vt store 8 int16_t data, and is expressed as: vd = ingenic_max_h (vs, vt); The equivalent operation is: vd0 := (vs0>vt0)?vs0:vt0; vd1 := (vs1>vt1)?vs1:vt1; …… vd7 := (vs7>vt7)?vs7:vt7.
5. The optimization method of convolution and pooling combination according to claim 4, characterized in that, a) the part of the feature map width and height divisible by 2: The comparison of sum_0 and sum_2 selects one of the larger data and stores it in sum_0, which is expressed as: sum_0 = ingenic_max_h (sum_0, sum_2), Where the input variables sum_0 and sum_2, and the output variable sum_0 store 8 int16_t data, and the input variables sum_0 and sum_2 store 8 int16_t data; The comparison of sum_0 and sum_4 selects one of the larger data and stores it in sum_0, which is expressed as: sum_0 = ingenic_max_h (sum_0, sum_4), Where the input variables sum_0 and sum_4, and the output variable sum_0 store 8 int16_t data, and the input variables sum_0 and sum_4 store 8 int16_t data; The comparison of sum_0 and sum_6 selects one of the larger data and stores it in sum_0, which is expressed as: sum_0 = ingenic_max_h (sum_0, sum_6), Where the input variables sum_0 and sum_6, and the output variable sum_0 store 8 int16_t data, and the input variables sum_0 and sum_6 store 8 int16_t data; The comparison of sum_1 and sum_3 selects one of the larger data and stores it in sum_1, which is expressed as: sum_1 = ingenic_max_h (sum_1, sum_3), Where the input variables sum_1 and sum_3, and the output variable sum_1 store 8 int16_t data, and the input variables sum_1 and sum_3 store 8 int16_t data; The comparison of sum_1 and sum_5 selects one of the larger data and stores it in sum_1, which is expressed as: sum_1= ingenic_max_h (sum_1, sum_5), Wherein, the input variable sum_1, sum_5, the output variable is that sum_1 stores 8 int16_t data, the input variable sum_1 and sum_5 store 8 int16_t data; The comparison sum_1, sum_7 is that one of the large data is selected and stored in sum_1, expressed as: Sum_1=ingenic_max_h (sum_1, sum_7), Wherein, the input variable sum_1, sum_7, the output variable is that sum_1 stores 8 int16_t data, the input variable sum_1 and sum_7 store 8 int16_t data; B) the remainder part of the width of the feature map divided by 2, and the part divided by 2 in height: Pooling processing: The comparison sum_0, sum_4 is that one of the large data is selected and stored in sum_0, expressed as: Sum_0=ingenic_max_h (sum_0, sum_4), Wherein, the input variable sum_0, sum_4, the output variable is that sum_0 stores 8 int16_t data, the input variable sum_0 and sum_4 store 8 int16_t data; The comparison sum_1, sum_5 is that one of the large data is selected and stored in sum_1, expressed as: Sum_1=ingenic_max_h (sum_1, sum_5), Wherein, the input variable sum_1, sum_5, the output variable is that sum_1 stores 8 int16_t data, the input variable sum_1 and sum_5 store 8 int16_t data; C) the part of the width of the feature map divided by 2, and the remainder part of the height divided by 2: The comparison sum_0, sum_2 is that one of the large data is selected and stored in sum_0, expressed as: Sum_0=ingenic_max_h (sum_0, sum_2); Wherein, the input variable sum_0, sum_2, the output variable is that sum_0 stores 8 int16_t data, the input variable sum_0 and sum_2 store 8 int16_t data; The comparison sum_1, sum_3 is that one of the large data is selected and stored in sum_1, expressed as: Sum_1=ingenic_max_h (sum_1, sum_3), Wherein, the input variable sum_1, sum_3, the output variable is that sum_1 stores 8 int16_t data, the input variable sum_1 and sum_3 store 8 int16_t data.
Citation Information
Patent Citations
Neural network processing method and processing system
CN107766292A
Point cloud sharp feature normal vector estimation method based on depth feature classification and neighborhood optimization
CN112561977A