A method for implementing weight-released convolution of a WRAM

Abstract

The application provides a method for implementing convolution of WRAM weight release, and the method comprises the following steps: S1, data storage: setting a storage mode of a feature map; setting a storage mode of a weight; S2, loading all data from a DDR to a WRAM using a SIMD instruction, loading 32 data each time; moving data in the DDR to an ORAM using an ORAM data moving instruction; S3, implementing convolution calculation. Through the setting method of the FRAM width, the data moving method from the ORAM to the FRAM and the corresponding new convolution calculation method, the application realizes the calculation of small input feature maps and small weights, accelerates and improves the efficiency.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for implementing WRAM-based convolution with weights. Background Technology

[0002] The Beijing Junzheng Integrated Circuit Co., Ltd.'s T40 chip (hereinafter referred to as the Beijing Junzheng T40 chip) is a low-power chip for deep learning in AI. It features an independent convolution computation unit and unique SIMD instructions. It has one ORAM (Output RAM), one WRAM (Wide RAM) for storing weights, and one FRAM (Front RAM) for storing input data. Its implementation requires data to be stored in the WRAM and FRAM before convolution computation can be performed. The sizes of the ORAM, WRAM, and FRAM are given by the chip's specifications. For example, the WRAM size is... FRAM is The ORAM size is The data based on these assumptions will be used in the calculations below.

[0003] All data is stored in DDR. Data needs to be moved to ORAM using DMA instructions, or loaded into a designated register using SIMD instructions, and then moved to WRAM or FRAM using special instructions.

[0004] Because this is a new chip, while common algorithms can be implemented, they are very inefficient. Furthermore, existing methods cannot utilize the unique computing units and instructions. Different input feature maps of different sizes and different weights require different implementation methods; using inappropriate algorithms leads to a sharp drop in efficiency.

[0005] In addition, the following are commonly used terms in the prior art:

[0006] 1. Convolution Kernel: A convolution kernel is a matrix used in image processing, representing the parameters used to perform operations with the original image. A convolution kernel is typically composed of a column matrix (e.g., ...). The matrix (a region containing cells) assigns a weight value to each cell. The matrix shape is typically 1×1, 3×3, 5×5, 7×7, 1×3, 3×1, 2×2, 1×5, 5×1, ...

[0007] 2. Convolution: The center of the convolution kernel is placed on the pixel to be calculated. The product of each element in the kernel and the image pixel value it covers is calculated one by one and then summed. The resulting structure is the new pixel value at that position. This process is called convolution.

[0008] 3. Feature Map: The result obtained after input data is processed by convolution is called a feature map. The result generated after data is processed by fully connected layers is also called a feature map. The size of a feature map is generally expressed as length × width × depth, or 1 × depth.

[0009] 4. FRAM (Feature RAM): This is a type of memory used to store all or part of the feature map, directly supplying the hardware computing unit for computation. It belongs to the storage portion of the computing unit. To use the computing unit, the feature map data must first be placed into the FRAM.

[0010] 5. WRAM (Weight RAM): This is a type of memory used to store all or part of the weights, directly supplied to the hardware computing unit for calculation. It belongs to the storage portion of the computing unit. Before using the computing unit, the weight data must first be placed into FRAM.

[0011] 6. ORAM (Oblivious RAM): A type of random access memory that enables fast read and write access to dynamic random access memory (DRAM). For any input X, Y, the generated series of instructions have the same probability distribution. Summary of the Invention

[0012] To address the problems in the prior art, the purpose of this application is to design a special calculation method based on specific circumstances, particularly for calculating small input feature maps and small weights on the Beijing Junzheng T40 chip.

[0013] Specifically, the present invention provides a method for implementing convolutions that can accommodate weights using WRAM, the method comprising the following steps:

[0014] S1, Data storage:

[0015] The feature map storage method is set as follows: feature map data storage order is 32, W, H, N; where 32 represents a portion of the depth, W is the width, H is the height, and N is the number of 32 elements at the depth. The depth of the feature map; the data is continuous in 32, then in width, then in height, and finally in the number of depths / 32.

[0016] The weights are stored using the following method: The sequence is as follows: first, continuous along the width of the convolution kernel; then continuous along the height of the convolution kernel; then continuous along the number of input depths / 32; and finally continuous along the number of output depths / 32. Before processing, the usual input depths need to be continuous, then the width and height of the convolution kernels need to be stored, and finally the output depths of the convolution kernels need to be stored in the required order.

[0017] S2 uses SIMD instructions to load all data from DDR to WRAM, loading 32 data items at a time:

[0018] S2.1, use SIMD instructions to load all data from DDR to WRAM, loading 32 data at a time: set the initial address of the weight data to widthdata;

[0019] Use SIMD load data instructions to load data into VR0 and VR1;

[0020] Use the WRAM load data instruction to load data into WRAM;

[0021] Since the weights are stored in the required order and the data size can fit entirely into the wram, they can be stored directly in the default order until all the data is stored. This method cannot be used when the wram cannot hold the data.

[0022] S2.2, Use the ORAM data transfer instruction to move data from DDR to ORAM: Let the initial address of the feature map be ddr_id, the number of bytes in the feature map be count, and the initial address of the ORAM be oram_id;

[0023] ingenic_ddr2oram(ddr_id, oram_id, count, 1);

[0024] Since the feature maps are stored in the required order and the data size can fit entirely into the ORAM, they can be stored directly in the default order until all data is stored. This method cannot be used when the ORAM cannot hold the data or when fram_w cannot hold the smallest computed pixel.

[0025] S3, Implementation of convolution calculation:

[0026] S3.1, To compute the convolution, the data needs to be moved from ORAM to FRAM before it can be used for convolution computation; the weights have all been loaded into WRAM, so there is no need to consider the weight transfer situation; however, FRAM cannot hold all the feature maps, so the amount of input data to be transferred from ORAM to FRAM is required.

[0027] S3.2, Convolution computation first requires loading data from ORAM into FRAM; only then can convolution computation be performed using FRAM and WRAM. Initial addresses for ORAM and WRAM must be provided, both initially set to 0. Let the input feature map depth be... `in_ic32` is a multiple of the input depth, `in_width` is the input width, and `in_height` is the input height; the output feature map depth is... `out_ic32` is a multiple of the output depth, `out_width` is the output width, and `out_height` is the input height; the kernel width is `kernel_w`, and the height is `kernel_h`; the kernel width stride is `stride_w`, and the kernel height stride is `stride_h`; the relationship between the output feature map width and the input feature map width... The relationship between the height of the output feature map and the height of the input feature map If they are not equal, the input feature maps need to be padded with zeros according to the specific convolution requirements to make them equal in width and height; the generated results are saved in vrd.

[0028] To reduce the number of times ORAM is loaded into FRAM, all results in the same depth direction are generated simultaneously. Therefore, when designing the loop order, the outermost loop is the height of the output feature map, followed by the width of the output feature map, then the depth of the output feature map / 32, and finally the convolutional computation unit.

[0029] Let fram_h be the number of rows generated each time, where fram_h = fram_count / fram_w; the more rows loaded, the fewer times they are repeatedly loaded in the height direction.

[0030] The method is applicable under the following conditions: the feature map size can be accommodated in ORAM, the weights are relatively small, the number of weights is less than or equal to wRAM, WRAM can accommodate them, the bit width is 8 bits, and the feature map data required to generate 8 pixels each time can be completely placed in FRAM; the length or width of the convolution kernel does not exceed 3; at the same time, the input depth is required to be a multiple of 32, and the output depth is also a multiple of 32; if the input depth of some layers in the model is not a multiple of 32, it needs to be padded to a multiple of 32; the corresponding weights are also padded.

[0031] The instructions used in the method are as follows:

[0032] a) Convolution calculation command:

[0033] ingenic_conv_bit8(fram_id,wram_id,ic32_num,kernel_w,kernel_h,stride_x,stride_y,feature_w,feature_h,vrd);

[0034] The input variables are: fram_id (starting address of fram), wram_id (starting address of wram), ic32_num (number of computations), kernel_w (width of convolution kernel), kernel_h (height of convolution kernel), stride_x (stride in x-direction of convolution computation), stride_y (stride in y-direction of convolution computation), feature_w (width of input feature map), feature_h (height of input feature map); and vrd (generated result).

[0035] Instructions for use: Calculates 4 pixels at a time; the calculation unit is depth 32, and the generated result is also 32, producing 4 pixels; if ic32_num=1, it calculates the input depth as 32x1 and generates 4 pixels with an output depth of 32; if ic32_num=3, it calculates the input depth as 32x3 and generates 4 pixels with an output depth of 32; if ic32_num=2, it calculates the input depth as 32x2 and generates 4 pixels with an output depth of 32; the minimum input depth is 32, the minimum output depth is 32, and the minimum number of output pixels is 4.

[0036] Setting the width of the FRAM, which is the number of pixels loaded from the input feature map, is a parameter setting of the convolution calculation instruction;

[0037] b) SIMD data loading instructions:

[0038] ingenic_load (indata, VR0, m)

[0039] Input the data to be loaded. Currently, it is the pointer to the data, indata. Starting from the memory location m pointed to by indata, 128 bits of data are loaded. If it is 8-bit data, 16 bits are loaded; if it is 16-bit data, 8 bits are loaded; if it is 32-bit data, 4 bits are loaded. The data is loaded into the variable vrd register. Here, m is calculated in bytes, i.e., 8 bits per unit. VR0 is the VR register of SIMD, which can store a maximum of 512 bits of data.

[0040] c) FRAM data loading instructions:

[0041] ingenic_vr2fram(VR0,fram_load_id,num)

[0042] Input variables: VR0 is the input data, fram_load_id is the starting address loaded into fram, and num is 0 or 1. When 0, the fram_load_id data remains unchanged after the instruction ends. When 1, fram_load_id = fram_load_id + 32 after the instruction ends.

[0043] d) WRAM data loading instructions:

[0044] ingenic_vr2wram(VR0,wram_load_id,num)

[0045] Input variables: VR0 is the input data, wram_load_id is the starting address loaded into wram, and num is 0 or 1. When 0, the data of wram_load_id remains unchanged after the instruction ends. When 1, wram_load_id = fram_load_id + 64 after the instruction ends.

[0046] f) ORAM data transfer instructions:

[0047] ingenic_ddr2oram(ddr_id,oram_id,count,num)

[0048] Input variables: ddr_id is the address where data begins to be loaded in the ddr, oram_id is the address where data begins to be loaded in the oram, and count is the number of bytes loaded; num is 0 or 1. When 0, the data of ddr_id and oram_id remain unchanged after the instruction ends. When 1, the data of ddr_id and oram_id are incremented by count after the instruction ends.

[0049] The width of the FRAM is set as follows:

[0050] Let the total number of bytes in the FRAM be fram_count, the width of fram be fram_w, and the number of rows of input feature maps loaded in each processing be fram_h. The value of fram_w is the number of pixels of the input feature map required to generate at least 8 pixels. The minimum number of pixels to be calculated in the generated result is 4. In practice, to generate the first 4 pixels, the feature map data of the next 4 pixels needs to be loaded, so the minimum number of pixels to be generated is 8.

[0051] The following formula is given.

[0052] (1)

[0053] in It uses integer operations and returns integers; the entire formula ensures that fram is a multiple of 4 and can generate 8 pixels.

[0054] Since each data load is also 4 pixels, for convolutions with a kernel size greater than 1, there will be an overlap between loaded and used data. To resolve this, the extra data is processed according to multiples of 4, that is, fram_w is added to (kernel_w-1) and rounded up (calculation formula). ),have

[0055] (2)

[0056] Through formulas (1) and (2), we have

[0057] (3)

[0058] Formula (3) cannot be combined because it is a rounding operation, and there will be inequalities after combining.

[0059] The number of rows of the input feature map loaded each time is: fram_h = fram_count / fram_w.

[0060] In step S2.1,

[0061] The data is loaded into VR0 and VR1 using SIMD load instructions, denoted as follows:

[0062] ingenic_load (widthdata, VR0,0)

[0063] ingenic_load (widthdata, VR0,1)

[0064] ingenic_load (widthdata, VR1,0)

[0065] ingenic_load (widthdata, VR1,1)

[0066] The WRAM load data instruction is used to load data into WRAM, denoted as ;

[0067] ingenic_vr2wram(VR0,wram_load_id,1)

[0068] ingenic_vr2wram(VR1,wram_load_id,1).

[0069] The specific implementation of the convolution calculation in step S3 is as follows:

[0070] S3.1, Initialization: denoted as

[0071] wram_id=0;

[0072] oram_id=0;

[0073] fram_id=0;

[0074] wr_fram_id=0;

[0075] rd_fram_id=0;

[0076] ;

[0077] S3.2, each time the result of generating the fram_h line is processed;

[0078] Let ydir_i be the height position of the generated result, with an initial value of ydir_i = 0. Let the initial number of processed rows be fram_h_ori, then fram_h = fram_h_ori. If ydir_i < out_height is true, continue executing this step. After this step is completed, ydir_i += fram_h, and check if ydir_i < out_height is true. If it is true, continue executing this step, and repeat the loop. If it is not true, check if ydir_i < (out_height + fram_h - 1) is true (here, out_height may not be an integer multiple of fram_h (when fram_h is greater than 1), so there is a remainder, which needs to be checked again). If it is true, then fram_h = ydir_i - out_height; execute this step. If it is not true, exit this step loop. This is denoted as:

[0079] for(int ydir_i =0; ydir_i<(out_height+ fram_h-1); ydir_i += fram_h);

[0080] if (ydir_i >=out_height) fram_h= ydir_i-out_height;

[0081] Execution: Read data continuously from DDR into ORAM, with each read row number being fram_h;

[0082] Execution: Initialize the address for reading from Oram to 0, and initialize the address for writing to Fram to 0; denoted as:

[0083] int rd_oram_idx = 0;

[0084] int wr_fram_idx = 0;

[0085] Execution: S3.2.1, generates 4 pixels in the width direction each time, and generates a fram_h line in the height direction;

[0086] Let the initial xdir_i = 0. If xdir_i < out_width + 3 (since there may be remainders when out_width is not a multiple of 4, and these remainders also need to be calculated and loaded according to the smallest operational unit, we need to round out_width up by dividing out_width by 4, which is equivalent to out_width + 3. The storage space for storing the input feature map will be slightly larger than the actual feature map space to prevent errors caused by data not being read), then continue executing this step. After this step is completed, xdir_i += 4; check if xdir_i < out_width + 3 is true. If it is true, repeat this step; if it is not true, exit this step and return to S3.2, denoted as:

[0087] for(int xdir_i =0; xdir_i < out_width+3; xdir_i+=4);

[0088] Execute step S3.2.1.1: If xdir_i > 1 is true, denoted as if(xdir_i > 1), then continue executing this step; otherwise, execute step S3.2.1.2.

[0089] Execution: Read data from ORAM into FRAM, with each read in the loop body... This section loads the data needed for the convolution calculation below. To form a loop, the minimum loading unit is 4 pixels, so the first load will be larger. ;

[0090] implement: ;

[0091] implement: ;

[0092] Execution: Loop body (5);

[0093] Execute step S3.2.1.2. If xdir_i == 0 is true, continue executing this step, denoted as: if(xdir_i == 0); if not true, continue to step S3.2.1.3.

[0094] Execution: Read the data from ORAM into FRAM. The length read by the subsequent loop body is... Therefore, the length of the first read is Read data as ;

[0095] Execute: rd_oram_idx = rd_oram_idx;

[0096] Execution: wr_fram_idx = wr_fram_idx;

[0097] Execution: Loop body (6);

[0098] Execution: In xdir_i > 1, each time it is added is When xdir_i=0, the width of the loaded data is... Therefore, after loading the data with xdir_i=0, the following value should be added:

[0099] ;

[0100] ;

[0101] Execution: S3.2.1.3,

[0102] Execution: int wram_id=0;

[0103] Execution: Each processing generates ;

[0104] Execution: Loop body (7);

[0105] Among them, loop body (5) loads data under normal conditions; loop body (6) loads data for the first time; loop body (7) is the specific implementation of convolution, which first calculates the result generated by sharing a weight, that is, first executes the result in height, and then calculates the result in depth / 32 direction, and calculates all the results in sequence.

[0106] In the method,

[0107] The loop body (5) is as follows:

[0108] Execution: Step (5)1, set the initial iccnum_i = 0, check whether icnum_i < in_ic32 is true. If it is true, continue the current loop step. After the current loop step, icnum_i++, and check whether icnum_i < in_ic32 is true again. Repeat this loop. If it is not true, exit the loop body.

[0109] Recorded as: for(int iccnum_i =0;icnum_i< in_ic32; icnum_i++)

[0110] implement: ;

[0111] Execute: ;

[0112] Execute: Step (5)2. Set the initial fh_i = 0. Determine whether fh_i < fram_h holds. If it holds, continue with this loop step. After this loop step, increment fh_i by 1, and then determine whether fh_i < fram_h holds again, and loop in sequence; if it does not hold, jump out of this step and return to execute Step (5)1;

[0113] Record as: for(int fh_i = 0; fh_i < fram_h; fh_i++)

[0114] Execute: ;

[0115] Execute: ;

[0116] Execute: Step (5)3. Set the initial fw_i = 0. Determine whether holds. If it holds, continue with this loop step. After this loop step, increment fw_i by 4, and then determine whether holds, and loop in sequence; if it does not hold, jump out of this step and return to execute Step (5)2;

[0117] Record as:

[0118] Execute: Load data into VR0 and VR1 using the SIMD load data instruction; Record as:

[0119] ingenic_load (rd_oram, VR0, 0);

[0120] ingenic_load (rd_oram, VR0, 1);

[0121] ingenic_load (rd_oram, VR1, 0);

[0122] ingenic_load (rd_oram, VR1, 1);

[0123] Execute: Use the FRAM load data instruction to load data into the FRAM; Record as:

[0124] ingenic_vr2fram(VR0, wr_fram, 1);

[0125] ingenic_vr2fram(VR1, wr_fram, 1);

[0126] The loop body (6) is as follows:

[0127] Execute: Step (6)1. Set the initial iccnum_i = 0, and determine whether icnum_i < in_ic32 holds. If it does not hold, jump out of the loop body (6). If it holds, continue to execute this loop step. After this loop step, icnum_i++. Then, determine whether it holds again. If it holds, loop sequentially; otherwise, jump out of this loop.

[0128] Recorded as: for(int iccnum_i =0;icnum_i< in_ic32; icnum_i++)

[0129] Execute: ;

[0130] Execute: ;

[0131] Execute: Step (6)2. Set the initial fh_i = 0, and determine whether fh_i < fram_h holds. If it does not hold, jump out of Step (6)2. If it holds, continue to execute this loop step. After this loop step, fh_i++. Then, determine whether it holds again. If it holds, loop sequentially; otherwise, jump out of this loop. Recorded as:

[0132] for(int fh_i=0;fh_i<fram_h; fh_i++)

[0133] Execute: ;

[0134] Execute: ;

[0135] Execute: Step (6)3. Set the initial fw_i = 0, and determine whether holds. If it does not hold, jump out of Step (6)3. If it holds, continue to execute this loop step. After this loop step, fw_i += 4. Then, determine whether it holds again. If it holds, loop sequentially; otherwise, jump out of this loop. Recorded as:

[0136] Execute: Load data into VR0 and VR1 using the SIMD load data instruction. Recorded as:

[0137] ingenic_load (rd_oram, VR0,0);

[0138] ingenic_load (rd_oram, VR0,1);

[0139] ingenic_load (rd_oram, VR1,0);

[0140] ingenic_load (rd_oram, VR1, 1);

[0141] Execute: Use the FRAM load data instruction to load data into the FRAM; Denote as:

[0142] ingenic_vr2fram(VR0, wr_fram, 1);

[0143] ingenic_vr2fram(VR1, wr_fram, 1);

[0144] The loop body (7) is as follows:

[0145] Execute step (7) 1: Set the initial ocnum_i = 0, and judge whether ocnum_i < out_ic32 holds. If it does not hold, jump out of step (7) 1. If it holds, continue to execute this loop step. After this loop step, ocnum_i++. Then judge whether it holds again. If it holds, loop sequentially; otherwise, jump out of this loop; Denote as:

[0146] Denote as: for(int ocnum_i = 0; ocnum_i < out_ic32; ocnum_i++)

[0147] Execute: Each time generate

[0148] Execute: fram_id = 0;

[0149] Execute: ;

[0150] Execute step (7) 2: Set the initial fh_i = 0, and judge whether fh_i < fram_h holds. If it does not hold, jump out of step (7) 2. If it holds, continue to execute this loop step. After this loop step, fh_i++. Then judge whether it holds again. If it holds, loop sequentially; otherwise, jump out of this loop; Denote as:

[0151] Denote as: for(int fh_i = 0; fh_i < fram_h; fh_i++)

[0152] Execute: ;

[0153] Execute: ingenic_conv_bit8(fram_id, wram_id, ic32_num, kernel_w,

[0154] kernel_h, stride_x, stride_y, vrd);

[0155] Execution: Retrieve the generated results for further processing and saving;

[0156] implement: .

[0157] Therefore, the advantages of this application are: a novel design method applicable to chips such as the Beijing Junzheng T40 chip, which first loads the feature map data from ddr to oram, and then from oram to fram, to achieve fast feature map loading and avoid the bandwidth limitation caused by loading feature maps; for the calculation of small weights, the weight data is directly saved to WRAM, and the data is not loaded repeatedly, which makes efficient use of the data and avoids the problem of data loading competing for bandwidth, thereby accelerating the process and improving efficiency. Attached Figure Description

[0158] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0159] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0160] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0161] This invention relates to a method for implementing convolution with weights that can be accommodated in WRAM, and the applicable requirements of the method. This is used on the Ingenic T40 chip, where the feature map size can be accommodated in ORAM, the weights are relatively small (the number of weights is less than or equal to WRAM), and WRAM can accommodate them. All weights are 8 bits, and the feature map data needed to generate 8 pixels each time can be completely placed in FRAM.

[0162] The method includes the following instructions:

[0163] a) Convolution calculation command:

[0164] ingenic_conv_bit8(fram_id,wram_id,ic32_num,kernel_w,kernel_h,stride_x,stride_y,feature_w,feature_h,vrd);

[0165] The input variables `fram_id` is the starting address used by `fram`, `wram_id` is the starting address used by `wram`, `ic32_num` is the number of computations, `kernel_w` is the width of the convolution kernel, `kernel_h` is the height of the convolution kernel, `stride_x` is the stride in the x-direction of the convolution computation, `stride_y` is the stride in the y-direction of the convolution computation, `feature_w` is the width of the input feature map, and `feature_h` is the height of the input feature map. `vrd` is the generated result.

[0166] Instructions: Calculates 4 pixels at a time. The calculation unit is depth 32, and the generated result is also 32, producing 4 pixels. If ic32_num=1, it calculates the input depth as 32x1, generating 4 pixels with an output depth of 32. If ic32_num=3, it calculates the input depth as 32x3, generating 4 pixels with an output depth of 32. The minimum input depth is 32, the minimum output depth is 32, and the minimum number of output pixels is 4.

[0167] Setting the width of the fram, which determines how many pieces are loaded from the input feature map, is a parameter setting for the convolution calculation instruction.

[0168] How to set the width of the frame:

[0169] Let the total number of bytes in the fram be fram_count, the width of the fram be fram_w, and the number of rows of input feature maps loaded in each processing iteration be fram_h. The fram_w value is set to the number of pixels of the input feature map required to generate at least 8 pixels. The minimum number of pixels to be generated is 4. In practice, generating the first 4 pixels requires loading the feature map data for the next 4 pixels, so the minimum number of pixels generated is 8. The following formula is used.

[0170] (1)

[0171] in It uses integer operations, returning the integer value. The entire formula ensures that `fram` is a multiple of 4 and generates 8 pixels.

[0172] Since each data load also consists of 4 pixels, for convolutions with a kernel size greater than 1, there will be an overlap between loaded and used data. To resolve this, the extra data is processed according to multiples of 4, i.e., adding (kernel_w-1) to fram_w and rounding up (calculation formula). ,have

[0173] (2)

[0174] Through formulas (1) and (2), we have

[0175] (3)

[0176] Formula (3) cannot be combined because it is a rounding operation, and there will be inequalities after combining.

[0177] The number of rows of the input feature map loaded each time is: fram_h = fram_count / fram_w.

[0178] b) Simd data loading instructions:

[0179] ingenic_load(indata, VR0, m)

[0180] The input data to be loaded is currently a pointer to the data, `indata`. Starting from the memory location `m` pointed to by `indata`, 128 bits of data are loaded. If the data is 8 bits, 16 bits are loaded; if it's 16 bits, 8 bits are loaded; and if it's 32 bits, 4 bits are loaded. The data is loaded into the variable `vrd` register. Here, `m` is calculated in bytes, i.e., 8 bits per unit. `VR0` is the SIMD VR register, which can store a maximum of 512 bits of data.

[0181] c) FRAM data loading instructions:

[0182] ingenic_vr2fram(VR0,fram_load_id,num)

[0183] Input variables: VR0 is the input data, fram_load_id is the starting address of the data loaded into fram, and num is 0 or 1. When 0, the fram_load_id data remains unchanged after the instruction ends. When 1, fram_load_id = fram_load_id + 32 after the instruction ends.

[0184] d) WRAM data loading instructions:

[0185] ingenic_vr2wram(VR0,wram_load_id,num)

[0186] Input variables: VR0 is the input data, wram_load_id is the starting address of the data loaded into the wram, and num is 0 or 1. When 0, the wram_load_id data remains unchanged after the instruction ends. When 1, wram_load_id = wram_load_id + 64 after the instruction ends.

[0187] f) ORAM data transfer commands:

[0188] ingenic_ddr2oram(ddr_id,oram_id,count,num)

[0189] Input variables: ddr_id is the address where data begins to be loaded from the ddr, oram_id is the address where data begins to be loaded from the oram, and count is the number of bytes loaded. num is 0 or 1; when 0, ddr_id and oram_id remain unchanged after the instruction ends; when 1, ddr_id and oram_id are incremented by count after the instruction ends.

[0190] The convolution calculation method described above is suitable for cases where the input feature map is relatively small, the weights are relatively small, and the length or width of the convolution kernel does not exceed 3. It also requires that the input depth and output depth be multiples of 32. If the input depth of some layers in the model is not a multiple of 32, it needs to be padded to a multiple of 32. The corresponding weights are also padded accordingly.

[0191] like Figure 1 As shown, the steps of this method include the following:

[0192] S1, Data storage:

[0193] Configure the storage method for the feature maps;

[0194] Set the storage method for the weights;

[0195] S2 uses SIMD instructions to load all data from DDR to WRAM, loading 32 data items at a time:

[0196] S2.1 uses SIMD instructions to load all data from DDR to WRAM, loading 32 data items at a time;

[0197] S2.2, Use the ORAM data transfer instruction to move data from DDR to ORAM: Let the initial address of the feature map be ddr_id, the number of bytes in the feature map be count, and the initial address of the ORAM be oram_id;

[0198] ingenic_ddr2oram(ddr_id, oram_id, count, 1);

[0199] Since the feature maps are stored in the required order and the data size can fit entirely into the ORAM, they can be stored directly in the default order until all data is stored. This method cannot be used when the ORAM cannot hold the data or when fram_w cannot hold the smallest computed pixel.

[0200] S3, Implementation of convolution calculation:

[0201] S3.1, To compute the convolution, the data needs to be moved from ORAM to FRAM before it can be used for convolution computation; the weights have all been loaded into WRAM, so there is no need to consider the weight transfer situation; however, FRAM cannot hold all the feature maps, so the amount of input data to be transferred from ORAM to FRAM is required.

[0202] S3.2, Convolution computation first requires loading data from ORAM into FRAM; only then can convolution computation be performed using FRAM and WRAM. Initial addresses for ORAM and WRAM must be provided, both initially set to 0. Let the input feature map depth be... `in_ic32` is a multiple of the input depth, `in_width` is the input width, and `in_height` is the input height; the output feature map depth is... `out_ic32` is a multiple of the output depth, `out_width` is the output width, and `out_height` is the input height; the kernel width is `kernel_w`, and the height is `kernel_h`; the kernel width stride is `stride_w`, and the kernel height stride is `stride_h`; the relationship between the output feature map width and the input feature map width... The relationship between the height of the output feature map and the height of the input feature map If they are not equal, the input feature maps need to be padded with zeros according to the specific convolution requirements to make them equal in width and height; the generated results are saved in vrd.

[0203] To reduce the number of times ORAM is loaded into FRAM, all results in the same depth direction are generated simultaneously. Therefore, when designing the loop order, the outermost loop is the height of the output feature map, followed by the width of the output feature map, then the depth of the output feature map / 32, and finally the convolutional computation unit.

[0204] Let fram_h be the number of rows generated each time, where fram_h = fram_count / fram_w; the more rows loaded, the fewer times they are repeatedly loaded in the height direction.

[0205] In step S2.1,

[0206] The data is loaded into VR0 and VR1 using SIMD load instructions, denoted as follows:

[0207] ingenic_load (widthdata, VR0,0)

[0208] ingenic_load (widthdata, VR0,1)

[0209] ingenic_load (widthdata, VR1,0)

[0210] ingenic_load (widthdata, VR1,1)

[0211] The WRAM load data instruction is used to load data into WRAM, denoted as ;

[0212] ingenic_vr2wram(VR0,wram_load_id,1)

[0213] ingenic_vr2wram(VR1,wram_load_id,1).

[0214] The specific implementation of the convolution calculation in step S3 is as follows:

[0215] S3.1, Initialization: denoted as

[0216] wram_id=0;

[0217] oram_id=0;

[0218] fram_id=0;

[0219] wr_fram_id=0;

[0220] rd_fram_id=0;

[0221] ;

[0222] S3.2, each time the result of generating the fram_h line is processed;

[0223] Let ydir_i be the height position of the generated result, with an initial value of ydir_i = 0. Let the initial number of processed rows be fram_h_ori, then fram_h = fram_h_ori. If ydir_i < out_height is true, continue executing this step. After this step is completed, ydir_i += fram_h, and check if ydir_i < out_height is true. If it is true, continue executing this step, and repeat the loop. If it is not true, check if ydir_i < (out_height + fram_h - 1) is true (here, out_height may not be an integer multiple of fram_h (when fram_h is greater than 1), so there is a remainder, which needs to be checked again). If it is true, then fram_h = ydir_i - out_height; execute this step. If it is not true, exit this step loop. This is denoted as:

[0224] for(int ydir_i =0; ydir_i<(out_height+ fram_h-1); ydir_i += fram_h);

[0225] if (ydir_i >=out_height) fram_h= ydir_i-out_height;

[0226] Execution: Read data continuously from DDR into ORAM, with each read row number being fram_h;

[0227] Execution: Initialize the address for reading from Oram to 0, and initialize the address for writing to Fram to 0; denoted as:

[0228] int rd_oram_idx = 0;

[0229] int wr_fram_idx = 0;

[0230] Execution: S3.2.1, generates 4 pixels in the width direction each time, and generates a fram_h line in the height direction;

[0231] Let the initial xdir_i = 0. If xdir_i < out_width + 3 (since there may be remainders when out_width is not a multiple of 4, and these remainders also need to be calculated and loaded according to the smallest operational unit, we need to round out_width up by dividing out_width by 4, which is equivalent to out_width + 3. The storage space for storing the input feature map will be slightly larger than the actual feature map space to prevent errors caused by data not being read), then continue executing this step. After this step is completed, xdir_i += 4; check if xdir_i < out_width + 3 is true. If it is true, repeat this step; if it is not true, exit this step and return to S3.2, denoted as:

[0232] for(int xdir_i =0; xdir_i < out_width+3; xdir_i+=4);

[0233] Execute step S3.2.1.1: If xdir_i > 1 is true, denoted as if(xdir_i > 1), then continue executing this step; otherwise, execute step S3.2.1.2.

[0234] Execution: Read data from ORAM into FRAM, with each read in the loop body... This section loads the data needed for the convolution calculation below. To form a loop, the minimum loading unit is 4 pixels, so the first load will be larger. ;

[0235] implement: ;

[0236] implement: ;

[0237] Execution: Loop body (5);

[0238] Execute step S3.2.1.2. If xdir_i == 0 is true, continue executing this step, denoted as: if(xdir_i == 0); if not true, continue to step S3.2.1.3.

[0239] Execution: Read the data from ORAM into FRAM. The length read by the subsequent loop body is... Therefore, the length of the first read is Read data as ;

[0240] Execute: rd_oram_idx = rd_oram_idx;

[0241] Execution: wr_fram_idx = wr_fram_idx;

[0242] Execution: Loop body (6);

[0243] Execution: In xdir_i > 1, each time it is added is When xdir_i=0, the width of the loaded data is... Therefore, after loading the data with xdir_i=0, the following value should be added:

[0244] ;

[0245] ;

[0246] Execution: S3.2.1.3,

[0247] Execution: int wram_id=0;

[0248] Execution: Each processing generates ;

[0249] Execution: Loop body (7);

[0250] Among them, loop body (5) loads data under normal conditions; loop body (6) loads data for the first time; loop body (7) is the specific implementation of convolution, which first calculates the result generated by sharing a weight, that is, first executes the result in height, and then calculates the result in depth / 32 direction, and calculates all the results in sequence.

[0251] In the method,

[0252] The loop body (5) is as follows:

[0253] Execution: Step (5)1, set the initial iccnum_i = 0, check whether icnum_i < in_ic32 is true. If it is true, continue the current loop step. After the current loop step, icnum_i++, and check whether icnum_i < in_ic32 is true again. If it is true, loop in sequence; if it is not true, exit the loop body (5).

[0254] Recorded as: for(int iccnum_i =0;icnum_i< in_ic32; icnum_i++)

[0255] implement: ;

[0256] implement: ;

[0257] Execute: Step (5)2. Set the initial value of fh_i to 0. Check if fh_i < fram_h holds. If it holds, continue with this loop step. After this loop step, increment fh_i by 1. Then check again if fh_i < fram_h holds. If so, loop sequentially; if not, jump out of this step and return to execute Step (5)1.

[0258] Denoted as: for(int fh_i = 0; fh_i < fram_h; fh_i++)

[0259] Execute: ;

[0260] Execute: ;

[0261] Execute: Step (5)3. Set the initial value of fw_i to 0. Check if it holds. If it holds, continue with this loop step. After this loop step, increment fw_i by 4. Then check if it holds. If it holds, loop sequentially; if not, jump out of this step and return to execute Step (5)2.

[0262] Denoted as:

[0263] Execute: Load data into VR0 and VR1 using the SIMD load data instruction. Denoted as:

[0264] ingenic_load (rd_oram, VR0, 0);

[0265] ingenic_load (rd_oram, VR0, 1);

[0266] ingenic_load (rd_oram, VR1, 0);

[0267] ingenic_load (rd_oram, VR1, 1);

[0268] Execute: Use the FRAM load data instruction to load data into the FRAM. Denoted as:

[0269] ingenic_vr2fram(VR0, wr_fram, 1);

[0270] ingenic_vr2fram(VR1, wr_fram, 1);

[0271] The loop body (6) is as follows:

[0272] Execute: Step (6)1. Set the initial iccnum_i = 0, and judge whether icnum_i < in_ic32 holds. If it does not hold, jump out of the loop body (6). If it holds, continue to execute this loop step. After this loop step, icnum_i++. Then judge whether it holds again. If it holds, loop in sequence; otherwise, jump out of this loop;

[0273] Denoted as: for(int iccnum_i =0;icnum_i< in_ic32; icnum_i++)

[0274] Execute: ;

[0275] Execute: ;

[0276] Execute: Step (6)2: Set the initial fh_i = 0, and judge whether fh_i < fram_h holds. If it does not hold, jump out of Step (6)2. If it holds, continue to execute this loop step. After this loop step, fh_i++. Then judge whether it holds again. If it holds, loop in sequence; otherwise, jump out of this loop; Denoted as:

[0277] for(int fh_i=0;fh_i<fram_h; fh_i++)

[0278] Execute: ;

[0279] Execute: ;

[0280] Execute: Step (6)3. Set the initial fw_i = 0, and judge whether it holds. If it does not hold, jump out of Step (6)3. If it holds, continue to execute this loop step. After this loop step, fw_i += 4. Then judge whether it holds again. If it holds, loop in sequence; otherwise, jump out of this loop; Denoted as:

[0281] Execute: Load data into VR0 and VR1 using the SIMD load data instruction; Denoted as:

[0282] ingenic_load (rd_oram,VR0,0);

[0283] ingenic_load (rd_oram,VR0,1);

[0284] ingenic_load (rd_oram,VR1,0);

[0285] ingenic_load (rd_oram, VR1, 1);

[0286] Execute: Use the FRAM load data instruction to load data into the FRAM; Denote as:

[0287] ingenic_vr2fram(VR0, wr_fram, 1);

[0288] ingenic_vr2fram(VR1, wr_fram, 1);

[0289] The loop body (7) is as follows:

[0290] Execute step (7) 1: Set the initial ocnum_i = 0, and judge whether ocnum_i < out_ic32 holds. If it does not hold, then jump out of step (7) 1. If it holds, then continue to execute this loop step. After this loop step, ocnum_i++. Then judge whether it holds again. If it holds, loop sequentially; otherwise, jump out of this loop; Denote as:

[0291] Denote as: for(int ocnum_i = 0; ocnum_i < out_ic32; ocnum_i++)

[0292] Execute: Each time generate

[0293] Execute: fram_id = 0;

[0294] Execute: ;

[0295] Execute step (7) 2: Set the initial fh_i = 0, and judge whether fh_i < fram_h holds. If it does not hold, then jump out of step (7) 2. If it holds, then continue to execute this loop step. After this loop step, fh_i++. Then judge whether it holds again. If it holds, loop sequentially; otherwise, jump out of this loop; Denote as:

[0296] Denote as: for(int fh_i = 0; fh_i < fram_h; fh_i++)

[0297] Execute: ;

[0298] Execute: ingenic_conv_bit8(fram_id, wram_id, ic32_num, kernel_w,

[0299] kernel_h, stride_x, stride_y, vrd);

[0300] Execution: Retrieve the generated results for further processing and saving;

[0301] implement: .

[0302] Therefore, the advantages of this application are as follows: This novel design method first loads the feature map data from DDR to ORAM, and then from ORAM to FRAM, achieving fast feature map loading and avoiding bandwidth limitations caused by loading feature maps; for the calculation of small weights, the weight data is directly saved to WRAM, and the data is not loaded repeatedly, making efficient use of the data and eliminating the problem of bandwidth contention during data loading, thus accelerating the process and improving efficiency.

[0303] Specifically, this method can also be described as follows:

[0304] S1, data storage.

[0305] Feature map storage method: The data storage order of the feature map is 32, W, H, N. Where 32 represents a portion of the depth, W is the width, H is the height, and N is the number of 32-bit segments along the depth. This represents the depth of the feature map. The data is continuous at 32, then continuous at width, then continuous at height, and finally continuous at depth / 32.

[0306] The weights are stored using... The sequence is as follows: first, continuous along the width of the convolution kernel; then continuous along the height of the convolution kernel; then continuous along the number of kernel depths (input depth / 32); and finally continuous along the number of output depths (output depth / 32). Before processing, the sequence of continuous input depths, kernel width and height, and finally kernel output depth needs to be stored in the required order.

[0307] S2 uses simd to load all data from ddr to wram, loading 32 data items at a time.

[0308] a) Use SIMD to load all data from DDR to WRAM, loading 32 data points at a time. Let the initial address of the weight data be widthdata.

[0309] Use the simd load data command to load data into VR0 and VR1.

[0310] ingenic_load (widthdata, VR0,1)

[0311] ingenic_load (widthdata, VR0,1)

[0312] ingenic_load (widthdata, VR1,1)

[0313] ingenic_load (widthdata, VR1,1)

[0314] Use the wram load data instruction to load data into wram.

[0315] ingenic_vr2wram(VR0,wram_load_id,1)

[0316] ingenic_vr2wram(VR1,wram_load_id,1)

[0317] Since the weights are stored in the required order and the data size can be fully stored in the wram, they can be stored directly in the default order until all data is stored. This method cannot be used when the wram cannot hold the data.

[0318] b) Use the ORAM data transfer instruction to move the data from the DDR to ORAM. Let the initial address of the feature map be ddr_id, the number of bytes in the feature map be count, and the initial address of ORAM be oram_id.

[0319] ingenic_ddr2oram(ddr_id,oram_id,count, 1);

[0320] Since the feature maps are stored in the required order and the data size can be fully accommodated in oram, they can be stored directly in the default order until all data is stored. This method cannot be used when oram cannot hold the data or fram_w cannot hold the smallest computed pixel.

[0321] S3, the implementation of convolution calculation.

[0322] To compute a convolution, data needs to be moved from ORAM to FRAM before it can be used for computation. Since all weights are already loaded into WRAM, there's no need to consider weight transfer. However, FRAM cannot hold all feature maps; therefore, the amount of input data needed depends on how much data is transferred from ORAM to FRAM.

[0323] Convolution calculations first require loading data from ORAM into FRAM. Only then can convolution calculations be performed using FRAM and WRAM. Initial addresses for both ORAM and WRAM need to be provided, both initialized to 0. Let the input feature map depth be... `in_ic32` is a multiple of the input depth, `in_width` is the input width, and `in_height` is the input height; the output feature map depth is... `out_ic32` is a multiple of the output depth, `out_width` is the output width, and `out_height` is the input height; the kernel width is `kernel_w`, and the height is `kernel_h`; the kernel stride in the width direction is `stride_w`, and the kernel stride in the height direction is `stride_h`. The relationship between the output feature map width and the input feature map width is... The relationship between the height of the output feature map and the height of the input feature map If they are not equal, the input feature maps need to be padded with zeros according to the specific convolution requirements to make them equal in width and height. The generated results are saved to vrd.

[0324] To reduce the number of times ORAM is loaded into FRM, this can be achieved by generating all results in the same depth direction simultaneously. Therefore, when designing the loop order, the outermost loop is the height of the output feature map, followed by the width of the output feature map, then the depth / 32 of the output feature map, and finally the convolutional computation unit.

[0325] Let fram_h be the number of rows generated each time (where fram_h = fram_count / fram_w). The more rows loaded, the fewer times they are repeatedly loaded in the height direction.

[0326] The specific implementation is as follows:

[0327] wram_id=0;

[0328] oram_id=0;

[0329] fram_id=0;

[0330] wr_fram_id=0;

[0331] rd_fram_id=0;

[0332] ;

[0333] Each processing generates the result of the fram_h line.

[0334] for(int ydir_i =0; ydir_i<(out_height+ fram_h-1); ydir_i += fram_h){

[0335] if (ydir_i >=out_height) fram_h= ydir_i-out_height;

[0336] Data is continuously read from the ddr into the oram, with each read reaching a height of fram_h.

[0337] The address to read from oram is initialized to 0, and the address to write to fram is initialized to 0.

[0338] int rd_oram_idx = 0;

[0339] int wr_fram_idx = 0;

[0340] Each processing step generates 4 pixels in the width direction and generates 4 lines in the height direction.

[0341] for(int xdir_i =0; xdir_i < out_width+3; xdir_i+=4){

[0342] if(xdir_i >1){

[0343] Read data from ORAM into FRM, and read it in the loop body each time. This section loads the data needed for the convolution calculation below. To form a loop, the minimum unit loaded each time is 4 pixels, so the first time it might load more.

[0344] ;

[0345] ;

[0346] for(int iccnum_i =0;icnum_i< in_ic32; icnum_i++){…(5)

[0347] ;

[0348] ;

[0349] for(int fh_i=0;fh_i <fram_h; fh_i++){

[0350] ;

[0351] ;

[0352]

[0353] Use the simd load data command to load data into VR0 and VR1.

[0354] ingenic_load (rd_oram, VR0,0)

[0355] ingenic_load (rd_oram, VR0,1)

[0356] ingenic_load (rd_oram, VR1,0)

[0357] ingenic_load (rd_oram, VR1,1)

[0358] Use the `fram` command to load data into `fram`.

[0359] ingenic_vr2frame(VR0, wr_frame,1)

[0360] ingenic_vr2fram(VR1, wr_fram,1)

[0361] }

[0362] }

[0363] }

[0364] }else if(xdir_i==0){

[0365] The data in oram is read into fram, and the length read in the subsequent loop body is... Therefore, the length of the first read is Read data as .

[0366] rd_oram_idx = rd_oram_idx;

[0367] wr_fram_idx = wr_fram_idx;

[0368] for(int iccnum_i =0;icnum_i< in_ic32; icnum_i++){…(6)

[0369] ;

[0370] ;

[0371] for(int fh_i=0;fh_i <fram_h; fh_i++){

[0372] ;

[0373] ;

[0374]

[0375] Use the simd load data command to load data into VR0 and VR1.

[0376] ingenic_load (rd_oram, VR0,0)

[0377] ingenic_load (rd_oram, VR0,1)

[0378] ingenic_load (rd_oram, VR1,0)

[0379] ingenic_load (rd_oram, VR1,1)

[0380] Use the `fram` command to load data into `fram`.

[0381] ingenic_vr2frame(VR0, wr_frame,1)

[0382] ingenic_vr2fram(VR1, wr_fram,1)

[0383] }

[0384] }

[0385] }

[0386] In xdir_i > 1, what is added each time is... When xdir_i=0, the width of the loaded data is... Therefore, after loading the data with xdir_i=0, the following value should be added:

[0387] ;

[0388] ;

[0389] }

[0390] int wram_id=0;

[0391] Each processing generates

[0392] for(int ocnum_i =0;ocnum_i< out_ic32; ocnum_i++){…(7)

[0393] Each processing generates

[0394] int fram_id=0;

[0395] ;

[0396] for(int fh_i=0;fh_i <fram_h; fh_i++){

[0397] ;

[0398] ingenic_conv_bit8(fram_id,wram_id,ic32_num,kernel_w,

[0399] kernel_h, stride_x, stride_y, vrd);

[0400] Extract the generated results for further processing and saving.

[0401] ;

[0402] }

[0403] }

[0404] }

[0405] }

[0406] (5) Looping is loading data under normal conditions. (6) is loading data for the first time. (7) is the specific implementation of convolution, which first calculates the result generated with a shared weight, that is, first executes the result in height, and then calculates the result in depth / 32 direction, and calculates all the results in sequence.

[0407] In summary, the key points of this application are:

[0408] 1. How to set the FRAM width and how to implement convolution.

[0409] 2. Data transfer method from ORAM to FRAM.

[0410] 3. Specific convolution calculation methods.

[0411] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for implementing convolution with weights that can be stored in WRAM, characterized in that, The method includes the following steps: S1, Data storage: The feature map storage method is set as follows: feature map data storage order is 32, W, H, N; where 32 represents a portion of the depth, W is the width, H is the height, and N is the number of 32 elements at the depth. The depth of the feature map; the data is continuous in 32, then in width, then in height, and finally in the number of depths / 32. The weights are stored using the following method: The sequence is as follows: first, continuous along the width of the convolution kernel; then continuous along the height of the convolution kernel; then continuous along the number of input depths / 32; and finally continuous along the number of output depths / 32. Before processing, the usual input depths need to be continuous, then the width and height of the convolution kernels need to be stored, and finally the output depths of the convolution kernels need to be stored in the required order. S2 uses SIMD instructions to load all data from DDR to WRAM, loading 32 data items at a time: S2.1, use SIMD instructions to load all data from DDR to WRAM, loading 32 data at a time: set the initial address of the weight data to widthdata; Use SIMD load data instructions to load data into VR0 and VR1; Use the WRAM load data instruction to load data into WRAM; Since the weights are stored in the required order and the data size can fit entirely into the wram, they can be stored directly in the default order until all the data is stored. This method cannot be used when the wram cannot hold the data. S2.2, Use the ORAM data transfer instruction to move data from DDR to ORAM: Let the initial address of the feature map be ddr_id, the number of bytes in the feature map be count, and the initial address of the ORAM be oram_id; ingenic_ddr2oram(ddr_id, oram_id, count, 1); Since the feature maps are stored in the required order and the data size can fit entirely into the ORAM, they can be stored directly in the default order until all data is stored. This method cannot be used when the ORAM cannot hold the data or when fram_w cannot hold the smallest computed pixel. S3, Implementation of convolution calculation: S3.1, To compute the convolution, the data needs to be moved from ORAM to FRAM before it can be used for convolution computation; the weights have all been loaded into WRAM, so there is no need to consider the weight transfer situation; however, FRAM cannot hold all the feature maps, so the amount of input data to be transferred from ORAM to FRAM is required. S3.2, Convolution computation first requires loading data from ORAM into FRAM; only then can convolution computation be performed using FRAM and WRAM. Initial addresses for ORAM and WRAM must be provided, both initially set to 0. Let the input feature map depth be... `in_ic32` is a multiple of the input depth, `in_width` is the input width, and `in_height` is the input height; the output feature map depth is... `out_ic32` is a multiple of the output depth, `out_width` is the output width, and `out_height` is the input height; the kernel width is `kernel_w`, and the height is `kernel_h`; the kernel width stride is `stride_w`, and the kernel height stride is `stride_h`; the relationship between the output feature map width and the input feature map width... The relationship between the height of the output feature map and the height of the input feature map If they are not equal, the input feature maps need to be padded with zeros according to the specific convolution requirements to make them equal in width and height; the generated results are saved in vrd. To reduce the number of times ORAM is loaded into FRAM, all results in the same depth direction are generated simultaneously. Therefore, when designing the loop order, the outermost loop is the height of the output feature map, followed by the width of the output feature map, then the depth of the output feature map / 32, and finally the convolutional computation unit. Let fram_h be the number of rows generated each time, where fram_h = fram_count / fram_w; the more rows loaded, the fewer times they are repeatedly loaded in the height direction.

2. The method for implementing WRAM-capable convolution with weights according to claim 1, characterized in that, The method requires that the feature map size can be accommodated in ORAM, the number of weights is less than or equal to wram, WRAM can accommodate them, the bit width is 8 bits, and the feature map data required to generate 8 pixels each time can be completely placed in FRAM. The convolution kernel length or width is no more than 3; at the same time, the input depth is required to be a multiple of 32, and the output depth is also a multiple of 32; if the input depth of some layers in the model is not a multiple of 32, it needs to be padded to a multiple of 32; the corresponding weights are also padded.

3. The method for implementing WRAM-capable convolution with weights according to claim 1, characterized in that, The instructions used in the method are as follows: a) Convolution calculation command: ingenic_conv_bit8(fram_id,wram_id,ic32_num,kernel_w,kernel_h,stride_x,stride_y,feature_w,feature_h,vrd); The input variables are: fram_id (starting address of fram), wram_id (starting address of wram), ic32_num (number of computations), kernel_w (width of convolution kernel), kernel_h (height of convolution kernel), stride_x (stride in x-direction of convolution computation), stride_y (stride in y-direction of convolution computation), feature_w (width of input feature map), feature_h (height of input feature map); and vrd (generated result). Instructions for use: Calculate the result for 4 pixels each time; The calculation unit is 32 pixels deep, and the generated result is also 32 pixels, producing 4 pixels. If ic32_num=1, it calculates an input depth of 32x1 and generates 4 output pixels with a depth of 32; if ic32_num=3, it calculates an input depth of 32x3 and generates 4 output pixels with a depth of 32; if ic32_num=2, it calculates an input depth of 32x2 and generates 4 output pixels with a depth of 32. The minimum input depth is 32, the minimum output depth is 32, and the minimum number of output pixels is 4. Setting the width of the FRAM, which is the number of pixels loaded from the input feature map, is a parameter setting of the convolution calculation instruction; b) SIMD data loading instructions: ingenic_load (indata, VR0, m) Input the data to be loaded. Currently, it is the pointer to the data, indata. Starting from the memory location m pointed to by indata, 128 bits of data are loaded. If it is 8-bit data, 16 bits are loaded; if it is 16-bit data, 8 bits are loaded; if it is 32-bit data, 4 bits are loaded. The data is loaded into the variable vrd register. Here, m is calculated in bytes, i.e., 8 bits per unit. VR0 is the VR register of SIMD, which can store a maximum of 512 bits of data. c) FRAM data loading instructions: ingenic_vr2fram(VR0,fram_load_id,num) Input variables: VR0 is the input data, fram_load_id is the starting address loaded into fram, and num is 0 or 1. When 0, the fram_load_id data remains unchanged after the instruction ends. When 1, fram_load_id = fram_load_id + 32 after the instruction ends. d) WRAM data loading instructions: ingenic_vr2wram(VR0,wram_load_id,num) Input variables: VR0 is the input data, wram_load_id is the starting address loaded into wram, and num is 0 or 1. When 0, the data of wram_load_id remains unchanged after the instruction ends. When 1, wram_load_id = fram_load_id + 64 after the instruction ends. f) ORAM data transfer instructions: ingenic_ddr2oram(ddr_id,oram_id,count,num) Input variables: ddr_id is the address where data begins to be loaded in the ddr, oram_id is the address where data begins to be loaded in the oram, and count is the number of bytes loaded; num is 0 or 1. When 0, the data of ddr_id and oram_id remain unchanged after the instruction ends. When 1, the data of ddr_id and oram_id are incremented by count after the instruction ends.

4. The method for implementing WRAM-capable convolution with weights according to claim 3, characterized in that, The width of the FRAM is set as follows: Let the total number of bytes in the FRAM be fram_count, the width of fram be fram_w, and the number of rows of input feature maps loaded in each processing be fram_h. The value of fram_w is the number of pixels of the input feature map required to generate at least 8 pixels. The minimum number of pixels to be calculated in the generated result is 4. In practice, to generate the first 4 pixels, the feature map data of the next 4 pixels needs to be loaded, so the minimum number of pixels to be generated is 8. The following formula is given. （1） in It uses integer operations and returns integers; the entire formula ensures that fram is a multiple of 4 and can generate 8 pixels. Since each data load is also 4 pixels, for convolutions with a kernel size greater than 1, there will be an overlap between loaded and used data. To resolve this, the extra data is processed according to multiples of 4, i.e., fram_w plus (kernel_w-1) and rounded up. The calculation formula is... ,have （2） Through formulas (1) and (2), we have （3） Formula (3) cannot be combined because it is a rounding operation, and there will be inequalities after combining. The number of rows of the input feature map loaded each time is: fram_h = fram_count / fram_w.

5. The method for implementing WRAM-capable convolution with weights according to claim 4, characterized in that, In S2.1, The data is loaded into VR0 and VR1 using SIMD load instructions, denoted as follows: ingenic_load (widthdata, VR0,0) ingenic_load (widthdata, VR0,1) ingenic_load (widthdata, VR1,0) ingenic_load (widthdata, VR1,1) The WRAM load data instruction is used to load data into WRAM, denoted as ; ingenic_vr2wram(VR0,wram_load_id,1) ingenic_vr2wram(VR1,wram_load_id,1).

6. The method for implementing WRAM-capable convolution with weights according to claim 5, characterized in that, The specific implementation of the S3 convolution calculation is as follows: S3.1, Initialization: denoted as wram_id=0; oram_id=0; fram_id=0; wr_fram_id=0; rd_fram_id=0; ； S3.2, each time the result of generating the fram_h line is processed; Let ydir_i be the height position of the generated result, with an initial value of ydir_i = 0. Let the initial number of processed rows be fram_h_ori, then fram_h = fram_h_ori. If ydir_i < out_height is true, continue executing this step. After this step, ydir_i += fram_h, and check if ydir_i < out_height is true. If true, continue executing this step, and repeat this loop. If not true, check if ydir_i < (out_height + fram_h - 1). Here, out_height may not be an integer multiple of fram_h, where fram_h is greater than 1, so there is a remainder, which needs to be checked again. If true, then fram_h = ydir_i - out_height; execute this step. If not true, exit this step loop. This is denoted as: for(int ydir_i =0; ydir_i<(out_height+ fram_h-1); ydir_i += fram_h); if (ydir_i >=out_height) fram_h= ydir_i-out_height; Execution: Read data continuously from DDR into ORAM, with each read row number being fram_h; Execution: Initialize the address to be read by oram to 0, and initialize the address to be written by fram to 0; Recorded as: int rd_oram_idx = 0; int wr_fram_idx = 0; Execution: S3.2.1, generates 4 pixels in the width direction each time, and generates a fram_h line in the height direction; Let the initial xdir_i = 0. If xdir_i < out_width + 3 is true, here, since there is a case where out_width is not a multiple of 4, there is a remainder. This remainder also needs to be calculated and loaded according to the smallest unit of computation, so out_width needs to be divided by 4 and rounded up, which is equivalent to out_width + 3. The storage space for storing the input feature map will be larger than the actual feature map space to prevent errors caused by data not being read; then continue to execute this step. After this step is executed, xdir_i += 4; check whether xdir_i < out_width + 3 is true. If it is true, repeat this step; if it is not true, jump out of this step and return to S3.2, denoted as: for(int xdir_i =0; xdir_i < out_width+3; xdir_i+=4); Execute step S3.2.1.1: If xdir_i > 1 is true, denoted as if(xdir_i > 1), then continue executing this step; otherwise, execute step S3.2.1.

2. Execution: Read data from ORAM into FRAM, with each read in the loop body... This section loads the data needed for the convolution calculation below. To form a loop, the minimum loading unit is 4 pixels, so the first load will be larger. ; implement: ; implement: ; Execution: Loop body (5); Execute: S3.2.1.2, if xdir_i == 0 holds, then continue to execute this step, denoted as: if(xdir_i == 0); if not, then continue to step S3.2.1.3; Execution: Read the data from ORAM into FRAM. The length read by the subsequent loop body is... Therefore, the length of the first read is Read data as ; Execute: rd_oram_idx = rd_oram_idx; Execute: wr_fram_idx = wr_fram_idx; Execute: loop body (6); Execution: In xdir_i > 1, each time it is added is When xdir_i=0, the width of the loaded data is... Therefore, after loading the data with xdir_i=0, the following value should be added: ； ； Execute: S3.2.1.3, Execute: int wram_id = 0; Execution: Each processing generates ; Execute: loop body (7); Among them, loop body (5) is to load data under normal circumstances; loop body (6) is to load data for the first time; loop body (7) is the specific implementation of convolution. It first calculates the generated results sharing the same weight, that is, first calculates the results in the height direction, and then calculates the results in the depth / 32 direction, and calculates all the results in turn.

7. The method for implementing WRAM-capable convolution with weights according to claim 6, characterized in that, In the said method, The said loop body (5) is as follows: Execute: Step (5)1, set the initial iccnum_i = 0, judge whether icnum_i < in_ic32 holds. If it holds, then continue this loop step. After this loop step, icnum_i++, and then judge whether icnum_i < in_ic32 holds, and loop in turn; if not, then jump out of loop body (5), that is, the loop ends; Denoted as: for(int iccnum_i = 0; icnum_i < in_ic32; icnum_i++) implement: ; implement: ; Execute: Step (5)2, set the initial fh_i = 0, judge whether fh_i < fram_h holds. If it holds, then continue this loop step. After this loop step, fh_i++, and then judge whether fh_i < fram_h holds, and loop in turn; if not, then jump out of this step and return to execute Step (5)1; Denoted as: for(int fh_i = 0; fh_i < fram_h; fh_i++) implement: ; implement: ; Execution: Step (5)3, set the initial fw_i=0, and determine If true, continue the current loop. After this loop step, fw_i += 4, then check again. Check if it is true or false, and repeat the process; if it is not true, skip this step and return to step (5)2. Recorded as: ; Execute: Use the SIMD data loading instruction to load data into VR0 and VR1; denoted as: ingenic_load (rd_oram, VR0, 0); ingenic_load (rd_oram, VR0, 1); ingenic_load (rd_oram, VR1, 0); ingenic_load (rd_oram, VR1, 1); Execute: Use the FRAM data loading instruction to load data into FRAM; denoted as: ingenic_vr2fram(VR0, wr_fram, 1); ingenic_vr2fram(VR1, wr_fram, 1); The said loop body (6) is as follows: Execute: Step (6)1, set the initial iccnum_i = 0, and determine whether icnum_i < in_ic32 holds. If it does not hold, jump out of the loop body (6). If it holds, continue to execute this loop step. After executing this loop step, icnum_i++. Then, determine whether it holds again. If it holds, loop sequentially; otherwise, jump out of this loop and execute the subsequent steps; Denoted as: for(int iccnum_i = 0; icnum_i < in_ic32; icnum_i++) implement: ; implement: ; Execute: Step (6)2: Set the initial fh_i = 0, and determine whether fh_i < fram_h holds. If it does not hold, jump out of Step (6)2. If it holds, continue to execute this loop step. After executing this loop step, fh_i++. Then, determine whether it holds again. If it does not hold, jump out of this loop and execute the subsequent steps; Denoted as: for(int fh_i = 0; fh_i < fram_h; fh_i++) implement: ; implement: ; Execution: Step (6)3, set the initial fw_i=0, and determine If the condition is not met, then step (6)3 is skipped. If the condition is met, then the current loop is executed. After the current loop, fw_i+=4 is executed, and then the condition is checked again. If the condition is met, the loop is repeated. Otherwise, the current loop is skipped and the next step is executed. Denoted as: Execution: Load data into VR0 and VR1 using SIMD load instructions; Denoted as: ingenic_load (rd_oram, VR0, 0); ingenic_load (rd_oram, VR0, 1); ingenic_load (rd_oram, VR1, 0); ingenic_load (rd_oram, VR1, 1) Execute: Use the FRAM data loading instruction to load data into FRAM; Denoted as: ingenic_vr2fram(VR0, wr_fram, 1); ingenic_vr2fram(VR1, wr_fram, 1); The loop body (7) is as follows: Execute Step (7)1: Set the initial ocnum_i = 0, and determine whether ocnum_i < out_ic32 holds. If it does not hold, jump out of Step (7)1. If it holds, continue to execute this loop step. After this loop step, ocnum_i++. Then, determine whether it holds again. Loop sequentially; otherwise, jump out of this loop and execute the subsequent steps: Denoted as: for(int ocnum_i = 0; ocnum_i < out_ic32; ocnum_i++) Execution: Each processing generates ; Execute: fram_id = 0; implement: ; Execute Step (7)2: Set the initial fh_i = 0, and determine whether fh_i < fram_h holds. If it does not hold, jump out of Step (7)2. If it holds, continue to execute this loop step. After this loop step, fh_i++. Then, determine whether it holds again. If it holds, loop sequentially; otherwise, jump out of this loop; Denoted as: Denoted as: for(int fh_i = 0; fh_i < fram_h; fh_i++) implement: ; Execute: ingenic_conv_bit8(fram_id, wram_id, ic32_num, kernel_w, kernel_h, stride_x, stride_y, vrd); Execution: Retrieve the generated results for further processing and saving; implement: .

Images