A method for implementing a dilated convolution based on a chip that does not support the dilated convolution
By splitting dilated convolution into multiple 1*1 ordinary convolutions and accelerating the computation on the T32 chip, the problem that the T32 chip cannot directly implement dilated convolution is solved, and support for dilated convolution is realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEFEI JUNZHENG TECH CO LTD
- Filing Date
- 2024-12-23
- Publication Date
- 2026-06-23
AI Technical Summary
The existing T32 model chip does not support dilated convolution acceleration and cannot directly implement dilated convolution operations because it does not have a register to represent the number of zeros filled between adjacent elements of the convolution kernel.
The dilated convolution is split into multiple 1*1 ordinary convolutions. Each 1*1 ordinary convolution is accelerated by using a T32 chip, and the results are accumulated to obtain the result of the dilated convolution.
It achieves dilated convolution acceleration for T32 chips, expands their convolution operation capabilities, and supports dilated convolution operations with kernel sizes greater than 7*7.
Smart Images
Figure CN122263967A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of neural network computing technology, and specifically relates to a method for implementing dilated convolution based on a chip that does not support dilated convolution. Background Technology
[0002] In existing technologies, neural networks are mathematical models that mimic the structure and function of biological neural networks. By learning the inherent patterns in training sample data, they gain the ability to analyze or represent sample data, and can be applied to various fields such as object detection, scene classification, and character recognition. Among these, NNA (Neural Network Accelerator) is frequently used. Its single operation instruction can multiply and accumulate the input image data and the convolution kernel data, thus improving the speed of convolution operations. Convolution, for each pixel in the input image data, calculates the product of its neighboring pixels and the corresponding element of the convolution kernel, and accumulates the products to obtain the pixel value of the corresponding position in the output image. By continuously sliding the convolution kernel, the output image data is obtained. Dilated convolution fills the adjacent elements of the convolution kernel with zeros, and the dilation coefficient controls the size of the expanded convolution kernel, thereby extracting multi-scale information. The larger the dilation coefficient, the larger the expanded convolution kernel.
[0003] In the prior art, input image data is written to FRAM, convolution kernel data is written to WRAM, NNA register is configured, and NNA is started to obtain output image data.
[0004] In addition, the T32 chip independently developed by Beijing Junzheng Integrated Circuit Co., Ltd. (hereinafter referred to as Junzheng) supports convolution acceleration, but the T32 chip only supports ordinary convolution acceleration with a kernel size of less than or equal to 7*7. It sets the kernel size by configuring the mac_kernel register. The register width is 8 bits, where bit[3:0] represents the kernel width and bit[7:4] represents the kernel height.
[0005] However, chips that do not support dilated convolution acceleration, such as the Ingenic T32 chip, cannot directly implement dilated convolution. For example, the T32 chip itself does not support dilated convolution, lacks the relevant registers to configure the dilation coefficient, and only supports ordinary convolution. This is because dilated convolution requires padding adjacent elements of the kernel with zeros, but the T32 chip lacks registers to indicate how many zeros to use between adjacent elements of the kernel.
[0006] In addition, the terminology commonly used in this technology includes:
[0007] dilate: The extent of kernel expansion, also called the dilation coefficient, is divided into dilate_x and dilate_y, which represent the extent of horizontal and vertical expansion of the kernel.
[0008] FRAM: The on-chip RAM inside the NNA stores the input image data. Its size is N KB, where N is determined according to the specific chip definition and is generally less than or equal to 128KB.
[0009] WRAM: On-chip RAM inside the NNA that stores convolution kernel data. Its size is M KB, where M is determined by the specific chip definition and is generally less than or equal to 64KB.
[0010] kernel: The size of the convolution kernel, divided into kernel_x and kernel_y, representing the horizontal and vertical sizes of the convolution kernel.
[0011] stride: The stride of the convolution kernel, divided into stride_x and stride_y, representing the stride of the convolution kernel in the horizontal and vertical directions. Summary of the Invention
[0012] To address the aforementioned issues, the purpose of this application is as follows: Chips like the T32 cannot directly implement dilated convolution; the T32 only supports acceleration of ordinary convolutions with kernel sizes less than or equal to 7*7. This application aims to solve the problem of enabling the T32 to support dilated convolution acceleration. Dilated convolution and ordinary convolution are different types of convolution. The solution proposed in this application is to split the expanded kernel of the dilated convolution into kh*kw 1*1 ordinary convolutions. The T32 is then used to accelerate the calculation of each 1*1 ordinary convolution, and the results of the kh*kw convolutions are summed to obtain the result of the dilated convolution.
[0013] Specifically, the present invention provides a method for implementing dilated convolution based on a chip that does not support dilated convolution, the method comprising:
[0014] If the dilated convolution is broken down into kh*kw ordinary convolutions of size 1*1, the calculation process is as follows:
[0015]
[0016] The kernel sizes are kh and kw, the strides are sh and sw, and the dilation coefficients are dy and dx.
[0017] b[p][q] is the data of the non-zero-filled part of the expanded convolutional kernel, and a[(p-1)*dy+1][(q-1)*dx+1] is the corresponding data of the expanded convolutional kernel on the input image;
[0018] The process iterates kh*kw times, writing the corresponding input image data to FRAM and the corresponding convolution kernel data to WRAM each time. The mac_kernel register is configured to set the convolution kernel size to 1*1. NNA is started to obtain the current 1*1 convolution output result. The convolution output results of kh*kw times are accumulated to obtain the output image data.
[0019] The process involves accumulating the output results of kh*kw convolutions. Specifically, the first 1*1 ordinary convolution result is stored in a pre-allocated buffer. Subsequently, the previously accumulated result is retrieved, added to the current 1*1 ordinary convolution result, and stored in the same buffer. The T32 has registers that can configure the buffer storage address of the convolution result. It also supports reading data from a specified buffer address, accumulating it with the current convolution result, and continuing to store it in the specified buffer address until the output image data is obtained.
[0020] The method further includes:
[0021] S1, assuming the input image size is NxN, the original convolution kernel size kh=m, kw=m, stride sh=1, sw=1, dilation coefficient dy=n, dx=n, and the dilated convolution kernel size is d_kh and d_kw: d_kh=n*(m-1)+1=mn-n+1;
[0022] d_kw=n*(m-1)+1=mn-n+1;
[0023] The input image data is shown in Table a below:
[0024] Table a:
[0025]
[0026]
[0027] The original convolution kernel data is shown in Table b below:
[0028] Table b:
[0029] K11 K12 ... K1m K21 K22 ... K2m ... ... ... Km1 Km2 ... Kmm
[0030] The dilated convolution kernel data is padded with (dilation coefficient - 1) zeros between adjacent elements of the original convolution kernel. When calculating the multiplication and accumulation of corresponding elements of the dilated convolution kernel and the input image, the zero-filled parts of the dilated convolution kernel data are not calculated. The biggest difference between dilated convolution and ordinary convolution is that the corresponding element positions of the non-zero parts of the convolution kernel data in the input image are different, so the multiplication and accumulation results are different; as shown in Table c below:
[0031] Table c:
[0032] K11 0 ... K12 0 ... K1m 0 0 ... 0 0 ... 0 ... ... ... ... ... ... ... K21 0 ... K22 0 ... K2m 0 0 ... 0 0 ... 0 ... ... ... ... ... ... ... Km1 0 ... Km2 0 ... Kmm
[0033] S2, the dilated convolution is split into kh*kw ordinary convolutions of size 1*1 to obtain the first point of the output image data. Other points are obtained by sliding the dilated convolution kernel to the right / down. The calculation process is as follows:
[0034] The loop iterates kh*kw times. On the first iteration, a11 is written to FRAM and K11 is written to WRAM. The mac_kernel register is configured to set the convolution kernel size to 1*1. Once NNA is started, the first 1*1 convolution output result can be obtained.
[0035] For the kwth time, a1k is written to FRAM, K1m is written to WRAM, and the mac_kernel register is configured to set the convolution kernel size to 1*1; NNA is started, and the kwth 1*1 convolution output result can be obtained;
[0036] And so on...
[0037] The kh*kwth time, akk is written to FRAM and Kmm is written to WRAM. The mac_kernel register is configured to set the convolution kernel size to 1*1. NNA is started, and the kh*kwth 1*1 convolution output result can be obtained.
[0038] The outputs of kh*kw convolutions are summed to obtain the first point of the output image data. Other points are obtained by sliding the dilated convolution kernel to the right / down.
[0039] y=a11*K11+...+a1k*K1m+...+ak1*Km1+...+akk*Kmm.
[0040] In the method described, the difference between dilated convolution and ordinary convolution is that in ordinary convolution, assuming the kernel size is m*m and there are no zeros between adjacent elements, dilated convolution expands the original kernel and fills the spaces between adjacent elements with zeros. The number of zeros is determined by the dilation coefficient: dx-1 zeros are filled between horizontally adjacent elements, and dy-1 zeros are filled between vertically adjacent elements. The size of the expanded kernel is d_kh*d_kw. The expanded kernel is multiplied and summed with the corresponding elements of the input image to obtain the first point of the dilated convolution output image data. As the expanded kernel slides right / down on the input image, all points of the dilated convolution output image are obtained; where k = 1,...,d_kh.
[0041] The method further includes:
[0042] S1, assuming the input image size is 8x8, the original convolutional kernel size kh = 3, kw = 3, stride sh = 1, sw = 1, dilation coefficient dy = 2, dx = 2, and the dilated convolutional kernel size is d_kh = 5, d_kw = 5; the input image data is shown in Table 1-1 below:
[0043] Table 1-1:
[0044] a11 a13 a15 a31 a33 a35 a51 a53 a55
[0045] The original convolution kernel data are shown in Table 1-2 below:
[0046] Table 1-2:
[0047] K11 K12 K13 K21 K22 K23 K31 K32 K33
[0048] The expanded convolutional kernel data are shown in Table 1-3 below:
[0049] Table 1-3:
[0050] K11 0 K12 0 K13 0 0 0 0 0 K21 0 K22 0 K23 0 0 0 0 0 K31 0 K32 0 K33
[0051] S2, the dilated convolution is split into 3*3 ordinary convolutions of size 1*1. To obtain the first point of the output image data, the calculation process is as follows:
[0052] The loop repeats 9 times. On the first iteration, a11 is written to FRAM, K11 is written to WRAM, and the mac_kernel register is configured to set the convolution kernel size to 1*1. Once NNA is started, the first 1*1 convolution output result can be obtained.
[0053] The second time, a13 is written to FRAM, K12 is written to WRAM, and the mac_kernel register is configured to set the convolution kernel size to 1*1; NNA is started, and the second 1*1 convolution output result can be obtained;
[0054] And so on...
[0055] The a55 is written to FRAM for the ninth time, the K33 is written to WRAM, and the mac_kernel register is configured to set the convolution kernel size to 1*1; NNA is started, and the ninth 1*1 convolution output result can be obtained;
[0056] The outputs of the nine convolutions are summed to obtain the output image data:
[0057] y=a11*K11+a13*K12+a15*K13+a31*K21+a33*K22+a35*K23+a51*K31+a53*K32+a55*K33.
[0058] The ordinary convolution:
[0059] Assuming the input image data is a, the convolution kernel data is b, the output image data is y, the convolution kernel sizes are kh and kw, and the strides are sh and sw, the calculation process is as follows:
[0060]
[0061] i and j are the indices of each element of the convolution kernel. Ordinary convolution is the multiplication and summation of the convolution kernel and the corresponding elements of the input image.
[0062] Write the input image data to FRAM, write the convolution kernel data to WRAM, configure the mac_kernel register to set the convolution kernel size. The register has a width of 8 bits, where bits [3:0] represent the kernel width and bits [7:4] represent the kernel height. Start NNA to obtain the output image data.
[0063] The dilated convolution:
[0064] Assume the input image data is a, the convolution kernel data is b, the output image data is y, the convolution kernel sizes are kh and kw, the stride is sh and sw, and the dilation coefficients are dy and dx. The original convolution kernel is padded with dy-1 zeros between vertically adjacent elements and with dx-1 zeros between horizontally adjacent elements. The dilated convolution kernel data is d_b, and the dilated convolution kernel sizes are d_kh and d_kw.
[0065] d_kh=dy*(kh-1)+1
[0066] d_kw=dx*(kw-1)+1
[0067] The calculation process is as follows:
[0068]
[0069] The chips that do not support dilated convolution include the Ingenic T32 chip. The T32 chip supports ordinary convolution operations with a kernel size of less than or equal to 7x7. It sets the kernel size by configuring the mac_kernel register, which has a bit width of 8 bits. Bits [3:0] represent the kernel width and bits [7:4] represent the kernel height. The T32 can only represent the kernel size of ordinary convolution. For dilated convolution, since it is necessary to fill (diffusion coefficient - 1) zeros between adjacent elements of the original convolution kernel, and the T32 does not have a register to represent the difusion coefficient, it only supports ordinary convolution and does not support dilated convolution.
[0070] Therefore, the advantage of this application is:
[0071] By splitting dilated convolution into ordinary convolution and using NNA to accelerate ordinary convolution calculations, the dilated convolution calculation is accelerated. Since chips like the T32 cannot directly implement dilated convolution, as dilated convolution and ordinary convolution are different types of convolution, for example, the T32 only supports acceleration of ordinary convolution with a kernel size of less than or equal to 7*7. This invention splits the expanded convolution kernel of dilated convolution into kh*kw 1*1 ordinary convolutions, accelerates the calculation of each 1*1 ordinary convolution using the T32, and accumulates the results of the kh*kw convolutions to obtain the result of dilated convolution, thus enabling the T32 chip to support dilated convolution acceleration. Attached Figure Description
[0072] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.
[0073] Figure 1 This is a flowchart illustrating the method. Detailed Implementation
[0074] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.
[0075] First, ordinary convolution: Assuming the input image data is a, the convolution kernel data is b, the output image data is y, the convolution kernel sizes are kh and kw, and the stride is sh and sw, the calculation process is as follows:
[0076]
[0077] Where i and j are the indices of each element of the convolution kernel, and ordinary convolution is the multiplication and summation of the convolution kernel and the corresponding elements of the input image;
[0078] Write the input image data to FRAM, write the convolution kernel data to WRAM, configure the mac_kernel register to set the convolution kernel size. The register has a width of 8 bits, where bits [3:0] represent the kernel width and bits [7:4] represent the kernel height. Start NNA to obtain the output image data.
[0079] For example, if the kernel size kh = 3, kw = 3, stride sh = 1, sw = 1, and the input image size is 8x8, the input image data is shown in Table 1 below:
[0080] Table 1:
[0081]
[0082] The convolution kernel data is shown in Table 2 below:
[0083] Table 2:
[0084] K11 K12 K13 K21 K22 K23 K31 K32 K33
[0085] The first point of the output image data is the sum of the multiplication of the convolution kernel data and the input image data (shaded area in Table 1); the convolution kernel slides to the right with a stride of 1 to obtain the data of the horizontal points of the output image; the convolution kernel slides down with a stride of 1 to obtain the data of the vertical points of the output image. As the convolution kernel slides horizontally and vertically, the data of all points in the output image are obtained.
[0086] Secondly, dilated convolution: Assume the input image data is a, the kernel data is b, the output image data is y, the kernel sizes are kh and kw, the stride is sh and sw, and the dilation coefficients are dy and dx. The original kernel is padded with dy-1 zeros between vertically adjacent elements and with dx-1 zeros between horizontally adjacent elements. The dilated kernel data is d_b, and the dilated kernel sizes are d_kh and d_kw.
[0087] d_kh=dy*(kh-1)+1
[0088] d_kw=dx*(kw-1)+1
[0089] The calculation process is as follows:
[0090]
[0091] Examples of dilated convolution:
[0092] If the input image size is 8x8, the original convolutional kernel size kh = 3, kw = 3, stride sh = 1, sw = 1, dilation coefficient dy = 2, dx = 2, and the dilated convolutional kernel size is d_kh = 5, d_kw = 5. The input image data is shown in Table 3 below:
[0093] Table 3:
[0094]
[0095] The original convolution kernel data is shown in Table 4 below:
[0096] Table 4:
[0097] K11 K12 K13 K21 K22 K23 K31 K32 K33
[0098] The expanded convolutional kernel data are shown in Table 5 below:
[0099] Table 5:
[0100] K11 0 K12 0 K13 0 0 0 0 0 K21 0 K22 0 K23 0 0 0 0 0 K31 0 K32 0 K33
[0101] The first point of the output image data is the sum of the multiplication of the dilated convolution kernel data and the input image data (shaded area in Table 3). The dilated convolution kernel slides to the right with a stride of 1 to obtain the data of the horizontal points of the output image. The dilated convolution kernel slides down with a stride of 1 to obtain the data of the vertical points of the output image. As the dilated convolution kernel slides horizontally and vertically, the data of all points in the output image are obtained.
[0102] However, since the T32 chip cannot directly implement dilated convolution, it only supports acceleration of ordinary convolution with a kernel size of 7*7 or less. The problem this application aims to solve is how to enable the T32 to support dilated convolution acceleration. Dilated convolution and ordinary convolution are different types of convolution. Therefore, the proposed solution is to split the expanded kernel of dilated convolution into kh*kw 1*1 ordinary convolutions. The T32 is used to accelerate the calculation of each 1*1 ordinary convolution. The results of the kh*kw convolutions are accumulated to obtain the result of dilated convolution. This is independent of the dilation coefficient. The T32 itself does not support dilated convolution and has no related registers to configure the dilation coefficient. It only supports ordinary convolution. Dilated convolution requires filling (dilation coefficient - 1) zeros between adjacent elements of the original convolution kernel, but the T32 has no register to represent the dilation coefficient.
[0103] In summary, this application proposes a method for implementing dilated convolution based on a chip that does not support dilated convolution, such as the T32 chip, and further includes:
[0104] To convert dilated convolution to regular convolution, the T32 only supports regular convolution and does not have a register to represent the dilation coefficient, so it does not support dilated convolution. Therefore, it is split by decomposing the dilated convolution into multiple 1x1 regular convolutions:
[0105] If the dilated convolution is broken down into kh*kw ordinary convolutions of size 1*1, the calculation process is as follows: The formula for ordinary convolution...
[0106]
[0107] Formula for 1x1 ordinary convolution
[0108]
[0109] The dilated convolution is decomposed into kh*kw 1*1 ordinary convolutions, and the results of these kh*kw 1*1 ordinary convolutions are summed to obtain the result of the dilated convolution, as shown in the formula below:
[0110]
[0111] The kernel sizes are kh and kw, the strides are sh and sw, and the dilation coefficients are dy and dx.
[0112] b[p][q] is the data of the non-zero-filled part of the expanded convolutional kernel, and a[(p-1)*dy+1][(q-1)*dx+1] is the corresponding data of the expanded convolutional kernel on the input image;
[0113] The loop iterates kh*kw times, writing the corresponding input image data to FRAM and the corresponding convolution kernel data to WRAM each time. The convolution kernel size is 1*1, which is the data of the non-zero-filled part of the expanded convolution kernel, that is, the original convolution kernel data. Dilated convolution is the expansion of the convolution kernel, filling (dilation coefficient - 1) zeros between adjacent elements of the original convolution kernel. Configure the mac_kernel register to set the convolution kernel size to 1*1. Start NNA to get the current 1*1 convolution output result. Accumulate the convolution output results of kh*kw times, that is, store the first 1*1 ordinary convolution result in a pre-allocated buffer. In each subsequent iteration, retrieve the previous accumulated result, add it to the current 1*1 ordinary convolution result, and store it in the same buffer. The T32 has registers to configure the buffer storage address of the convolution result, and also supports reading data at a specified buffer address, accumulating it with the current convolution result, and continuing to store it in a specified buffer address to obtain the output image data.
[0114] like Figure 1 As shown, it further includes:
[0115] S1, assuming the input image size is NxN, the original convolution kernel size kh=m, kw=m, stride sh=1, sw=1, dilation coefficient dy=n, dx=n, and the dilated convolution kernel size is d_kh and d_kw: d_kh=n*(m-1)+1=mn-n+1;
[0116] d_kw=n*(m-1)+1=mn-n+1;
[0117] The input image data is shown in Table a below:
[0118] Table a:
[0119]
[0120]
[0121] The original convolution kernel data is shown in Table b below:
[0122] Table b:
[0123] K11 K12 ... K1m K21 K22 ... K2m ... ... ... Km1 Km2 ... Kmm
[0124] The expanded convolutional kernel data is shown in Table c below:
[0125] Table c:
[0126] K11 0 ... K12 0 ... K1m 0 0 ... 0 0 ... 0 ... ... ... ... ... ... ... K21 0 ... K22 0 ... K2m 0 0 ... 0 0 ... 0 ... ... ... ... ... ... ... Km1 0 ... Km2 0 ... Kmm
[0127] S2, the dilated convolution is split into kh*kw ordinary convolutions of size 1*1 to obtain the first point of the output image data. As the dilated convolution kernel slides to the right / down on the input image, the data of all output points is obtained.
[0128] The calculation process for obtaining the first point of the output image data is as follows:
[0129] The loop iterates kh*kw times. On the first iteration, a11 is written to FRAM and K11 is written to WRAM. The mac_kernel register is configured to set the convolution kernel size to 1*1. Once NNA is started, the first 1*1 convolution output result can be obtained.
[0130] For the kwth time, a1k is written to FRAM, K1m is written to WRAM, and the mac_kernel register is configured to set the convolution kernel size to 1*1; NNA is started, and the kwth 1*1 convolution output result can be obtained;
[0131] And so on...
[0132] The kh*kwth time, akk is written to FRAM and Kmm is written to WRAM. The mac_kernel register is configured to set the convolution kernel size to 1*1. NNA is started, and the kh*kwth 1*1 convolution output result can be obtained.
[0133] The first point of the output image data is obtained by summing the output results of kh*kw 1*1 convolutions: y=a11*K11+...+a1k*K1m+...+ak1*Km1+...+akk*Kmm.
[0134] For example: If the input image size is 8x8, the original convolution kernel size kh=3, kw=3, stride sh=1, sw=1, dilation coefficient dy=2, dx=2, the dilated convolution kernel size is d_kh=5, d_kw=5;
[0135] The input image data is shown in Table 1-1 below:
[0136] Table 1-1:
[0137] a11 a13 a15 a31 a33 a35 a51 a53 a55
[0138] The original convolution kernel data are shown in Table 1-2 below:
[0139] Table 1-2:
[0140] K11 K12 K13 K21 K22 K23 K31 K32 K33
[0141] The expanded convolutional kernel data are shown in Table 1-3 below:
[0142] Table 1-3:
[0143] K11 0 K12 0 K13 0 0 0 0 0 K21 0 K22 0 K23 0 0 0 0 0 K31 0 K32 0 K33
[0144] The dilated convolution is split into 3*3 ordinary convolutions of size 1*1. The original convolution kernel size of the dilated convolution is 3*3. After expansion, the adjacent elements of the original convolution kernel are filled with 0 (fill with the dilation coefficient - 1 0). The T32 chip does not have a register to represent the dilation coefficient and does not support dilated convolution. Therefore, the expanded convolution kernel is split into kh*kw (that is, 3*3) 1*1 ordinary convolutions for calculation. The results of these kh*kw (9) times are accumulated to obtain the first point of the dilated convolution output. As the expanded convolution kernel slides to the right / down on the input image, the data of all output points are obtained.
[0145] To obtain the first point of the output image data, the calculation process is as follows:
[0146] The loop repeats 9 times. On the first iteration, a11 is written to FRAM, K11 is written to WRAM, and the mac_kernel register is configured to set the convolution kernel size to 1*1. Once NNA is started, the first 1*1 convolution output result can be obtained.
[0147] The second time, a13 is written to FRAM, K12 is written to WRAM, and the mac_kernel register is configured to set the convolution kernel size to 1*1; NNA is started, and the second 1*1 convolution output result can be obtained;
[0148] And so on...
[0149] The a55 is written to FRAM for the ninth time, the K33 is written to WRAM, and the mac_kernel register is configured to set the convolution kernel size to 1*1; NNA is started, and the ninth 1*1 convolution output result can be obtained;
[0150] The output results of nine 1*1 convolutions are summed to obtain the first point of the output image data: y = a11*K11 + a13*K12 + a15*K13 + a31*K21 + a33*K22 + a35*K23 + a51*K31 + a53*K32 + a55*K33.
[0151] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for implementing dilated convolution based on a chip that does not support dilated convolution, characterized in that, The method includes: If the dilated convolution is broken down into kh*kw ordinary convolutions of size 1*1, the calculation process is as follows: The original convolutional kernel sizes are kh and kw, the stride is sh and sw, and the dilation coefficients are dy and dx; b[p][q] is the data of the non-zero-filled part of the dilated convolutional kernel, and a[(p-1)*dy+1][(q-1)*dx+1] is the corresponding data of the dilated convolutional kernel on the input image; The process iterates kh*kw times, writing the corresponding input image data to FRAM and the corresponding convolution kernel data to WRAM each time. The mac_kernel register is configured to set the convolution kernel size to 1*1. NNA is started to obtain the current 1*1 convolution output result. The convolution output results of kh*kw times are accumulated to obtain the output image data.
2. The method for implementing dilated convolution based on a chip that does not support dilated convolution according to claim 1, characterized in that, The process involves accumulating the output results of kh*kw convolutions. Specifically, the first 1*1 ordinary convolution result is stored in a pre-allocated buffer. Subsequently, the previously accumulated result is retrieved, added to the current 1*1 ordinary convolution result, and stored in the same buffer. The chip has registers that can configure the buffer storage address of the convolution result. It also supports reading data from a specified buffer address, accumulating it with the current convolution result, and continuing to store it in the specified buffer address until the output image data is obtained.
3. The method for implementing dilated convolution based on a chip that does not support dilated convolution according to claim 1, characterized in that, The method further includes: S1, assuming the input image size is NxN, the original convolution kernel size kh=m, kw=m, stride sh=1, sw=1, dilation coefficient dy=n, dx=n, and the size of the dilated convolution kernel is d_kh and d_kw: d_kh=n*(m-1)+1=mn-n+1; d_kw=n*(m-1)+1=mn-n+1; The input image data is shown in Table a below: Table a: The original convolution kernel data is shown in Table b below: Table b: The dilated convolution kernel data is padded with (dilation coefficient - 1) zeros between adjacent elements of the original convolution kernel. When calculating the multiplication and accumulation of corresponding elements of the dilated convolution kernel and the input image, the zero-filled parts of the dilated convolution kernel data are not calculated. The biggest difference between dilated convolution and ordinary convolution is that the corresponding element positions of the non-zero parts of the convolution kernel data in the input image are different, so the multiplication and accumulation results are different; as shown in Table c below: Table c: S2, the dilated convolution is split into kh*kw ordinary convolutions of size 1*1. First, the first point of the output image data is obtained, and the other points are obtained by sliding the dilated convolution kernel to the right / down. The calculation process is as follows: The loop iterates kh*kw times. On the first iteration, a11 is written to FRAM and K11 is written to WRAM. The mac_kernel register is configured to set the convolution kernel size to 1*1. Once NNA is started, the first 1*1 convolution output result can be obtained. For the kwth time, a1k is written to FRAM, K1m is written to WRAM, and the mac_kernel register is configured to set the convolution kernel size to 1*1; NNA is started, and the kwth 1*1 convolution output result can be obtained; And so on... The kh*kwth time, akk is written to FRAM and Kmm is written to WRAM. The mac_kernel register is configured to set the convolution kernel size to 1*1. NNA is started, and the kh*kwth 1*1 convolution output result can be obtained. The outputs of kh*kw convolutions are summed to obtain the first point of the output image data. Other points are obtained by sliding the dilated convolution kernel to the right / down. y=a11*K11+...+a1k*K1m+...+ak1*Km1+...+akk*Kmm.
4. The method for implementing dilated convolution based on a chip that does not support dilated convolution according to claim 3, characterized in that, In the method described, the difference between dilated convolution and ordinary convolution is that in ordinary convolution, assuming the kernel size is m*m and there are no zeros between adjacent elements, dilated convolution expands the original kernel and fills the spaces between adjacent elements with zeros. The number of zeros is determined by the dilation coefficient: dx-1 zeros are filled between horizontally adjacent elements, and dy-1 zeros are filled between vertically adjacent elements. The size of the expanded kernel is d_kh*d_kw. The expanded kernel is multiplied and summed with the corresponding elements of the input image to obtain the first point of the dilated convolution output image data. As the expanded kernel slides right / down on the input image, all points of the dilated convolution output image are obtained; where k = 1,...,d_kh.
5. A method for implementing dilated convolution based on a chip that does not support dilated convolution according to claim 3, characterized in that, The method further includes: S1, assuming the input image size is 8x8, the original convolutional kernel size kh = 3, kw = 3, stride sh = 1, sw = 1, dilation coefficient dy = 2, dx = 2, and the dilated convolutional kernel size is d_kh = 5, d_kw = 5; the input image data is shown in Table 1-1 below: Table 1-1: The original convolution kernel data are shown in Table 1-2 below: Table 1-2: The expanded convolutional kernel data are shown in Table 1-3 below: Table 1-3: S2, the dilated convolution is split into 3*3 ordinary convolutions of size 1*1. To obtain the first point of the output image data, the calculation process is as follows: The loop repeats 9 times. On the first iteration, a11 is written to FRAM, K11 is written to WRAM, and the mac_kernel register is configured to set the convolution kernel size to 1*1. Once NNA is started, the first 1*1 convolution output result can be obtained. The second time, a13 is written to FRAM, K12 is written to WRAM, and the mac_kernel register is configured to set the convolution kernel size to 1*1; NNA is started, and the second 1*1 convolution output result can be obtained; And so on... The a55 is written to FRAM for the ninth time, the K33 is written to WRAM, and the mac_kernel register is configured to set the convolution kernel size to 1*1; NNA is started, and the ninth 1*1 convolution output result can be obtained; The outputs of the nine convolutions are summed to obtain the output image data: y=a11*K11+a13*K12+a15*K13+a31*K21+a33*K22+a35*K23+a51*K31+a53*K32+a55*K33.
6. The method for implementing dilated convolution based on a chip that does not support dilated convolution according to claim 1, characterized in that, The ordinary convolution: Assuming the input image data is a, the convolution kernel data is b, the output image data is y, the convolution kernel sizes are kh and kw, and the strides are sh and sw, the calculation process is as follows: i and j are the indices of each element of the convolution kernel. Ordinary convolution is the multiplication and summation of the convolution kernel and the corresponding elements of the input image. Write the input image data to FRAM, write the convolution kernel data to WRAM, configure the mac_kernel register to set the convolution kernel size. The register has a width of 8 bits, where bits [3:0] represent kw and bits [7:4] represent kh. Start NNA to obtain the output image data.
7. A method for implementing dilated convolution based on a chip that does not support dilated convolution according to claim 1, characterized in that, The dilated convolution: Assume the input image data is a, the convolution kernel data is b, the output image data is y, the convolution kernel sizes are kh and kw, the stride is sh and sw, and the dilation coefficients are dy and dx. The original convolution kernel is padded with dy-1 zeros between vertically adjacent elements and with dx-1 zeros between horizontally adjacent elements. The dilated convolution kernel data is d_b, and the dilated convolution kernel sizes are d_kh and d_kw. d_kh=dy*(kh-1)+1 d_kw=dx*(kw-1)+1 The calculation process is as follows:
8. A method for implementing dilated convolution based on a chip that does not support dilated convolution according to claim 1, characterized in that, The chips that do not support dilated convolution include the Ingenic T32 chip. The T32 chip supports ordinary convolution operations with a kernel size of less than or equal to 7x7. It sets the kernel size by configuring the mac_kernel register, which has a bit width of 8 bits. Bits [3:0] represent the kernel width and bits [7:4] represent the kernel height. The T32 chip does not have a register to represent the dilation coefficient, so it only supports ordinary convolution and does not support dilated convolution.