Efficient fault-tolerant convolution method for memory

Abstract

The invention relates to a high-efficiency fault-tolerant convolution method for a memory, which comprises the following steps of: (1) memory optimization: performing memory optimization conversion on an input feature tensor I to generate an intermediate tensor L; (2) block matrix parallel multiplication: dividing the intermediate tensor L and the convolution kernel tensor F into sub-matrixes according to the convolution kernel height kh and the output height oh, and executing matrix multiplication GEMM in parallel to calculate an output tensor O; (3) multi-batch processing: reconstructing an intermediate tensor L in multi-batch convolution by adopting a batch mixed arrangement mode; and (4) fault-tolerant check and error correction: dividing the input feature tensor I into in element groups, calculating checksums Co1 and Co2 for error detection and error correction and output checksums So1 and So2, and positioning and correcting error element values by comparing Co1 with So1 and comparing Co2 with So2. According to the method, the memory consumption can be remarkably reduced while the calculation rate is guaranteed, error elements caused by attacks can be accurately detected and corrected, and the method is suitable for deep learning model reasoning adopting multi-batch convolution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a memory-efficient fault-tolerant convolution method. BACKGROUND

[0002] The reason why the convolution calculation is low in efficiency is that the convolution kernel involves a large number of cross-row accesses when sliding on the input features, which leads to a decrease in cache hit rate, and further affects the overall calculation performance. Since the convolution operation has certain similarity with matrix multiplication in data access and calculation mode, matrix multiplication has more advantages in cache utilization, and there are various optimization methods and efficient parallel BLAS (Basic Linear Algebra Subprograms) libraries available for its acceleration. Therefore, the Im2col method realizes the conversion of the convolution operation into matrix multiplication, so as to use the highly optimized BLAS library to improve the calculation rate. The mainstream deep learning frameworks such as PyTorch, Darknet and TensorFlow all adopt the Im2col convolution method, which accelerates the convolution operation through the optimized general matrix multiplication, thereby improving the execution speed of the deep learning model in the training and inference stages.

[0003] The ABFT (Algorithm-Based Fault Tolerance) based method is a system-level fault-tolerant mechanism, which realizes the detection and correction of errors in the calculation process by encoding data and designing algorithms that can process the encoded data. This method is particularly suitable for matrix calculation tasks such as matrix addition, multiplication, LU decomposition and transposition operations. By designing a reasonable encoding and verification mechanism, the algorithm can accurately detect the specific element position of the error in the convolution calculation, and realize fine-grained error positioning and correction of the convolution operation.

[0004] In the prior art, the optimization scheme of the three common convolution methods is as follows:

[0005] I. Convolution optimization technology based on Im2col method. Im2col is a method of converting convolution operation into matrix multiplication, which flattens image data into a matrix, so that convolution calculation is converted into a matrix multiplication problem, thereby using the existing matrix multiplication library GEMM (General Matrix Multiply) to accelerate the calculation. The core advantage of this method is that it can fully utilize the efficient matrix multiplication implementation to significantly improve the efficiency of convolution operation. In recent years, to further optimize the deficiencies of Im2col in memory occupation and access efficiency, Lu et al. proposed the Im2win variant method. Im2win restructures the sliding window structure of the input image to avoid redundant data duplication, realizes continuous memory access and higher data reuse rate, and is particularly suitable for SIMD architecture, which reduces the memory occupation while improving the overall calculation performance.

[0006] II. Optimization techniques based on Winograd method. Winograd method reduces computational complexity by reducing the number of multiplications in convolution operations, thereby achieving acceleration. This method is particularly suitable for small convolution kernels (such as 3x3) and can significantly reduce the amount of calculation. The core idea of Winograd method is to transform convolution calculation into fewer multiplications and more additions through mathematical conversion. In recent years, to overcome the limitations of Winograd method in precision, universality and hardware adaptation, Huang et al. proposed the DWM (Decomposable Winograd Method) variant method, which expands the scope of the method by decomposing large convolution kernels into multiple small convolution kernels.

[0007] III. Optimization techniques based on memory-efficient convolution (MEC) method. MEC method optimizes memory access and data layout during convolution process, reduces memory occupation and improves calculation speed. Its core ideas include block calculation, compact memory layout design and reuse of intermediate calculation results. MEC method significantly reduces the memory requirements of convolution operation by reducing the use of memory bandwidth and optimizing memory access mode, and is particularly suitable for resource-constrained edge devices. The advantage of MEC method is that it can maximize the reduction of memory occupation while ensuring the calculation speed, meeting the inference requirements of low delay and low power consumption.

[0008] Large Kernel Convolution (LKC) has gradually become a research hotspot in the field of deep learning. Although traditional small convolution kernels (such as 3x3 or 5x5) can improve the learning ability of local features, they have limited receptive fields and need to be stacked through multiple layers to capture global features, which increases computational complexity and causes gradient vanishing problems. In contrast, large convolution kernels can cover a larger receptive field in a single convolution layer, enhancing the modeling ability of global features and reducing network depth, thereby reducing computational complexity. RepLKNet introduces a reparameterization technique to enhance feature capture ability through a multi-path convolution structure during the training phase and combines it into an equivalent large convolution kernel during the inference phase, significantly improving computational speed while avoiding overfitting risks. In addition, RepLKNet also combines dynamic receptive field design and sparsification techniques to further optimize performance, especially in high-resolution tasks. Wang R. applied this design to the action recognition task, combined with dynamic receptive field and sparsification methods, significantly improved the capture ability of spatio-temporal features, solved the long-distance dependency problem, and achieved significant improvement on the UCF101 and HMDB51 datasets. Although large convolution kernels have high computational complexity and large parameter quantity, current research is optimizing computational costs through factorization, combining large convolution kernels with attention mechanisms, and other methods, which are expected to play an important role in more tasks in the future.

[0009] Fault Injection Attack (FIA) is an attack method that introduces abnormal faults or physical interference to maliciously manipulate the running behavior of the system. When deploying deep learning models on edge devices, FIA can be used to tamper with the inference process and output results of the model, thereby threatening the security and robustness of the system. According to different implementation methods, FIA can be divided into two categories: soft fault attacks and physical attacks. Among them, Rowhammer attack is a typical soft fault method, which works by accessing a certain row in DRAM at a high frequency, causing bit flips in adjacent rows. This fault may cause model parameters or intermediate state data to be modified, affecting the correctness of inference. In contrast, physical attacks such as laser irradiation can directly act on DRAM, causing storage content errors, or even when the CPU is not encapsulated, some instructions may be skipped, thereby seriously interfering with the control flow of model inference.

[0010] To reduce the resource overhead caused by hardware fault tolerance, researchers have proposed various algorithm-based fault tolerance mechanisms in recent years. The ABFT scheme first proposed by Huang and Abraham is designed specifically for matrix multiplication, which can effectively detect and correct soft errors without relying on additional hardware. With further research, Zhao et al. further proposed the FT-CNN method, which applies the ABFT scheme to the convolutional neural network (CNN). FT-CNN introduces a lightweight check mechanism in the convolution operation to achieve real-time detection and correction of hardware faults, and improves the fault tolerance capability of the system in actual deployment without significantly increasing the computational burden.

[0011] The existing three convolution methods have their own advantages and limitations in improving the efficiency of convolutional neural networks. The first convolution optimization method based on the Im2col method converts convolution into matrix multiplication to speed up computation, but generates a large intermediate tensor during the conversion process, resulting in high memory usage, especially when processing high-resolution images or large convolution kernels. This problem is particularly prominent on resource-constrained devices and becomes a performance bottleneck. The second convolution optimization method based on the Winograd method effectively improves the speed of convolution calculation by reducing multiplication operations, especially suitable for small convolution kernels and model inference that requires a large number of convolution calculations. However, since each different convolution kernel requires the development of a corresponding conversion matrix, when the size, shape or other parameters of the convolution kernel change, the Winograd method cannot be directly applied and must be redesigned for each convolution kernel. This makes the Winograd method unsuitable for convolution calculations with arbitrary kernel sizes, especially in dynamic convolution or adaptive convolution applications, where Im2col is more efficient and flexible. The third convolution optimization method based on the memory-efficient convolution (MEC) method reduces the size of the intermediate tensor, reducing memory usage. However, it still has some problems. First, in some cases, the MEC method may generate a larger intermediate tensor than Im2col. Second, the division of small matrices in the MEC method depends on the output tensor height o h , which means at least o h times GEMM operation is required, which is prone to efficiency differences under different convolution parameters, affecting the consistency and stability of overall performance.

[0012] Existing algorithm-based fault-tolerant ABFT solutions are mainly designed for traditional matrix operations. In CNN inference, in order to utilize mature matrix multiplication acceleration techniques, engineering practices usually convert the input feature tensor into a matrix form by using the Im2col method, thereby converting the convolution operation into standard matrix multiplication. However, the Im2col method generates a large-scale intermediate tensor, which not only replicates and rearranges a large amount of input data, resulting in a significant increase in memory consumption, but also directly affects the memory occupation and calculation time of the fault-tolerant convolution method based on the input tensor I. Specifically, when performing error detection and correction, the fault-tolerant method needs to perform additional checksum calculation on the intermediate tensor. The larger the matrix size, the more checksum storage and calculation resources are required, thereby significantly increasing the memory burden and operation overhead. On resource-constrained edge devices, such high memory occupation not only reduces the inference efficiency, but also may cause the system to fail to operate normally, greatly limiting the practical application of the ABFT solution in edge intelligent scenarios. SUMMARY

[0013] The purpose of the present application is to provide a memory-efficient fault-tolerant convolution method that can significantly reduce memory consumption while ensuring calculation speed and can be effectively applied to multi-batch convolution.

[0014] To achieve the above purpose, the technical solution adopted by the present application is:

[0015] (1) Memory optimization: memory optimization conversion is performed on the input feature tensor I to generate an intermediate tensor L, so that the size of L is the same as and smaller than the size of the intermediate tensor L obtained by Im2col conversion, thereby reducing the memory space required for storing the intermediate tensor L;

[0016] (2) Block matrix parallel multiplication: according to the convolution kernel height k h and the output tensor height o h , the intermediate tensor L and the convolution kernel tensor F are divided into sub-matrices, and the output tensor O is calculated by performing parallel matrix multiplication GEMM; when the step size s = 1, k h times GEMM operation is called, and when s > 1, o h × k h times GEMM operation is called;

[0017] (3) Multi-batch processing: in the case of multi-batch convolution, the intermediate tensor L is reconstructed by using a batch mixed arrangement method, so that the elements of different batches are arranged alternately, thereby avoiding the influence of the change of the input batch size i n on the calculation speed;

[0018] (4) Fault-tolerant checksum and error correction: the input feature tensor I is divided into i n element groups (i nCompute checksum C for error detection and correction o1 , C o2 and output checksum S o1 , S o2 , by comparing C o1 with S o1 , C o2 with S o2 , locate and correct the erroneous element values.

[0019] Further, the intermediate tensor L is obtained by vertically arranging the local blocks of the input feature tensor I and deleting redundant elements, and the size of L is OMEC size , which is calculated by the following formula:

[0020] OMEC size = i n ×mec h ×o w ×k w ×i c Formula (1)

[0021] mec h = s> = k h ? o h ×k h : (o h -1)×s+k h Formula (2)

[0022] where OMEC size is the size of the intermediate tensor L, i n is the input batch size, mec h is the memory-optimized height calculated by the stride s and the convolution kernel height k h , o w is the output tensor width, k w is the convolution kernel width, i c is the input channel number, s is the stride, k h is the convolution kernel height, and o h is the output tensor height.

[0023] Further, in step (1), the generation process of the intermediate tensor L includes:

[0024] extracting local blocks from the input feature tensor I using a sliding window with a size of k w ×i c ;

[0025] flattening the extracted local blocks into row vectors and vertically arranging them, and deleting redundant blocks containing duplicate elements.

[0026] Further, in step (2), the block matrix multiplication includes two execution schemes:

[0027] When the step size s = 1, k h GEMM operations are performed, and the number of calculations is reduced by merging the sub-matrices;

[0028] When the step size s > 1, o h ×k h GEMM operations are performed, and each operation multiplies a sub-matrix of the intermediate tensor L with a sub-matrix of the convolution kernel tensor F.

[0029] Further, in step (3), the batch mixing arrangement is specifically:

[0030] The elements of the multi-batch input feature tensor I are alternately arranged using a sliding window of k w ×i c size to generate an intermediate tensor L with a height of mec h ×o w ×i n and a width of k w ×i c .

[0031] Further, in step (4), the input feature tensor I is divided into i n element groups D, each D has i h ×i w ×i c elements; C d1 and C d2 are calculated by formula (3) and formula (4):

[0032]

[0033] where D n is the nth element group; L d1 and L d2 are obtained by OMEC intermediate tensor conversion of C d1 and C d2 , and then the checksum C o1 for error detection and correction is calculated by the OMEC convolution method, C d1 = l o2 × F and C d2 = L o1 × F; where C o2 is the sum of the results obtained after convolution operation of each D group, and C o2 is the sum of the product of the convolution operation result of each D group and the coordinates of the corresponding D group.

[0034] For the output tensor O, it is divided into i n element groups P, each P has oh ×o w ×o c The output check sum S is calculated by formula (5) and formula (6) o1 and S o2 :

[0035]

[0036] Wherein, S o1 is the sum of corresponding elements in each P group, S o2 is the sum of the product of the corresponding element value of each P group and the coordinate of the corresponding P group.

[0037] Then, by comparing whether the elements in C o1 and S o1 are equal, the serial number j corresponding to the error element is identified; after determining the serial number j of the error element, the group number i of the error element is further located by using formula (7):

[0038]

[0039] Finally, the error element value is corrected by formula (8):

[0040] O[i×o h ×o w ×o c +j]=O[i×o h ×o w ×o c +j]+C o1 [j]-S o1 [j] formula (8).

[0041] Further, the data format of the input feature tensor I is NHWC, the data format of the convolution kernel tensor F is RSCK, in the single batch convolution case, the data format of the output tensor O is NPQK, and in the multi-batch convolution case, the data format of the output tensor O is converted from PQNK to NPQK.

[0042] Compared with the prior art, the present application has the following beneficial effects: the present application proposes a memory-efficient fault-tolerant convolution method OMEC ft(Optimization of Memory-Efficient Convolutions with FaultResistance), which realizes optimization in the following four aspects: first, it overcomes the problem of incomplete space saving in the MEC method, significantly reduces the memory overhead through memory optimization; second, it avoids the problem that the number of convolution GEMM in the MEC method is affected by the height of the output tensor, improves the stability and consistency of the calculation process. Finally, in order to avoid the negative impact of batch size change on the calculation rate, a multi-batch processing mechanism based on layout improvement is introduced, through batch mixed arrangement, the overall calculation performance is improved under the condition of multi-batch convolution. In addition, the fault tolerance mechanism of the input tensor I is introduced into the OMEC method, and the OMEC method optimizes the data layout and conversion mode, significantly reduces the size of the intermediate tensor, so that the conversion matrix used for error detection and correction is significantly smaller than the matrix converted by the Im2col method. This optimization not only effectively reduces the memory occupation of the fault tolerance convolution method based on the input tensor I, but also reduces the calculation time required in the checksum and error correction process. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a conversion process diagram of the intermediate tensor L in the embodiment of the application;

[0044] Figure 2 is an example diagram of single-batch single-step OMEC convolution in the embodiment of the application;

[0045] Figure 3 is an example diagram of single-batch and step-2 OMEC convolution in the embodiment of the application;

[0046] Figure 4 is an example diagram of multi-batch OMEC convolution in the embodiment of the application;

[0047] Figure 5 is a fault tolerance convolution method OMEC based on the input tensor I in the embodiment of the application ft example diagram;

[0048] Figure 6 is a comparison diagram of MEC_B, Winograd and OMEC small convolution kernel convolution speed-up ratio in the embodiment of the application;

[0049] Figure 7 is a comparison diagram of MEC_B and OMEC large convolution kernel convolution speed-up ratio in the embodiment of the application;

[0050] Figure 8 is a comparison diagram of multi-batch MEC_A, MEC_B and OMEC convolution speed-up ratio in the embodiment of the application;

[0051] Figure 9 is an embodiment of the present application OMEC ft and Im2col ft acceleration ratio comparison chart. DETAILED DESCRIPTION

[0052] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0053] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0054] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0055] The present application proposes a memory-efficient fault-tolerant convolution method OMEC ft . This method not only solves the problem of incomplete space saving in traditional MEC method, but also avoids the problem that the number of convolution GEMM is affected by the height of output tensor o h . On this basis, the OMEC method also introduces a multi-batch processing mechanism based on layout improvement, which effectively avoids the negative impact of batch size i n change on the calculation rate and improves the overall calculation performance. In terms of result fault tolerance, the fault tolerance mechanism of the input tensor I is introduced into the OMEC method, which significantly improves the detection and correction performance. The OMEC ft method can not only accurately detect all real attacks, but also effectively correct the results to ensure the correctness of the output. Finally, on the ARM heterogeneous edge device, three different data sets are used for comprehensive testing to verify the effectiveness and correctness of the OMEC method and the present method OMEC ft .

[0056] To ensure consistency of the data mentioned below, first list the symbol explanation table 1.

[0057] Table 1 Symbol Explanation

[0058]

[0059] As shown in Table 1, in the context of input tensor I, the four-dimensional tensor is usually denoted as N (batch size), C (number of input channels), H (input height), and W (input width). In contrast, for the convolution kernel tensor F, the tensors are denoted as K (number of output channels), C (number of input channels), R (height of the convolution kernel), and S (width of the convolution kernel). As for the output tensor O, the tensors are denoted as N (batch size), K (number of output channels), P (height of the output tensor), and Q (width of the output). If the data format of the input tensor I is NHWC, the data format of the convolution kernel F is RSCK, and the data format of the output tensor O is NPQK. If the data format of the input tensor I is NCHW, the data format of the convolution kernel F is KCRS, and the data format of the output tensor O is NKPQ. Subsequent methods newly proposed by the present application are based on the NHWC data format.

[0060] 1. OMEC method

[0061] The following will be described from four aspects: (1) memory optimization of the OMEC method, (2) OMEC method design in the case of single batch convolution, (3) OMEC method design in the case of multiple batch convolution, and (4) detailed description of the OMEC method:

[0062] (1) Memory optimization of the OMEC method: This optimization makes the size of the intermediate tensor L after OMEC conversion the same and smaller than the intermediate tensor L after Im2col conversion, thereby reducing the memory space required for storing the intermediate tensor L.

[0063] Figure 1 The implementation of the intermediate conversion tensor L of OMEC is shown, which divides the intermediate tensor L based on Im2col into multiple small blocks with a size of o h ×k w i c = 2 x 3 = 6. By vertically arranging these blocks and deleting the redundant orange blocks containing the same elements, the OMEC conversion tensor shown in the last row will be obtained. Therefore, the size of the intermediate tensor L of OMEC is the same and smaller than the intermediate tensor L after Im2col conversion, and OMEC size can be calculated by formula (1).

[0064] OMEC size = i n ×mec h ×o w ×k w ×i c Formula (1)

[0065] wherein mec h is calculated by formula (2).

[0066] mec h =s>=k h ? o h ×k h :(o h -1)×s+k h Formula (2)

[0067] like Figure 1 As shown, the gray area in I represents the Im2col window. To generate the transformed tensor L, the Im2col-based transformation uses a window of size k. h ×k w i c The window extracts local blocks from the input tensor I. The window slides from left to right and from top to bottom with a stride s. In each step, the block within the window is flattened into a row vector, and all row vectors are concatenated to form a tensor L. In contrast, OMEC uses a tensor of size 1×k. w i c The window slides from left to right and from top to bottom, each local block is flattened into row vectors, and all these row vectors are arranged to construct tensor L.

[0068] (2) OMEC method design in the case of single batch convolution: Since OMEC performs differently in the GEMM-based convolution method when s=1 and s>1, it will be described in different cases.

[0069] Case 1: s = 1

[0070] Figure 2 This demonstrates a case study of single-stride, single-batch OMEC convolution under the NHWC data format. The input tensor I is arranged in the format NHWC(i n =1,i h =i w =4,i c =2). The data type of the convolution kernel F is RSCK(k) h =k w =2,i c =2,o c =2). OMEC uses a size of k w ×i c A sliding window (=2×2=4) (e.g.) Figure 2 As shown in the blue box area of ​​tensor I, elements are extracted from the original tensor I sequentially from left to right and top to bottom to generate the transformed tensor L. The height and width of the transformed tensor L are i and i', respectively. n ×mec h ×o w (=1×4×3=12) and k w ×i c(= 2 x 2 = 4). OMEC divides the tensor L into mec h (= 4) of size o w x k w x i c = 3 x 2 x 2 = 12, labeled a, b, c and d, respectively. Similarly, the convolution kernel F is divided into k h (= 2) segments, labeled C and D, each of which has a height of k w i c (= 2 x 2) and a width of o c (= 2). Assume that the output tensor O is divided into o h (= 3) blocks, labeled E, G and H. Next, OMEC performs i n x o h x k h = 1 x 3 x 2 = 6 GEMMs to obtain the output tensor O, where O = (E, G, H) = ((a x C + b x D), (b x C + c x D), (c x C + d x D)), as shown in Scheme 1 in Figure 2

[0071] In Scheme 1, sub-matrices a, b and c are multiplied by C, and sub-matrices b, c and d are multiplied by D. Blocks a, b and c are contiguous in the transformed tensor L, and so are blocks b, c and d. Therefore, Scheme 2 merges a, b and c into a single block A, and merges b, c and d into block B. Thus, only i n x k h = 1 x 2 = 2 GEMMs are needed to directly compute the output O, i.e., O = A x C + B x D. Through the above analysis, in the case of a single batch, when s = 1, at least k h GEMM operations are called to compute the output O. If the sub-matrices are too large, they can be further divided into smaller sub-matrices, so that k h x o h GEMM operations are called to compute the output O. This process can be flexibly selected according to specific circumstances. In performing large matrix multiplication, OMEC adopts Scheme 1 to divide L into smaller sub-matrices to compute the output tensor O, thereby using o h x k h small matrix GEMMs in parallel to speed up the convolution. In contrast, in performing small and medium matrix operations, OMEC uses Scheme 2 to directly divide L into k h large blocks to compute O, reducing the number of GEMM operations to k h and effectively speeding up the computation.

[0072] Case 2: s > 1

[0073] Figure 3 ​This demonstrates a case study of OMEC convolution with a single batch and a stride of 2 under the NHWC data format. The input tensor I is arranged in the format NHWC(i n =1,i n =i w =4,i c =2). The data type of the convolution kernel F is RSCK(k) h =k w =2,i c =2,o c =2). OMEC uses a size of k w ×i c A sliding window (=2×2=4) (e.g.) Figure 3 As shown in the blue box area of ​​tensor I, elements are extracted from the original tensor I sequentially from left to right and top to bottom to generate the transformed tensor L. The height and width of the transformed tensor L are i and i', respectively. n ×mec h ×o w (=1×4×2=8) and k w ×i c (=2×2=4). The convolution kernel F is divided into k h There are 2 parts (C, D), each with a height of k. w ×i c (=2×2=4), width is o c (=2). Tensor L is partitioned into mec h (=4) of size o w ×k w ×i c The output tensor O is divided into blocks of 2 × 2 × 2 = 8, labeled a, b, c, and d. h There are 2 blocks, labeled E and G. Then, the corresponding parts of F and L are multiplied by matrix and summed to get the output O, that is: O = (E, G) = (a × C + b × D, c × C + d × D).

[0074] In summary, under the condition of single batch and s=1, the MEC method requires o h Unlike previous GEMM calls, the OMEC method reduces the number of GEMM calls to k by optimizing the data layout. h Next. Because usually o h >k h And o hThe OMEC method exhibits better stability due to its greater variability. When performing calculations on small to medium-sized matrices, multiple GEMM calls can be time-consuming, while OMEC has a significant advantage in these cases by reducing the number of GEMM calls. For large matrix calculations, the OMEC method also allows the matrix to be further divided into smaller blocks than MEC, thereby increasing the number of GEMM calls to k. h ×o h This effectively avoids memory bottlenecks that may occur when the matrix is ​​very large, and OMEC can better utilize hardware performance, especially when memory bandwidth is limited. When s > 1, the number of GEMM calls to the OMEC method is o. h ×k h Furthermore, in some cases, OMEC saves more memory than the original MEC. Therefore, the OMEC method is generally superior to the MEC method in matrix computations of various sizes.

[0075] (3) OMEC method design for multi-batch convolution: In the case of multi-batch convolution of small and medium matrices, the computational cost is easily affected by the batch size i. n The influence of this can lead to multiple GEMM calls for small matrices, potentially reducing efficiency. To address this issue, in the case of multi-batch convolution, the OMEC method introduces a layout-based improvement, employing a batch hybrid arrangement so that the transformed tensor L is no longer processed individually for each batch, but rather arranged in an alternating pattern. This design can avoid the influence of i n The changes in [the data] negatively impact computing speed, thus improving overall computing speed.

[0076] Figure 4 This example illustrates a single-stride OMEC convolution with a batch size of 2 in the NHWC data format. The input tensor I is arranged as NHWC(i n =2,i h =i w =4,i c =1). The data type of the convolution kernel F is RSCK(k) h =k w =2,i c =1,o c =2). OMEC uses a size of k w ×i c A sliding window (e.g., =2×1=2) Figure 4 As shown in the blue box area of ​​tensor I, elements are extracted from the original tensor I sequentially from left to right and top to bottom, back and forth between different batches, ultimately generating the transformed tensor L. The height and width of the transformed tensor L are mec and mec, respectively. h ×o w ×i n(= 4 x 3 x 2 = 24) and k w x i c (= 2 x 1 = 2). The convolution kernel F is divided into k h (= 2) parts (C, D) with height k w x i c (= 2 x 1 = 2) and width o c (= 2). The tensor L is also divided into m h (= 4) small matrices (a, b, c, d) with height o w x i n (= 3 x 2 = 6) and width k w x i c (= 2 x 1 = 2). By matrix multiplication and accumulation on the small matrices, we get O = (E, G, H) = ((a x C + b x D), (b x C + c x D), (c x C + d x D)), which requires k h x o h (= 2 x 3 = 6) GEMM calls to compute the output O'( Figure 4 , scheme one), which has data format PQNK, not the desired NPQK, so one more data format conversion is needed to get O.

[0077] In addition, we can further merge the small matrices a, b, c, d in L into k h larger matrices (A = (a, b, c), B = (b, c, d)) with height o h x o w x i n (= 3 x 3 x 2 = 18) and width k w x i c (= 2 x 1 = 2). Then, we perform matrix multiplication and accumulation on the corresponding k h parts in F and L to get the output O', i.e., O' = A x C + B x D, and finally perform data format conversion to get O. It can be seen that when i n >= 1, only k h GEMM calls (scheme two in Figure 4 ) are needed. Through the above analysis, in the case of multiple batches, when s = 1, at least k h GEMM operations are needed to compute the output O. If the sub-matrix is too large, we can further divide it into smaller sub-matrices, so that k h x o h GEMM operations are needed to compute the output O. When s > 1, o h x k hSub-GEMM. In OMEC, the number of GEMM calls is not affected by i n , and is more suitable for small and medium-sized matrix computation in batches.

[0078] (4) Detailed description of the OMEC method:

[0079] Algorithm 1: Implementation of the OMEC method based on GEMM

[0080]

[0081]

[0082] The implementation of the OMEC method is shown in Algorithm 1. In Algorithm 1, the first line allocates the memory space of the intermediate tensor L, whose dimensions are mec h o w i n ×k w i c , where mec h o w i n is the height of the intermediate tensor L, and k w i c is the width of the intermediate tensor L. Lines 2-5 outline the process of extracting the matrix L, which iteratively copies k w i c consecutive elements from the input matrix I to the target matrix L, saves the elements of each batch together, and these copies can be completed in parallel. Line 7 controls whether to choose scheme one or scheme two to effectively implement the GEMM-based convolution. When s = 1, if it involves a large matrix convolution operation, T = True can be set, and scheme one can be executed to obtain faster convolution operation. If it involves a small and medium-sized matrix operation, T = False can be set, and scheme two is executed to reduce the number of GEMM operations and obtain faster operation effect.

[0083] Lines 8-15 represent scheme one, which requires o h ×k h GEMM operations to calculate L × F = O. The parameter t in line 8 represents the number of sub-blocks between the starting positions of the corresponding regions in L used to calculate two adjacent rows in O when calculating this convolution. As shown in Figure 4 , E starts from a and G starts from b, with a difference of t = 1 sub-block (the distance between a and b). Each GEMM operation multiplies a small matrix in L (such as a) with a small matrix in F (such as C), as shown in Figure 4 Lines 17-23 represent scheme two, which requires k h GEMM operations to calculate L × F = O. In the k h loop, each time oh ×o w ×i n ×o c k elements, and each time k h The calculated values ​​are added together to obtain O in PQNK data format. Finally, line 29 converts O from PQNK data format to NPQK data format.

[0084] 2.This method OMEC ft

[0085] This invention introduces a fault-tolerant mechanism for the input tensor I into the OMEC method, realizing a fault-tolerant convolution method OMEC based on OMEC. ft This method treats each batch as a group and introduces input and output checksums to detect and correct errors in the calculation results. For example... Figure 5 As shown, the input I is divided into i n There are (=2) element groups D, each D contains i h ×i w ×i c (=3×3×1=9) elements (as shown in the green box). C is calculated using formulas (3) and (4). d1 and C d2 Then, perform OMEC intermediate tensor transformation on them to obtain L. d1 and L d2 Next, the checksum C, used for error detection and correction, is calculated using OMEC's ​​convolution method. o1 =L d1 ×F and C o2 =L d2 ×F. Where, C o1 It is the sum of the results obtained after the convolution operation of each D group, because the elements in different D groups are independent and do not interfere with each other; while C o2 It is the sum of the products of the convolution result of each D group and the coordinates of the corresponding D group.

[0086]

[0087] For the output O, divide it into i n There are 2 elements P, each P consisting of o h ×o w ×o c It consists of 8 elements (2 × 2 × 2 = 8) (as shown in the red box). S is obtained by calculating using formulas (5) and (6). o1 and S o2 Among them, S o1 It is the sum of the corresponding elements in each P group, while S o2is the sum of the product of the corresponding element value of each P-group and the coordinate of the corresponding P-group. The coordinates of P-group and D-group here are the same because they are both according to i n The C o1 and S o1 are compared to identify the sequence number j of the error element. Once the sequence number j of the error element is determined, the group number i of the error element is further located by using formula (7). Finally, the error element value is corrected by using formula (8).

[0088]

[0089] O[i×o h ×o w ×o c +j]=O[i×o h ×o w ×o c +j]+C o1 [j]-S o1 [j] Formula (8)

[0090] Algorithm 2 shows the specific implementation of obtaining C d1 and C d2 . Then, L d1 and L d2 are obtained by performing OMEC intermediate tensor conversion on C d1 and C d2 . Finally, the checksums C o1 = L d1 × F and C o2 = L d2 × F for error detection and correction are calculated by performing matrix operations on L d1 , L d2 and F, i.e. OMEC convolution method. Algorithm 3 shows the specific implementation of obtaining S o1 and S o2 . Then, the error correction operation is performed by algorithm 4 to ensure the correctness of the calculation result.

[0091] Algorithm 2 obtains C d1 and C d2 Algorithm

[0092]

[0093]

[0094] Algorithm 3 obtains S o1 and S o2 Algorithm

[0095]

[0096] Algorithm 4 error correction algorithm

[0097]

[0098] 3. Devices and environments

[0099] The present application is implemented using a Rock 5B development board as an edge device. The Rock 5B is equipped with an 8-core CPU (4 Cortex A55 and 4 Cortex A76). The operating system installed on the Rock 5B is Ubuntu 20.04.6 LTS, and the running memory is 16 GB.

[0100] 4. Experimental methods and benchmark convolution

[0101] The C versions of MEC, OMEC, and Im2col and Winograd methods were tested on the Rock 5B platform. Three different datasets and tests were used to comprehensively evaluate the computational performance of the OMEC method. First, based on the convolution instances provided by the deep learning model implemented based on the Darknet framework, the practicality of the OMEC method was verified. Second, in order to further analyze the performance of the OMEC method in large convolution kernel convolution calculation, the convolution instances provided by the deep learning model implemented based on the RepLKNet framework were used for further performance testing. Finally, the performance of the OMEC method in multi-batch convolution calculation was evaluated to test its computational performance improvement in multi-batch convolution. The benchmark convolution test examples are shown in Table 2, where cv6-10 correspond to the multi-batch parameters of cv1-5. The symbols representing the related methods in the experiment are shown in Table 3. The experiment was conducted in single-thread and multi-thread tests, and the number of threads in multi-thread was set to 4 to balance the computational resources and thread management overhead, avoiding excessive thread switching.

[0102] Table 2 Convolution operation examples

[0103]

[0104] It should be noted that MEC_A and MEC_B are two solutions from the code in the original MEC paper. The Winograd method is derived from the FALCON library mentioned in the MEC paper, and the entire C language implementation is implemented using OpenBLAS and OpenMP.

[0105] Table 3 Symbols of related methods

[0106] Name Meaning MEC_A Scheme A of MEC convolution method, implemented for batch processing, using OpenBLAS implementation MEC_B Scheme B of MEC convolution method, using OpenBLAS implementation OMEC OMEC convolution method, using OpenBLAS implementation Im2col Im2col convolution method, using OpenBLAS implementation

[0107] 5. Memory optimization results

[0108] The intermediate memory of the above 10 benchmark convolution instances is tested, mainly comparing the intermediate tensor size of the OMEC, Im2col and Winograd three methods. In order to measure the memory saving effect, the memory saving rate R of OMEC relative to the Im2col method is calculated by formula (9) Im2col , and the memory saving R of OMEC relative to the Winograd method is calculated by formula (10) Winograd .

[0109]

[0110] Table 4 shows the comparison of the intermediate tensor size of different convolution methods (OMEC, Im2col and Winograd) under small convolution kernel convolution. As can be seen from the data in Table 4, compared with the Im2col method, the OMEC method saves at least 64.81% and at most 65.76% of the intermediate tensor memory. Compared with the Winograd method, the OMEC method saves 20.83% to 22.97% of the intermediate tensor memory. Table 5 shows the comparison of the intermediate tensor size of OMEC and Im2col methods under large convolution kernel convolution. The data shows that the OMEC method saves at least 48.38% and at most 88.23% of the intermediate tensor memory compared with the Im2col method under large convolution kernel convolution.

[0111] Table 4 Comparison table of intermediate tensor size of small convolution kernel convolution method (unit: element number)

[0112] Convolution OMEC size ]]> im2col size ]] Winograd size ]]> [R Im2col ]]> [R Winograd ]]> cv1 2,159,616 6,308,352 2,803,712 65.76％ 22.97％ cv2 1,050,624 2,985,984 1,327,104 64.81％ 20.83％

[0113] Table 5 Comparison table of intermediate tensor size of large convolution kernel convolution method (unit: element number)

[0114] Convolution OMEC size ]]> im2col size ]]> Winograd size ]]> [R Im2col ]]> [R Winograd ]]> cv3 507,904 984,064 - 48.38％ - cv4 516,096 4,064,256 - 87.30％ - cv5 557,056 4,734,976 - 88.23％ -

[0115] Table 6 shows the comparison of the intermediate tensor size of different convolution methods (OMEC, Im2col and Winograd) under multi-batch convolution. As can be seen from the data in Table 6, compared with the Im2col method, the OMEC method saves at least 48.38% and at most 88.23% of the intermediate tensor memory. Compared with the Winograd method, the OMEC method saves 20.83% to 22.97% of the intermediate tensor memory.

[0116] Table 6 Comparison table of intermediate tensor size of multi-batch convolution method (unit: element number)

[0117]

[0118]

[0119] In summary, the OMEC method can effectively reduce memory occupation in both small and large convolution kernel convolutions, especially in large convolution kernel convolutions.

[0120] 6. Speedup ratio results

[0121] (1) Speedup ratio of small convolution kernel convolution

[0122] Figure 6 The calculation results of convolution instances from deep learning models implemented by the Darknet framework are shown, comparing the speedup ratios of the MEC_B and OMEC methods compared with the Im2col and Winograd methods, covering single-thread and multi-thread test results. The x-axis represents different benchmark convolution instances, and the y-axis is the speedup ratio. The red dotted line represents the comparison benchmark (speedup ratio is 1.00x).

[0123] In a single-thread environment, OMEC shows speedup effects compared with MEC_B and Im2col, but in the cases of cv1 and cv2, it runs slower than the Winograd method. Specifically, the speedup ratio of OMEC compared with Im2col is 1.08x to 1.56x, and compared with Winograd is 0.56x. The speedup ratio of MEC_B compared with Im2col is between 0.89x and 1.56x, and compared with Winograd is between 0.42x and 0.62x. In a multi-thread environment, OMEC is always better than Im2col and faster than MEC_B in the cases of cv1 and cv2. Specifically, the speedup ratio of OMEC compared with Im2col is between 1.24x and 1.40x, and compared with Winograd is 1.05x to 1.29x. The speedup ratio of MEC_B compared with Im2col is between 0.78x and 1.32x, and compared with Winograd is between 0.86x and 1.04x.

[0124] These results show that the OMEC method can effectively shorten the calculation time in small convolution kernel convolution. Compared with the Im2col method, it always achieves less execution time, and in multi-thread testing, it also shows obvious advantages compared with the Winograd method. In addition, the OMEC method is better than the MEC_B method in all benchmark convolution tests, showing better stability and faster execution speed.

[0125] (2) Speedup ratio of large convolution kernel convolution

[0126] Figure 7The computational results of the convolution instances from the deep learning model implemented in the RepLKNet framework are shown, comparing the speedup ratios of the MEC_B and OMEC methods compared to the Im2col method, covering both single-threaded and multi-threaded test results. The x-axis represents different benchmark convolution instances, and the y-axis represents the speedup ratio. The red dotted line represents the comparison baseline (speedup ratio of 1.00x).

[0127] Due to the complexity of the Winograd method requiring redesign for different convolution kernels, it is not compared with Winograd for large convolution kernels. In a single-threaded environment, OMEC is always superior to Im2col and MEC_B, with a speedup ratio of 1.07x to 1.90x compared to Im2col. MEC_B has poor stability and is slower than Im2col, with a speedup ratio ranging from 0.58x to 1.34x. In a multi-threaded environment, OMEC is still always superior to Im2col, with a speedup ratio of up to 3.05x and a minimum of 2.84x. In contrast, the speedup ratio of MEC_B compared to Im2col is between 1.34x and 1.81x, which is not as effective as OMEC.

[0128] Therefore, the OMEC method exhibits more significant execution efficiency in large convolution kernel convolution, especially in multi-threaded tests, where the speedup effect is particularly evident. Moreover, in all benchmark convolution tests, the execution time of the OMEC method is superior to that of the Im2col method and the MEC_B method, demonstrating higher stability and faster execution speed.

[0129] (3) Multi-batch convolution speedup ratio

[0130] Figure 8 The computational results from the batch-optimized convolution instances are shown, comparing the speedup ratios of the MEC_A, MEC_B, and OMEC methods compared to the Im2col method, covering both single-threaded and multi-threaded test results. The x-axis represents different benchmark convolution instances, and the y-axis represents the speedup ratio. The red dotted line represents the comparison baseline (speedup ratio of 1.00x).

[0131] In the single-threaded environment, the execution speed of the three methods is lower than that of Im2col in some cases, but overall, the performance of OMEC is better than that of MEC_A and MEC_B. Specifically, the speedup ratio of OMEC compared to Im2col is between 0.99x and 2.40x, the speedup ratio of MEC_A compared to Im2col is between 0.80x and 2.38x, and the speedup ratio of MEC_B compared to Im2col is between 0.47x and 1.90x. In the multi-threaded environment, the execution speed of the three methods is also lower than that of Im2col in some cases, but the overall performance of OMEC is still better than that of MEC_A and MEC_B. Specifically, the speedup ratio of OMEC compared to Im2col is between 0.90x and 1.49x, the speedup ratio of MEC_A compared to Im2col is between 0.82x and 1.32x, and the speedup ratio of MEC_B compared to Im2col is between 0.35x and 1.09x.

[0132] Therefore, in the case of multi-batch convolution, the OMEC method is still better than the MEC method and the Im2col method, with faster execution speed and higher stability. Whether in single-threaded or multi-threaded testing, the speedup ratio of OMEC is higher than that of the MEC method and the Im2col method.

[0133] 7. Fault-tolerant performance results

[0134] Based on the convolution operation example in Table 2, the following experiments were conducted, mainly testing the speed comparison and memory usage between the fault-tolerant Im2col method based on input tensor I and the fault-tolerant OMEC method based on input tensor I. The symbols representing the relevant methods in the experiment are shown in Table 7. OMEC ft and Im2col ft both use fault-tolerant mechanisms based on input tensor I, and the difference mainly lies in the intermediate tensor conversion method. Both methods can achieve error detection and correction.

[0135] Table 7 shows the symbols of the relevant methods

[0136] Name Meaning OMEC ft ]]> Fault-tolerant OMEC convolution method based on input tensor I im2col ft ]]> Fault-tolerant Im2col convolution method based on input tensor I

[0137] (1) Memory optimization results

[0138] Table 8 shows the memory changes of fault-tolerant convolution based on input tensor I after using OMEC, focusing on the memory usage of the error correction part. It can be seen that by using the OMEC method, the memory requirement of the error correction scheme is greatly reduced, saving between 37.98% and 91.81% of memory.

[0139] Table 8: Comparison of additional memory usage of OMEC and Im2col error correction schemes (unit: number of elements)

[0140] Convolution im2col ft ]]> OMEC ft ]]> Percentage of memory saved cv1 19,702,784 11,405,312 42.11％ cv2 9,365,504 5,494,784 41.32％ cv3 2,496,512 1,548,288 37.98％ cv4 14,647,296 1,200,128 91.81％ cv5 9,666,560 1,310,720 86.41％ cv6 19,702,784 11,405,312 42.11％ cv7 9,365,504 5,494,784 41.32％ cv8 2,496,512 1,548,288 37.98％ cv9 14,647,296 1,200,128 91.81％ cv10 9,666,560 1,310,720 86.41％

[0141] (2) Speedup ratio analysis

[0142] Figure 9 It is shown that, taking the test data set shown in Table 2 as an example, OMEC ft achieves acceleration compared with Im2col ft . The x-axis represents different benchmark convolution instances, and the y-axis represents the speedup ratio. The red dotted line represents the comparison benchmark (speedup ratio is 1.00x).

[0143] From Figure 9 the above results, it can be seen that, in all benchmark convolution tests, OMEC ft achieves acceleration compared with Im2col ft . Specifically, OMEC ft achieves significant performance improvement in tests such as cv1, cv2, cv5, cv9 and cv10, and especially in the test of cv5, the speedup ratio reaches 3.21x. In the remaining test instances, the speedup ratio of OMEC ft is also higher than 1.00x, and the lowest speedup ratio is 1.07x, which shows that for the tested benchmark convolution, OMEC ft can bring performance improvement.

[0144] The above experimental results show that, in the tested benchmark convolution, OMEC ft is superior to Im2col ft . For large kernel convolution and small kernel convolution and their multi-batch convolution, OMEC ft can bring performance improvement, and in some test convolutions, significant acceleration effect is obtained.

[0145] (3) Fault tolerance performance analysis

[0146] The fault tolerance of OMEC ft is evaluated by cv1, and the method of randomly attacking any multiple elements is used to measure the accuracy of error correction. In the cv1 task, the output tensor contains 1,401,856 elements, and the experimental results are shown in Table 9. In Table 9, the "number of detected attacks" refers to the number of successfully detected real attacks, excluding false positives (i.e. falsely detecting non-attack behavior). The "accuracy" in Table 9 represents the proportion of detection results that are completely consistent with the original results after error correction. It can be seen that under different attack quantities, OMEC ft can accurately detect all real attacks, and the detection results are correct after error correction, and the accuracy is always 100%.

[0147] Table 9 Error correction rate of OMEC ft ​

[0148] Number of attacks Number of attacks detected Accuracy rate 10 10 100.00％ 100 100 100.00％ 1000 1000 100.00％ 10000 10000 100.00％ 100000 100000 100.00％

[0149] The above results show that, compared with Im2col ft , OMEC ft can effectively accelerate the calculation, and the highest acceleration ratio reaches 3.21x, while the memory occupation is saved by 91.81% at most. In addition, in terms of error correction, OMEC ft can accurately detect all real attacks under different attack quantity tests, and the detection result after error correction is correct, and the accuracy is always 100%.

[0150] The application proposes a memory-efficient fault-tolerant convolution method OMEC ft , which provides a faster and more reliable solution for memory optimization, acceleration and fault tolerance of convolution on resource-constrained edge devices. The OMEC method improves the MEC method and fixes its defects in memory management, and realizes the optimization of convolution of different step lengths. Compared with MEC, OMEC adopts different data layouts when calculating small and medium-sized matrices, reduces the number of blocks, and achieves faster convolution calculation speed. At the same time, its block method is based on the height of the convolution kernel, not the height of the output tensor, so as to improve the stability. In large-scale matrix calculation, OMEC divides more small blocks, more fully utilizes parallel optimization, and speeds up the convolution calculation process. In the case of multi-batch convolution, OMEC introduces layout-based improvement, adopts batch mixed arrangement, so that the converted tensor L is no longer processed separately for each batch, but is arranged in an alternating mode, effectively avoiding the negative impact of the change of batch size on the calculation rate, and improving the overall calculation rate. Therefore, compared with the classical convolution method (Im2col and Winograd), the OMEC method can obtain smaller memory occupation and faster convolution calculation speed. In addition, the fault-tolerant mechanism of the input tensor I is introduced into the OMEC method, and a fault-tolerant convolution method OMEC ft based on OMEC is realized. Compared with the fault-tolerant Im2col method based on the input tensor I, the method not only can effectively accelerate the calculation and significantly reduce the memory requirement of convolution fault tolerance, but also has excellent fault-tolerant performance. Under different attack quantity tests, OMEC ft can accurately detect all real attacks, and the false detection rate is zero. After error correction, the detection result is completely consistent with the original result, and the accuracy always remains at 100%.

[0151] The above merely describes preferred embodiments of the present application, but does not limit the present application in other forms, and any skilled person in the art can modify or change the above disclosed technical content into equivalent embodiments with equivalent changes. However, any simple modification, equivalent change and modification made on the above embodiments without departing from the technical solution content of the present application and according to the technical essence of the present application still belongs to the protection scope of the technical solution of the present application.

Claims

1. A memory-efficient, fault-tolerant convolution method, characterized in that, Includes the following steps: (1) Memory optimization: The input feature tensor I is transformed by memory optimization to generate an intermediate tensor L, so that the size of L is the same as and smaller than the size of the intermediate tensor L obtained by Im2col transformation, so as to reduce the memory space required to store the intermediate tensor L. (2) Block matrix parallel multiplication: based on the convolution kernel height k h and output tensor height o h The intermediate tensor L and the convolution kernel tensor F are divided into submatrices, and the output tensor O is obtained by parallel matrix multiplication GEMM. When step size s = 1, call k. h The next GEMM operation calls o when s > 1. h ×k h The next GEMM operation; (3) Multi-batch processing: In the case of multi-batch convolution, the intermediate tensor L is reconstructed by batch hybrid arrangement, so that the elements of different batches are arranged alternately, avoiding the input batch size i n The impact of changes on computation speed; (4) Fault tolerance and error correction: Divide the input feature tensor I into i n Group of elements (i) n Calculate the checksum C for error detection and correction (where the batch size is the input). o1 C o2 and output checksum S o1 S o2 By comparing C o1 With S o1 C o2 With S o2 Locate and correct the erroneous element value.

2. According to claim 1, a memory-efficient fault-tolerant convolution method, in step (1), the intermediate tensor L is obtained by vertically arranging local blocks of the input feature tensor I and deleting redundant elements, and its size is OMEC. size Calculated using the following formula: OMEC size = i n ×mec h ×o w ×k w ×i c Formula (1) mec h = s >= k h ? o h × k h : (o h -1) × s + k h Formula (2) in, OMEC size Let i be the size of the intermediate tensor L. n For the input batch size, mec h To use stride s and kernel height k h The calculated memory optimization height, o w To output the tensor width, k w i is the kernel width. c s is the number of input channels, s is the step size, and k is the input channel. h O is the height of the convolution kernel. h To output the tensor height.

3. The memory-efficient fault-tolerant convolution method according to claim 1, wherein in step (1), the generation process of the intermediate tensor L includes: Use size k w ×i c A sliding window extracts local blocks from the input feature tensor I; The extracted local blocks are flattened into row vectors and arranged vertically, and redundant blocks containing duplicate elements are removed.

4. In the memory-efficient, fault-tolerant convolution method according to claim 1, in step (2), the block matrix multiplication includes two execution schemes: When step size s = 1, execute k h The GEMM operation reduces the number of calculations by merging submatrices; When step size s > 1, execute o. h ×k h Each GEMM operation multiplies a submatrix of the intermediate tensor L with a submatrix of the convolution kernel tensor F.

5. In the memory-efficient fault-tolerant convolution method according to claim 1, in step (3), the batch mixing arrangement is as follows: Use k w ×i c A sliding window of size I alternately arranges the elements of multiple batches of input feature tensor I, generating a sequence of heights mec. h ×o w ×i n Width is k w ×i c The intermediate tensor L.

6. In the memory-efficient, fault-tolerant convolution method according to claim 1, in step (4), the input feature tensor I is divided into i n There are 3 elements D, each D containing i. h ×i w ×i c One element; Calculate C using formulas (3) and (4). d1 and C d2 : in, D n For the nth element group; set C d1 and C d2 Perform OMEC intermediate tensor transformation to obtain L d1 and L d2 Then, the checksum C for error detection and correction is calculated using the OMEC convolution method. o1 =L d1 ×F and C o2 =L d2 ×F; where C o1 C is the sum of the results obtained after the convolution operation for each D group. o2 The sum of the products of the convolution result for each D group and the coordinates of the corresponding D group; For the output tensor O, it is divided into i n There are 1 element group P, each P consisting of o. h ×o w ×o c Composed of several elements; the output checksum S is calculated using formulas (5) and (6). o1 and S o2 : Among them, S o1 S is the sum of the corresponding elements in each P group. o2 The sum of the products of the corresponding element value and the coordinates of the corresponding P group for each P group; Then, by comparing C o1 and S o1 To determine if the elements in the set are equal, identify the index j of the erroneous element; after determining the index j of the erroneous element, use formula (7) to further locate the group number i of the erroneous element: Finally, the incorrect element values ​​are corrected using formula (8): O[i×o h ×o w ×o c +j]=O[i×o h ×o w ×o c +j]+C o1 [j]-S o1 [j] Formula (8).

7. The memory-efficient fault-tolerant convolution method according to claim 1, wherein the data format of the input feature tensor I is NHWC, and the data format of the convolution kernel tensor F is RSCK; in the case of single-batch convolution, the data format of the output tensor O is NPQK; in the case of multi-batch convolution, the data format of the output tensor O is converted from PQNK to NPQK.

Images