A multi-channel convolution optimization method and system for DSP

Abstract

The application discloses a kind of multi-channel convolution optimization method and system for DSP, the method of the present application includes the input size CI*(FH*FW) input image F of input is rearranged, obtain the size CI*(KH*KW*OH*OW) rearranged image F', wherein CI is input channel number, FW is the width of input image, FH is the height of input image, KH is the height of convolution kernel, KW is the width of convolution kernel, OH is the height of rearranged image, OW is the width of rearranged image;Matrix multiplication K*F' is executed to obtain the size CO*(OH*OW) calculation result OU as convolution calculation result, wherein CO is output channel number, K indicates convolution kernel, and * indicates matrix multiplication operation.The application can realize the vectorization of multi-channel convolution in deep learning by converting convolution into matrix multiplication operation, to optimize calculation operation to the greatest extent, reduce calculation amount.

Description

Technical Field

[0001] This invention relates to the field of hardware acceleration optimization technology for convolutional neural networks, specifically to a multi-channel convolution optimization method and system for DSPs. Background Technology

[0002] Convolutional neural networks (CNNs) are the most classic model structure in computer vision technology and are often used to analyze visual images. They are mainly composed of stacked network layers such as convolution, pooling, activation, and fully connected layers. The convolutional layer is the core layer for building a CNN, and it generates most of the computation in the network, so it is necessary to optimize the convolution algorithm. Starting with the most basic image convolution operation, since images can be single-channel (grayscale) or multi-channel (RGB), convolution can also be divided into single-channel convolution and multi-channel convolution. In single-channel convolution, for the pixel matrix of an image, the convolution operation involves scanning the pixel matrix row by row and column by column using a convolution kernel, multiplying it element-wise to obtain a new pixel matrix. The convolution kernel is also called a filter, and the area scanned by the filter on the input pixel matrix is ​​called the receptive field. Multi-channel convolution, on the other hand, uses a different convolution kernel for each channel. Each channel's pixel matrix is ​​convolved with the corresponding kernel channel's value, resulting in a single output convolution result for each channel. Multiple convolution results are summed at corresponding positions to obtain the final convolution result, which is the combined result of all channels of the original image. The computational cost of multi-channel convolution is several times that of single-channel convolution. This high computational cost leads to slow algorithm execution and low efficiency, so it is necessary to optimize the computational operations of the convolutional layer as much as possible. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a multi-channel convolution optimization method and system for DSP, which addresses the above-mentioned problems in the prior art. This invention converts convolution into matrix multiplication operations, thereby realizing the vectorization of multi-channel convolution in deep learning, thus optimizing the computational operations to the greatest extent and reducing the amount of computation.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0005] A multi-channel convolution optimization method for DSP includes:

[0006] S101, rearrange the input image F of size CI*(FH*FW) to obtain a rearranged image F' of size CI*(KH*KW*OH*OW), where CI is the number of input channels, FW is the width of the input image, FH is the height of the input image, KH is the height of the convolution kernel, KW is the width of the convolution kernel, OH is the height of the rearranged image, and OW is the width of the rearranged image;

[0007] S102, perform matrix multiplication K*F' to obtain the calculation result OU of size CO*(OH*OW) as the convolution calculation result, where CO is the number of output channels, K represents the convolution kernel of size CO*CI*KH*KW used in the convolution calculation, and * represents matrix multiplication operation.

[0008] Optionally, the rearrangement of the input image F of size CI*(FH*FW) in step S101 includes:

[0009] S201, initialize the value of loop variable i to 1;

[0010] S202, for the i-th input channel of the input image F with size CI*(FH*FW), the SG transfer mode is adopted. According to the index Addr of the SG transfer mode, the data in the source address Src_Addr is moved from memory to the destination address Dst_Addr in the DSP core; then, ordinary point-to-point DMA transfer is used to move the data from the destination address Dst_Addr in the DSP core back to the memory space outside the core to generate a temporary matrix as the rearranged sub-image with size (KH*KW*OH*OW) for the i-th input channel;

[0011] S203, determine whether the value of the loop variable i is equal to the number of input channels CI. If not, increment the value of the loop variable i by 1 and jump to step S202 to continue iterating; otherwise, obtain a rearranged image of size CI*(KH*KW*OH*OW) composed of rearranged sub-images corresponding to all input channels.

[0012] Optionally, in step S102, performing matrix multiplication K*F' to obtain a calculation result OU of size CO*(OH*OW) as the convolution calculation result includes:

[0013] S301, initialize the first column of the convolution kernel K as the current column, and rearrange the first row of the image F' as the current row;

[0014] S302 uses ordinary point-to-point DMA transmission to traverse the current column of the convolution kernel K to obtain an 8*M submatrix K' and pass it into the scalar space SM, where M is the block parameter;

[0015] S303, a submatrix F” of size M*48 obtained from traversing the current row of the rearranged image F' is transferred to the array memory AM via ordinary point-to-point DMA transfer;

[0016] S304, calculate a matrix multiplication of size (8*M)*(M*48) for submatrix K' in scalar space SM and submatrix F” in array memory AM;

[0017] S305, determine whether the submatrix F” has traversed a whole row of data in the rearranged image F’. If it has traversed a whole row of data in the rearranged image F’, then jump to step S306; otherwise, jump to step S303.

[0018] S306, determine whether the submatrix K' has traversed a whole column of data in the convolution kernel K. If it has traversed a whole column of data in the convolution kernel K, then jump to step S307; otherwise, jump to step S302.

[0019] S307, determine whether the convolution kernel K and the rearranged image F' have been traversed. If they have not been traversed, shift the convolution kernel K by one column and the rearranged image F' by one row, and jump to step S302; otherwise, use the calculated result OU of size CO*(OH*OW) as the convolution calculation result.

[0020] Optionally, when performing matrix multiplication K*F' in step S102 to obtain a calculation result OU of size CO*(OH*OW) as the convolution calculation result, a double buffering mechanism is also adopted for the scalar space SM and the array memory AM. In steps S302 and S303, one buffer of the double buffer of the scalar space SM and the array memory AM is used in turn to store the submatrix K' and the submatrix F”, so that the data transmission overhead and the calculation overhead overlap when performing matrix multiplication of size (8*M)*(M*48) for the submatrix K' in the scalar space SM and the submatrix F” in the array memory AM in step S304.

[0021] Optionally, when performing matrix multiplication K*F' in step S102 to obtain a calculation result OU of size CO*(OH*OW) as the convolution calculation result, it also includes soft pipelining optimization for the DSP, so that the matrix multiplication of size (8*M)*(M*48) for submatrix K' in scalar space SM and submatrix F” in array memory AM in any i-th execution of step S304, and the memory access operation of the rearranged image F' and the broadcast operation of convolution kernel K corresponding to the matrix multiplication of size (8*M)*(M*48) for submatrix K' in scalar space SM and submatrix F” in array memory AM in the (i+1)-th execution of step S304 are partially overlapped in sequence.

[0022] Optionally, when performing matrix multiplication K*F' in step S102 to obtain a calculation result OU of size CO*(OH*OW) as the convolution calculation result, it also includes performing parallel execution of unfolding for steps S302 to S306.

[0023] Optionally, in step S304, the matrix multiplication of size (8*M)*(M*48) for submatrix K' in scalar space SM and submatrix F” in vector space is performed using the matrix multiplication function GEMM8_R_48.

[0024] Optionally, the block parameter M is set to 512 bits.

[0025] Furthermore, the present invention also provides a multi-channel convolution optimization system for DSPs, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the multi-channel convolution optimization method for DSPs.

[0026] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program for being programmed or configured by a microprocessor to execute the DSP-oriented multichannel convolution optimization method.

[0027] Compared with existing technologies, the present invention has the following advantages: The method of the present invention includes rearranging an input image F of size CI*(FH*FW) to obtain a rearranged image F' of size CI*(KH*KW*OH*OW), where CI is the number of input channels, FW is the width of the input image, FH is the height of the input image, KH is the height of the convolution kernel, KW is the width of the convolution kernel, OH is the height of the rearranged image, and OW is the width of the rearranged image; performing matrix multiplication K*F' yields a calculation result OU of size CO*(OH*OW) as the convolution calculation result, where CO is the number of output channels, K represents the convolution kernel, and * represents matrix multiplication. By converting convolution into matrix multiplication, the present invention can realize the vectorization of multi-channel convolution in deep learning, thereby maximizing the optimization of computational operations and reducing the computational load. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram illustrating the basic principle of the method in an embodiment of the present invention.

[0030] Figure 3 This is a schematic diagram illustrating the principle of rearrangement in an embodiment of the present invention.

[0031] Figure 4 This is a schematic diagram of the rearrangement process in an embodiment of the present invention.

[0032] Figure 5 This is a schematic diagram illustrating the principle of data transmission using the SG transmission mode in an embodiment of the present invention.

[0033] Figure 6 This is a schematic diagram of the process of performing matrix multiplication K*F' in an embodiment of the present invention.

[0034] Figure 7 This is an example of the matrix-vector multiplication logic in an embodiment of the present invention.

[0035] Figure 8 This is an example of the matrix-vector multiplication logic in an embodiment of the present invention.

[0036] Figure 9 This is a time sequence diagram for comparison in an embodiment of the present invention without the use of a double buffering mechanism.

[0037] Figure 10 This is a time sequence diagram showing the use of a double buffering mechanism in an embodiment of the present invention.

[0038] Figure 11 This is a schematic diagram of the front half of the soft water flow arrangement in the present invention and embodiments.

[0039] Figure 12 This is a schematic diagram of the latter half of the soft water flow arrangement in the present invention and embodiments. Detailed Implementation

[0040] like Figure 1 and Figure 2 As shown, the multi-channel convolution optimization method for DSP in this embodiment includes:

[0041] S101, rearrange the input image F of size CI*(FH*FW) to obtain a rearranged image F' of size CI*(KH*KW*OH*OW), where CI is the number of input channels, FW is the width of the input image, FH is the height of the input image, KH is the height of the convolution kernel, KW is the width of the convolution kernel, OH is the height of the rearranged image, and OW is the width of the rearranged image;

[0042] S102, perform matrix multiplication K*F' to obtain the calculation result OU of size CO*(OH*OW) as the convolution calculation result, where CO is the number of output channels, K represents the convolution kernel of size CO*CI*KH*KW used in the convolution calculation, and * represents matrix multiplication operation.

[0043] It should be noted that the method in this embodiment supports batch processing of input images F. Taking N=1 (N is the number of images) as an example, the convolutional kernel is K[CO][CI][KH][KW], and the feature map is F[1][CI][FH][FW], where CO is the number of output channels, CI is the number of input channels, KH is the height of the convolutional kernel, KW is the width of the convolutional kernel, FH is the height of the feature map, and FW is the width of the feature map. For the input image F, the value is obtained by sliding according to the size of the convolutional kernel, and C... I*(FH*FW) is rearranged (excluding padding) into a temporary matrix F'[1][CI][KH][KW][OH][OW] of (CI*KH*KW)*(OH*OW), where K does not need to be rearranged. OH represents the height of the rearranged image, and OW represents the width of the rearranged image. Matrix multiplication C = A*B (C is the output OU, A is the convolution kernel K, and B is the rearranged feature map F') yields an output OU[1][CO][OH][OW] of size CO*(OH*OW). [1] represents the first image. For example... Figure 3 In this case, the 3x3 matrix is ​​rearranged into a one-dimensional result, as shown below. Figure 3 The solid-line box in the image is shown.

[0044] like Figure 4 As shown, step S101, rearranging the input image F of size CI*(FH*FW), includes:

[0045] S201, initialize the value of loop variable i to 1;

[0046] S202, for the i-th input channel of the input image F with size CI*(FH*FW), the SG (Scatter Gather) transfer mode is adopted. According to the index Addr of the SG transfer mode, the data in the source address Src_Addr is moved from memory to the destination address Dst_Addr in the DSP core; then, ordinary point-to-point DMA transfer is used to move the data from the destination address Dst_Addr in the DSP core back to the memory space outside the core to generate a temporary matrix as the rearranged sub-image of size (KH*KW*OH*OW) for the i-th input channel;

[0047] S203, determine whether the value of the loop variable i is equal to the number of input channels CI. If not, increment the value of the loop variable i by 1 and jump to step S202 to continue iterating; otherwise, obtain a rearranged image of size CI*(KH*KW*OH*OW) composed of rearranged sub-images corresponding to all input channels.

[0048] Scatter Gather (SG) transfer mode is a DMA transfer mode. When the source address (Src_Addr) of the data is irregular, DMA cannot generate it and must generate it using the source base address and source index (addr). The source address index is stored in off-core memory and needs to be retrieved via DMA transfer. Figure 5 As shown, each source address Src_Addr( Figure 5 The data includes three source addresses (addr0 to addr3), each corresponding to a 64-bit word. The source address Src_Addr must be 64-bit aligned. The index Addr is a 32-bit integer that stores the offset of the number to be transmitted relative to the source address Src_Addr. For example, if addr[0] = 0, then Src_Addr[0] will be transmitted to the address of Dst_Addr[0]; if addr[1] = 2*8, then Src_Addr[2] will be transmitted to the address of Dst_Addr[1]. Through memory, it can be transmitted to the destination address Dst_Addr( Figure 5 This includes three destination address indices: dst0 to dst3. Figure 3 For example, padding is 0, stride is 2, kernel size K is 3*3, input image size F is 9*9, and input channels CI are 3. It should be noted that in step S202, when moving data from the destination address Dst_Addr within the DSP kernel back to the memory space outside the kernel to generate a temporary matrix, if zero padding is needed, the number before the source address can be set to 0, the source address can be shifted forward by one word, and the index of the position where zero padding is needed can be set to 0. To satisfy multi-channel convolution vectorization, the data to be calculated needs to be moved to the array memory AM. However, since the data is stored in one dimension in memory, the row and column sizes need to be specified to achieve partial matrix movement. Before moving, the entire matrix needs to be rearranged. The size of the calculation matrix is ​​determined by the size of the convolution kernel, and whether zero padding is needed for parts that are less than a multiple of the stride is determined by the corresponding padding. After zero padding, the size of the calculation matrix should not be affected, and the data in each row and column should not be changed due to zero padding. Matrix multiplication is performed after the movement.

[0049] Generally, rearranged convolutional data is too large to be processed in the kernel space (SM / AM) all at once. For example... Figure 6 As shown, in step S102 of this embodiment, performing matrix multiplication K*F' to obtain a calculation result OU of size CO*(OH*OW) as the convolution calculation result includes:

[0050] S301, initialize the first column of the convolution kernel K as the current column, rearrange the first row of the image F' as the current row, this step is in Figure 6The omission in the middle can be regarded as the initialization operation in "beginning of matrix multiplication";

[0051] S302 uses ordinary point-to-point DMA transfer to obtain an 8*M submatrix K' from the current column of the convolution kernel K and input it into the scalar space SM. Figure 6 (represented as SM space), where M is the block parameter;

[0052] S303, a submatrix F” of size M*48 obtained from traversing the current row of the rearranged image F' is transferred to the array memory AM via ordinary point-to-point DMA transfer. Figure 6 (represented as AM space);

[0053] S304, calculate a matrix multiplication of size (8*M)*(M*48) for submatrix K' in scalar space SM and submatrix F” in array memory AM;

[0054] S305, determine whether the submatrix F” has traversed a whole row of data in the rearranged image F’. If it has traversed a whole row of data in the rearranged image F’, then jump to step S306; otherwise, jump to step S303.

[0055] S306, determine whether the submatrix K' has traversed a whole column of data in the convolution kernel K. If it has traversed a whole column of data in the convolution kernel K, then jump to step S307; otherwise, jump to step S302.

[0056] S307, determine whether the convolution kernel K and the rearranged image F' have been traversed. If they have not been traversed, shift the convolution kernel K by one column and the rearranged image F' by one row, and jump to step S302; otherwise, use the calculated result OU of size CO*(OH*OW) as the convolution calculation result.

[0057] Step S303 involves transferring an M*48 submatrix F” obtained by traversing the current row of the rearranged image F' to the array memory AM via ordinary point-to-point DMA transfer; in this embodiment, the block parameter M is 512 bits. That is, matrix multiplication is converted into multiplication of multiple submatrices, treating each submatrix as an element, resulting in a formula similar to matrix multiplication. The convolution kernel K is divided into blocks smaller than 8*512, and the rearranged image F' is divided into blocks smaller than 512*48. Considering the kernel space size and the utilization rate of the FT3000 chip, the convolution kernel K is divided into 8*512 blocks, and the rearranged image F' is divided into 512*48 blocks.

[0058] In this embodiment, the matrix multiplication of size (8*M)*(M*48) for submatrix K' in scalar space SM and submatrix F” in vector space in step S304 is implemented using the matrix multiplication function GEMM8_R_48. GEMM8_R_48 can calculate the matrix multiplication of C[8]

[48] =C[8]

[48] +A[8][R]*B[R]

[48] , where R is any value. Assuming that the elements of the broadcast matrix A are a(0,0),a(1,0),……a(M-1,0),a(0,1),……, then the calculation process that this element needs to participate in is c(i,n)=c(i,n)+a(i,j)*b(j,n),n∈[0,N]. In the chip, the Vector Processing Unit (VPU) is a scalable vector operation cluster structure, which mainly handles computationally intensive parallel tasks and consists of 16 isomorphic Vector Processing Units (VPUs). It consists of Element (VPE), and each VPE is designed with three vector multiply-accumulate (MAC) units and one fixed-point arithmetic unit (IEU) to support large-scale parallel computing of MAC.

[0059] Therefore, according to Figure 7 In the calculation example shown, the element a in matrix A is... 0,0 The broadcast yields a vector `vector_a`. Matrix B can take 16 numbers at once for operations. Each VPE is designed with three vector multiply-add units (MACs), thus enabling three parallel multiply-add operations. 0,0 ,b 0,1 ...b 0,15 For vectors vector_b1, b 0,16 ,b 0,17 ...b 0,31 For vector_b2, b 0,32 ,b 0,17 ...b 0,47 Given a vector_b3, a can be calculated in the time it takes to perform a single multiplication-addition operation. 0,0 *b 0,0 a 0,0 *b 0,1 ...a 0,0 *b 0,47 There are 48 multiplication and addition operations in total, and each subsequent line is calculated using the same method. For example... Figure 8 In the computational example shown, when matrix A completes the broadcast of one column of data, matrix B only needs to access one row of data, while matrix C needs to complete the cumulative update of all position elements once. This method can maximize the parallelism of floating-point multiplication and addition. Figure 7 and Figure 8In the diagram, the direction at matrix A indicates the order in which elements are taken from matrix A for broadcasting; VPE1 M1 to VPE16 M1 and VPE2 M2 on the upper side of matrices B and C respectively represent vector operation units in the DSP.

[0060] In this embodiment, when performing matrix multiplication K*F' in step S102 to obtain a calculation result OU of size CO*(OH*OW) as the convolution calculation result, a double buffering mechanism is also adopted for the scalar space SM and the array memory AM. Furthermore, in steps S302 and S303, one buffer from the double buffer of the scalar space SM and the array memory AM is used alternately to store submatrices K' and F', so that in step S304, when calculating a matrix multiplication of size (8*M)*(M*48) for submatrices K' in the scalar space SM and F' in the array memory AM, the data transfer overhead and computational overhead overlap. This embodiment uses a double buffering mechanism to hide data transfer and computation, thereby maximizing computational efficiency. In the matrix multiplication algorithm, the main overhead can be divided into two parts: the data transfer overhead between the off-core DDR space and the on-core space SM / AM, and the computational overhead of calling GEMM8_R_48 block matrix multiplication. The data transfer overhead is mainly the overhead of DMA operations. To reduce the total overhead of the matrix multiplication algorithm, the overlap between data transfer overhead and computational overhead should be maximized. Figure 9 This is a time sequence diagram for comparison in this embodiment without the use of double buffering. Figure 10 This is a time sequence diagram of the double-buffering mechanism used in this embodiment, showing the calculation of the Corepac Gem GEMM8_R_48 block matrix multiplication, compared to... Figure 7 and Figure 8 It can be seen that by using the double buffering mechanism, the scalar space SM and the array memory AM in the kernel space can be divided into two parts. When the upper part is used for data transmission, the lower part is used for calculation. When the upper part is used for transmission and the lower part is used for calculation, the lower part is used for data transmission of the other part, and the upper part is used for calculation.

[0061] The main execution process within the loop includes memory access to matrix B, broadcasting of matrix A, and multiplication and addition calculation of matrix C. Clearly, there is no data dependency between memory access and broadcasting, allowing them to execute concurrently, while the calculation depends on the results of memory access and broadcasting. Furthermore, because cumulative calculations are required between different loops, the data writes from memory access and broadcasting in the (i+1)th loop cannot be earlier than the data reads from the multiplication and addition calculation in the ith loop. Figure 11 and Figure 12 In this embodiment of the invention, the soft pipeline layout is shown. Each column represents one cycle. The blank space in the middle of the instruction represents the number of cycles required to execute the instruction, called a delay slot. The presence of delay slots in the instruction will cause a blank cycle. To improve efficiency, delay slots should be hidden as much as possible. Each row in the figure represents the parallel instructions in each cycle. Through analysis... Figure 11 and Figure 12 It can be observed that the parallelism that may exist between different loops is achieved by fine-grained arrangement so that the i-th multiply-add calculation and the (i+1)-th memory access and broadcasting operations overlap, while strictly ensuring the order of data reading and writing. Therefore, in step S102 of this embodiment, when performing matrix multiplication K*F' to obtain the calculation result OU of size CO*(OH*OW) as the convolution calculation result, soft pipelining optimization is also performed on the DSP. This ensures that the matrix multiplication of size (8*M)*(M*48) for submatrix K' in scalar space SM and submatrix F” in array memory AM in any i-th execution of step S304, and the memory access operation of the rearranged image F' and the broadcasting operation of convolution kernel K corresponding to the matrix multiplication of size (8*M)*(M*48) for submatrix K' in scalar space SM and submatrix F” in array memory AM in the (i+1)-th execution of step S304, partially overlap in sequence.

[0062] In this embodiment, when performing matrix multiplication K*F' in step S102 to obtain the calculation result OU of size CO*(OH*OW) as the convolution calculation result, the process also includes expanding and parallelizing steps S302 to S306. Expanding the core loop reduces the total number of branch instructions, increases instruction parallelism, and allows parallel execution of instructions that meet the VLIW instruction rules, improving hardware utilization. Furthermore, it leverages the memory access instruction bandwidth advantage provided by the architecture to maximize memory access efficiency. The offsets in single-word and double-word memory access instructions do not need to be modified because the units of the offset register offset and the immediate offset are the same as the memory access granularity.

[0063] To verify the effectiveness of the method in this embodiment, code data before and after optimization by this method were compared for a certain version of the code. The experimental results are shown in Table 1.

[0064] Table 1: A comparative diagram of code data before and after optimization using this implementation method.

[0065]

[0066] In Table 1, K_size represents the convolution kernel size, CI is the number of input channels, CO is the number of output channels, and F_size represents the height and width of the input image (width = height). Comparing the time before and after optimization, it can be seen that the method in this embodiment can effectively improve the time of convolution operations, reduce the computational load, and improve computational efficiency.

[0067] In summary, the method of this embodiment includes rearranging an input image F of size CI*(FH*FW) to obtain a rearranged image F' of size CI*(KH*KW*OH*OW), where CI is the number of input channels, FW is the width of the input image, FH is the height of the input image, KH is the height of the convolution kernel, KW is the width of the convolution kernel, OH is the height of the rearranged image, and OW is the width of the rearranged image; performing matrix multiplication K*F' yields a calculation result OU of size CO*(OH*OW) as the convolution calculation result, where CO is the number of output channels, K represents the convolution kernel, and * represents the matrix multiplication operation. This invention, by converting convolution into a matrix multiplication operation, enables the vectorization of multi-channel convolution in deep learning, thereby maximizing the optimization of computational operations and reducing the computational load.

[0068] Furthermore, this embodiment also provides a DSP-oriented multi-channel convolution optimization system, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the DSP-oriented multi-channel convolution optimization method. This embodiment also provides a computer-readable storage medium storing a computer program for being programmed or configured by the microprocessor to execute the DSP-oriented multi-channel convolution optimization method.

[0069] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0070] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multi-channel convolution optimization method for DSP, characterized in that, include: S101, rearrange the input image F of size CI * (FH * FW) to obtain a rearranged image F' of size CI * (KH * KW * OH * OW), where CI is the number of input channels, FW is the width of the input image, FH is the height of the input image, KH is the height of the convolution kernel, KW is the width of the convolution kernel, OH is the height of the rearranged image, and OW is the width of the rearranged image; S102, perform matrix multiplication K*F' to obtain the calculation result OU of size CO*(OH*OW) as the convolution calculation result, where CO is the number of output channels, K represents the convolution kernel of size CO * CI * KH * KW used in the convolution calculation, and * represents matrix multiplication operation; Step S101, which involves rearranging the input image F of size CI * (FH * FW), includes: S201, initialize the value of loop variable i to 1; S202, for the i-th input channel of the input image F with size CI * (FH * FW), the SG transfer mode is adopted. According to the index Addr of the SG transfer mode, the data in the source address Src_Addr is moved from memory to the destination address Dst_Addr in the DSP core. Then, ordinary point-to-point DMA transfer is used to move the data from the destination address Dst_Addr in the DSP core back to the memory space outside the core to generate a temporary matrix as the rearranged sub-image with size (KH * KW * OH * OW) for the i-th input channel. When moving the data from the destination address Dst_Addr in the DSP core back to the memory space outside the core to generate a temporary matrix, if zero padding is required, the number before the source address is set to 0, the source address is shifted forward by one word, and the index of the position where zero padding is required is set to 0. S203, determine whether the value of the loop variable i is equal to the number of input channels CI. If not, increment the value of the loop variable i by 1 and jump to step S202 to continue the iteration; otherwise, obtain a rearranged image of size CI * (KH * KW * OH * OW) composed of rearranged sub-images corresponding to all input channels. In step S102, performing matrix multiplication K*F' yields a calculation result OU of size CO*(OH*OW), which is used as the convolution calculation result. S301, initialize the first column of the convolution kernel K as the current column, and rearrange the first row of the image F' as the current row; S302 uses ordinary point-to-point DMA transmission to traverse the current column of the convolution kernel K to obtain an 8*M submatrix K' and pass it into the scalar space SM, where M is the block parameter; S303, a submatrix F'' of size M*48 obtained from traversing the current row of the rearranged image F' is transferred to the array memory AM via ordinary point-to-point DMA transfer; S304, calculate a matrix multiplication of size (8*M)*(M*48) for submatrix K' in scalar space SM and submatrix F'' in array memory AM; S305, determine whether the submatrix F'' has traversed a whole row of data in the rearranged image F'. If it has traversed a whole row of data in the rearranged image F', then jump to step S306; otherwise, jump to step S303. S306, determine whether the submatrix K' has traversed a whole column of data in the convolution kernel K. If it has traversed a whole column of data in the convolution kernel K, then jump to step S307; otherwise, jump to step S302. S307, determine whether the convolution kernel K and the rearranged image F' have been traversed. If they have not been traversed, shift the convolution kernel K by one column and the rearranged image F' by one row, and jump to step S302; otherwise, use the calculated result OU of size CO*(OH*OW) as the convolution calculation result. In step S102, when performing matrix multiplication K*F' to obtain a calculation result OU of size CO*(OH*OW) as the convolution calculation result, a double buffering mechanism is also adopted for the scalar space SM and the array memory AM. In steps S302 and S303, one buffer of the double buffer of the scalar space SM and the array memory AM is used in turn to store the submatrix K' and the submatrix F'', so that the data transfer overhead and the calculation overhead overlap when calculating the matrix multiplication of size (8*M)*(M*48) for the submatrix K' in the scalar space SM and the submatrix F'' in the array memory AM in step S304. In step S102, when performing matrix multiplication K*F' to obtain the calculation result OU of size CO*(OH*OW) as the convolution calculation result, soft pipelining optimization is also performed on the DSP. This ensures that the matrix multiplication of size (8*M)*(M*48) performed in step S304 for submatrix K' in scalar space SM and submatrix F'' in array memory AM, and the memory access operation of the rearranged image F' and the broadcast operation of convolution kernel K corresponding to the matrix multiplication of size (8*M)*(M*48) performed in step S304 for submatrix K' in scalar space SM and submatrix F'' in array memory AM are performed in a partially overlapping order. The data write of memory access and broadcast in the (i+1)th loop cannot be earlier than the data read of multiply-add calculation in the ith loop.

2. The multi-channel convolution optimization method for DSP according to claim 1, characterized in that, When performing matrix multiplication K*F' in step S102 to obtain a calculation result OU of size CO*(OH*OW) as the convolution calculation result, it also includes expanding and parallel execution of steps S302 to S306.

3. The multi-channel convolution optimization method for DSP according to claim 1, characterized in that, In step S304, the matrix multiplication of size (8*M)*(M*48) for submatrix K' in scalar space SM and submatrix F'' in vector space is performed using the matrix multiplication function GEMM8_R_48.

4. The multi-channel convolution optimization method for DSP according to claim 1, characterized in that, The block parameter M is set to 512 bits.

5. A multi-channel convolution optimization system for DSPs, comprising interconnected microprocessors and memory, characterized in that, The microprocessor is programmed or configured to execute the DSP-oriented multichannel convolution optimization method according to any one of claims 1 to 4.

6. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to be programmed or configured by a microprocessor to execute the DSP-oriented multichannel convolution optimization method according to any one of claims 1 to 4.

Images