A half-precision deconvolution method based on a vector processor and related components

Abstract

The application discloses a half-precision deconvolution method based on a vector processor and related components, applied to the technical field of image processing, and comprises the following steps: loading half-precision input feature maps and half-precision filter data into on-chip scalar memory and on-chip array memory respectively; loading an input feature map subblock Fbsm, determining the effective value distribution characteristics of the Fbsm after deconvolution transformation based on the transformation principle of the deconvolution calculation; based on the effective value distribution characteristics, performing vectorized deconvolution Deconv4x4 calculation on the effective data at the corresponding position in the Fbsm and the half-precision filter data at the corresponding position, so as to obtain the calculation result of the Fbsm; and sequentially loading each Fbsm until the calculation result of each Fbsm is obtained, and then determining the output feature map result of the half-precision input feature map. The scheme disclosed by the application can effectively improve the execution efficiency of Deconv4x4 and obtain the output feature map result of the half-precision input feature map.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a half-precision deconvolution method based on a vector processor and related components. Background Technology

[0002] Deconvolution is one of the core operations of neural networks, and its concept is widely used in signal processing and image processing techniques. Deconv4×4 is a common deconvolution specification. Currently, Deconv4×4 computation is often inefficient on general-purpose platforms, making optimized Deconv4×4 computation crucial.

[0003] In conclusion, how to effectively improve the execution efficiency of Deconv4×4 is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a half-precision deconvolution method and related components based on a vector processor, so as to effectively improve the execution efficiency of Deconv4×4.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] A half-precision deconvolution method based on a vector processor includes:

[0007] The half-precision input feature map is loaded into the on-chip scalar memory, and the half-precision filter data is loaded into the on-chip array memory.

[0008] One input feature map sub-block Fbsm is loaded from the on-chip scalar memory, and the effective value distribution characteristics of the currently loaded input feature map sub-block Fbsm after deconvolution transformation are determined based on the transformation principle of deconvolution calculation.

[0009] Based on the effective value distribution characteristics, the effective data at the corresponding position in the input feature map sub-block Fbsm and the half-precision filter data at the corresponding position loaded from the on-chip array memory are subjected to vectorized deconvolution Deconv4×4 calculation to obtain the calculation result corresponding to the currently loaded input feature map sub-block Fbsm;

[0010] Return to the operation of loading one input feature map sub-block Fbsm from the on-chip scalar memory, load each input feature map sub-block Fbsm in sequence until the calculation result of each input feature map sub-block Fbsm is obtained, which is used as the determined output feature map result corresponding to the half-precision input feature map, and transfer the output feature map result to the off-chip storage space;

[0011] The data format of the input feature map sub-block Fbsm is [inCb, 2, inw, n], where inCb is the size of the sub-block in the input channel dimension, inw is the width of the half-precision input feature map, and n is the number of half-precision data that can be computed by a single register in the vector processor.

[0012] In one embodiment, loading the half-precision input feature map into the on-chip scalar memory and loading the half-precision filter data into the on-chip array memory includes:

[0013] Divide the half-precision input feature map F into n inCb Each input sub-block Fb is loaded into the on-chip scalar memory in batches.

[0014] Divide the half-precision filter data Q into n outCb ×n inCb Each filter data sub-block Qb is divided into n outCb ×n inCb Each of the filter data sub-blocks Qb is then loaded into the on-chip array memory in sequence;

[0015] The data format of the half-precision input feature map F is [inC, inh, inw, n], where inC is the number of input channels, inh is the height of the half-precision input feature map, inw is the width of the half-precision input feature map, and n is the number of half-precision data that can be calculated by a single register in the vector processor.

[0016] The data format of the half-precision filter data Q is [inC, outC, fs, fs, b], where inC is the number of input channels, outC is the number of output channels, fs is the filter side length, and b is the size that the half-precision filter data needs to be expanded and copied during the calculation.

[0017] outCb is the size of the sub-blocks in the output channel dimension, n outCb The number of blocks in the output channel dimension and inCb is the size of the sub-blocks in the input channel dimension, n inCb The number of blocks in the input channel dimension and

[0018] The data format of the input sub-block Fb is [inCb, inh, inw, n], the data format of the input feature map sub-block Fbsm is [inCb, 2, inw, n], and the data format of the filter data sub-block Qb is [fs, fs, inCb, outCb, b].

[0019] In one implementation, an input feature map sub-block Fbsm is loaded from the on-chip scalar memory, and based on the transformation principle of deconvolution calculation, the effective value distribution characteristics of the currently loaded input feature map sub-block Fbsm after deconvolution transformation are determined, including:

[0020] Initialize p h =0; p h For the input feature map sub-block Fb sm The initial position of [inCb,2,inw,n] is the index on the inh dimension of the half-precision input feature map F;

[0021] Initialize p ic =0; p ic For the input feature map sub-block Fb sm [inCb,2,inw,n] represents the input feature map F after being divided into blocks, with n elements. inCb Block indexes on dimensions;

[0022] Initialize p oc =0; p oc For the filter data sub-block Qb, after dividing the filter data Q into blocks n outCb Block indexes on dimensions;

[0023] Load the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm :[inCb,2,inw,n], load the corresponding filter data sub-block Qb:[fs,fs,inCb,outCb,b] from the on-chip array memory;

[0024] Initialize p w =0; p w The initial position of the input feature map sub-block Fbsm is the index in the inw dimension of the half-precision input feature map F;

[0025] From the currently loaded input feature map sub-block Fbsm, read inCb×2×2×n data points into a register, which will be used as the currently read data block F. rg Based on the transformation principle of deconvolution calculation, the data arrangement after deconvolution transformation of the currently loaded input feature map sub-block Fbsm is simulated, and the starting address is determined to be F. rg(inCb,0,pw,n) The 2×2 feature map is converted into a 4×4 pixel window and its layout type is determined, and the boundary information flag_a, flag_b, flag_c and flag_d are loaded.

[0026] Here, flag_a represents the result of determining whether it is the first row after being converted to a 4×4 pixel window, flag_b represents the result of determining whether it is the first column after being converted to a 4×4 pixel window, flag_c represents the result of determining whether it is the last row after being converted to a 4×4 pixel window, and flag_b represents the result of determining whether it is the last column after being converted to a 4×4 pixel window.

[0027] The layout types include the defined layout types A to D. Layout type A indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 1st row, 3rd row, 1st column, and 3rd column; layout type B indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 1st row, 3rd row, 2nd column, and 4th column; layout type C indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 2nd row, 4th row, 1st column, and 3rd column; and layout type D indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 2nd row, 4th row, 2nd column, and 4th column.

[0028] In one implementation, based on the effective value distribution characteristics, the effective data at the corresponding position in the input feature map sub-block Fbsm and the half-precision filter data at the corresponding position loaded from the on-chip array memory are subjected to vectorized deconvolution Deconv4×4 calculation to obtain the calculation result corresponding to the currently loaded input feature map sub-block Fbsm; the operation of loading one input feature map sub-block Fbsm from the on-chip scalar memory is returned, and each input feature map sub-block Fbsm is loaded sequentially until the calculation result of each input feature map sub-block Fbsm is obtained, which is used as the determined output feature map result corresponding to the half-precision input feature map, and the output feature map result is transferred to off-chip storage space, specifically including:

[0029] Initialize p k =0; p k For data block F rg Indexes on the inCb dimension;

[0030] Initialize i = 0; i is the data block F rg Indexes on the inh dimension;

[0031] Initialize j = 0, where j is the data block F. rg Indexes on the inw dimension;

[0032] Initialize each vector register storing the output data;

[0033] Determine p ic Is it 0? If p icIf the result is not 0, the intermediate Deconv4×4 product result temporarily stored in the on-chip array memory will be loaded into each of the vector registers.

[0034] If p ic If the value is 0, then a scalar load instruction with a granularity of n×2 bytes is used to read data block F in a single operation. rg The first n half-precision data are written to a scalar register, and the data in the scalar register are converted from scalars to a set of identical vector data and stored in the corresponding vector register using a broadcast instruction;

[0035] Based on the determined layout type of the 4×4 pixel window, the position of the corresponding data on the filter data sub-block Qb is determined, and the filter data block read during the calculation is denoted as Q. rg Q rg All data in the outCb dimension are loaded into p vector registers; among them, the filter data block Q read during the calculation is... rg The data format is [fs, fs, pk, outCb, b], where p represents the number of vector operation units in the vector processor;

[0036] data blocks The vectorized data are respectively compared with The data is multiplied by a dot, and the result of the dot product is then added to the corresponding values ​​in each of the vector registers. The S vector processing units perform the calculations synchronously and in parallel, and the intermediate calculation results are stored in each of the vector registers in sequence.

[0037] Let j = j + 1;

[0038] Determine if j is less than 2. If j is less than 2, return to execute the operation of using the scalar load instruction with a granularity of n×2 bytes;

[0039] If j is not less than 2, then let i = i + 1;

[0040] Determine if i is less than 2. If i is less than 2, return to execute the operation of using the scalar load instruction with a granularity of n×2 bytes;

[0041] If i is not less than 2, then let p k =p k +1;

[0042] Determine if pk is less than inCb. If pk is less than inCb, return to execute the initialization operation i=0. If pk is not less than inCb, then for F... rg In Q rg The deconvolution calculation of inCb input channels is completed to obtain 2×2×outCb×n intermediate result data;

[0043] Based on the loaded boundary information flag_a, flag_b, flag_c, and flag_d, data clearing is performed; specifically, when the judgment result represented by flag_a is yes, the first row of data is cleared; when the judgment result represented by flag_b is yes, the last row of data is cleared; when the judgment result represented by flag_c is yes, the first column of data is cleared; and when the judgment result represented by flag_d is yes, the last column of data is cleared.

[0044] The intermediate results on the same output channel are summed to obtain the final result of deconvolution, which is then saved to each of the aforementioned vector registers.

[0045] After completing data block F rg After the Deconv4×4 calculation is performed on the corresponding position data in the filter data sub-block Qb, the deconvolution results saved in each of the vector registers are stored in the on-chip array memory;

[0046] After calculating the pixel image of layout type B or the pixel image of layout type D, return to execute the operation of using the scalar loading instruction with a granularity of n×2 bytes;

[0047] After calculating the pixel image of layout type A or the pixel image of layout type C, let p w =p w +1;

[0048] Determine p w Is it less than inw? If p w If the value is less than inw, the process will return to executing the step of reading inCb×2×2×n data from the currently loaded input feature map sub-block Fbsm into a register, which will be used as the currently read data block F. rg Operation;

[0049] If p w If p is not less than inw, then let p ic =p ic +1;

[0050] Determine p ic Is it less than n? inCb If p ic Less than n inCb Then return to loading the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm The operation of [inCb,2,inw,n];

[0051] If p ic Not less than n inCbThis confirms that all input channels have been traversed in the deconvolution calculation, and a block of size n is allocated on the on-chip array memory. outCb ×inw×2×outCb×b's space AM temp The deconvolution results stored in each of the vector registers are then stored in the spatial AM. temp In, and let p oc =p oc +1;

[0052] Determine p oc Is it less than n? outCb If p oc Less than n outCb Then return to loading the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm The operation of [inCb,2,inw,n];

[0053] If p oc Not less than n outCb This confirms that all output channels have been traversed in the deconvolution calculation, resulting in n. outCb The output result is of size ×inw×2×outCb×b;

[0054] Space AM temp The data format is [n outCb The deconvolution result of [inw×2,outCb,n] is transposed to obtain [n outCb The result in the format [outCb,inw×2,n] is saved to the space AM. temp middle;

[0055] Space AM temp The output feature map results stored in the memory are stored in a designated location in the first memory.

[0056] Let p h =p h +1;

[0057] Determine p h Is it less than inh×2? If p h If the value is less than inh×2, then return to execute the initialization p. ic =0 operation;

[0058] If p h If the value is not less than inh×2, then the Deconv4×4 calculation of the half-precision input feature map F and the half-precision filter data Q is completed, the output feature map result corresponding to the half-precision input feature map F is obtained, and the output feature map result is transmitted to the off-chip storage space.

[0059] A half-precision deconvolution system based on a vector processor includes:

[0060] The loading module is used to load the half-precision input feature map into the on-chip scalar memory and load the half-precision filter data into the on-chip array memory.

[0061] The effective value distribution feature determination module is used to load an input feature map sub-block Fbsm from the on-chip scalar memory and determine the effective value distribution features of the currently loaded input feature map sub-block Fbsm after deconvolution transformation based on the transformation principle of deconvolution calculation.

[0062] The calculation module is used to perform vectorized deconvolution Deconv4×4 calculation on the effective data at the corresponding position in the input feature map sub-block Fbsm and the half-precision filter data at the corresponding position loaded from the on-chip array memory, based on the effective value distribution characteristics, to obtain the calculation result corresponding to the currently loaded input feature map sub-block Fbsm;

[0063] The output feature map result determination module is used to trigger the effective value distribution feature determination module until the calculation result of each input feature map sub-block Fbsm is obtained, which is used as the determined output feature map result corresponding to the half-precision input feature map, and the output feature map result is transmitted to the off-chip storage space.

[0064] The data format of the input feature map sub-block Fbsm is [inCb, 2, inw, n], where inCb is the size of the sub-block in the input channel dimension, inw is the width of the half-precision input feature map, and n is the number of half-precision data that can be computed by a single register in the vector processor.

[0065] In one implementation, the loading module is specifically used for:

[0066] Divide the half-precision input feature map F into n inCb Each input sub-block Fb is loaded into the on-chip scalar memory in batches.

[0067] Divide the half-precision filter data Q into n outCb ×n inCb Each filter data sub-block Qb is divided into n outCb ×n inCb Each of the filter data sub-blocks Qb is then loaded into the on-chip array memory in sequence;

[0068] The data format of the half-precision input feature map F is [inC, inh, inw, n], where inC is the number of input channels, inh is the height of the half-precision input feature map, inw is the width of the half-precision input feature map, and n is the number of half-precision data that can be calculated by a single register in the vector processor.

[0069] The data format of the half-precision filter data Q is [inC, outC, fs, fs, b], where inC is the number of input channels, outC is the number of output channels, fs is the filter side length, and b is the size that the half-precision filter data needs to be expanded and copied during the calculation.

[0070] outCb is the size of the sub-blocks in the output channel dimension, n outCb The number of blocks in the output channel dimension and inCb is the size of the sub-blocks in the input channel dimension, n inCb The number of blocks in the input channel dimension and

[0071] The data format of the input sub-block Fb is [inCb, inh, inw, n], the data format of the input feature map sub-block Fbsm is [inCb, 2, inw, n], and the data format of the filter data sub-block Qb is [fs, fs, inCb, outCb, b].

[0072] In one implementation, the effective value distribution characteristic determination module is specifically used for:

[0073] Initialize p h =0; p h For the input feature map sub-block Fb sm The initial position of [inCb,2,inw,n] is the index on the inh dimension of the half-precision input feature map F;

[0074] Initialize p ic =0; p ic For the input feature map sub-block Fb sm [inCb,2,inw,n] represents the input feature map F after being divided into blocks, with n elements. inCb Block indexes on dimensions;

[0075] Initialize p oc =0; p oc For the filter data sub-block Qb, after dividing the filter data Q into blocks n outCb Block indexes on dimensions;

[0076] Load the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm:[inCb,2,inw,n], load the corresponding filter data sub-block Qb:[fs,fs,inCb,outCb,b] from the on-chip array memory;

[0077] Initialize p w =0; p w The initial position of the input feature map sub-block Fbsm is the index in the inw dimension of the half-precision input feature map F;

[0078] From the currently loaded input feature map sub-block Fbsm, read inCb×2×2×n data points into a register, which will be used as the currently read data block F. rg Based on the transformation principle of deconvolution calculation, the data arrangement after deconvolution transformation of the currently loaded input feature map sub-block Fbsm is simulated, and the starting address is determined to be F. rg(inCb,0,pw,n) The 2×2 feature map is converted into a 4×4 pixel window and its layout type is determined, and the boundary information flag_a, flag_b, flag_c and flag_d are loaded.

[0079] Here, flag_a represents the result of determining whether it is the first row after being converted to a 4×4 pixel window, flag_b represents the result of determining whether it is the first column after being converted to a 4×4 pixel window, flag_c represents the result of determining whether it is the last row after being converted to a 4×4 pixel window, and flag_b represents the result of determining whether it is the last column after being converted to a 4×4 pixel window.

[0080] The layout types include the defined layout types A to D. Layout type A indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 1st row, 3rd row, 1st column, and 3rd column; layout type B indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 1st row, 3rd row, 2nd column, and 4th column; layout type C indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 2nd row, 4th row, 1st column, and 3rd column; and layout type D indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 2nd row, 4th row, 2nd column, and 4th column.

[0081] In one implementation, the calculation module and the output feature map result determination module are specifically used for:

[0082] Initialize p k =0; p k For data block F rg Indexes on the inCb dimension;

[0083] Initialize i = 0; i is the data block F rg Indexes on the inh dimension;

[0084] Initialize j = 0, where j is the data block F. rg Indexes on the inw dimension;

[0085] Initialize each vector register storing the output data;

[0086] Determine p ic Is it 0? If p ic If the result is not 0, the intermediate Deconv4×4 product result temporarily stored in the on-chip array memory will be loaded into each of the vector registers.

[0087] If p ic If the value is 0, then a scalar load instruction with a granularity of n×2 bytes is used to read data block F in a single operation. rg The first n half-precision data are written to a scalar register, and the data in the scalar register are converted from scalars to a set of identical vector data and stored in the corresponding vector register using a broadcast instruction;

[0088] Based on the determined layout type of the 4×4 pixel window, the position of the corresponding data on the filter data sub-block Qb is determined, and the filter data block read during the calculation is denoted as Q. rg Q rg All data in the outCb dimension are loaded into p vector registers; among them, the filter data block Q read during the calculation is... rg The data format is [fs, fs, pk, outCb, b], where p represents the number of vector operation units in the vector processor;

[0089] Data block F rg(pk,i,j,n) The vectorized data are respectively compared with Q rg(fs,fs,pk,outCb,b) The data is multiplied by a dot, and the result of the dot product is then added to the corresponding values ​​in each of the vector registers. The S vector processing units perform the calculations synchronously and in parallel, and the intermediate calculation results are stored in each of the vector registers in sequence.

[0090] Let j = j + 1;

[0091] Determine if j is less than 2. If j is less than 2, return to execute the operation of using the scalar load instruction with a granularity of n×2 bytes;

[0092] If j is not less than 2, then let i = i + 1;

[0093] Determine if i is less than 2. If i is less than 2, return to execute the operation of using the scalar load instruction with a granularity of n×2 bytes;

[0094] If i is not less than 2, then let p k =p k +1;

[0095] Determine if pk is less than inCb. If pk is less than inCb, return to execute the initialization operation i=0. If pk is not less than inCb, then for F... rg In Q rg The deconvolution calculation of inCb input channels is completed to obtain 2×2×outCb×n intermediate result data;

[0096] Based on the loaded boundary information flag_a, flag_b, flag_c, and flag_d, data clearing is performed; specifically, when the judgment result represented by flag_a is yes, the first row of data is cleared; when the judgment result represented by flag_b is yes, the last row of data is cleared; when the judgment result represented by flag_c is yes, the first column of data is cleared; and when the judgment result represented by flag_d is yes, the last column of data is cleared.

[0097] The intermediate results on the same output channel are summed to obtain the final result of deconvolution, which is then saved to each of the aforementioned vector registers.

[0098] After completing data block F rg After the Deconv4×4 calculation is performed on the corresponding position data in the filter data sub-block Qb, the deconvolution results saved in each of the vector registers are stored in the on-chip array memory;

[0099] After calculating the pixel image of layout type B or the pixel image of layout type D, return to execute the operation of using the scalar loading instruction with a granularity of n×2 bytes;

[0100] After calculating the pixel image of layout type A or the pixel image of layout type C, let p w =p w +1;

[0101] Determine p w Is it less than inw? If p w If the value is less than inw, the process will return to executing the step of reading inCb×2×2×n data from the currently loaded input feature map sub-block Fbsm into a register, which will be used as the currently read data block F. rg Operation;

[0102] If p w If p is not less than inw, then let p ic =p ic +1;

[0103] Determine p ic Is it less than n? inCb If p ic Less than ninCb Then return to loading the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm The operation of [inCb,2,inw,n];

[0104] If p ic Not less than n inCb This confirms that all input channels have been traversed in the deconvolution calculation, and a block of size no is allocated on the on-chip array memory. utCb ×inw×2×outCb×b's space AM temp The deconvolution results stored in each of the vector registers are then stored in the spatial AM. temp In, and let p oc =p oc +1;

[0105] Determine p oc Is it less than n? outCb If p oc Less than n outCb Then return to loading the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm The operation of [inCb,2,inw,n];

[0106] If p oc Not less than n outCb This confirms that all output channels have been traversed in the deconvolution calculation, resulting in n. outCb The output result is of size ×inw×2×outCb×b;

[0107] Space AM temp The data format is [n outCb The deconvolution result of [inw×2,outCb,n] is transposed to obtain [n outCb The result in the format [outCb,inw×2,n] is saved to the space AM. temp middle;

[0108] Space AM temp The output feature map results stored in the memory are stored in a designated location in the first memory.

[0109] Let p h =p h +1;

[0110] Determine p h Is it less than inh×2? If p h If the value is less than inh×2, then return to execute the initialization p. ic =0 operation;

[0111] If p hIf the value is not less than inh×2, then the Deconv4×4 calculation of the half-precision input feature map F and the half-precision filter data Q is completed, the output feature map result corresponding to the half-precision input feature map F is obtained, and the output feature map result is transmitted to the off-chip storage space.

[0112] A vector processor-based half-precision deconvolution device includes:

[0113] Memory, used to store computer programs;

[0114] A processor for executing the computer program to implement the steps of the vector processor-based half-precision deconvolution method as described above.

[0115] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the vector processor-based half-precision deconvolution method described above.

[0116] Applying the technical solution provided by the embodiments of the present invention, considering that half-precision deconvolution calculation can be implemented based on a vector processor, and since the solution of this application extracts the position of the non-zero value in the transformed data input feature map during calculation, that is, extracts the effective data, and performs convolution calculation with the non-zero value and the data at the corresponding position in the filter, the actual computation amount is significantly reduced compared with the theoretical computation amount of deconvolution.

[0117] Specifically, the half-precision input feature map is loaded into on-chip scalar memory, and the half-precision filter data is loaded into on-chip array memory. Then, one input feature map sub-block, Fbsm, can be loaded from the on-chip scalar memory. Based on the principle of deconvolution, two rows in the 4×4 feature map are padded with zeros. Therefore, only two rows of the original input data need to be loaded. Thus, the data format of a single-loaded Fbsm is [inCb, 2, inw, n], where inCb is the sub-block size in the input channel dimension, inw is the width of the half-precision input feature map, and n is the number of half-precision data that a single register in the vector processor can compute. Furthermore, based on the transformation principle of deconvolution, the effective value distribution characteristics of the currently loaded input feature map sub-block Fbsm after deconvolution transformation are determined. It can be understood that the effective value distribution characteristics reflect the positions of non-zero values. Therefore, based on the effective value distribution characteristics, the effective data at the corresponding positions in the input feature map sub-block Fbsm can be vectorized and deconvolved (Deconv4×4) with the half-precision filter data at the corresponding positions loaded from the on-chip array memory to obtain the calculation result corresponding to the currently loaded input feature map sub-block Fbsm. This reduces the computational cost of the proposed solution by 3 / 4 compared to the theoretical computational cost of deconvolution. After obtaining the calculation results of each input feature map sub-block Fbsm, the output feature map result of the half-precision input feature map can be determined. Attached Figure Description

[0118] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0119] Figure 1 This is a flowchart illustrating the implementation of a half-precision deconvolution method based on a vector processor in this invention.

[0120] Figure 2 This is a schematic diagram of the structure of a vector processor in a specific embodiment of the present invention;

[0121] Figure 3 A schematic diagram of the original input feature map of size 1×4×4 in the example of calculating Deconv4×4 for deconvolution;

[0122] Figure 4 A schematic diagram of the std-padded feature map in the example of calculating Deconv4×4 for deconvolution;

[0123] Figure 5In the example of deconvolution calculation of Deconv4×4, the distribution of pixels of four categories ABCD after std padding is shown.

[0124] Figure 6 In the example of calculating Deconv4×4 for deconvolution, after std padding, a schematic diagram of 4×4 convolution of the A-class pixel distribution map is shown.

[0125] Figure 7 A schematic diagram of the new feature map after std+pad padding in the example of deconvolution calculation of Deconv4×4;

[0126] Figure 8 In the example of calculating Deconv4×4 for deconvolution, the data size is inCb×2×2×n, and the F-value is... rg ;

[0127] Figure 9 In the example of calculating Deconv4×4 for deconvolution, the theoretically transformed new feature map is simulated under the conditions of std=2 and pad=2.

[0128] Figure 10 A schematic diagram illustrating the specific operation of scalar loading in the embodiment of deconvolution calculation of Deconv4×4;

[0129] Figure 11 A schematic diagram of the vectorization processing of input feature map data in the embodiment of deconvolution calculation of Deconv4×4;

[0130] Figure 12 A schematic diagram illustrating the specific operation of the broadcast command in the embodiment of deconvolution calculation of Deconv4×4;

[0131] Figure 13 In the example of deconvolution calculation of Deconv4×4, a flowchart is shown showing the effective value location and loading of filter data before calculation;

[0132] Figure 14 A schematic diagram illustrating the specific operation of vector loading in the embodiment of deconvolution calculation of Deconv4×4;

[0133] Figure 15 A schematic diagram of vector multiplication and addition in the embodiment of deconvolution calculation of Deconv4×4;

[0134] Figure 16 A schematic diagram of the data structure, loading, and vector multiplication and addition in the embodiment of deconvolution computation Deconv4×4;

[0135] Figure 17 A schematic diagram of the data structure, loading, and vector multiplication and addition in the embodiment of deconvolution computation Deconv4×4;

[0136] Figure 18 This is a schematic diagram illustrating the accumulation of intermediate results from the same output channel in the example of deconvolution calculation of Deconv4×4.

[0137] Figure 19 In the example of calculating Deconv4×4 for deconvolution, immediately following Figure 9 The subsequent pixel distribution map of class A;

[0138] Figure 20 This is a schematic diagram of the structure of a half-precision deconvolution system based on a vector processor in this invention. Detailed Implementation

[0139] The core of this invention is to provide a half-precision deconvolution method based on a vector processor, which can effectively improve the execution efficiency of Deconv4×4 and obtain the output feature map result of the half-precision input feature map.

[0140] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0141] Please refer to Figure 1 , Figure 1 The following is a flowchart illustrating an implementation of a vector processor-based half-precision deconvolution method according to the present invention. This vector processor-based half-precision deconvolution method may include the following steps:

[0142] Step S101: Load the half-precision input feature map into the on-chip scalar memory and load the half-precision filter data into the on-chip array memory.

[0143] Specifically, in the solution of this application, half-precision deconvolution calculation can be implemented based on a vector processor. The specific structure of the vector processor can be set and adjusted according to actual needs. For example, in a specific embodiment of the present invention, see [reference needed]. Figure 2 This is a schematic diagram of the structure of a vector processor in one specific implementation. The vector processor can employ a sequential execution architecture based on Very Long Instruction Words (VLE), for example... Figure 2Specifically, it consists of 32 DSP cores and 16 ARMv8 CPU cores. Each DSP core can contain a scalar processing unit (SPU) and a vector processing unit (VPU). The SPU can include a scalar processing unit (SPE) and an on-chip scalar memory (SM). The VPU can consist of S vector processing units (VPEs) and an on-chip array memory (AM). The VPEs can cooperate in a single instruction multiple data (SIMD) manner. A single VPE can integrate multiple vector processing units. For example, in one specific case, a single VPE can integrate three vector processing units to support both fixed-point and floating-point vector operations simultaneously. The DSP core can also include a direct memory access (DMA) unit to implement data transfer between the SM and DDR (Double Data Rate Synchronous Dynamic Random Access Memory), as well as between the AM and DDR.

[0144] Vector processors can simultaneously support FP64, FP32, and FP16 operations. A single VPE can process one 8-byte data unit (e.g., FP64, Int64), two 4-byte data units (e.g., FP32, Int32), or four 2-byte data units (e.g., FP16) at a time. The precision of half-precision floating-point is sufficient for inference computations in most applications in the field of artificial intelligence.

[0145] Furthermore, the principle of deconvolution is introduced. Deconvolution can be understood as the inverse operation of convolution. It reverses the steps in the convolution transformation process, transposes the filter, and then performs a convolution operation again with the result of the previous convolution. Deconvolution calculation involves three tensors: input feature map, filter, and output feature map. The calculation parameters can be set as follows: let the deconvolution stride be std, the filter side length be fs, and the padding size be pad. In the traditional scheme, the specific deconvolution calculation process is as follows: ① Take the original convolution result as the input feature map, and perform a zero-padding operation according to the parameter std, that is, pad std-1 zeros between each element of the feature map to obtain a new feature map. ② On the basis of the expanded feature map, pad the boundaries again according to the parameter pad, each time padding fs-pad-1 zeros. ③ Obtain the transposed filter data, and perform a regular convolution on the new feature map to obtain the final deconvolution result.

[0146] In the scheme of this application, the half-precision deconvolution calculation based on the vector processor is also applicable to the deconvolution calculation of fs=4, std=2, pad=1, that is, the calculation of Deconv4×4. In the following text, the 4×4 deconvolution calculation will also be referred to as Deconv4×4.

[0147] Specifically, the half-precision input feature map F can be stored in DDR, and then a direct memory access operation can be called to load the half-precision input feature map F from DDR into the on-chip scalar memory SM. Similarly, the half-precision filter data Q can be stored in DDR, and then a direct memory access operation can be called to load the half-precision filter data Q from DDR into the on-chip array memory AM.

[0148] Furthermore, in one specific embodiment of the present invention, considering that the storage space of the on-chip scalar memory SM and the on-chip array memory AM is limited, a batch loading method can be adopted. That is, after loading a batch of data, the calculation of this part of the data is performed, and then the next batch of data is loaded, until the complete loading of the half-precision input feature map F and the half-precision filter data Q is completed.

[0149] In one specific embodiment of the present invention, step S101 may specifically include:

[0150] Divide the half-precision input feature map F into n inCb Each input sub-block Fb is loaded into the on-chip scalar memory in batches.

[0151] Divide the half-precision filter data Q into n outCb ×n inCb Each filter data sub-block Qb is divided into n outCb ×n inCb Each filter data sub-block Qb is loaded into the on-chip array memory in turn.

[0152] In this implementation, the data format of the half-precision filter data Q is [inC, outC, fs, fs, b], where inC is the number of input channels, outC is the number of output channels, fs is the filter side length (in this application, fs = 4), and b is the size of the half-precision filter data that needs to be expanded and copied during calculation. For b, given that the half-precision data bit length is 16, and for example, the bit length of a register in a vector processor is d, the number of half-precision data that can be calculated in one register is... In the specific implementation of deconvolution, Each input data point should operate on the same filter data, therefore the filter data needs to be copied. Each part is involved in the calculation, that is

[0153] The half-precision filter data Q can be divided into blocks along the input and output channel dimensions, resulting in a data format of [n] after conversion. outCb ,n inCbTherefore, we can obtain the filter data sub-block Qb: [fs, fs, inCb, outCb, b], that is, the data format of the filter data sub-block Qb is [fs, fs, inCb, outCb, b]. outCb is the sub-block size in the output channel dimension, n. outCb The number of blocks in the output channel dimension and Furthermore, outCb = S × P, where P represents the number of vector operation units in a single vector processing unit (VPE) within the vector processor architecture, for example, P = 3. inCb is the sub-block size in the input channel dimension, n inCb The number of blocks in the input channel dimension and In conjunction with vector processor architecture features, such as inCb, it can be set to 64.

[0154] As can be seen from the above block division principle, calling the direct memory access operation can divide the half-precision filter data Q into n blocks. outCb ×n inCb Then, according to the granularity Qb, each filter data sub-block Qb is loaded into the AM space in multiple batches.

[0155] The half-precision input feature map is denoted as F, and its data format is [inC, inh, inw, n], where inC is the number of input channels, inh is the height of the half-precision input feature map, inw is the width of the half-precision input feature map, and n is the number of half-precision data that a single register in the vector processor can compute. As analyzed above...

[0156] The output feature map obtained after deconvolution can be denoted as O, with the data format O:[outC,outh,outw,n], where outC is the number of output channels, outh is the height of the output feature map, and outw is the width of the output feature map. Since the on-chip scalar memory SM has limited space, the half-precision input feature map F needs to be divided into blocks. Given that the filter data sub-blocks inC dimension are divided into inCb blocks, the input feature map can also be divided into sub-blocks inCb. That is, the half-precision input feature map can be denoted as the sub-block Fb of F, with the data format [inCb, inh, inw, n], and the number of blocks is n. inCb That is, we get n inCb One input sub-block Fb.

[0157] Based on the above block division requirements, direct memory access operations can be invoked to divide the half-precision input feature map F into n blocks. inCbThe input sub-block Fb is loaded multiple times into the on-chip scalar memory SM space. However, it should be noted that, theoretically, four rows of input feature map data should be loaded each time the vectorization Deconv4×4 operation is performed, based on computational requirements. But since two rows in the 4×4 feature map are padded zeros, only two rows of the original input data are actually loaded. That is, only the data [inCb,2,inw,n] needs to be loaded at a time, which can be considered as Fb. sm [inCb,2,inw,n]. That is to say, after the blocks are divided, each input feature map sub-block Fbsm is loaded into the on-chip scalar memory in batches. The data format of the input feature map sub-block Fbsm is [inCb,2,inw,n].

[0158] Step S102: Load an input feature map sub-block Fbsm from the on-chip scalar memory, and determine the effective value distribution characteristics of the currently loaded input feature map sub-block Fbsm after deconvolution transformation based on the transformation principle of deconvolution calculation.

[0159] For easier understanding, please refer to the following: Figure 3 This is a schematic diagram of the original input feature map of size 1×4×4 in the Deconv4×4 example for deconvolution calculation. That is, if the original input feature map size is 1×4×4, and given fs=4, std=2, pad=2, the original input feature map would look like this. Figure 3 As shown.

[0160] Based on the principle of deconvolution, std-1 zeros are inserted between each element of the original input feature map, that is, one zero is padded between each element, resulting in the padded feature map. Figure 4 At this point, without considering padding, we temporarily obtain a 7x7 image. We then acquire the 4x4 filter data... Figure 2 Perform a standard 4×4 convolution calculation. When the filter is in a 4×4 window... Figure 2 When swiping from left to right and from top to bottom, it can be observed that the 4x4 pixel window it defines has a regularity: only four pixels within the 4x4 window are always valid values, with the rest being filled with zeros. The pixel distribution of these valid pixels within the 4x4 window can be categorized as follows: Figure 5 The four categories shown are A, B, C, and D.

[0161] In these four types of pixel windows, the positions of the valid input data values ​​are fixed, therefore the acquisition of the valid filter values ​​is also deterministic. Once we determine which type the 4×4 pixel window arrangement belongs to, we can perform calculations using only the four values ​​at specific positions out of the 16 pixel values / filter elements, thus reducing the actual computational load to one-quarter of the theoretical computational load. For example, for... Figure 5 For a Class A pixel window, you only need to obtain it in the filter. Figure 6 The filter data at the four positions shown can be used for subsequent convolution calculations.

[0162] Of course, in practice, based on the pad parameter, it is also necessary to pad the feature map after std padding with fs-pad-1 zeros on its boundaries, that is, to pad each of the four sides of the boundary with 2 zeros, to obtain the following: Figure 7 The diagram shows a new 11×11 feature map, which is then subjected to a 4×4 regular convolution, and finally the deconvolution calculation is completed.

[0163] Analysis of the deconvolution calculation process reveals that after padding, the effective value positions of the 4×4 pixel window convolved at the boundary have different arrangements than the four cases A, B, C, and D described above. However, they can all be regarded as variations of the four arrangements listed above, and their variation patterns are closely related to the boundary information. That is, after performing corresponding operations based on the boundary information, it can be equivalent to the four cases A, B, C, and D listed above.

[0164] Based on the above analysis, in the scheme of this application, after loading an input feature map sub-block Fbsm from the on-chip scalar memory, it does not directly fill it with 0 as in the traditional scheme. Instead, it determines the effective value distribution characteristics of the currently loaded input feature map sub-block Fbsm after the deconvolution transformation based on the transformation principle of deconvolution calculation.

[0165] For example, in one specific embodiment of the present invention, step S102 may specifically include:

[0166] Step 1: Initialize p h =0; p h For the input feature map sub-block Fb sm The initial position of [inCb,2,inw,n] is the index on the inh dimension of the half-precision input feature map F;

[0167] Step 2: Initialize p ic =0; p ic For the input feature map sub-block Fb sm [inCb,2,inw,n] represents the input feature map F after being divided into blocks, with n elements. inCb Block index on dimension

[0168] Step 3: Initialize p oc =0; p oc For the filter data sub-block Qb, after dividing the filter data Q into blocks n outCb Block indexes on dimensions;

[0169] Step 4: Load the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm[inCb,2,inw,n], loads the corresponding filter data sub-block Qb:[fs,fs,inCb,outCb,b] from the on-chip array memory;

[0170] Step 5: Initialize p w =0; p w The initial position of the input feature map sub-block Fbsm is the index in the inw dimension of the half-precision input feature map F;

[0171] Step 6: Read inCb×2×2×n data points from the currently loaded input feature map sub-block Fbsm into a register, which will be used as the currently read data block F. rg Based on the transformation principle of deconvolution, the data arrangement after deconvolution transformation of the currently loaded input feature map sub-block Fbsm is simulated, and the starting address is determined to be F. rg(inCb,0,pw,n) The 2×2 feature map is converted into a 4×4 pixel window and its layout type is determined, and the boundary information flag_a, flag_b, flag_c and flag_d are loaded.

[0172] Here, flag_a represents the result of determining whether it is the first row after being converted to a 4×4 pixel window, flag_b represents the result of determining whether it is the first column after being converted to a 4×4 pixel window, flag_c represents the result of determining whether it is the last row after being converted to a 4×4 pixel window, and flag_b represents the result of determining whether it is the last column after being converted to a 4×4 pixel window.

[0173] The layout types include the defined layout types A to D. Layout type A indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 1st row, 3rd row, 1st column, and 3rd column; layout type B indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 1st row, 3rd row, 2nd column, and 4th column; layout type C indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 2nd row, 4th row, 1st column, and 3rd column; and layout type D indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 2nd row, 4th row, 2nd column, and 4th column.

[0174] Specifically, in this implementation, steps 1 to 3 and step 5 can initialize the corresponding parameters, while step 4 can load the corresponding filter data sub-block Qb in AM, where fs is the filter size 4, and load the corresponding word block Fbsm in SM.

[0175] As can be seen from the analysis above, after pad conversion, the arrangement of the effective value positions of the 4×4 pixel window convolved by the boundary part can be different from the four cases A, B, C, and D described above. However, they can all be regarded as variations of the four arrangements listed above, and their variation pattern is closely related to the boundary information. That is, after performing corresponding operations based on the boundary information, it can be equivalent to the four cases A, B, C, and D listed above.

[0176] In this regard, this application takes into account the parameters flag_a, flag_b, flag_c, and flag_d that can be set to determine whether it is the first row, the first column, the last row, and the last column, respectively. For example, "yes" is represented by 0 and "no" is represented by 1.

[0177] When acquiring 4×4 pixel window data, the data address needs to be offset according to the boundary parameters. The specific rules are as follows: If it is the first row, the starting address of the input data needs to be offset forward by an address of the entire row size, which is the size of one frame length transferred from DMA to SM; if it is the first column, the starting address of the input data needs to be offset forward by an address of double precision length; if it is the last row or the last column, no offset is required.

[0178] After optimizing the convolution calculation according to the four arrangement methods mentioned above, the intermediate dot product results are further processed based on the boundary parameters as follows: if it is the first row, the dot product result of the first row is cleared to zero; if it is the last row, the dot product result of the last row is cleared to zero; if it is the first column, the dot product result of the first column is cleared to zero; if it is the last column, the dot product result of the last column is cleared to zero. Finally, the data is summed up according to the convolution calculation requirements.

[0179] Based on the above analysis, in this implementation method, when executing step 6, inCb×2×2×n data can be read from SM into the register, and this data block is named F. rg Then, you can simulate Fb. sm The data layout after stride+pad transformation, i.e., determining the starting address as F. rg(inCb,0,pw,n) After the 2×2 feature map is converted into a 4×4 pixel window, it belongs to which of the four categories A, B, C, and D? At the same time, the boundary information flag_a, flag_b, flag_c, and flag_d can be passed into the function.

[0180] In other words, read data block F rg Next, based on the transformation principle of deconvolution, it is necessary to simulate the data arrangement after deconvolution transformation of the currently loaded input feature map sub-block Fbsm, so as to determine the starting address F. rg(inCb,0,pw,n)The 2×2 feature map is converted into a 4×4 pixel window and its layout type is determined, and the boundary information flag_a, flag_b, flag_c and flag_d are loaded.

[0181] Furthermore, as described above, for the set layout type A, please refer to... Figure 5 This indicates that after conversion to a 4x4 pixel window, a valid value exists in row 1, row 3, column 1, and column 3. Similarly, for layout type B, this indicates that after conversion to a 4x4 pixel window, a valid value exists in row 1, row 3, column 2, and column 4. For layout type C, this indicates that after conversion to a 4x4 pixel window, a valid value exists in row 2, row 4, column 1, and column 3. For layout type D, this indicates that after conversion to a 4x4 pixel window, a valid value exists in row 2, row 4, column 2, and column 4.

[0182] For example, Figure 8 It is a data block F with size inCb×2×2×n. rg Under simulated conditions of std=2 and pad=2, the theoretically transformed new feature map is as follows: Figure 9 This is the B-type layout, where each square in the image contains n pixels.

[0183] Step S103: Based on the effective value distribution characteristics, the effective data at the corresponding position in the input feature map sub-block Fbsm and the half-precision filter data at the corresponding position loaded from the on-chip array memory are vectorized deconvolution Deconv4×4 to obtain the calculation result corresponding to the currently loaded input feature map sub-block Fbsm.

[0184] Step S104: Return to the operation of step S102, load each input feature map sub-block Fbsm in sequence until the calculation result of each input feature map sub-block Fbsm is obtained, and use it as the determined output feature map result corresponding to the half-precision input feature map, and transfer the output feature map result to the off-chip storage space.

[0185] Traditional methods require zero-padding and boundary padding of the input feature map to obtain a new feature map, which is then convolved with the filter data using conventional methods. However, in this application, no actual zero-padding or boundary padding is performed during deconvolution. Instead, the original input feature map and filter data are directly transmitted to the SM and AM spaces. A transformed new feature map is then simulated. During computation, the positions of non-zero values ​​in the new feature map are extracted, and these non-zero values ​​are convolved with the corresponding positions in the filter data. This significantly reduces the computational cost compared to traditional deconvolution.

[0186] In other words, after obtaining the effective value distribution characteristics, this application will perform vectorized deconvolution Deconv4×4 calculation on the effective data at the corresponding position in the input feature map sub-block Fbsm and the half-precision filter data at the corresponding position loaded from the on-chip array memory to obtain the calculation result corresponding to the currently loaded input feature map sub-block Fbsm.

[0187] By sequentially loading each input feature map sub-block Fbsm, the calculation results corresponding to each input feature map sub-block Fbsm can be obtained. After obtaining the calculation results of each input feature map sub-block Fbsm, the output feature map result of the half-precision input feature map is determined.

[0188] In one specific embodiment of the present invention, steps S103 and S104 may specifically include:

[0189] Step 7: Initialize p k =0; p k For data block F rg Indexing on the inCb dimension. It is understood that this step is performed after step 6 in the above implementation; therefore, step numbering begins here with step 7.

[0190] Step 8: Initialize i = 0; i is the data block F rg Indexes on the inh dimension;

[0191] Step 9: Initialize j = 0, where j is the data block F rg Indexes on the inw dimension;

[0192] Step 10: Initialize the vector registers used to store the output data. The vector registers used to store the output data can be preset; for example, in a specific scenario, the vector register storing the output data can be designated as VR. 11+i×j×p+j×p VR 12+i×j×p+j×p VR 11+2×2×p-1 These vector registers are initialized to 0. The subscript P indicates the number of vector operation units in a single vector processing unit (VPE) in the vector processor architecture. For example, when P = 3, in this case, the vector registers VR need to be initialized to 0. 11 To Vector Register VR 22 The value inside is cleared to zero.

[0193] Step 11: Determine p ic Is it 0? If p ic If the result is not 0, the intermediate Deconv4×4 product result temporarily stored in the on-chip array memory is loaded into each vector register. For example, in one scenario, if p icIf the value is not 0, the intermediate Deconv4×4 product result temporarily stored in the on-chip array memory AM can be loaded into the aforementioned vector registers VR. 11+i×j×p+j×p VR 12+i×j×p+j×p VR 11+2×2×p-1 In the middle. If p ic If the value is 0, then this step can be skipped and the next step can be executed directly.

[0194] Step 12: Use scalar load instructions with a granularity of n×2 bytes to read data block F in a single operation. rg The first n half-precision data are written to a scalar register, and the data in the scalar register are converted from scalars to a set of identical vector data and stored in the corresponding vector register using a broadcast instruction.

[0195] For example, in one scenario, using scalar load instructions with a granularity of n×2 bytes, F can be read in a single operation. rg(pk,i,j,n) Write the first n half-precision data values ​​into a scalar register, for example, read them into scalar register R. 30 Then, use the broadcast command to send R 30 Data is converted from a scalar into a set of identical vector data and stored in a vector register, such as VR. 10 .

[0196] For example, when i = 0, j = 0, p k When = 0, read F rg The first element of the feature map of the first input channel, i.e., A0, is stored in R. 30 Then the scalar register R 30 Data broadcast to vector register VR 10 ,like Figure 10 and 11 As shown, given n=4, cell A0 contains 4 half-precision data points, and a scalar instruction loads 4 data points into R in a single operation. 30 In the middle. Based on scalar register R 30 The half-precision data stored in the vector register VR is broadcast and stored in the vector register VR. 10 In this example, with each register having a bit length of 64 and S=8, a 64-bit length can store four half-precision data points. SS vector processing units (VPEs) store the same data. The specific broadcast operation is as follows: Figure 12 As shown.

[0197] Step 13: Based on the determined layout type of the 4×4 pixel window, determine the position of the corresponding data on the filter data sub-block Qb, and let the filter data block read during the calculation be Q. rg Q rgAll data in the outCb dimension are loaded into p vector registers; among them, the filter data block Q read during the calculation is... rg The data format is [fs, fs, pk, outCb, b], where p represents the number of vector operation units in the vector processor.

[0198] Specifically, based on the description in step 6 above, the layout type of the 4×4 pixel window can be determined, thereby determining the positions of non-zero pixel values. This also allows us to determine the position of the corresponding data on the filter data sub-block Qb in the AM space. Let Qb be the filter data block read by the register in the actual calculation. rg Q rg(fs,fs,pk,outCb,b) All data in dimension outCb is loaded into p vector registers VR0, VR1...VR p-1 In the context of vector loading instructions, the minimum number of loads per instruction is [not specified]. Half-precision data.

[0199] like Figure 13 Given F rg Theoretically, the transformed feature map of the data block is a type B pixel distribution map, meaning the layout type of the determined 4×4 pixel window is B. Based on this layout type, the data at the corresponding position on the filter is obtained. The vector loading instruction can load outCb channels of data into VR0, VR1...VR in a single operation. p-1 In the figure, each square contains outCb×b half-precision data.

[0200] like Figure 14 Taking a register with a bit length of 64 and S=8 as an example, the vector load instruction loads the filter data B1 into the vector register VR0. The instruction loads in a single operation. Data points.

[0201] Step 14: Transfer data block F rg(pk,i,j,n) The vectorized data are respectively compared with Q rg(fs,fs,pk,outCb,b) The data is multiplied by a dot product, and the result is then added to the corresponding values ​​in each vector register. The S vector processing units perform the calculations synchronously and in parallel, and the intermediate calculation results are stored in each vector register in turn.

[0202] Specifically, in a certain situation, such as Figure 15 As shown, F rg(pk,i,j,n) Vectorized data VR 10 respectively with Q rg(fs,fs,pk,outCb,b) Data VR0, VR1...VR p-1 Perform a dot product operation, and then multiply the result by the vector register VR. 11+i×j×p+j×p VR 12+i×j×p+j×p VR 11+2×2×p-1The values ​​are accumulated, and the S vector processing units perform synchronous and parallel calculations. The intermediate calculation results are stored sequentially in the vector register VR. 11+i×j×p+j×p VR 12+i×j×p+j×p VR 11+2×2×p-1 middle.

[0203] Step 15: Let j = j + 1;

[0204] Step 16: Determine if j is less than 2. If j is less than 2, return to the operation of using a scalar load instruction with a granularity of n×2 bytes, that is, return to step 12. If j is not less than 2, continue to the next step.

[0205] Step 17: Let i = i + 1;

[0206] Step 18: Determine if i is less than 2. If i is less than 2, return to the operation of using a scalar load instruction with a granularity of n×2 bytes, that is, return to step 12. If i is not less than 2, continue to the next step.

[0207] Understandably, at this point, the input channel index F is pk. rg(pk,i,j,n) The original 2×2 pixel window and Q rg(fs,fs,pk,outCb,b) The 4×4 filter vector multiplication and addition of the above outCb output channels is completed, and the intermediate results are stored sequentially in the vector register VR. 11+i×j×p+j×p VR 12+i×j×p+j×p VR 11+2×2×p-1 In the middle. At this time, as Figure 16 Therefore, the 2×2 pixel window traversal has ended, indicating that all four valid pixel positions have been calculated.

[0208] Step 19: Let p k =p k +1;

[0209] Step 20: Determine if pk is less than inCb. If pk is less than inCb, return to execute the initialization operation i=0, that is, return to step 8.

[0210] If pk is not less than inCb, then for F rg In Q rg The deconvolution calculation for the inCb input channels is completed, resulting in 2×2×outCb×n intermediate result data. At this point, as... Figure 17 As shown, the traversal of the inC dimension on the sub-block has ended, indicating that the data of the inCb input channels has been calculated, resulting in 2×2×outCb×n intermediate result data.

[0211] Step 21: Based on the loaded boundary information flag_a, flag_b, flag_c, and flag_d, perform data clearing.

[0212] Specifically, when the judgment result represented by flag_a is yes, that is, if flag_a = 0, then the first row of data is cleared to zero, which means the vector register VR is reset. 11+i×0×p+0×p VR 12+i×0×p+0×p VR 11+p-1+i×0×p+0×p Reset to zero.

[0213] When the judgment result represented by flag_b is yes, that is, if flag_b = 0, then the last row of data is cleared to zero, which means the vector register VR is cleared. 11+i×1×p+1×p VR 12+i×1×p+1×p VR 11+p-1+i×1×p+1×p Reset to zero.

[0214] When the judgment result represented by flag_c is yes, that is, if flag_c = 0, then the first column of data is cleared to zero, which means the vector register VR is reset. 11+0×j×p+j×p VR 12+0×j×p+j×p VR 11+p-1+0×j×p+0×p Reset to zero.

[0215] When the judgment result represented by flag_d is yes, the last column of data is cleared to zero, that is, the vector register VR is cleared. 11+1×j×p+j×p VR 12+1×j×p+j×p VR 11+p-1+1×j×p+1×p Reset to zero.

[0216] Step 22: Accumulate the intermediate results on the same output channel to obtain the final result of deconvolution, and save it to each vector register.

[0217] After clearing the output channels accordingly, the intermediate results on the same output channel are summed to obtain the final deconvolution result, which can then be saved to VR. 11 VR 12 VR 12+p-1 .middle.

[0218] like Figure 18 As shown, the feature map after the simulation transformation is not at the boundary of the original feature map, so the boundary parameters are all 1, and there is no need to clear them. It is only necessary to accumulate the data of the 4 pixels on the same output channel.

[0219] Step 23: After completing data block F rg After performing Deconv4×4 calculations on the corresponding data positions on the filter data sub-block Qb, the deconvolution results saved in each vector register are stored in the on-chip array memory.

[0220] At this time, due to data block F rg Once the Deconv4×4 calculation of the corresponding position data on the filter data sub-block Qb is completed, VR can be... 11 VR12 VR 12+p-1 The deconvolution results are stored in the AM space.

[0221] Step 24: After calculating the pixel image of layout type B or layout type D, return to execute the operation using a scalar load instruction with a granularity of n×2 bytes; after calculating the pixel image of layout type A or layout type C, let p w =p w +1.

[0222] It is known that pixel distribution maps of classes B and A exist in pairs; B is always followed by A, and the effective pixel values ​​of B are exactly the same as those of A, only their distribution positions differ. Therefore, after calculating the pixel map of class B, it is not necessary to reload the input feature map into SM; we can directly jump to step 12 and perform deconvolution calculation according to the distribution map of class A. The same applies to the pixel distribution maps of classes D and C. Of course, if the pixel map is neither class B nor class D, then p can be left as an option. w =p w +1, and continue to the next step.

[0223] by Figure 9 For example, according to the above embodiment, the feature map after the last transformation is... Figure 9 That is, the pixel distribution map of class B. According to the principle of deconvolution input feature map transformation, a 4×4 pixel window in the inw dimension is... Figure 19 This refers to the pixel distribution map of type A, whose effective data used in the calculation are... Figure 9 Similarly, there is no need to repeatedly load input data from SM; simply return to step 12.

[0224] Step 25: Determine p w Is it less than inw? If p w If the value is less than inw, execution will return to read inCb×2×2×n data points from the currently loaded input feature map sub-block Fbsm into a register, which will be used as the currently read data block F. rg The operation is to return to step 6.

[0225] If p w If p is not less than inw, then let p ic =p ic +1. If p w If the value is not less than inw, it means that the input feature map sub-block Fb on SM is not less than inw. sm The Deconv4×4 operation is completed with the corresponding position data on the AM filter data sub-block Qb, therefore p can be set... ic =p ic +1, and continue to the next step.

[0226] Step 26: Determine p ic Is it less than n? inCb If p ic Less than n inCb Then return to load the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm The operation [inCb,2,inw,n] returns to step 4.

[0227] If p ic Not less than n inCb This confirms that all input channels have been traversed in the deconvolution calculation, and that a block of size n has been allocated on the on-chip array memory AM. outCb ×inw×2×outCb×b's space AM temp This allows the deconvolution results stored in each vector register to be stored in the spatial AM. temp In, and let p oc =p oc +1. For example, the data stored in the vector register VR can be... 11 VR 12 VR 12+p-1 The final deconvolution result is stored in AM. temp Proceed to the next step.

[0228] Step 27: Determine p oc Is it less than n? outCb If p oc Less than n outCb Then return to load the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm The operation [inCb,2,inw,n] returns to step 4.

[0229] If p oc Not less than n outCb This confirms that all output channels have been traversed in the deconvolution calculation, resulting in n. outCb The output result of size ×inw×2×outCb×b can be used to continue to the next step.

[0230] Step 28: Place the space AM temp The data format is [n outCb The deconvolution result of [inw×2,outCb,n] is transposed to obtain [n outCb The result in the format [outCb,inw×2,n] is saved to the space AM. temp middle.

[0231] When executing step 28, AM is at this time temp The deconvolution result data format is [n outCbThe required output data format is [n, inw×2, outCb, n]. outCb Therefore, it needs to be transposed to obtain [n]. outCb The result is in the format [outCb,inw×2,n] and is also saved to AM. temp middle.

[0232] Step 29: AM Space temp The output feature map results stored in the memory are stored in a specified location in the first memory.

[0233] Direct memory access operations can be invoked to transfer the AM data from the AM space. temp The stored output feature map results are stored in a specified location in a double-rate synchronous dynamic random access memory.

[0234] Step 30: Let p h =p h +1;

[0235] Step 31: Determine p h Is it less than inh×2? If p h If the value is less than inh×2, then return to execute the initialization of p. ic =0 operation; if p h If the value is not less than inh×2, then the Deconv4×4 calculation of the half-precision input feature map F and the half-precision filter data Q is completed, the output feature map result corresponding to the half-precision input feature map F is obtained, and the output feature map result is transferred to the off-chip storage space.

[0236] p h When the value is less than inh×2, the initialization of p can be returned. ic If p = 0, return to step 2. h If the value is not less than inh×2, then the Deconv4×4 calculation of all input feature maps and all filter data is completed, that is, the Deconv4×4 calculation of half-precision input feature map F and half-precision filter data Q is completed.

[0237] Applying the technical solution provided by the embodiments of the present invention, considering that half-precision deconvolution calculation can be implemented based on a vector processor, and since the solution of this application extracts the position of the non-zero value in the transformed data input feature map during calculation, that is, extracts the effective data, and performs convolution calculation with the non-zero value and the data at the corresponding position in the filter, the actual computation amount is significantly reduced compared with the theoretical computation amount of deconvolution.

[0238] Specifically, the half-precision input feature map is loaded into on-chip scalar memory, and the half-precision filter data is loaded into on-chip array memory. Then, one input feature map sub-block, Fbsm, can be loaded from the on-chip scalar memory. Based on the principle of deconvolution, two rows in the 4×4 feature map are padded with zeros. Therefore, only two rows of the original input data need to be loaded. Thus, the data format of a single-loaded Fbsm is [inCb, 2, inw, n], where inCb is the sub-block size in the input channel dimension, inw is the width of the half-precision input feature map, and n is the number of half-precision data that a single register in the vector processor can compute. Furthermore, based on the transformation principle of deconvolution, the effective value distribution characteristics of the currently loaded input feature map sub-block Fbsm after deconvolution transformation are determined. It can be understood that the effective value distribution characteristics reflect the positions of non-zero values. Therefore, based on the effective value distribution characteristics, the effective data at the corresponding positions in the input feature map sub-block Fbsm can be vectorized and deconvolved (Deconv4×4) with the half-precision filter data at the corresponding positions loaded from the on-chip array memory to obtain the calculation result corresponding to the currently loaded input feature map sub-block Fbsm. This reduces the computational cost of the proposed solution by 3 / 4 compared to the theoretical computational cost of deconvolution. After obtaining the calculation results of each input feature map sub-block Fbsm, the output feature map result of the half-precision input feature map can be determined.

[0239] Corresponding to the above method embodiments, this invention also provides a half-precision deconvolution system based on a vector processor, which can be referred to in conjunction with the above.

[0240] See Figure 20 As shown, this vector processor-based half-precision deconvolution system may include:

[0241] The loading module 201 is used to load the half-precision input feature map into the on-chip scalar memory and load the half-precision filter data into the on-chip array memory.

[0242] The effective value distribution feature determination module 202 is used to load an input feature map sub-block Fbsm from the on-chip scalar memory and determine the effective value distribution features of the currently loaded input feature map sub-block Fbsm after deconvolution transformation based on the transformation principle of deconvolution calculation.

[0243] The calculation module 203 is used to perform vectorized deconvolution Deconv4×4 calculation on the effective data at the corresponding position in the input feature map sub-block Fbsm and the half-precision filter data at the corresponding position loaded from the on-chip array memory, based on the effective value distribution characteristics, to obtain the calculation result corresponding to the currently loaded input feature map sub-block Fbsm.

[0244] The output feature map result determination module 204 is used to trigger the effective value distribution feature determination module 202 until the calculation results of each input feature map sub-block Fbsm are obtained, which are used as the determined output feature map results corresponding to the half-precision input feature map, and the output feature map results are transferred to the off-chip storage space.

[0245] The data format of the input feature map sub-block Fbsm is [inCb, 2, inw, n], where inCb is the size of the sub-block in the input channel dimension, inw is the width of the half-precision input feature map, and n is the number of half-precision data that can be computed by a single register in the vector processor.

[0246] In one specific embodiment of the present invention, the loading module 201 is specifically used for:

[0247] Divide the half-precision input feature map F into n inCb Each input sub-block Fb is loaded into the on-chip scalar memory in batches.

[0248] Divide the half-precision filter data Q into n outCb ×n inCb Each filter data sub-block Qb is divided into n outCb ×n inCb Each filter data sub-block Qb is loaded into the on-chip array memory in turn;

[0249] The data format of the half-precision input feature map F is [inC, inh, inw, n], where inC is the number of input channels, inh is the height of the half-precision input feature map, inw is the width of the half-precision input feature map, and n is the number of half-precision data that can be computed by a single register in the vector processor.

[0250] The data format of the half-precision filter data Q is [inC, outC, fs, fs, b], where inC is the number of input channels, outC is the number of output channels, fs is the filter side length, and b is the size of the half-precision filter data that needs to be expanded and copied during the calculation.

[0251] outCb is the size of the sub-blocks in the output channel dimension, n outCb The number of blocks in the output channel dimension and inCb is the size of the sub-blocks in the input channel dimension, n inCb The number of blocks in the input channel dimension and

[0252] The data format of the input sub-block Fb is [inCb, inh, inw, n], the data format of the input feature map sub-block Fbsm is [inCb, 2, inw, n], and the data format of the filter data sub-block Qb is [fs, fs, inCb, outCb, b].

[0253] In one specific embodiment of the present invention, the effective value distribution characteristic determination module 202 is specifically used for:

[0254] Initialize p h =0; p h For the input feature map sub-block Fb sm The initial position of [inCb,2,inw,n] is the index on the inh dimension of the half-precision input feature map F;

[0255] Initialize p ic =0; p ic For the input feature map sub-block Fb sm [inCb,2,inw,n] represents the input feature map F after being divided into blocks, with n elements. inCb Block indexes on dimensions;

[0256] Initialize p oc =0; p oc For the filter data sub-block Qb, after dividing the filter data Q into blocks n outCb Block indexes on dimensions;

[0257] Load the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm [inCb,2,inw,n], loads the corresponding filter data sub-block Qb:[fs,fs,inCb,outCb,b] from the on-chip array memory;

[0258] Initialize p w =0; p w The initial position of the input feature map sub-block Fbsm is the index in the inw dimension of the half-precision input feature map F;

[0259] Read inCb×2×2×n data points from the currently loaded input feature map sub-block Fbsm into a register, which will be used as the currently read data block F. rg Based on the transformation principle of deconvolution, the data arrangement after deconvolution transformation of the currently loaded input feature map sub-block Fbsm is simulated, and the starting address is determined to be F. rg(inCb,0,pw,n) The 2×2 feature map is converted into a 4×4 pixel window and its layout type is determined, and the boundary information flag_a, flag_b, flag_c and flag_d are loaded.

[0260] Here, flag_a represents the result of determining whether it is the first row after being converted to a 4×4 pixel window, flag_b represents the result of determining whether it is the first column after being converted to a 4×4 pixel window, flag_c represents the result of determining whether it is the last row after being converted to a 4×4 pixel window, and flag_b represents the result of determining whether it is the last column after being converted to a 4×4 pixel window.

[0261] The layout types include the defined layout types A to D. Layout type A indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 1st row, 3rd row, 1st column, and 3rd column; layout type B indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 1st row, 3rd row, 2nd column, and 4th column; layout type C indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 2nd row, 4th row, 1st column, and 3rd column; and layout type D indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 2nd row, 4th row, 2nd column, and 4th column.

[0262] In one specific embodiment of the present invention, the calculation module 203 and the output feature map result determination module 204 are specifically used to perform the operations of steps 7 to 31 as described in the above embodiment.

[0263] Corresponding to the above methods and system embodiments, the present invention also provides a vector processor-based half-precision deconvolution device and a computer-readable storage medium, wherein the vector processor-based half-precision deconvolution device may include:

[0264] Memory, used to store computer programs;

[0265] A processor for executing a computer program to implement the steps of a vector processor-based half-precision deconvolution method as described in any of the above embodiments.

[0266] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the vector processor-based half-precision deconvolution method as described in any of the above embodiments. The computer-readable storage medium referred to herein includes random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0267] This application uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A half-precision deconvolution method based on a vector processor, characterized in that, include: The half-precision input feature map is loaded into the on-chip scalar memory, and the half-precision filter data is loaded into the on-chip array memory. One input feature map sub-block Fbsm is loaded from the on-chip scalar memory, and the effective value distribution characteristics of the currently loaded input feature map sub-block Fbsm after deconvolution transformation are determined based on the transformation principle of deconvolution calculation. Based on the effective value distribution characteristics, the effective data at the corresponding position in the input feature map sub-block Fbsm and the half-precision filter data at the corresponding position loaded from the on-chip array memory are subjected to vectorized deconvolution Deconv4×4 calculation to obtain the calculation result corresponding to the currently loaded input feature map sub-block Fbsm; Return to the operation of loading one input feature map sub-block Fbsm from the on-chip scalar memory, load each input feature map sub-block Fbsm in sequence until the calculation result of each input feature map sub-block Fbsm is obtained, which is used as the determined output feature map result corresponding to the half-precision input feature map, and transfer the output feature map result to the off-chip storage space; The data format of the input feature map sub-block Fbsm is [inCb, 2, inw, n], where inCb is the size of the sub-block in the input channel dimension, inw is the width of the half-precision input feature map, and n is the number of half-precision data that can be computed by a single register in the vector processor.

2. The half-precision deconvolution method based on a vector processor according to claim 1, characterized in that, The step of loading the half-precision input feature map into the on-chip scalar memory and loading the half-precision filter data into the on-chip array memory includes: Divide the half-precision input feature map F into n inCb Each input sub-block Fb is loaded into the on-chip scalar memory in batches. Divide the half-precision filter data Q into n outCb ×n inCb Each filter data sub-block Qb is divided into n... outCb ×n inCb Each of the filter data sub-blocks Qb is loaded into the on-chip array memory in turn; The data format of the half-precision input feature map F is [inC, inh, inw, n], where inC is the number of input channels, inh is the height of the half-precision input feature map, inw is the width of the half-precision input feature map, and n is the number of half-precision data that can be calculated by a single register in the vector processor. The data format of the half-precision filter data Q is [inC, outC, fs, fs, b], where inC is the number of input channels, outC is the number of output channels, fs is the filter side length, and b is the size that the half-precision filter data needs to be expanded and copied during the calculation. outCb is the size of the sub-blocks in the output channel dimension, n outCb The number of blocks in the output channel dimension and inCb is the size of the sub-blocks in the input channel dimension, n inCb The number of blocks in the input channel dimension and The data format of the input sub-block Fb is [inCb, inh, inw, n], the data format of the input feature map sub-block Fbsm is [inCb, 2, inw, n], and the data format of the filter data sub-block Qb is [fs, fs, inCb, outCb, b].

3. The half-precision deconvolution method based on a vector processor according to claim 2, characterized in that, One input feature map sub-block Fbsm is loaded from the on-chip scalar memory, and based on the transformation principle of deconvolution calculation, the effective value distribution characteristics of the currently loaded input feature map sub-block Fbsm after deconvolution transformation are determined, including: Initialize p h =0; p h For the input feature map sub-block Fb sm The initial position of [inCb,2,inw,n] is the index on the inh dimension of the half-precision input feature map F; Initialize p ic =0; p ic For the input feature map sub-block Fb sm [inCb,2,inw,n] represents the input feature map F after being divided into blocks, with n elements. inCb Block indexes on dimensions; Initialize p oc =0; p oc For the filter data sub-block Qb, after dividing the filter data Q into blocks n outCb Block indexes on dimensions; Load the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm :[inCb,2,inw,n], load the corresponding filter data sub-block Qb:[fs,fs,inCb,outCb,b] from the on-chip array memory; Initialize p w =0; p w The initial position of the input feature map sub-block Fbsm is the index in the inw dimension of the half-precision input feature map F; From the currently loaded input feature map sub-block Fbsm, read inCb×2×2×n data points into a register, which will be used as the currently read data block F. rg Based on the transformation principle of deconvolution calculation, the data arrangement after deconvolution transformation of the currently loaded input feature map sub-block Fbsm is simulated, and the starting address is determined to be F. rg(inCb,0,pw,n) The 2×2 feature map is converted into a 4×4 pixel window and its layout type is determined, and the boundary information flag_a, flag_b, flag_c and flag_d are loaded. Here, flag_a represents the result of determining whether it is the first row after being converted to a 4×4 pixel window, flag_b represents the result of determining whether it is the first column after being converted to a 4×4 pixel window, flag_c represents the result of determining whether it is the last row after being converted to a 4×4 pixel window, and flag_b represents the result of determining whether it is the last column after being converted to a 4×4 pixel window. The layout types include the defined layout types A to D. Layout type A indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 1st row, 3rd row, 1st column, and 3rd column; layout type B indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 1st row, 3rd row, 2nd column, and 4th column; layout type C indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 2nd row, 4th row, 1st column, and 3rd column; and layout type D indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 2nd row, 4th row, 2nd column, and 4th column.

4. The half-precision deconvolution method based on a vector processor according to claim 3, characterized in that, Based on the effective value distribution characteristics, the effective data at the corresponding position in the input feature map sub-block Fbsm and the half-precision filter data at the corresponding position loaded from the on-chip array memory are subjected to vectorized deconvolution Deconv4×4 calculation to obtain the calculation result corresponding to the currently loaded input feature map sub-block Fbsm; Returning to the operation of loading one input feature map sub-block Fbsm from the on-chip scalar memory, the operation of loading each input feature map sub-block Fbsm sequentially is repeated until the calculation result of each input feature map sub-block Fbsm is obtained, which is used as the determined output feature map result corresponding to the half-precision input feature map, and the output feature map result is transferred to the off-chip storage space, specifically including: Initialize p k =0; p k For data block F rg Indexes on the inCb dimension; Initialize i = 0; i is the data block F rg Indexes on the inh dimension; Initialize j = 0, where j is the data block F. rg Indexes on the inw dimension; Initialize each vector register storing the output data; Determine p ic Is it 0? If p ic If the result is not 0, the intermediate Deconv4×4 product result temporarily stored in the on-chip array memory will be loaded into each of the vector registers. If p ic If the value is 0, then a scalar load instruction with a granularity of n×2 bytes is used to read data block F in a single operation. rg The first n half-precision data are written to a scalar register, and the data in the scalar register are converted from scalars to a set of identical vector data and stored in the corresponding vector register using a broadcast instruction; Based on the determined layout type of the 4×4 pixel window, the position of the corresponding data on the filter data sub-block Qb is determined, and the filter data block read during the calculation is denoted as Q. rg Q rg All data in the outCb dimension are loaded into p vector registers; among them, the filter data block Q read during the calculation is... rg The data format is [fs, fs, pk, outCb, b], where p represents the number of vector operation units in the vector processor; Data block F rg(pk,i,j,n) The vectorized data are respectively compared with Q rg(fs,fs,pk,outCb,b) The data is multiplied by a dot, and the result of the dot product is then added to the corresponding values ​​in each of the vector registers. The S vector processing units perform the calculations synchronously and in parallel, and the intermediate calculation results are stored in each of the vector registers in sequence. Let j = j + 1; Determine if j is less than 2. If j is less than 2, return to execute the operation of using the scalar load instruction with a granularity of n×2 bytes; If j is not less than 2, then let i = i + 1; Determine if i is less than 2. If i is less than 2, return to execute the operation of using the scalar load instruction with a granularity of n×2 bytes; If i is not less than 2, then let p k =p k +1; Determine if pk is less than inCb. If pk is less than inCb, return to execute the initialization operation i=0. If pk is not less than inCb, then for F... rg In Q rg The deconvolution calculation of inCb input channels is completed to obtain 2×2×outCb×n intermediate result data; Based on the loaded boundary information flag_a, flag_b, flag_c, and flag_d, data clearing is performed; specifically, when the judgment result represented by flag_a is yes, the first row of data is cleared; when the judgment result represented by flag_b is yes, the last row of data is cleared; when the judgment result represented by flag_c is yes, the first column of data is cleared; and when the judgment result represented by flag_d is yes, the last column of data is cleared. The intermediate results on the same output channel are summed to obtain the final result of deconvolution, which is then saved to each of the aforementioned vector registers. After completing data block F rg After the Deconv4×4 calculation is performed on the corresponding position data in the filter data sub-block Qb, the deconvolution results saved in each of the vector registers are stored in the on-chip array memory; After calculating the pixel image of layout type B or the pixel image of layout type D, return to execute the operation of using the scalar loading instruction with a granularity of n×2 bytes; After calculating the pixel image of layout type A or layout type C, let p w =p w +1; Determine p w Is it less than inw? If p w If the value is less than inw, the process will return to executing the step of reading inCb×2×2×n data from the currently loaded input feature map sub-block Fbsm into a register, which will be used as the currently read data block F. rg Operation; If p w If p is not less than inw, then let p ic =p ic +1; Determine p ic Is it less than n? inCb If p ic Less than n inCb Then return to loading the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm The operation of [inCb,2,inw,n]; If p ic Not less than n inCb This confirms that all input channels have been traversed in the deconvolution calculation, and a block of size n is allocated on the on-chip array memory. outCb ×inw×2×outCb×b's space AM temp The deconvolution results stored in each of the vector registers are then stored in the spatial AM. temp In, and let p oc =p oc +1; Determine p oc Is it less than n? outCb If p oc Less than n outCb Then return to loading the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm The operation of [inCb,2,inw,n]; If p oc Not less than n outCb This confirms that all output channels have been traversed in the deconvolution calculation, resulting in n. outCb The output result is of size ×inw×2×outCb×b; Space AM temp The data format is [n outCb The deconvolution result of [inw×2,outCb,n] is transposed to obtain [n outCb The result in the format [outCb,inw×2,n] is saved to the space AM. temp middle; Space AM temp The output feature map results stored in the memory are stored in a designated location in the first memory. Let p h =p h +1; Determine p h Is it less than inh×2? If p h If the value is less than inh×2, then return to execute the initialization p. ic =0 operation; If p h If the value is not less than inh×2, then the Deconv4×4 calculation of the half-precision input feature map F and the half-precision filter data Q is completed, the output feature map result corresponding to the half-precision input feature map F is obtained, and the output feature map result is transmitted to the off-chip storage space.

5. A half-precision deconvolution system based on a vector processor, characterized in that, include: The loading module is used to load the half-precision input feature map into the on-chip scalar memory and load the half-precision filter data into the on-chip array memory. The effective value distribution feature determination module is used to load an input feature map sub-block Fbsm from the on-chip scalar memory and determine the effective value distribution features of the currently loaded input feature map sub-block Fbsm after deconvolution transformation based on the transformation principle of deconvolution calculation. The calculation module is used to perform vectorized deconvolution Deconv4×4 calculation on the effective data at the corresponding position in the input feature map sub-block Fbsm and the half-precision filter data at the corresponding position loaded from the on-chip array memory, based on the effective value distribution characteristics, to obtain the calculation result corresponding to the currently loaded input feature map sub-block Fbsm; The output feature map result determination module is used to trigger the effective value distribution feature determination module until the calculation result of each input feature map sub-block Fbsm is obtained, which is used as the determined output feature map result corresponding to the half-precision input feature map, and the output feature map result is transmitted to the off-chip storage space. The data format of the input feature map sub-block Fbsm is [inCb, 2, inw, n], where inCb is the size of the sub-block in the input channel dimension, inw is the width of the half-precision input feature map, and n is the number of half-precision data that can be computed by a single register in the vector processor.

6. The half-precision deconvolution system based on a vector processor according to claim 5, characterized in that, The loading module is specifically used for: Divide the half-precision input feature map F into n inCb Each input sub-block Fb is loaded into the on-chip scalar memory in batches. Divide the half-precision filter data Q into n outCb ×n inCb Each filter data sub-block Qb is divided into n... outCb ×n inCb Each of the filter data sub-blocks Qb is loaded into the on-chip array memory in turn; The data format of the half-precision input feature map F is [inC, inh, inw, n], where inC is the number of input channels, inh is the height of the half-precision input feature map, inw is the width of the half-precision input feature map, and n is the number of half-precision data that can be calculated by a single register in the vector processor. The data format of the half-precision filter data Q is [inC, outC, fs, fs, b], where inC is the number of input channels, outC is the number of output channels, fs is the filter side length, and b is the size that the half-precision filter data needs to be expanded and copied during the calculation. outCb is the size of the sub-blocks in the output channel dimension, n outCb The number of blocks in the output channel dimension and inCb is the size of the sub-blocks in the input channel dimension, n inCb The number of blocks in the input channel dimension and The data format of the input sub-block Fb is [inCb, inh, inw, n], the data format of the input feature map sub-block Fbsm is [inCb, 2, inw, n], and the data format of the filter data sub-block Qb is [fs, fs, inCb, outCb, b].

7. The half-precision deconvolution system based on a vector processor according to claim 6, characterized in that, The effective value distribution characteristic determination module is specifically used for: Initialize p h =0; p h For the input feature map sub-block Fb sm The initial position of [inCb,2,inw,n] is the index on the inh dimension of the half-precision input feature map F; Initialize p ic =0; p ic For the input feature map sub-block Fb sm [inCb,2,inw,n] represents the input feature map F after being divided into blocks, with n elements. inCb Block indexes on dimensions; Initialize p oc =0; p oc For the filter data sub-block Qb, after dividing the filter data Q into blocks n outCb Block indexes on dimensions; Load the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm :[inCb,2,inw,n], load the corresponding filter data sub-block Qb:[fs,fs,inCb,outCb,b] from the on-chip array memory; Initialize p w =0; p w The initial position of the input feature map sub-block Fbsm is the index in the inw dimension of the half-precision input feature map F; From the currently loaded input feature map sub-block Fbsm, read inCb×2×2×n data points into a register, which will be used as the currently read data block F. rg Based on the transformation principle of deconvolution calculation, the data arrangement after deconvolution transformation of the currently loaded input feature map sub-block Fbsm is simulated, and the starting address is determined to be F. rg(inCb,0,pw,n) The 2×2 feature map is converted into a 4×4 pixel window and its layout type is determined, and the boundary information flag_a, flag_b, flag_c and flag_d are loaded. Here, flag_a represents the result of determining whether it is the first row after being converted to a 4×4 pixel window, flag_b represents the result of determining whether it is the first column after being converted to a 4×4 pixel window, flag_c represents the result of determining whether it is the last row after being converted to a 4×4 pixel window, and flag_b represents the result of determining whether it is the last column after being converted to a 4×4 pixel window. The layout types include the defined layout types A to D. Layout type A indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 1st row, 3rd row, 1st column, and 3rd column; layout type B indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 1st row, 3rd row, 2nd column, and 4th column; layout type C indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 2nd row, 4th row, 1st column, and 3rd column; and layout type D indicates that after conversion to a 4×4 pixel window, there are valid values ​​in the 2nd row, 4th row, 2nd column, and 4th column.

8. The half-precision deconvolution system based on a vector processor according to claim 7, characterized in that, The calculation module and the output feature map result determination module are specifically used for: Initialize p k =0; p k For data block F rg Indexes on the inCb dimension; Initialize i = 0; i is the data block F rg Indexes on the inh dimension; Initialize j = 0, where j is the data block F. rg Indexes on the inw dimension; Initialize each vector register storing the output data; Determine p ic Is it 0? If p ic If the result is not 0, the intermediate Deconv4×4 product result temporarily stored in the on-chip array memory will be loaded into each of the vector registers. If p ic If the value is 0, then a scalar load instruction with a granularity of n×2 bytes is used to read data block F in a single operation. rg The first n half-precision data are written to a scalar register, and the data in the scalar register are converted from scalars to a set of identical vector data and stored in the corresponding vector register using a broadcast instruction; Based on the determined layout type of the 4×4 pixel window, the position of the corresponding data on the filter data sub-block Qb is determined, and the filter data block read during the calculation is denoted as Q. rg Q rg All data in the outCb dimension are loaded into p vector registers; among them, the filter data block Q read during the calculation is... rg The data format is [fs, fs, pk, outCb, b], where p represents the number of vector operation units in the vector processor; Data block F rg(pk,i,j,n) The vectorized data are respectively compared with Q rg(fs,fs,pk,outCb,b) The data is multiplied by a dot, and the result of the dot product is then added to the corresponding values ​​in each of the vector registers. The S vector processing units perform the calculations synchronously and in parallel, and the intermediate calculation results are stored in each of the vector registers in sequence. Let j = j + 1; Determine if j is less than 2. If j is less than 2, return to execute the operation of using the scalar load instruction with a granularity of n×2 bytes; If j is not less than 2, then let i = i + 1; Determine if i is less than 2. If i is less than 2, return to execute the operation of using the scalar load instruction with a granularity of n×2 bytes; If i is not less than 2, then let p k =p k +1; Determine if pk is less than inCb. If pk is less than inCb, return to execute the initialization operation i=0. If pk is not less than inCb, then for F... rg In Q rg The deconvolution calculation of inCb input channels is completed to obtain 2×2×outCb×n intermediate result data; Based on the loaded boundary information flag_a, flag_b, flag_c, and flag_d, data clearing is performed; specifically, when the judgment result represented by flag_a is yes, the first row of data is cleared; when the judgment result represented by flag_b is yes, the last row of data is cleared; when the judgment result represented by flag_c is yes, the first column of data is cleared; and when the judgment result represented by flag_d is yes, the last column of data is cleared. The intermediate results on the same output channel are summed to obtain the final result of deconvolution, which is then saved to each of the aforementioned vector registers. After completing data block F rg After the Deconv4×4 calculation is performed on the corresponding position data in the filter data sub-block Qb, the deconvolution results saved in each of the vector registers are stored in the on-chip array memory; After calculating the pixel image of layout type B or the pixel image of layout type D, return to execute the operation of using the scalar loading instruction with a granularity of n×2 bytes; After calculating the pixel image of layout type A or layout type C, let p w =p w +1; Determine p w Is it less than inw? If p w If the value is less than inw, the process will return to executing the step of reading inCb×2×2×n data from the currently loaded input feature map sub-block Fbsm into a register, which will be used as the currently read data block F. rg Operation; If p w If p is not less than inw, then let p ic =p ic +1; Determine p ic Is it less than n? inCb If p ic Less than n inCb Then return to loading the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm The operation of [inCb,2,inw,n]; If p ic Not less than n inCb This confirms that all input channels have been traversed in the deconvolution calculation, and a block of size n is allocated on the on-chip array memory. outCb ×inw×2×outCb×b's space AM temp The deconvolution results stored in each of the vector registers are then stored in the spatial AM. temp In, and let p oc =p oc +1; Determine p oc Is it less than n? outCb If p oc Less than n outCb Then return to loading the corresponding input feature map sub-block Fb from the on-chip scalar memory. sm The operation of [inCb,2,inw,n]; If p oc Not less than n outCb This confirms that all output channels have been traversed in the deconvolution calculation, resulting in n. outCb The output result is of size ×inw×2×outCb×b; Space AM temp The data format is [n outCb The deconvolution result of [inw×2,outCb,n] is transposed to obtain [n outCb The result in the format [outCb,inw×2,n] is saved to the space AM. temp middle; Space AM temp The output feature map results stored in the memory are stored in a designated location in the first memory. Let p h =p h +1; Determine p h Is it less than inh×2? If p h If the value is less than inh×2, then return to execute the initialization p. ic =0 operation; If p h If the value is not less than inh×2, then the Deconv4×4 calculation of the half-precision input feature map F and the half-precision filter data Q is completed, the output feature map result corresponding to the half-precision input feature map F is obtained, and the output feature map result is transmitted to the off-chip storage space.

9. A half-precision deconvolution device based on a vector processor, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the vector processor-based half-precision deconvolution method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the vector processor-based half-precision deconvolution method as described in any one of claims 1 to 4.

Images