Sparse matrix processing method and device in image recognition process, equipment and medium

Abstract

The invention discloses a sparse matrix processing method and device in an image recognition process, equipment and a medium, and relates to the field of computer image processing, and the method comprises the steps: inputting a feature image into a convolutional neural network; if the convolution kernel meets a preset sparse matrix condition, performing structured sparseness on the convolution kernel to obtain a sparse matrix, compressing the sparse matrix by using an improved compression sparse row format to obtain a compression matrix, and constructing an index matrix of the compression matrix; storing the feature image, the compression matrix, the index matrix and the preset bias matrix in a global memory, activating a target thread bundle in the computing core, and distributing the target thread bundle to a target data processing channel; loading the compression matrix and the index matrix to a shared memory corresponding to the target data processing channel by using the target thread bundle, and loading the feature image and a preset bias matrix to a register file; and executing the operation instruction to read the current matrix operation data from the shared memory and the register file to the matrix calculation unit to complete the matrix multiplication and addition operation.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing, and in particular to a sparse matrix processing method, device, equipment and medium in an image recognition process. Background Art

[0002] With the rapid development of information technology, the amount of data has shown an explosive growth. In many fields such as machine learning, big data analysis, graphics processing, scientific computing, network analysis, etc., matrix operations have become an indispensable part, especially in the fields of machine learning and deep learning. Operations such as weight update, feature extraction, and prediction in the model training process all involve large-scale matrix operations. Among these matrices, sparse matrices account for a considerable proportion. Traditional matrix multiplication algorithms traverse and multiply each element, including zero-value elements, which not only wastes a lot of computing resources, but also leads to unnecessary memory access, reduces cache hit rate, and increases computing latency. In addition, when faced with large or very large sparse matrices, the storage space required by traditional algorithms will also become a bottleneck.

[0003] In summary, how to improve the efficiency and performance of sparse matrix operations, thereby reducing unnecessary memory accesses and saving computer resources is a problem that needs to be solved. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a sparse matrix processing method, device, equipment and medium in the image recognition process, which can improve the efficiency and performance of sparse matrix operations in the image recognition process, thereby reducing unnecessary memory access and saving computer resources. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a sparse matrix processing method in an image recognition process, comprising:

[0006] The feature image obtained after preprocessing the original image is input into the convolutional neural network;

[0007] If the convolution kernel for convolution operation with the feature image satisfies a preset sparse matrix condition, the convolution kernel is structured sparsely processed to obtain a sparse matrix, the sparse matrix is ​​compressed using an improved compressed sparse row format to obtain a compressed matrix, and an index matrix of the compressed matrix is ​​constructed;

[0008] The feature image, the compression matrix, the index matrix and the preset bias matrix are stored in a global memory, and a computing core allocated for the current convolution operation is determined, and then a target thread warp in the computing core is activated, and the target thread warp is allocated to a target data processing channel;

[0009] Using the target thread warp to load the compression matrix and the index matrix from the global memory to a shared memory corresponding to the target data processing channel, and using the target thread warp to load the feature image and the preset bias matrix from the global memory to a register file;

[0010] The target thread warp is used to execute a computation instruction to read current matrix computation data from the shared memory and the register file to a preset matrix computation unit to complete a matrix multiplication and addition operation.

[0011] Optionally, performing structured sparse processing on the convolution kernel to obtain a sparse matrix, compressing the sparse matrix using an improved compressed sparse row format to obtain a compressed matrix, and constructing an index matrix of the compressed matrix, includes:

[0012] According to the rule that the number of zero-value elements in the matrix elements is the same as the number of non-zero-value elements, a pruning algorithm is used to perform structured sparseness on the convolution kernel to obtain a sparse matrix;

[0013] The sparse matrix is ​​compressed using an improved compressed sparse row format, and non-zero value elements of each row are rearranged to form a new row to obtain a compressed matrix;

[0014] Column indices of non-zero-valued elements in each row are extracted from the sparse matrix to construct an index matrix corresponding to the compressed matrix.

[0015] Optionally, the computing core includes at least four data processing channels, each of the data processing channels corresponds to an activated thread warp at the same time, and the thread warp includes at least eight threads;

[0016] Accordingly, activating the target warp in the computing core and allocating the target warp to the target data processing channel includes:

[0017] Determine and activate the target thread warp in descending order of task priority;

[0018] A target data processing channel is determined from all the data processing channels in order of the current load from small to large, and the target thread warp is allocated to the target data processing channel.

[0019] Optionally, the shared memory includes a value shared memory and an index shared memory;

[0020] Accordingly, the step of using the target thread warp to load the compression matrix and the index matrix from the global memory to a shared memory corresponding to the target data processing channel includes:

[0021] Using the target thread warp to read a first preset memory access instruction from a preset instruction cache;

[0022] Obtaining a first base address corresponding to the compressed matrix in the global memory and a first destination address in the numerical shared memory corresponding to the target data processing channel, so as to read the compressed matrix stored at the first base address from the global memory based on the first preset memory access instruction, and write the compressed matrix to the first destination address in the numerical shared memory;

[0023] Obtain a second base address corresponding to the index matrix in the global memory and a second destination address in the index shared memory corresponding to the target data processing channel, so as to read the index matrix stored at the second base address from the global memory based on the first preset memory access instruction, and write it to the second destination address in the index shared memory.

[0024] Optionally, the register file includes a first register file and a second register file;

[0025] Accordingly, the step of using the target thread warp to load the feature image and the preset bias matrix from the global memory to a register file includes:

[0026] Using the target thread warp to read a second preset memory access instruction from the preset instruction cache;

[0027] Obtaining a third base address corresponding to the feature image in the global memory and an address of the first register file, and obtaining the second destination address corresponding to the index matrix, so as to write a target element of the feature image read from the global memory according to the third base address and the second destination address into the first register file based on the second preset memory access instruction;

[0028] Using the target thread warp to read a word load instruction from the preset instruction cache;

[0029] A fourth base address corresponding to the preset bias matrix in the global memory and an address of the second register file are obtained, so as to read the preset bias matrix stored at the fourth base address from the global memory based on the word load instruction and write it into the second register file.

[0030] Optionally, writing the target element of the feature image read from the global memory according to the third base address and the second destination address into the first register file includes:

[0031] Based on a matrix product operation rule, using the second destination address to retrieve a row index address corresponding to the feature image from the index shared memory;

[0032] The target element of the feature image read from the global memory based on the third base address and the row index address is written into the first register file.

[0033] Optionally, the reading of current matrix operation data from the shared memory and the register file to a preset matrix calculation unit to complete the matrix multiplication and addition operation includes:

[0034] Reading a first target row element in the compressed matrix from the shared memory to a first local memory, reading a second target row element of the feature image from the register file to a second local memory, and reading a third target row element of the preset bias matrix from the register file to a third local memory; wherein the first target row element, the second target row element and the third target row element are combined to obtain current matrix operation data;

[0035] The data in the first local memory, the second local memory and the third local memory are input into a preset matrix calculation unit to complete a matrix multiplication and addition operation.

[0036] In a second aspect, the present application discloses a sparse matrix processing device in an image recognition process, comprising:

[0037] An image input module is used to input a feature image obtained by preprocessing the original image into a convolutional neural network;

[0038] A sparse matrix processing module, configured to perform structured sparse processing on the convolution kernel to obtain a sparse matrix if the convolution kernel for convolution operation with the feature image meets a preset sparse matrix condition, compress the sparse matrix using an improved compressed sparse row format to obtain a compressed matrix, and construct an index matrix of the compressed matrix;

[0039] a resource allocation module, configured to store the feature image, the compression matrix, the index matrix, and the preset bias matrix in a global memory, determine a computing core allocated for a current convolution operation, activate a target thread bundle in the computing core, and allocate the target thread bundle to a target data processing channel;

[0040] a data loading module, configured to load the compression matrix and the index matrix from the global memory to a shared memory corresponding to the target data processing channel by using the target thread warp, and to load the feature image and the preset bias matrix from the global memory to a register file by using the target thread warp;

[0041] The operation module is used to execute the operation instruction by using the target thread warp to read the current matrix operation data from the shared memory and the register file to a preset matrix calculation unit to complete the matrix multiplication and addition operation.

[0042] In a third aspect, the present application discloses an electronic device, comprising:

[0043] Memory, used to store computer programs;

[0044] The processor is used to execute the computer program to implement the steps of the sparse matrix processing method in the image recognition process disclosed above.

[0045] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the sparse matrix processing method in the image recognition process disclosed above are implemented.

[0046] It can be seen that the present application inputs the feature image obtained after preprocessing the original image into the convolutional neural network; if the convolution kernel for convolution operation with the feature image meets the preset sparse matrix condition, the convolution kernel is structured sparse to obtain a sparse matrix, and the sparse matrix is ​​compressed using the improved compressed sparse row format to obtain a compressed matrix, and an index matrix of the compressed matrix is ​​constructed; the feature image, the compressed matrix, the index matrix and the preset bias matrix are stored in the global memory, and the computing core assigned to the current convolution operation is determined, and then the target thread bundle in the computing core is activated, and the target thread bundle is assigned to the target data processing channel; the compressed matrix and the index matrix are loaded from the global memory to the shared memory corresponding to the target data processing channel by using the target thread bundle, and the feature image and the preset bias matrix are loaded from the global memory to the register file by using the target thread bundle; the target thread bundle is used to execute the operation instruction to read the current matrix operation data from the shared memory and the register file to the preset matrix calculation unit to complete the matrix multiplication and addition operation.

[0047] Beneficial effect: This application needs to preprocess the original image after acquiring it, and input the feature image obtained after preprocessing into the convolutional neural network for subsequent convolution operations. If the convolution kernel for convolution operation with the feature image meets the preset sparse matrix condition, the convolution kernel is first structured sparse to obtain a sparse matrix, and the sparse matrix is ​​compressed using the improved compressed sparse row format to obtain a compressed matrix, and an index matrix corresponding to the compressed matrix is ​​constructed. By improving the sparse row format, the purpose is to abandon the row pointer array in the sparse row format, further reduce the storage space required for the operation, and improve the reading efficiency of sparse matrix elements. Then in the storage stage, a hierarchical storage structure is adopted, including global memory, shared memory, register file, etc., and independent shared memory is allocated to each data processing channel, which can greatly reduce the probability of data access conflict and improve the memory access efficiency of matrix elements. Specifically, it is necessary to store the feature image, compression matrix, index matrix and preset bias matrix in the global memory, determine the computing core assigned to the current convolution operation, then activate the target thread bundle in the computing core, and assign the target thread bundle to the target data processing channel; further, use the target thread bundle to load the compression matrix and index matrix from the global memory to the shared memory corresponding to the target data processing channel, and load the feature image and preset bias matrix from the global memory to the register file. Finally, use the target thread bundle to execute the operation instruction to read the current matrix operation data from the shared memory and register file to the preset matrix calculation unit to complete the matrix multiplication and addition operation. In this way, this solution significantly improves the efficiency and performance of matrix multiplication operations by adopting improved compressed sparse row format, hierarchical storage, multi-channel processing and other technologies, thereby reducing unnecessary memory access and saving computer resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 A flow chart of a sparse matrix processing method in an image recognition process disclosed in this application;

[0050] Figure 2 A schematic diagram of a sparse-dense matrix multiplication operation process disclosed in this application;

[0051] Figure 3 A schematic diagram of sparse matrix compression preprocessing disclosed in this application;

[0052] Figure 4 A schematic diagram of hierarchical storage of matrix elements disclosed in this application;

[0053] Figure 5 A schematic diagram of the architecture of a matrix computing subunit disclosed in this application;

[0054] Figure 6 A schematic diagram of the structure of a sparse matrix processing device in an image recognition process disclosed in this application;

[0055] Figure 7 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0057] Currently, in the field of machine learning and deep learning, operations such as weight updating, feature extraction, and prediction in the model training process all involve large-scale matrix operations. Among these matrices, sparse matrices account for a considerable proportion. Traditional matrix multiplication algorithms traverse and multiply each element, including zero-value elements, which not only wastes a lot of computing resources, but also leads to unnecessary memory access, reduces cache hit rate, and increases computing latency. In addition, when faced with large or ultra-large sparse matrices, the storage space required by traditional algorithms will also become a bottleneck. The embodiments of the present application disclose a sparse matrix processing method, device, equipment, and medium in an image recognition process, which can improve the efficiency and performance of sparse matrix operations in the image recognition process, thereby reducing unnecessary memory access and saving computer resources.

[0058] See also Figure 1 and Figure 2 As shown, the embodiment of the present application discloses a sparse matrix processing method in an image recognition process, the method comprising:

[0059] Step S11: inputting the feature image obtained after preprocessing the original image into the convolutional neural network.

[0060] In this embodiment, the present application needs to preprocess the original image after acquiring it, and input the feature image obtained after the preprocessing into the convolutional neural network for subsequent convolution operations.

[0061] Step S12: If the convolution kernel for convolution operation with the feature image satisfies the preset sparse matrix condition, the convolution kernel is structured sparsely processed to obtain a sparse matrix, and the sparse matrix is ​​compressed using the improved compressed sparse row format to obtain a compressed matrix, and an index matrix of the compressed matrix is ​​constructed.

[0062] In this embodiment, if the convolution kernel for convolution operation with the feature image meets the preset sparse matrix condition, the convolution kernel is first structured sparse to obtain a sparse matrix, and the sparse matrix is ​​compressed using the improved compressed sparse row format to obtain a compressed matrix, and an index matrix corresponding to the compressed matrix is ​​constructed. By improving the sparse row format, the purpose is to abandon the row pointer array in the sparse row format, further reduce the storage space required for the operation, and improve the reading efficiency of the sparse matrix elements. In addition, the preset sparse condition here refers to the situation when there is certain redundant information in the data or some elements have little effect on the final result. In this case, sparseness is more meaningful. For the convolution kernel, if its elements are distributed more evenly and all play an important role in feature extraction and classification, blind sparseness may cause information loss and reduce model performance.

[0063] Among them, the improved compressed sparse row format is defined as the RCSR (Reduced Compressed SparseRow) compression format. This compression format is further simplified on the basis of the traditional CSR format, abandoning the row pointer Row_ptr array. Each row retains a fixed number of matrix elements according to the pruning algorithm, and only stores the column index Col.index and the corresponding matrix element Value value, further reducing the storage space required for calculations and improving the reading efficiency of sparse matrix elements.

[0064] In a specific implementation, the above-mentioned structured sparseness of the convolution kernel to obtain a sparse matrix, and the sparse matrix is ​​compressed using the improved compressed sparse row format to obtain a compressed matrix, and an index matrix of the compressed matrix is ​​constructed, including: according to the rule that the number of zero-value elements in the matrix elements is the same as the number of non-zero-value elements, the convolution kernel is structured sparse using a pruning algorithm to obtain a sparse matrix; the sparse matrix is ​​compressed using the improved compressed sparse row format, and the non-zero-value elements of each row are rearranged to form a new row to obtain a compressed matrix; the column index of the non-zero-value elements in each row of the sparse matrix is ​​extracted to construct an index matrix corresponding to the compressed matrix.

[0065] That is, assuming that the convolution kernel is an 8×8 matrix, first, according to the rule that the number of zero-value elements in the matrix elements is the same as the number of non-zero-value elements, that is, 1:1, the pruning algorithm is used to perform structured sparseness on the convolution kernel to obtain a sparse matrix, thereby obtaining an 8x8 sparse matrix A S, such as Figure 3 As shown. Then, the RCSR matrix compression format is used to compress the sparse matrix A S Compress and replace the non-zero value A of each row S .value are rearranged to form new rows, and the 8x4 compressed matrix A is obtained. At the same time, the sparse matrix A S The column index AS.col_index of the non-zero value in each row corresponds to the dense matrix A one-to-one, and is combined into a new 8x4 index matrix A I , such as Figure 3 shown.

[0066] Step S13: storing the feature image, the compression matrix, the index matrix and the preset bias matrix into the global memory, determining the computing core assigned to the current convolution operation, activating the target thread warp in the computing core, and assigning the target thread warp to the target data processing channel.

[0067] In this embodiment, after the sparse matrix is ​​compressed, the obtained compression matrix A and index matrix A are I The image is stored in the corresponding area of ​​the global memory, and the 8x8 feature image matrix B and the 8x8 preset bias matrix C are stored in the corresponding area of ​​the global memory to complete the preprocessing stage. Specifically, high-bandwidth, low-latency HBM (High Bandwidth Memory) is used as the global memory.

[0068] In the storage stage, the computing core assigned to the current convolution operation is determined, and then the target thread warp in the computing core is activated and assigned to the target data processing channel.

[0069] In a specific implementation, the computing core includes at least four data processing channels, each of the data processing channels corresponds to an activated thread bundle at the same time, and the thread bundle includes at least eight threads; accordingly, activating the target thread bundle in the computing core and allocating the target thread bundle to the target data processing channel includes: determining and activating the target thread bundle in order of task priority from high to low; determining the target data processing channel from all the data processing channels in order of current load from small to large, and allocating the target thread bundle to the target data processing channel.

[0070] It is understandable that this solution designs a multi-channel pipeline processing mechanism based on the parallel computing characteristics of GPGPU (General-purpose computing on graphics processing units). Specifically, there are four data processing channels C0~C3 in the computing core. Each channel can only activate one thread bundle at the same time. By default, each thread bundle contains 8 threads. Next, the target thread bundle can be determined and activated in order from high to low task priority. For example, if multiple thread bundles request channel resources at the same time, they may be preferentially allocated to the thread bundle that prepares data earlier or has a greater impact on the overall computing progress. Further, the target data processing channel is determined from all data processing channels in order from small to large current load. It is understandable that each pipeline channel in the computing core should bear the computing tasks as evenly as possible. Before allocating thread bundles, the system may evaluate the current task load of each channel, including the amount of data being processed, the computational complexity of the allocated thread bundles, etc. For example, by monitoring the memory access frequency of each channel, the busyness of the computing unit, and other indicators, the new thread bundle is assigned to the channel with a lighter load to ensure that the work progress of each channel is relatively consistent, avoiding the occurrence of a channel with processing delays due to overloaded tasks, which affects the overall computing efficiency. Finally, the target thread bundle is assigned to the target data processing channel, for example, the target thread bundle W0 is activated and assigned to the target data processing channel C0.

[0071] Step S14: using the target thread warp to load the compression matrix and the index matrix from the global memory to the shared memory corresponding to the target data processing channel, and using the target thread warp to load the feature image and the preset bias matrix from the global memory to the register file.

[0072] In this embodiment, the target thread bundle is used to load the compression matrix and the index matrix from the global memory to the shared memory corresponding to the target data processing channel, and the feature image and the preset bias matrix are loaded from the global memory to the register file. In this way, in the storage stage, the present application adopts a hierarchical storage structure, including global memory, shared memory, register file, etc., and allocates independent shared memory to each data processing channel, which can greatly reduce the probability of data access conflict and improve the memory access efficiency of matrix elements.

[0073] In a specific embodiment, the above-mentioned shared memory includes a numerical shared memory and an index shared memory; accordingly, the use of the target thread bundle to load the compressed matrix and the index matrix from the global memory to the shared memory corresponding to the target data processing channel includes: using the target thread bundle to read a first preset memory access instruction from a preset instruction cache; obtaining a first base address corresponding to the compressed matrix in the global memory and a first destination address pointing to the numerical shared memory corresponding to the target data processing channel, so as to read the compressed matrix stored at the first base address from the global memory based on the first preset memory access instruction, and write it to the first destination address in the numerical shared memory; obtaining a second base address corresponding to the index matrix in the global memory and a second destination address pointing to the index shared memory corresponding to the target data processing channel, so as to read the index matrix stored at the second base address from the global memory based on the first preset memory access instruction, and write it to the second destination address in the index shared memory.

[0074] Specifically, Figure 4 As shown, the target thread bundle W0 reads the customized first preset memory access instruction LDG2S (where LD is the abbreviation of Load, G represents global memory, 2 is the same as to, and S represents shared memory) from the preset instruction cache according to the PC (Program Counter) provided by the thread scheduler, and then parses through the decoding unit to obtain the first base address GEMM.srcA corresponding to the compression matrix A in the global memory and the first destination address C0.SMEM.dstA in the numerical shared memory V-SMEM corresponding to the target data processing channel C0, and then passes the address information to the LSU (Load / StoreUnit) unit, which reads the 4x4 compression matrix A according to the first base address and stores it in V-SMEM according to the first destination address. Repeat the execution of the first preset memory access instruction to obtain the second base address GEMM.srcA corresponding to the index matrix in the global memory. I and the second destination address C0.SMEM.dstA in the index shared memory I-SMEM corresponding to the target data processing channel C0 I , the LSU unit indexes the matrix A according to the address I Stored in I-SMEM.

[0075] In a specific embodiment, the register file includes a first register file and a second register file; accordingly, the use of the target thread bundle to load the feature image and the preset bias matrix from the global memory to the register file includes: using the target thread bundle to read the second preset memory access instruction from the preset instruction cache; obtaining the third base address corresponding to the feature image in the global memory and the address of the first register file, and obtaining the second destination address corresponding to the index matrix, so as to write the target element of the feature image read from the global memory according to the third base address and the second destination address into the first register file based on the second preset memory access instruction; using the target thread bundle to read the word load instruction from the preset instruction cache; obtaining the fourth base address corresponding to the preset bias matrix in the global memory and the address of the second register file, so as to read the preset bias matrix stored at the fourth base address from the global memory based on the word load instruction, and write it into the second register file.

[0076] Among them, the above-mentioned writing the target element of the feature image read from the global memory according to the third base address and the second destination address into the first register file includes: based on the matrix product operation rule, using the second destination address to take out the row index address corresponding to the feature image from the index shared memory; based on the third base address and the row index address, writing the target element of the feature image read from the global memory into the first register file.

[0077] That is, if Figure 4 As shown, the target thread warp W0 continues to fetch instructions from the preset instruction cache according to the new PC provided by the thread scheduler to execute the customized second preset memory access instruction LDGS2R (where LD is the abbreviation of Load, which means load; G: represents global memory; S: represents shared memory; 2: is the same as to; R: represents register file), and the decoding unit parses to obtain the third base address GEMM.srcB corresponding to the 8x8 feature image matrix B in the global memory, and obtains the address RF.rdB of the first register file and the second destination address C0.SMEM.srcA corresponding to the index matrix. I , then pass the address information to the LSU unit for address resolution, according to C0.SMEM.srcA IThe address is taken from the index shared memory I-SMEM corresponding to the C0 channel to obtain the row index address Row.index0 corresponding to the feature image matrix B, and then the entire row target element of the corresponding matrix B is read from the global memory according to the address GEMM.srcB+Row.index0 and written into the first register file RF.rdB. The default register width is 256 bits. Repeat the LDGS2R memory access instruction, and store all the remaining 3 rows of the matrix B corresponding to the row indexes Row.index1~Row.index3 into the corresponding first register file, and finally record them as R0~R3. Then the target thread bundle W0 continues to fetch the execution word load instruction (Load Word, LW), and after decoding and parsing, the fourth base address GEMM.srcC and the address RF.rdC of the second register file corresponding to the 8x8 preset bias matrix C in the global memory are obtained. Then the LSU unit takes out the first row element of the matrix C according to the address GEMM.srcC and stores it in the second register file RF.rdC, and records it as R4. At this point, the storage stage is completed, and the required data is prepared for the matrix operation.

[0078] Step S15: Utilize the target thread warp to execute the operation instruction, so as to read the current matrix operation data from the shared memory and the register file to a preset matrix calculation unit to complete the matrix multiplication and addition operation.

[0079] In this embodiment, Figure 4 As shown, the target thread warp is used to execute the operation instruction SDFMA (sparse-dense matrix multiplication operation instruction; where S represents sparse, Sparse; D represents dense, Dense; FMA represents floating-point multiplication and addition operation) to read the current matrix operation data from the shared memory and register file to the preset matrix calculation unit to complete the matrix multiplication and addition operation.

[0080] In a specific implementation, the above-mentioned reading of the current matrix operation data from the shared memory and the register file to a preset matrix calculation unit to complete the matrix multiplication and addition operation includes: reading the first target row element in the compressed matrix from the shared memory to the first local memory, and reading the second target row element of the feature image from the register file to the second local memory, and reading the third target row element of the preset bias matrix from the register file to the third local memory; wherein the first target row element, the second target row element and the third target row element are combined to obtain the current matrix operation data; and inputting the data in the first local memory, the second local memory and the third local memory into the preset matrix calculation unit to complete the matrix multiplication and addition operation.

[0081] Specifically, the first row of the 4x4 compressed matrix A is read from the numerical shared memory V-SMEM of C0 using the operand collection unit and is recorded as A 1x4 And store it in the first local memory LMEM-A, and read R0~R3 combination from the first register file RF.rdC respectively, record it as B 4x8 Stored in the second local memory LMEM-B, and read R4 from the second register file RF.rdC, recorded as C 1x8 The data in the first local memory, the second local memory and the third local memory are then input into the preset matrix calculation unit ME (Matrix Element) to complete the matrix multiplication and addition operation. Figure 5 As shown, the matrix calculation unit integrates 8 matrix calculation sub-units Sub-ME and 8 bias addition units. Each Sub-ME consists of 4 multiplication units and 3 addition units. The matrix data {A 1x4 , B 4x1}, the bias unit allocates the matrix data C1x1. After calculation, the calculation unit ME outputs the calculation result D 1x8 Write it back to the register file, and the minimum granularity sparse matrix multiplication operation is completed. Repeat the above steps until the 8x8 sparse matrix operation is completed.

[0082] It can be seen that the present application needs to pre-process the original image after obtaining it, and input the feature image obtained after pre-processing into the convolutional neural network for subsequent convolution operations. If the convolution kernel for convolution operation with the feature image meets the preset sparse matrix condition, the convolution kernel is first structured sparse to obtain a sparse matrix, and the sparse matrix is ​​compressed using the improved compressed sparse row format to obtain a compressed matrix, and the index matrix corresponding to the compressed matrix is ​​constructed. By improving the sparse row format, its purpose is to abandon the row pointer array in the sparse row format, further reduce the storage space required for the operation, and improve the reading efficiency of the sparse matrix elements. Then in the storage stage, a hierarchical storage structure is adopted, including global memory, shared memory, register file, etc., and independent shared memory is allocated to each data processing channel, which can greatly reduce the probability of data access conflict and improve the memory access efficiency of matrix elements. Specifically, it is necessary to store the feature image, compression matrix, index matrix and preset bias matrix in the global memory, determine the computing core assigned to the current convolution operation, then activate the target thread bundle in the computing core, and assign the target thread bundle to the target data processing channel; further, use the target thread bundle to load the compression matrix and index matrix from the global memory to the shared memory corresponding to the target data processing channel, and load the feature image and preset bias matrix from the global memory to the register file. Finally, use the target thread bundle to execute the operation instruction to read the current matrix operation data from the shared memory and register file to the preset matrix calculation unit to complete the matrix multiplication and addition operation. In this way, this solution significantly improves the efficiency and performance of matrix multiplication operations by adopting improved compressed sparse row format, hierarchical storage, multi-channel processing and other technologies, thereby reducing unnecessary memory access and saving computer resources.

[0083] See also Figure 6 As shown, the embodiment of the present application discloses a sparse matrix processing device in an image recognition process, the device comprising:

[0084] An image input module 11 is used to input a feature image obtained by preprocessing the original image into a convolutional neural network;

[0085] A sparse matrix processing module 12 is used for, if the convolution kernel for convolution operation with the feature image satisfies a preset sparse matrix condition, performing structured sparse operation on the convolution kernel to obtain a sparse matrix, compressing the sparse matrix using an improved compressed sparse row format to obtain a compressed matrix, and constructing an index matrix of the compressed matrix;

[0086] A resource allocation module 13, used to store the feature image, the compression matrix, the index matrix and the preset bias matrix in a global memory, determine a computing core allocated for a current convolution operation, then activate a target thread warp in the computing core, and allocate the target thread warp to a target data processing channel;

[0087] A data loading module 14, configured to load the compression matrix and the index matrix from the global memory to a shared memory corresponding to the target data processing channel by using the target thread warp, and to load the feature image and the preset bias matrix from the global memory to a register file by using the target thread warp;

[0088] The operation module 15 is used to execute the operation instruction by using the target thread warp to read the current matrix operation data from the shared memory and the register file to a preset matrix calculation unit to complete the matrix multiplication and addition operation.

[0089] It can be seen that the present application needs to pre-process the original image after obtaining it, and input the feature image obtained after pre-processing into the convolutional neural network for subsequent convolution operations. If the convolution kernel for convolution operation with the feature image meets the preset sparse matrix condition, the convolution kernel is first structured sparse to obtain a sparse matrix, and the sparse matrix is ​​compressed using the improved compressed sparse row format to obtain a compressed matrix, and the index matrix corresponding to the compressed matrix is ​​constructed. By improving the sparse row format, its purpose is to abandon the row pointer array in the sparse row format, further reduce the storage space required for the operation, and improve the reading efficiency of the sparse matrix elements. Then in the storage stage, a hierarchical storage structure is adopted, including global memory, shared memory, register file, etc., and independent shared memory is allocated to each data processing channel, which can greatly reduce the probability of data access conflict and improve the memory access efficiency of matrix elements. Specifically, it is necessary to store the feature image, compression matrix, index matrix and preset bias matrix in the global memory, determine the computing core assigned to the current convolution operation, then activate the target thread bundle in the computing core, and assign the target thread bundle to the target data processing channel; further, use the target thread bundle to load the compression matrix and index matrix from the global memory to the shared memory corresponding to the target data processing channel, and load the feature image and preset bias matrix from the global memory to the register file. Finally, use the target thread bundle to execute the operation instruction to read the current matrix operation data from the shared memory and register file to the preset matrix calculation unit to complete the matrix multiplication and addition operation. In this way, this solution significantly improves the efficiency and performance of matrix multiplication operations by adopting improved compressed sparse row format, hierarchical storage, multi-channel processing and other technologies, thereby reducing unnecessary memory access and saving computer resources.

[0090] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part refer to the description of the embodiments of the method part, which will not be described here. In addition, it has the same beneficial effects as the sparse matrix processing method in the image recognition process mentioned above.

[0091] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the sparse matrix processing method in the image recognition process performed by the electronic device disclosed in any of the aforementioned embodiments.

[0092] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0093] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0094] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0095] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows, Unix, Linux, etc. In addition to including a computer program that can be used to complete the sparse matrix processing method in the image recognition process performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to data transmitted from an external device received by the electronic device, the data 223 can also include data collected by its own input and output interface 25.

[0096] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the sparse matrix processing method in the image recognition process disclosed in any of the aforementioned embodiments are implemented.

[0097] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0098] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0099] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.

[0100] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0101] The above is a detailed introduction to the sparse matrix processing method, device, equipment and storage medium in the image recognition process provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A sparse matrix processing method in an image recognition process, characterized in that: include: The feature image obtained after preprocessing the original image is input into the convolutional neural network; If the convolution kernel for convolution operation with the feature image satisfies a preset sparse matrix condition, the convolution kernel is structured sparsely processed to obtain a sparse matrix, the sparse matrix is ​​compressed using an improved compressed sparse row format to obtain a compressed matrix, and an index matrix of the compressed matrix is ​​constructed; The feature image, the compression matrix, the index matrix and the preset bias matrix are stored in a global memory, and a computing core allocated for the current convolution operation is determined, and then a target thread warp in the computing core is activated, and the target thread warp is allocated to a target data processing channel; Using the target thread warp to load the compression matrix and the index matrix from the global memory to a shared memory corresponding to the target data processing channel, and using the target thread warp to load the feature image and the preset bias matrix from the global memory to a register file; The target thread warp is used to execute a computation instruction to read current matrix computation data from the shared memory and the register file to a preset matrix computation unit to complete a matrix multiplication and addition operation.

2. The sparse matrix processing method in the image recognition process according to claim 1, characterized in that: The step of performing structured sparse processing on the convolution kernel to obtain a sparse matrix, compressing the sparse matrix using an improved compressed sparse row format to obtain a compressed matrix, and constructing an index matrix of the compressed matrix includes: According to the rule that the number of zero-value elements in the matrix elements is the same as the number of non-zero-value elements, a pruning algorithm is used to perform structured sparseness on the convolution kernel to obtain a sparse matrix; The sparse matrix is ​​compressed using an improved compressed sparse row format, and non-zero value elements of each row are rearranged to form a new row to obtain a compressed matrix; Column indices of non-zero-valued elements in each row are extracted from the sparse matrix to construct an index matrix corresponding to the compressed matrix.

3. The sparse matrix processing method in the image recognition process according to claim 1, characterized in that: The computing core includes at least four data processing channels, each of the data processing channels corresponds to an activated thread warp at the same time, and the thread warp includes at least eight threads; Accordingly, activating the target warp in the computing core and allocating the target warp to the target data processing channel includes: Determine and activate the target thread warp in descending order of task priority; A target data processing channel is determined from all the data processing channels in order of the current load from small to large, and the target thread warp is allocated to the target data processing channel.

4. The sparse matrix processing method in the image recognition process according to claim 1, characterized in that: The shared memory includes a value shared memory and an index shared memory; Accordingly, the step of using the target thread warp to load the compression matrix and the index matrix from the global memory to a shared memory corresponding to the target data processing channel includes: Using the target thread warp to read a first preset memory access instruction from a preset instruction cache; Obtaining a first base address corresponding to the compressed matrix in the global memory and a first destination address in the numerical shared memory corresponding to the target data processing channel, so as to read the compressed matrix stored at the first base address from the global memory based on the first preset memory access instruction, and write the compressed matrix to the first destination address in the numerical shared memory; Obtain a second base address corresponding to the index matrix in the global memory and a second destination address in the index shared memory corresponding to the target data processing channel, so as to read the index matrix stored at the second base address from the global memory based on the first preset memory access instruction, and write it to the second destination address in the index shared memory.

5. The sparse matrix processing method in the image recognition process according to claim 4, characterized in that: The register file comprises a first register file and a second register file; Accordingly, the step of using the target thread warp to load the feature image and the preset bias matrix from the global memory to a register file includes: Using the target thread warp to read a second preset memory access instruction from the preset instruction cache; Obtaining a third base address corresponding to the feature image in the global memory and an address of the first register file, and obtaining the second destination address corresponding to the index matrix, so as to write a target element of the feature image read from the global memory according to the third base address and the second destination address into the first register file based on the second preset memory access instruction; Using the target thread warp to read a word load instruction from the preset instruction cache; A fourth base address corresponding to the preset bias matrix in the global memory and an address of the second register file are obtained, so as to read the preset bias matrix stored at the fourth base address from the global memory based on the word load instruction and write it into the second register file.

6. The sparse matrix processing method in the image recognition process according to claim 5, characterized in that: Writing the target element of the feature image read from the global memory according to the third base address and the second destination address into the first register file comprises: Based on a matrix product operation rule, using the second destination address to retrieve a row index address corresponding to the feature image from the index shared memory; The target element of the feature image read from the global memory based on the third base address and the row index address is written into the first register file.

7. The sparse matrix processing method in the image recognition process according to any one of claims 1 to 6, characterized in that: The step of reading the current matrix operation data from the shared memory and the register file to a preset matrix calculation unit to complete the matrix multiplication and addition operation includes: Reading a first target row element in the compressed matrix from the shared memory to a first local memory, reading a second target row element of the feature image from the register file to a second local memory, and reading a third target row element of the preset bias matrix from the register file to a third local memory; wherein the first target row element, the second target row element and the third target row element are combined to obtain current matrix operation data; The data in the first local memory, the second local memory and the third local memory are input into a preset matrix calculation unit to complete a matrix multiplication and addition operation.

8. A sparse matrix processing device in an image recognition process, characterized in that: include: An image input module is used to input a feature image obtained by preprocessing the original image into a convolutional neural network; A sparse matrix processing module, configured to perform structured sparse processing on the convolution kernel to obtain a sparse matrix if the convolution kernel for convolution operation with the feature image meets a preset sparse matrix condition, compress the sparse matrix using an improved compressed sparse row format to obtain a compressed matrix, and construct an index matrix of the compressed matrix; a resource allocation module, configured to store the feature image, the compression matrix, the index matrix, and the preset bias matrix in a global memory, determine a computing core allocated for a current convolution operation, activate a target thread bundle in the computing core, and allocate the target thread bundle to a target data processing channel; a data loading module, configured to load the compression matrix and the index matrix from the global memory to a shared memory corresponding to the target data processing channel by using the target thread warp, and to load the feature image and the preset bias matrix from the global memory to a register file by using the target thread warp; The operation module is used to execute the operation instruction by using the target thread warp to read the current matrix operation data from the shared memory and the register file to a preset matrix calculation unit to complete the matrix multiplication and addition operation.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the sparse matrix processing method in the image recognition process as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the sparse matrix processing method in the image recognition process as described in any one of claims 1 to 7 are implemented.

Images