Unstructured sparse neural network model-oriented reasoning optimization method and system

Abstract

The invention discloses an inference optimization method and system for an unstructured sparse neural network model, and belongs to the technical field of image processing, and the method comprises the steps: reconstructing a convolution kernel data flow in a preprocessing stage, and employing sparse representation to code non-zero data to reduce the redundant zero calculation; in the compiling stage, optimization such as sparse operator implementation, tensor kernel cyclic matching and cyclic boundary fixation based on a predefined sketch is completed by utilizing a TVM optimization rule so as to adapt to different edge hardware; in the reasoning stage, the memory multiplexing relation is comprehensively considered for feature map data flow reconstruction, and frequent addressing in the calculation process is reduced. Compared with an existing sparse acceleration scheme, the method has the advantages that the model reasoning real-time performance is improved while the reasoning precision is guaranteed, and the method can be efficiently adapted to various reasoning hardware.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically to an inference optimization method and system for an unstructured sparse neural network model. Background Art

[0002] In recent years, Convolutional Neural Networks (CNN) have been widely used in many fields, such as image classification, object detection, semantic segmentation, etc. With the continuous evolution of the model, today's CNN model has become very complex, and the storage and computing requirements have increased significantly, which makes the deployment of CNN models on back-end hardware platforms face huge challenges. Taking the VGG16 model as an example, it has 130 million parameters and occupies 500MB of storage space. It requires 30 billion floating-point operations to complete an image recognition task. Studies have shown that there are a large number of redundant parameters in the neural network. Therefore, many studies have begun to try to prune the convolutional layer to reduce the overall size of the model and achieve more efficient reasoning.

[0003] Structured and unstructured pruning are typical pruning methods. Structured pruning compresses the model by pruning unimportant convolution kernels (including the entire convolution kernel or some unimportant channels in the convolution kernel) or network layers to achieve inference acceleration, but in order to ensure the accuracy of the network, its pruning rate is often not high. Unstructured pruning compresses the network by deleting unimportant neurons (i.e., convolution kernel parameters in the convolution kernel). Compared with structured pruning, there is a finer pruning granularity, so unstructured pruning can relatively reduce the model size more significantly while ensuring the original accuracy of the network. Typical unstructured model pruning methods can remove more than 90% of the parameters in the model with almost no loss of accuracy, and reduce the computational requirements of the sparse model after pruning to 10% before pruning. The model accuracy loss after unstructured pruning is smaller, but due to the randomness of the pruned convolution kernel object, the pruned parameters are irregular, which brings irregular sparsity to the model. When deploying the model, the irregular sparsity of the model makes it impossible for the hardware platform to perform efficient parallel acceleration. It is necessary to design a dedicated sparse library to support it through the collaboration of software and hardware, which leads to heavy development costs and difficulties in migrating methods.

[0004] The acceleration of sparse model reasoning after unstructured pruning faces two main challenges: First, the data flow of unstructured sparse neural network models is irregular, and irregular data flows are difficult to match the parallel computing mode of hardware, resulting in low computing and memory access efficiency. Second, sparse computing methods require specific scheduling strategies for different hardware platforms to achieve the best performance of model reasoning, which requires high labor costs.

[0005] Therefore, for the reasoning of unstructured sparse neural network models in image recognition, how to effectively reconstruct the data flow to achieve efficient parallel reasoning and improve the reasoning performance of unstructured sparse neural network models on multiple platforms is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0006] In view of this, the present invention provides an inference optimization method and system for unstructured sparse neural network models that at least solve some of the above-mentioned technical problems. The method and system are applied to image recognition to perform inference acceleration of unstructured sparse neural network models. The method is mainly divided into three parts according to the process: the preprocessing stage reconstructs the convolution kernel data stream and uses sparse representation to encode non-zero data to reduce redundant zero value calculations; the compilation stage uses TVM optimization rules to complete sparse operator implementation, tensor kernel loop matching, loop boundary fixing based on predefined sketches and other optimizations to adapt to different edge hardware; the inference stage comprehensively considers the memory reuse relationship to reconstruct the feature map data stream to reduce frequent addressing during the calculation process; compared with the existing sparse acceleration scheme, the present invention can effectively reconstruct the data stream to achieve efficient parallel reasoning, which not only improves the real-time performance of model reasoning while ensuring the reasoning accuracy, but also can be efficiently adapted to a variety of reasoning hardware, thereby improving the reasoning performance of unstructured sparse neural network models on multiple platforms.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is:

[0008] In a first aspect, the present invention provides an inference optimization method for an unstructured sparse neural network model, the method comprising:

[0009] In the preprocessing stage, the model convolution kernel data stream is reconstructed and the non-zero data of the convolution kernel is encoded using sparse representation;

[0010] Parallel parameter tuning is performed during the compilation phase, and the TVM framework is used to complete hardware backend adaptation optimization;

[0011] In the inference phase, the input feature map data stream is reconstructed to convert the three-dimensional input feature map in the convolution into a two-dimensional intermediate matrix form.

[0012] Preferably, when reconstructing the model convolution kernel data stream, the convolution kernel is first reduced to two dimensions; then the CSR sparse representation of the convolution kernel non-zero data is used; and finally, the convolution kernel is matched with the corresponding position of the input feature map through a mapping function.

[0013] Preferably, when reconstructing the model convolution kernel data flow, a bucket sorting method is also used to solve the problem of unbalanced task loads between multi-threads / computing units, specifically including:

[0014] Sort each row of the two-dimensional matrix after sparse representation by the number of non-zero data from large to small and regard the number of threads / parallel calculations supported by the hardware as buckets;

[0015] Add the sorted workload from largest to smallest to the bucket with the least load.

[0016] Preferably, during parallel parameter tuning, sparse operator implementation, tensor core loop matching, and loop boundary fixing based on predefined sketches are completed to adapt to different edge hardware.

[0017] Preferably, the sparse operators include: a sparse convolution kernel data processing operator, a feature map conversion operator and a sparse convolution operator, wherein: the sparse convolution kernel data processing operator is used in the preprocessing stage and not in the compilation stage; the feature map conversion operator and the sparse convolution operator are compiled and optimized in TVM.

[0018] Preferably, in the process of fixing the loop boundary based on the predefined sketch, each sparse operator is pre-labeled, and the conditional function is used to judge the operator during the tuning process. If the operator is judged to be a labeled sparse operator, the custom sketch tuning space construction is executed; the application function is used in the custom sketch to determine the tuning space.

[0019] Preferably, when reconstructing the input feature map data stream, the feature map is reconstructed using a data aggregation method to aggregate the feature map data required by the sparse convolution kernel; then the aggregated results are processed using data reuse in the vertical direction; finally, the multi-channel feature map is reduced in dimension using a channel-first approach, and the converted matrix presents a continuous arrangement.

[0020] In a second aspect, the present invention further provides a data stream reconstruction system for an unstructured sparse neural network model, which is applied to the above-mentioned inference optimization method for an unstructured sparse neural network model to perform inference acceleration of the unstructured sparse neural network model, and the system comprises:

[0021] The convolution kernel data stream reconstruction module is used to reconstruct the model convolution kernel data stream in the preprocessing stage and use sparse representation to encode the convolution kernel non-zero data;

[0022] The parallel parameter tuning module is used to perform parallel parameter tuning during the compilation phase and use the TVM framework to complete backend adaptation optimization;

[0023] The feature map data stream reconstruction module is used to reconstruct the input feature map data stream in the inference stage and convert the three-dimensional input feature map in the convolution into a two-dimensional intermediate matrix form.

[0024] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned inference optimization method for unstructured sparse neural network models.

[0025] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0026] The present invention provides an inference optimization method and system for unstructured sparse neural network models. In the preprocessing stage, the present invention reconstructs the convolution kernel data stream, and uses sparse representation to encode non-zero data to reduce redundant zero value calculations; in the compilation stage, the TVM optimization rules are used to complete sparse operator implementation, tensor kernel loop matching, loop boundary fixing based on predefined sketches and other optimizations to adapt to different edge hardware; in the inference stage, the memory reuse relationship is comprehensively considered to reconstruct the feature map data stream to reduce frequent addressing in the calculation process; compared with the existing sparse acceleration scheme, the present invention can effectively reconstruct the data stream to achieve efficient parallel reasoning, which not only improves the real-time performance of model reasoning while ensuring the reasoning accuracy, but also can be efficiently adapted to a variety of reasoning hardware, thereby improving the reasoning performance of unstructured sparse neural network models on multiple platforms.

[0027] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0028] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0031] Figure 1 A schematic flow chart of an inference optimization method for an unstructured sparse neural network model provided in an embodiment of the present invention.

[0032] Figure 2A schematic diagram of Im2col data stream reconstruction provided by an embodiment of the present invention.

[0033] Figure 3 A schematic diagram of a vertical data expansion process provided by an embodiment of the present invention.

[0034] Figure 4 A schematic diagram of a multi-channel data stream reconstruction process provided by an embodiment of the present invention.

[0035] Figure 5 A schematic diagram of the convolution kernel data stream reconstruction process provided in an embodiment of the present invention.

[0036] Figure 6 A schematic diagram of a bucket sorting process provided by an embodiment of the present invention.

[0037] Figure 7 A schematic diagram of sparse operator replacement provided in an embodiment of the present invention.

[0038] Figure 8 A schematic diagram of a calling template provided in an embodiment of the present invention.

[0039] Fig. 9 A schematic diagram of the comparison of inference latency on an X86 CPU provided in an embodiment of the present invention.

[0040] Fig.10 A schematic diagram of the comparison of inference latency on an ARM CPU provided in an embodiment of the present invention.

[0041] Fig.11 A schematic diagram of inference latency comparison on the RTX3090 GPU provided for an embodiment of the present invention.

[0042] Fig.12 A schematic diagram of the comparison of inference latency on a V100 GPU provided in an embodiment of the present invention.

[0043] Fig.13 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0045] In the description of the present invention, it should be noted that: in some processes described in the specification and drawings of this application, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be performed in the order in which they appear in this article or may be performed in parallel. In addition, various serial numbers, etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. It should be noted that the word "comprising" does not exclude the existence of components or steps not listed in the claims. The word "one" or "an" preceding a component does not exclude the existence of multiple such components. The present invention can be implemented by means of hardware including several different components and by means of a suitably programmed computer.

[0046] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Embodiment 1:

[0048] See also Figure 1 As shown, an embodiment of the present invention provides an inference optimization method for unstructured sparse neural network models, which is applied to image recognition to perform inference acceleration of unstructured sparse neural network models; the method is mainly divided into three parts according to the process: the preprocessing stage includes model convolution kernel data flow reconstruction, the compilation stage includes parallel parameter tuning, and the inference stage includes input feature map data flow reconstruction.

[0049] In the embodiment of the present invention, in order to better illustrate the connection between the methods used in the present invention, the following will first introduce the reconstruction of the input feature map data stream and the reconstruction of the model convolution kernel data stream, and finally introduce the multi-platform adaptation and tuning.

[0050] (I) Input feature map data stream reconstruction:

[0051] The present invention is mainly used in the field of image recognition, and performs convolutional neural network reasoning acceleration (specifically, to enable unstructured sparse neural network models to achieve efficient reasoning on multiple platforms). The convolutional neural network contains many layers, each layer is composed of an input feature map, an output feature map, and a convolution kernel. The output feature map of the current layer is the input feature map of the next layer. Therefore, the data that needs to be optimized in the present invention is the input feature map and convolution kernel data in the convolutional neural network reasoning process.

[0052] In the reconstruction part of the model input feature map data stream, the present invention converts the three-dimensional input feature map in the convolution into a new two-dimensional intermediate matrix form. The feature map is reconstructed by using the data aggregation method, and the feature map data required by the sparse convolution kernel is aggregated together. Then the aggregated result is further processed by using data reuse in the vertical direction. Finally, the multi-channel feature map is reduced in dimension by channel priority, and the converted matrix presents a continuous arrangement. The specific implementation method is as follows:

[0053] 1. Data aggregation:

[0054] In this embodiment, the data stream reconstruction method Im2col is used to aggregate the feature graph data that needs to be calculated for each sliding window, thereby increasing the continuity of data access. Figure 2 As shown, it is a standard 3×3 convolution process, the input feature map size (I h is the input feature map height, I w The input feature width) is 7×7 (for ease of presentation, the channel dimension is ignored), and the output feature map size (O h is the output feature map height, O w The output feature width is consistent with the input feature map size, the convolution kernel size is 3×3, the padding is 1, and the stride is 1. The right side is the two-dimensional intermediate matrix after the input feature map data stream is reconstructed. Specifically, Figure 2 The gray part of the input feature map is the data that needs to be calculated at the (0,0) position in the convolution kernel. First, it is aggregated into the first row of the two-dimensional intermediate matrix (shown in gray, a total of O h ×O w data). Then, the input feature map data required for the remaining positions of the convolution kernel are reconstructed one by one. For example, the dotted box in the input feature map is the data required by the convolution kernel at position (2,1), which is gathered in the 8th row of the two-dimensional intermediate matrix. Finally, when all the input feature maps are converted, the size of the obtained two-dimensional intermediate matrix is ​​K h ×K w ×O h ×O w (3×3×7×7).

[0055] 2. Vertical data expansion and reconstruction:

[0056] After the aggregation process is completed, many redundant values ​​appear. The present invention increases the vertical data reuse in the process of the aggregation method. Figure 3As shown in the figure, after completing the arrangement of the calculation data required for the position (0,0) in the convolution kernel, the data is arranged in the vertical direction (the shaded part in the input feature map), and finally data A is formed. Slide to the right with a step size of 1 to obtain the next partition B in the feature map, that is, after completing the arrangement of the calculation data (green frame) required for the position (0,1) in the convolution kernel, the data is arranged in the vertical direction. Finally, the arrangement of the calculation data (purple frame) required for the position (0,1) in the convolution kernel is completed, and the data is arranged in the vertical direction. In this example, three groups of reconstructed data {A, B, C} are finally formed. Compared with the data aggregation method, the data storage capacity can be further reduced. The size of the three groups of reconstructed data obtained is 3×7×9. Compared with the data aggregation method, the data storage capacity (3×3×7×7) is greatly reduced. When the step size is not equal to 1 (such as the step size is 2), the reuse of vertical data will be different from when the step size is 1. Specifically, after completing the arrangement of the calculation data required for the position (0,0) in the convolution kernel, the data is arranged in the vertical direction. Next, the data arrangement required for the convolution kernel position (1,0) needs to be completed and the data arrangement in the vertical direction needs to be continued. Next, the data arrangement required for the convolution kernel position (0,1) needs to be completed and the data arrangement in the vertical direction needs to be continued. Similarly, the data arrangement required for the convolution kernel positions (1,1), (0,2), and (1,2) needs to be completed and the data arrangement in the vertical direction needs to be continued.

[0057] 3. Reconstruction of feature map data stream with input channel priority:

[0058] For common multi-channel input feature maps, the present invention adopts a feature map data stream reconstruction strategy with input channel priority. Figure 4 As shown, Figure 4 The input channels in the are three channels (C i , i=1,2,3), where each input channel C i Will form 3 sets of reconstruction data {A i , B i , C i For these 9 sets of data, the final data stream reconstruction order is A0~A3, B0~B3, C0~C3. The input channel priority arrangement can meet the computational requirements of the convolution kernel with high energy consumption, making data acquisition more continuous.

[0059] The following will introduce how to reconstruct the convolution kernel data flow so that the reconstructed convolution kernel data accurately matches the reconstructed input feature map data, and utilize the sparse characteristics of the convolution kernel to significantly reduce the amount of calculation while efficiently utilizing the parallel capabilities of the hardware.

[0060] (II) Model convolution kernel data flow reconstruction:

[0061] In an embodiment of the present invention, the convolution kernel is first reduced to two dimensions; secondly, CSR is used to sparsely represent the non-zero data of the convolution kernel, effectively reducing the redundant zero value storage of the convolution kernel; finally, a mapping function is used to match the convolution kernel with the corresponding position of the input feature map, thereby improving the addressing efficiency in the calculation process.

[0062] For detailed procedures, see Figure 5 As shown in the figure, in the process of reducing the sparse weight matrix of the convolution kernel into a two-dimensional matrix, the data of each convolution kernel is a row, and the channel priority principle is used to arrange the data inside each convolution kernel, that is, the arrangement order is HWC. In order to reduce the storage amount of non-zero values, the present invention adopts the CSR sparse representation method, which is represented as follows Figure 5 Finally, in order to enable the convolution calculation to be calculated using efficient addressing, the present invention uses the following formula to map the convolution kernel data CSR index to an index matrix. Figure 2 dimensional intermediate matrix for mapping.

[0063]

[0064] Among them, I in the index matrix row and I col The input feature map data required for the convolution kernel non-zero data is in the feature Figure 2 The starting position in the intermediate matrix, for example, (0,0) corresponds to the feature Figure 2 The gray box in the 3-dimensional intermediate matrix. (3,10) corresponds to the feature Figure 2 The green box in the 3D intermediate matrix. The entire convolution kernel data flow reconstruction process is completed in the compilation phase before inference, which does not take up additional inference time. This ultimately solves the problem of low addressing efficiency in unstructured sparse computing.

[0065] Furthermore, in the embodiment of the present invention, a bucket sort algorithm is used to solve the problem of unbalanced task load between multi-threads / computing units in the sparse computing process. The details are as follows:

[0066] like Figure 5 As shown in part (b), the non-zero value calculation tasks in the convolution kernel (each row in the two-dimensional matrix) are performed. At this time, there are two parallel methods. One is to divide the tasks within the convolution kernel, that is, to assign the non-zero value calculation tasks in each convolution kernel to different threads / computing units, but this allocation will cause write conflicts to varying degrees, which is not conducive to system maintenance. The second is to divide the tasks according to the number of convolution kernels, that is, to assign all non-zero value calculation tasks in the convolution kernel to one computing unit, and form parallelism between the convolution kernels. At this time, since the number of non-zero values ​​of each convolution kernel is different, it causes an unbalanced task load between multiple threads / computing units.

[0067] In a specific embodiment, the present invention uses bucket sorting to solve the problem of unbalanced task loads among multiple threads / computing units. The basic process is: first, sort each row in the two-dimensional matrix after sparse representation from large to small according to the number of non-zero data and regard the number of threads / parallel calculations supported by the hardware as buckets. Secondly, sort the sorted workload from large to small to add the work to the bucket with the least load. Figure 6 As shown, A, B, C, and D are Figure 5 In part (b), the workload of each row of the two-dimensional matrix is ​​2, 3, 1, and 1, respectively, measured by the number of non-zero values. After sorting from largest to smallest, their order is B, A, C, and D. Assuming that the hardware supports dual threads (two buckets), first add the task B with the largest workload to the bucket of thread 1 with the least workload. At this time, the bucket with the least workload becomes the bucket of thread 2. Next, assign the current task A with the largest workload to thread 2, and the task assignment is considered complete until the last task D is put into the bucket.

[0068] (III) Parallel parameter tuning:

[0069] In the embodiment of the present invention, in order to make the algorithm adaptable to a variety of hardware platforms, the present invention adopts the TVM framework to implement backend adaptation optimization. The present invention performs multiple operations to meet the optimization rules of TVM, including sparse operator implementation, tensor core loop matching, loop boundary fixing based on predefined sketches, etc.

[0070] The specific method is as follows:

[0071] 1. Sparse operator implementation:

[0072] In an embodiment of the present invention, a sparse convolution kernel data processing operator (KernelTransform), a feature map conversion operator (FeatureTransform) and a sparse convolution operator (SparseConv) are implemented. The sparse convolution kernel data processing operator is used in the preprocessing stage and is not used in the TVM compilation stage. The feature map conversion operator (FeatureTransform) and the sparse convolution operator (SparseConv) need to be compiled and optimized in TVM. After completing the definition of the FeatureTransform operator and the SparseConv operator, the present invention implements a traversal function (ExprFunctor, a functional function that reads the model layer by layer, such as a ten-layer network, from the first layer to the tenth layer), and after reading the model into the form of TVM's Relay Graph, the dense operator is replaced with a sparse convolution operator accordingly, and the FeatureTransform operator is inserted at the corresponding position. Figure 7The difference between the computation graphs before and after the conversion is shown. The present invention converts a conv2d (intensive computation operator) into a sparse convolution operator (SparseConv), adds FeatureTransform to it after the input, reconstructs it and converts it into two dimensions.

[0073] 2. Tensor Core Cycle Matching:

[0074] Before performing automatic tuning, TVM needs to make the outer loop of the operator to be optimized meet the NCHW order, that is, the outer loop is the dimension of the output tensor. Therefore, the sparse operator needs to be rewritten into the template format required by TVM, where the template is defined as a TVM built-in function with the following function interface:

[0075] te.compute(operator shape, operator implementation function, tag = "name (user-defined)");

[0076] For specific template format, see Figure 8 As shown, it is an example of calling the template te.compute() of the present invention. The present invention has encapsulated this operation, and the user does not need to implement it specifically. It is only for demonstration. The third line can specify the operator shape, and the 4th to 16th lines are the operator implementation functions, and the name is represented by name here.

[0077] 3. Loop boundary fixing and predefined sketches:

[0078] In the embodiment of the present invention, the sparse operator tuning process generally includes constructing a tuning space and performing scheduling tuning. The present invention completes the construction of the tuning space through a predefined sketch. Finally, the scheduling tuning algorithm of TVM is called to implement the tuning of the sparse operator to match the optimal code implementation of different hardware backends.

[0079] TVM mainly forms a tuning space by splitting the loop axis, and searches for the optimal implementation of the code in the tuning space. The boundary of the loop axis is fixed to determine the search space and search strategy. If there is a double index (such as a[b[i]]), TVM cannot be tuned. In the sparse calculation process of the present invention, due to the randomness of the sparse value of the convolution kernel, the phenomenon of determining the starting calculation position of the feature map according to the non-zero convolution kernel position during addressing will occur, resulting in the problem of double indexing due to the unfixed loop boundary. In order to fix the loop boundary, the present invention is based on a predefined sketch, using specified conditional functions and application functions to form a tuning space, and finally calls TVM scheduling and tuning to complete the tuning of the composite index. The specific process is as follows:

[0080] First, each sparse operator is pre-labeled, and the conditional function is used to judge the operator during the tuning process. If the operator is determined to be a label sparse operator, the custom sketch tuning space construction of the present invention is executed. The application function is used in the custom sketch to determine the tuning space. The present invention first extracts the cyclic axis used for sparse operator calculation, defines the upper and lower bounds and split granularity of the cyclic axis according to the sparsity of the convolution kernel (using the TVM default split granularity), and then uses the Split statement to complete the axis splitting according to the upper and lower bounds and split granularity, and uses the Reorder statement to complete the axis reconstruction work, forming a scheduling space for sparse operators. Finally, the scheduling tuning algorithm of TVM is called to realize the tuning of sparse operators to match the optimal code implementation of different hardware backends.

[0081] Finally, in order to improve the usability of the present invention, the present invention integrates the above process into Pass, which is a conversion tool formed by TVM inheriting the LLVM concept. After using Pass to integrate the optimization of the present invention into the TVM framework, it only needs to read the model in the front end and automatically convert it into an intermediate representation, and then call the corresponding Pass function to complete all optimizations and finally obtain an optimized model that can be deployed.

[0082] From the description of the above embodiments, those skilled in the art can know that: in order to enable the unstructured sparse neural network model to achieve efficient reasoning on multiple platforms, the present invention provides an optimization method (data flow reconstruction method), the core of which is to reduce the memory access and parallelism problems caused by the non-zero value reducing the amount of calculation in the sparse case. In the present invention, the input data is reconstructed by aggregation and reuse methods to solve the memory access problem caused by sparsity, the weight data is reconstructed according to the mapping relationship for advance addressing, the parallelism is improved by bucket row, and the TVM sparse scheduling is used through predefined sketches and other methods to solve the multi-platform adaptation problem. Through this method, the unstructured sparse neural network model can obtain reasoning acceleration effect on multiple platforms.

[0083] In order to verify the effectiveness of the present invention, the ResNet50, VGG19, and YOLOv8 models after unstructured sparseness are used. The overall sparsity of these models is between 70% and 90%, and the accuracy is similar to the original model. Table 1 shows the extracted comprehensive benchmark test set consisting of 13 convolutional layers, including the commonly used convolution sizes of 3×3 and 1×1, various convolution parameter arrangements (p={0,1}, s={1,2}), and multiple output channel configurations.

[0084] Table 1 Benchmark test sets

[0085] name Input Parameters Convolution kernel parameters Padding, stride Sparsity CV1 179,14,14 179,179,3,3 1,1 0.827797 CV2 179,14,14 716,179,1,1 0,1 0.831005 CV3 179,28,28 179,179,3,3 1,2 0.827987 CV4 2048,7,7 358,2048,1,1 0,1 0.881186 CV5 358,28,28 716,358,1,1 0,2 0.829097 CV6 358,7,7 2048,358,1,1 0,1 0.881133 CV7 44,56,56 44,44,1,1 0,1 0.871384 CV8 716,14,14 2048,716,1,1 0,2 0.879547 CV9 89,28,28 89,89,3,3 1,1 0.832793 CV10 512,14,14 512,512,3,3 1,1 0.900000 CV11 192,40,40 192,192,3,3 1,1 0.890501 CV12 288,20,20 288,288,3,3 1,1 0.878399 CV13 96,80,80 96,96,3,3 1,1 0.928204

[0086] In the embodiment of the present invention, the experimental platform includes two types of CPU architectures and two types of GPUs. On the CPU, the present invention mainly compares the mature methods on TVM and integration. Since TVM-S is mainly designed for X86, it does not run well on ARM. On the GPU, the present invention not only compares the TVM method, but also compares the general framework PyTorch, the sparse computing library Cusparse and sparTA (sparse model accelerator).

[0087] See also Fig. 9 As shown in FIG. 1 , it is shown that on an x86 CPU, the invention (SPT) can achieve a 1.3-3.6X acceleration compared to TVM-D. In the case of 1x1 size, the acceleration is more than 1.4X compared to TVM-S. Fig.10 The results show that SPT has a 1.5x acceleration over TVM-D on an ARM CPU. In the case of 1x1 with a large number of channels, SPT can achieve a 3x acceleration, and the highest (CV4) can reach a 5.6x acceleration. Fig.11 Results on RTX3090 are shown. In the case of 1x1 where the input channel is much smaller than the output channel (cv4), SPT is slightly inferior to TVM-D and sparTA. In other cases, SPT is 1.1-5x faster than TVM-D and 1-3.6x faster than sparTA. In all cases, the speedup is more than 1.1x compared to PyTorch, and the highest speedup is 33x compared to CuSparse (CV5). Fig.12 Results are shown on V100. The SPT effect on CV4 is still not ideal, which is the opposite of the result on CPU, but it is understandable. Large-scale input leads to more threads, small number of input channels leads to greater data conflicts, and constant cache access failures are fatal to GPU parallelism. In other cases, SPT can achieve 1-4.4x acceleration compared to TVM-D, 1-12x acceleration compared to PyTorch, more than 3x acceleration compared to CuSparse, and 1.2x acceleration compared to sparTA.

[0088] Convolution is the most critical component in the ResNet network. In order to approximately evaluate the overall inference time, the present invention summarizes the time results of all pruned convolutional layers in ResNet50. Table 2 shows that on four different platforms, SPT can achieve 1.53x to 2.44x speedup on TVM and 3.10x to 4.11x speedup on GPU compared with PyTorch.

[0089] Table 2 Comparison of inference performance of unstructured sparse model (ResNet50) on four platforms

[0090]

[0091] In summary, in image recognition detection tasks, compared with existing solutions, it is verified that the method of the present invention can improve the reasoning performance of unstructured sparse neural network models on multiple platforms.

[0092] Embodiment 2:

[0093] The present invention also provides a data stream reconstruction system for an unstructured sparse neural network model, which is applied to the inference optimization method for an unstructured sparse neural network model described in the above embodiment 1 to perform inference acceleration of the unstructured sparse neural network model. The system includes:

[0094] The convolution kernel data stream reconstruction module is used to reconstruct the model convolution kernel data stream in the preprocessing stage and use sparse representation to encode the convolution kernel non-zero data;

[0095] The parallel parameter tuning module is used to perform parallel parameter tuning during the compilation phase and use the TVM framework to complete backend adaptation optimization;

[0096] The feature map data stream reconstruction module is used to reconstruct the input feature map data stream in the inference stage and convert the three-dimensional input feature map in the convolution into a two-dimensional intermediate matrix form.

[0097] An embodiment of the present invention provides a data stream reconstruction system for an unstructured sparse neural network model, and its implementation principle and technical effects are the same as those of the aforementioned method embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment, which will not be repeated here.

[0098] Example 3

[0099] Reference Fig.13 As shown, an embodiment of the present invention further provides an electronic device, which may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, and the processor executes the computer program to implement an inference optimization method for an unstructured sparse neural network model in the above-mentioned embodiment 1.

[0100] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory 11, and calling data stored in the memory 11.

[0101] The memory 11 here is a computer-readable storage medium, including: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, DVD-ROM, Blu-ray or optical disk memory, hard disk drive (HDD), solid state drive (SSD), card memory (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage system, optical data storage system, hard disk, solid state disk and any other system, any of the above-mentioned any other system is configured to store computer programs and any associated data, data files and data structures in a non-temporary manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent system, a server, etc. In addition, in one example, the computer program and any associated data, data files, and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0102] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products, etc. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0104] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for inference optimization of unstructured sparse neural network models, applied to image recognition, to perform inference acceleration of unstructured sparse neural network models; characterized in that: The method includes: In the preprocessing stage, the model convolution kernel data stream is reconstructed and the non-zero data of the convolution kernel is encoded using sparse representation; Parallel parameter tuning is performed during the compilation phase, and the TVM framework is used to complete hardware backend adaptation optimization; In the inference phase, the input feature map data stream is reconstructed to convert the three-dimensional input feature map in the convolution into a two-dimensional intermediate matrix form.

2. The inference optimization method for an unstructured sparse neural network model according to claim 1, characterized in that: When reconstructing the model convolution kernel data stream, the convolution kernel is first reduced to two dimensions; then the non-zero data of the convolution kernel is sparsely represented using CSR; finally, the convolution kernel is matched with the corresponding position of the input feature map through a mapping function.

3. The inference optimization method for unstructured sparse neural network model according to claim 2, characterized in that: When reconstructing the model convolution kernel data flow, the bucket sorting method is also used to solve the problem of unbalanced task load between multi-threads / computing units, including: Sort each row of the two-dimensional matrix after sparse representation by the number of non-zero data from large to small and regard the number of threads / parallel calculations supported by the hardware as buckets; Add the sorted workload from largest to smallest to the bucket with the least load.

4. The inference optimization method for unstructured sparse neural network model according to claim 1, characterized in that: When tuning parallel parameters, sparse operator implementation, tensor core loop matching, and loop boundary fixing based on predefined sketches are completed to adapt to different edge hardware.

5. The inference optimization method for unstructured sparse neural network model according to claim 4, characterized in that: Sparse operators include: sparse convolution kernel data processing operator, feature map conversion operator and sparse convolution operator. Among them: the sparse convolution kernel data processing operator is used in the preprocessing stage but not in the compilation stage; the feature map conversion operator and sparse convolution operator are compiled and optimized in TVM.

6. The inference optimization method for unstructured sparse neural network model according to claim 4, characterized in that: In the process of fixing the loop boundary based on the predefined sketch, each sparse operator is pre-labeled, and the conditional function is used to judge the operator during the tuning process. If the operator is judged to be a labeled sparse operator, the custom sketch tuning space construction is executed; the application function is used in the custom sketch to determine the tuning space.

7. The inference optimization method for unstructured sparse neural network model according to claim 1, characterized in that: When reconstructing the input feature map data stream, the feature map is reconstructed by using the data aggregation method to aggregate the feature map data required by the sparse convolution kernel. The aggregated results are then processed using vertical data reuse. Finally, the multi-channel feature map is reduced in dimension using a channel-first approach, and the converted matrix presents a continuous arrangement.

8. A data stream reconstruction system for unstructured sparse neural network models, characterized in that: When applied, the method for optimizing the inference of an unstructured sparse neural network model according to any one of claims 1 to 7 is executed to accelerate the inference of the unstructured sparse neural network model. The system comprises: The convolution kernel data stream reconstruction module is used to reconstruct the model convolution kernel data stream in the preprocessing stage and use sparse representation to encode the convolution kernel non-zero data; The parallel parameter tuning module is used to perform parallel parameter tuning during the compilation phase and use the TVM framework to complete backend adaptation optimization; The feature map data stream reconstruction module is used to reconstruct the input feature map data stream in the inference stage and convert the three-dimensional input feature map in the convolution into a two-dimensional intermediate matrix form.

9. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement an inference optimization method for an unstructured sparse neural network model as described in any one of claims 1 to 7.

Images