Neural network acceleration processing method, device, computer equipment and storage medium

Abstract

The present disclosure provides a method, apparatus, computer device and storage medium for accelerating processing of a neural network, wherein the method comprises: obtaining compressed sparse feature data streams and compressed sparse coordinate indexes corresponding to a plurality of channel feature maps respectively; the sparse coordinate index is used to indicate the position of non-zero features in the compressed sparse feature data stream in the channel feature map; based on the compressed sparse coordinate index and at least one resolution information corresponding to a network layer in a sparse convolutional neural network, respectively determining a compressed coordinate index model for each resolution information; based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each channel feature map, determining a compressed and arranged first sparse feature map output by the sparse convolutional neural network.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and more specifically, to a method, apparatus, computer device, and storage medium for accelerating processing of neural networks. Background Art

[0002] Deep convolutional neural networks are widely used in modern computer vision, and ensuring the real-time and efficient inference and prediction of deep convolutional neural networks is crucial for their commercialization. This is especially true for scenarios where input feature data is sparse and performance requirements, such as latency, are high. Efficient and high-performance acceleration methods for these scenarios are crucial. Accelerating deep convolutional neural networks using sparse feature maps has important applications in many fields, such as autonomous driving, object detection, and 3D reconstruction.

[0003] However, in many application scenarios, due to the large amount of feature map data and its real-time updates, existing deep convolutional neural network accelerators often suffer from low data access efficiency and prediction latency. Furthermore, deep convolutional neural network accelerators designed for specific applications have low versatility in their deep learning algorithms and chip resources (i.e., they are only suitable for specific application scenarios) and are difficult to deploy. Summary of the Invention

[0004] The embodiments of the present disclosure at least provide a method, apparatus, computer device, and storage medium for accelerating processing of a neural network.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for accelerating processing of a neural network, comprising:

[0006] Obtaining compressed sparse feature data streams and compressed sparse coordinate indexes corresponding to a plurality of channel feature maps, respectively; the compressed sparse coordinate indexes are used to indicate the positions of non-zero features in the compressed sparse feature data streams in the channel feature maps;

[0007] Determine a compressed coordinate index model for each resolution information based on the compressed sparse coordinate index and at least one resolution information corresponding to a network layer in the sparse convolutional neural network;

[0008] Based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each of the channel feature maps, a first sparse feature map of the compressed arrangement output by the sparse convolutional neural network is determined.

[0009] In this regard, the use of compressed data formats of compressed sparse feature data streams and compressed sparse coordinate indexes can reduce the amount of data stored and calculated, improve the real-time performance of predictive neural network inference calculations, and enable predictive neural networks to have better energy efficiency. In addition, based on the determined multiple compressed coordinate index models and the compressed sparse feature data streams corresponding to each of the channel feature maps, the compression calculation process of the compressed arrangement of the first sparse feature map output by the sparse convolutional neural network is determined, which can improve the data processing efficiency of the sparse convolutional neural network and reduce prediction latency. In addition, the accelerated processing flow of the sparse feature map is decoupled into multiple stages. For example, one stage is: based on the compressed sparse coordinate index and at least one resolution information corresponding to the network layer in the sparse convolutional neural network, the compressed coordinate index model of each resolution information is determined respectively; another stage is: based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each channel feature map, the first sparse feature map of the compressed arrangement output by the sparse convolutional neural network is determined; this processing method can solve the flexibility and versatility problems of using sparse convolutional neural networks for accelerated processing of compressed sparse feature data streams, and can improve the ease of deployment of hardware resources in sparse convolutional neural networks for different application scenarios.

[0010] In an optional embodiment, determining the compressed arrangement of the first sparse feature map output by the sparse convolutional neural network based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each of the channel feature maps includes:

[0011] Based on the compressed sparse coordinate index and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, each of the compressed sparse feature data streams is aligned to obtain an alignment feature map corresponding to each of the compressed sparse feature data streams; the minimum two-dimensional concurrent granularity is the data range indicated by the compressed sparse coordinate index;

[0012] Based on the compressed coordinate index model corresponding to each resolution information and each of the aligned feature maps, a first sparse feature map of the compressed arrangement output by the sparse convolutional neural network is determined.

[0013] This implementation method, in accordance with the processing form of feature data by the sparse convolutional neural network, needs to align the compressed sparse feature data stream into an aligned feature map, and use the compressed sparse coordinate index and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to perform alignment processing on each compressed sparse feature data stream respectively. It can reduce the influence of invalid feature data in the aligned feature map while retaining non-zero feature data. Processing the aligned feature map can not only improve the model operation efficiency, but also improve the accuracy of the output compressed arranged first sparse feature map.

[0014] In an optional embodiment, determining the compressed arrangement of the first sparse feature map output by the sparse convolutional neural network based on the compressed coordinate index model corresponding to each resolution information and each of the aligned feature maps includes:

[0015] For a first network layer in the sparse convolutional neural network, based on resolution information corresponding to the first network layer, screening out a first index model corresponding to the first network layer from the determined multiple compressed coordinate index models;

[0016] Determining a compressed and permuted second sparse feature map output by the first network layer based on the first index model and each of the aligned feature maps;

[0017] For any network layer other than the first network layer in the sparse convolutional neural network, based on resolution information corresponding to the network layer, screening out a second index model corresponding to the network layer from the determined multiple compressed coordinate index models;

[0018] Determine a compressed and arranged second sparse feature map output by the network layer based on a second sparse feature map output by a previous network layer of the network layer and a second index model corresponding to the network layer;

[0019] The compressed and arranged second sparse feature map output by the last network layer in the sparse convolutional neural network is used as the compressed and arranged first sparse feature map output by the sparse convolutional neural network.

[0020] In this embodiment, the processing of each aligned feature map and the subsequent processing of the second sparse feature map output by any network layer all utilize their corresponding index models, namely the first index model and the second index model. Here, the first index model can more accurately locate the position of the non-zero features in the aligned feature map in the corresponding channel feature map. Therefore, by utilizing the first network layer to process the non-zero features in each aligned feature map, an accurate compressed and arranged second sparse feature map can be output more quickly. Similarly, the second index model can more accurately locate the position of the non-zero features in the second sparse feature map output by the previous network layer in the corresponding channel feature map. Therefore, by utilizing the current network layer to process the non-zero features in the second sparse feature map output by the previous network layer, an accurate compressed and arranged second sparse feature map can be output more quickly.

[0021] In an optional embodiment, determining the compressed and arranged second sparse feature map output by the first network layer based on the first index model and each of the aligned feature maps includes:

[0022] For each aligned feature map, based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network and the first index model, restore the aligned feature map to obtain a corresponding sparsely arranged third sparse feature map;

[0023] Based on the sparsely arranged third sparse feature map, the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network and the parameter information corresponding to the first network layer, the compressed arranged second sparse feature map output by the first network layer is determined.

[0024] In this embodiment, the first index model can more accurately locate the position of the non-zero features in the alignment feature map in the corresponding channel feature map. Therefore, by using the first index model to restore the alignment feature map, the sparsely arranged third sparse feature map corresponding to the alignment feature map can be accurately restored. Afterwards, the first network layer configured with parameter information is used to process the more accurate third sparse feature map. For example, the non-zero features in the third sparse feature map are processed at the minimum two-dimensional concurrent granularity, and the accurate compressed arranged second sparse feature map is output more quickly.

[0025] In an optional embodiment, the method of aligning each of the compressed sparse feature data streams based on the compressed sparse coordinate index and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to obtain an aligned feature map corresponding to each of the compressed sparse feature data streams includes:

[0026] Reorganize the compressed sparse coordinate index using the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to obtain at least one coordinate index block;

[0027] For each of the compressed sparse feature data streams, alignment processing is performed on the compressed sparse feature data stream based on the coordinate position indicated by each coordinate index block to obtain an alignment feature map corresponding to the compressed sparse feature data stream.

[0028] This implementation method utilizes the compressed data form of the compressed sparse coordinate index and the compressed data form of the compressed sparse feature data stream to relatively quickly implement preprocessing of the compressed sparse feature data stream. At the same time, the compressed sparse coordinate index is reorganized using the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to obtain at least one coordinate index block. The coordinate index block can relatively accurately indicate the coordinate position of the non-zero feature data in the compressed sparse feature data stream, and obtain an alignment feature map corresponding to the compressed sparse feature data stream.

[0029] In an optional embodiment, the resolution information includes a first resolution ratio; the first resolution ratio is used to indicate that the ratio between the resolution of the input feature map of the network layer and the resolution of the output feature map is 1;

[0030] The step of determining a compressed coordinate index model for each resolution information based on the compressed sparse coordinate index and at least one resolution information corresponding to a network layer in the sparse convolutional neural network comprises:

[0031] For the first resolution ratio, based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, align the compressed sparse coordinate index to obtain a first aligned coordinate index corresponding to the compressed sparse coordinate index;

[0032] Based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the first alignment coordinate index is pooled to determine a compressed coordinate index model corresponding to the first resolution ratio.

[0033] In this embodiment, invalid coordinates are not present in the compressed sparse coordinate index. Alignment processing is performed on the valid coordinate indexes, i.e., the compressed sparse coordinate indexes, using the minimum two-dimensional concurrent granularity, to obtain a first aligned coordinate index with valid coordinates. Pooling processing is performed on the first aligned coordinate index using the minimum two-dimensional concurrent granularity to obtain a relatively accurate compressed coordinate index model that matches the first resolution ratio.

[0034] In an optional implementation, the resolution information further includes a second resolution ratio; the second resolution ratio is used to indicate that the ratio between the resolution of the input feature map and the resolution of the output feature map of the network layer is not 1;

[0035] The step of determining a compressed coordinate index model for each resolution information based on the compressed sparse coordinate index and at least one resolution information corresponding to a network layer in the sparse convolutional neural network comprises:

[0036] For each second resolution ratio, performing downsampling processing on the compressed sparse coordinate index based on the second resolution ratio;

[0037] Based on the compressed sparse coordinate index after downsampling processing and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, a compressed coordinate index model corresponding to the second resolution ratio is determined.

[0038] This embodiment downsamples the compressed sparse coordinate index to obtain a downsampled compressed sparse coordinate index that matches the second resolution ratio. Processing the downsampled compressed sparse coordinate index at the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network can obtain a relatively accurate compressed coordinate index model that matches the second resolution ratio.

[0039] In an optional embodiment, determining the compressed coordinate index model corresponding to the second resolution ratio based on the compressed sparse coordinate index after downsampling processing and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network includes:

[0040] Performing corrosion and expansion processing on the compressed sparse coordinate index after downsampling processing to obtain the corrosion and expansion coordinate index;

[0041] Based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the erosion and expansion coordinate index is aligned to obtain a second alignment coordinate index corresponding to the erosion and expansion coordinate index;

[0042] Based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the second alignment coordinate index is pooled to determine a compressed coordinate index model corresponding to the second resolution ratio.

[0043] This implementation performs erosion and dilation on the compressed sparse coordinate index after downsampling, expanding the valid coordinates in the compressed sparse coordinate index to obtain an eroded and dilated coordinate index. The compressed coordinate index model determined by the second aligned coordinate index corresponding to the eroded and dilated coordinate index can indicate the valid coordinates of more feature data in the sparse feature map during subsequent calculations, thereby improving the accuracy of the sparse feature map processing results.

[0044] In a second aspect, an embodiment of the present disclosure further provides a neural network acceleration processing device, comprising:

[0045] An acquisition module is used to obtain compressed sparse feature data streams corresponding to multiple channel feature maps, and compressed sparse coordinate indexes; the compressed sparse coordinate indexes are used to indicate the positions of non-zero features in the compressed sparse feature data streams in the channel feature maps;

[0046] A first determining module is configured to determine a compressed coordinate index model for each resolution information based on the compressed sparse coordinate index and at least one resolution information corresponding to a network layer in the sparse convolutional neural network;

[0047] The second determination module is used to determine the compressed and arranged first sparse feature map output by the sparse convolutional neural network based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each of the channel feature maps.

[0048] In an optional embodiment, the second determination module is configured to perform alignment processing on each of the compressed sparse feature data streams based on the compressed sparse coordinate index and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, to obtain an alignment feature map corresponding to each of the compressed sparse feature data streams;

[0049] Based on the compressed coordinate index model corresponding to each resolution information and each of the aligned feature maps, a first sparse feature map of the compressed arrangement output by the sparse convolutional neural network is determined.

[0050] In an optional embodiment, the second determination module is configured to, for a first network layer in the sparse convolutional neural network, filter out a first index model corresponding to the first network layer from the determined multiple compressed coordinate index models based on resolution information corresponding to the first network layer;

[0051] Determining a compressed and permuted second sparse feature map output by the first network layer based on the first index model and each of the aligned feature maps;

[0052] For any network layer other than the first network layer in the sparse convolutional neural network, based on resolution information corresponding to the network layer, screening out a second index model corresponding to the network layer from the determined multiple compressed coordinate index models;

[0053] Determine a compressed and arranged second sparse feature map output by the network layer based on a second sparse feature map output by a previous network layer of the network layer and a second index model corresponding to the network layer;

[0054] The compressed and arranged second sparse feature map output by the last network layer in the sparse convolutional neural network is used as the compressed and arranged first sparse feature map output by the sparse convolutional neural network.

[0055] In an optional embodiment, the second determination module is configured to perform a restoration process on each aligned feature map based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network and the first index model to obtain a corresponding sparsely arranged third sparse feature map;

[0056] Based on the sparsely arranged third sparse feature map, the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network and the parameter information corresponding to the first network layer, the compressed arranged second sparse feature map output by the first network layer is determined.

[0057] In an optional embodiment, the second determining module is configured to reorganize the compressed sparse coordinate index using the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to obtain at least one coordinate index block;

[0058] For each of the compressed sparse feature data streams, alignment processing is performed on the compressed sparse feature data stream based on the coordinate position indicated by each coordinate index block to obtain an alignment feature map corresponding to the compressed sparse feature data stream.

[0059] In an optional embodiment, the resolution information includes a first resolution ratio; the first resolution ratio is used to indicate that the ratio between the resolution of the input feature map of the network layer and the resolution of the output feature map is 1;

[0060] The first determining module is configured to perform alignment processing on the compressed sparse coordinate index based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network for the first resolution ratio, to obtain a first aligned coordinate index corresponding to the compressed sparse coordinate index;

[0061] Based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the first alignment coordinate index is pooled to determine a compressed coordinate index model corresponding to the first resolution ratio.

[0062] In an optional implementation, the resolution information further includes a second resolution ratio; the second resolution ratio is used to indicate that the ratio between the resolution of the input feature map and the resolution of the output feature map of the network layer is not 1;

[0063] The first determining module is configured to perform downsampling processing on the compressed sparse coordinate index based on the second resolution ratio for each second resolution ratio;

[0064] Based on the compressed sparse coordinate index after downsampling processing and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, a compressed coordinate index model corresponding to the second resolution ratio is determined.

[0065] In an optional implementation, the first determining module is configured to perform erosion and dilation processing on the compressed sparse coordinate index after downsampling processing to obtain an erosion and dilation coordinate index;

[0066] Based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the erosion and expansion coordinate index is aligned to obtain a second alignment coordinate index corresponding to the erosion and expansion coordinate index;

[0067] Based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the second alignment coordinate index is pooled to determine a compressed coordinate index model corresponding to the second resolution ratio.

[0068] In a third aspect, an embodiment of the present disclosure further provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the acceleration processing method of the first aspect or any possible neural network aspect in the first aspect are performed.

[0069] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes the steps of the accelerated processing method of the above-mentioned first aspect, or any possible neural network aspect in the first aspect.

[0070] For a description of the effects of the acceleration processing device, computer equipment and storage medium of the above-mentioned neural network method, please refer to the description of the acceleration processing x method of the above-mentioned neural network method, which will not be repeated here.

[0071] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0073] Figure 1 A flowchart of a method for accelerating processing of a neural network provided by an embodiment of the present disclosure is shown;

[0074] Figure 2A schematic diagram illustrating a method of restoring non-zero features in a compressed sparse feature data stream into a sparsely arranged feature map using a compressed coordinate indexing model provided by an embodiment of the present disclosure is shown;

[0075] Figure 3 A schematic diagram of a process for performing alignment processing on a compressed sparse feature data stream provided by an embodiment of the present disclosure to obtain an alignment feature map corresponding to the compressed sparse feature data stream is shown;

[0076] Figure 4 A schematic diagram illustrating a method of restoring a compressed aligned feature map to a sparsely arranged feature map using a compressed coordinate indexing model provided by an embodiment of the present disclosure is shown;

[0077] Figure 5 A schematic diagram of a process for restoring a compressed and arranged second sparse feature map output by a previous network layer using a second index model provided by an embodiment of the present disclosure is shown;

[0078] Figure 6 A schematic diagram of a process for accelerating inference processing for feature maps of a partial region provided by an embodiment of the present disclosure is shown;

[0079] Figure 7 A schematic diagram showing a specific process flow of accelerating processing of a neural network provided by an embodiment of the present disclosure is shown;

[0080] Figure 8 A schematic diagram of a process for converting a compressed sparse coordinate index provided by an embodiment of the present disclosure into a compressed coordinate index model is shown;

[0081] Figure 9 A schematic diagram of a process for aligning an eroded and dilated coordinate index to obtain a second aligned coordinate index corresponding to the eroded and dilated coordinate index, and performing pooling on the second aligned coordinate index to obtain a compressed coordinate index model corresponding to a second resolution ratio, provided by an embodiment of the present disclosure;

[0082] Figure 10 A schematic diagram of the process of preprocessing a compressed sparse coordinate index provided by an embodiment of the present disclosure is shown;

[0083] Figure 11 A schematic diagram of a neural network acceleration processing device provided by an embodiment of the present disclosure is shown;

[0084] Figure 12 A schematic structural diagram of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0086] In addition, the terms "first," "second," and the like in the description and claims of the embodiments of the present disclosure and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments described herein can be practiced in an order other than that shown or described herein.

[0087] In this document, "multiple or several" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0088] Research has found that using deep convolutional neural network accelerators to process large, real-time, sparse feature maps often suffers from low data access efficiency and prediction latency. Furthermore, deep convolutional neural network accelerators are often less versatile and difficult to deploy.

[0089] Based on the above research, the present disclosure provides a method for accelerating the processing of a neural network. By utilizing the compressed data form of a compressed sparse feature data stream and a compressed sparse coordinate index, the amount of data stored and calculated can be reduced, the real-time performance of the predictive neural network inference calculation can be improved, and the predictive neural network can have a better energy efficiency ratio. In addition, based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each of the channel feature maps, the compression calculation process of the compressed arrangement of the first sparse feature map output by the sparse convolutional neural network is determined, which can improve the data processing efficiency of the sparse convolutional neural network and reduce the prediction delay. In addition, the accelerated processing flow of the sparse feature map is decoupled into multiple stages. For example, one stage is: based on the compressed sparse coordinate index and at least one resolution information corresponding to the network layer in the sparse convolutional neural network, the compressed coordinate index model of each resolution information is determined respectively; another stage is: based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each channel feature map, the first sparse feature map of the compressed arrangement output by the sparse convolutional neural network is determined; this processing method can solve the flexibility and versatility problems of using sparse convolutional neural networks for accelerated processing of compressed sparse feature data streams, and can improve the ease of deployment of hardware resources in sparse convolutional neural networks for different application scenarios.

[0090] The defects in the above solutions are the results obtained by the inventors after practice and careful research. Therefore, the process of discovering the above problems and the solutions proposed by this disclosure for the above problems below should be the contributions made by the inventors to this disclosure during the disclosure process.

[0091] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0092] The following are explanations of special terms in the embodiments of the present disclosure:

[0093] 1. The Split function returns a one-dimensional array with a subscript starting from zero. The split function contains a specified number of substrings.

[0094] To facilitate understanding of this embodiment, we first provide a detailed introduction to a neural network acceleration processing method disclosed in this embodiment. The neural network acceleration processing method provided in this embodiment is generally executed by a computer device with certain computing capabilities. In some possible implementations, this neural network acceleration processing method can be implemented by a processor invoking computer-readable instructions stored in a memory.

[0095] The following describes the accelerated processing method of a neural network provided by an embodiment of the present disclosure, taking a computer device as an example of an execution subject.

[0096] See also Figure 1 FIG. 1 is a flowchart of a method for accelerating processing of a neural network according to an embodiment of the present disclosure, wherein the method includes steps S101 to S103, wherein:

[0097] S101: Obtain compressed sparse feature data streams and compressed sparse coordinate indexes corresponding to multiple channel feature maps respectively; the compressed sparse coordinate indexes are used to indicate the positions of non-zero features in the compressed sparse feature data stream in the channel feature map.

[0098] In this step, the plurality of channel feature maps may be obtained by preprocessing the original image acquired by the acquisition device. The compressed sparse feature data stream may be a data stream obtained by compressing and arranging the channel feature maps, and the data stream includes data of non-zero features in the channel feature maps.

[0099] The acquisition device can be a sensor or other device capable of acquiring images, such as a visual sensor, camera, lidar, etc.

[0100] Taking the autonomous driving scenario as an example, a visual sensor is used to capture raw images, and preprocessing is performed to obtain a compressed data stream of feature maps from multiple channels, also known as a compressed sparse feature data stream. The number of channels here can be determined based on the number of channels in the neural network during preprocessing, and this embodiment of the disclosure does not specifically limit this.

[0101] In this step, the compressed sparse coordinate index can be determined based on the locations of non-zero features in multiple channel feature maps. Here, since the multiple channel feature maps are obtained based on the same original image preprocessing, the locations of non-zero features in each channel feature map are the same, and the resulting sparse coordinate index is the same. The sparse coordinate index is compressed and arranged to obtain a compressed sparse coordinate index.

[0102] Exemplarily, a compressed sparse coordinate index may include multiple compressed arranged coordinate values ​​in the form of {x, y}, and each coordinate may be used to indicate the position of a non-zero feature in the compressed sparse feature data stream in the corresponding channel feature map.

[0103] S102: Based on the compressed sparse coordinate index and at least one resolution information corresponding to the network layer in the sparse convolutional neural network, determine a compressed coordinate index model for each resolution information respectively.

[0104] In this step, the sparse convolutional neural network can be a deep convolutional neural network that processes sparse feature maps. For example, a Split-based two-dimensional convolutional computing unit (called SPConv2D) can be used. The hardware of the sparse convolutional neural network can be a convolutional neural network inference accelerator.

[0105] The resolution information may include the resolution ratio of the network layer in the sparse convolutional neural network. The resolution ratio of the network layer is determined based on the network architecture of the network layer. For example, the network layer may be a convolutional layer, and the resolution ratio of the network layer may be the ratio between the resolution of the input feature map corresponding to the convolutional layer and the resolution of the output feature map. The input feature map corresponding to the convolutional layer may be the feature map input to the convolutional layer, and the output feature map corresponding to the convolutional layer may be the feature map output after convolution processing of the input feature map.

[0106] Since a sparse convolutional neural network may include multiple different convolutional layers, the resolution ratio corresponding to each convolutional layer may be the same or different. Therefore, different types of resolution information corresponding to the convolutional layers in the sparse convolutional neural network can be obtained.

[0107] Here, since the sparse convolutional neural network is pre-trained and the resolution ratios corresponding to each network layer are known, at least one resolution information corresponding to the network layer in the sparse convolutional neural network can be directly obtained during the preprocessing of the compressed sparse coordinate index.

[0108] The preprocessing process of the compressed sparse coordinate index can be specifically input into the sparse coordinate index accelerator. The sparse coordinate index accelerator processes the compressed sparse coordinate index according to at least one resolution information corresponding to the network layer in the sparse convolutional neural network, and obtains a compressed coordinate index model for each resolution information in a compressed arrangement.

[0109] S103: Based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each channel feature map, determine a first sparse feature map of the compressed arrangement output by the sparse convolutional neural network.

[0110] In specific implementation, the compressed coordinate index model can restore the non-zero features in the compressed sparse feature data stream into a sparsely arranged feature map. Afterwards, the sparse convolutional neural network is used to perform inference acceleration processing on the sparsely arranged feature map to obtain the first sparse feature map of the compressed arrangement.

[0111] Exemplarily, the compressed sparse feature data stream is restored to a sparsely arranged feature map using the valid coordinate positions indicated by the compressed coordinate index model. Since the valid coordinate positions indicated by the compressed coordinate index model may correspond to invalid features (invalid features may be zero features), the feature map restored to the sparse arrangement may include invalid features.

[0112] Taking the compressed coordinate index model of any resolution information as an example, you can refer to Figure 2 As shown in FIG, it is a schematic diagram of using a compressed coordinate index model to restore non-zero features in a compressed sparse feature data stream to a sparsely arranged feature map. In FIG (a), non-zero features in a compressed sparse feature data stream are shown. In FIG (b), valid coordinate positions are the positions of the black circles in the figure, and invalid coordinate positions are the positions of the white circles in the figure. Here, a valid coordinate position indicates 2×2 pixels (such as the four pixels indicated by 21 in FIG (c)). According to the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, such as the concurrent granularity of 2×2 or 4×4, the compressed sparse feature data stream is restored to a sparsely arranged feature map (as shown in FIG (c)). Here, not every 2×2 pixel indicated by the valid coordinate position is a non-zero feature. For example, 211 in FIG (c) is a non-zero feature, that is, 211 in the compressed sparse feature data stream is a non-zero feature, 212 is an invalid feature, 213 is an invalid feature, and 214 is an invalid feature; 221 is a non-zero feature, and 231 is a non-zero feature. Therefore, the feature maps restored to sparse arrangement include invalid features.

[0113] The above S101 to S103, using the compressed data form of compressed sparse feature data stream and compressed sparse coordinate index, can reduce the amount of data stored and calculated, improve the real-time performance of the predictive neural network reasoning calculation, and make the predictive neural network have better energy efficiency. In addition, based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each of the channel feature maps, the compression calculation process of the first sparse feature map of the compressed arrangement output by the sparse convolutional neural network is determined, which can improve the data processing efficiency of the sparse convolutional neural network and reduce the prediction delay. In addition, decoupling the accelerated processing process of the sparse feature map into multiple stages, such as the preprocessing stage of the compressed sparse coordinate index and the reasoning acceleration stage of the sparse feature map, can solve the flexibility and versatility problems of accelerated processing of compressed sparse feature data streams using sparse convolutional neural networks, and can improve the ease of deployment of hardware resources in sparse convolutional neural networks for different application scenarios.

[0114] In some embodiments, for S103, a compressed sparse feature data stream preprocessing stage can also be included. Specifically, based on the compressed sparse coordinate index and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, each compressed sparse feature data stream can be aligned to obtain an aligned feature map corresponding to each compressed sparse feature data stream.

[0115] Here, the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network is the data range indicated by the compressed sparse coordinate index, which can be determined based on the hardware architecture corresponding to the sparse convolutional neural network. For example, when the sparse convolutional neural network processes compressed sparse feature data streams, it aligns each compressed sparse feature data stream according to the 2×2 or 4×4 concurrent granularity and compressed sparse coordinate index to obtain an aligned feature map corresponding to each compressed sparse feature data stream.

[0116] Taking a compressed sparse feature data stream as an example, when it is determined that the compressed sparse coordinate index includes {1,1}, {2,2}, and {3,3}, the coordinate position of the first non-zero feature in the compressed sparse feature data stream in the channel feature map is determined to be {1,1}, the coordinate position of the second non-zero feature in the channel feature map is {2,2}, and the coordinate position of the third non-zero feature in the channel feature map is {3,3}. The compressed sparse feature data stream is aligned according to a 2×2 concurrent granularity to obtain an aligned feature map corresponding to the compressed sparse feature data stream. The aligned feature map includes a first aligned sub-feature map and a second aligned sub-feature map with a granularity of 2×2. The first aligned sub-feature map includes a non-zero feature at a coordinate position of {1,1}, an invalid feature at a coordinate position of {1,2}, an invalid feature at a coordinate position of {2,1}, and a non-zero feature at a coordinate position of {2,2}. The second alignment sub-feature map includes non-zero features at coordinates {3,3}, invalid features at coordinates {3,4}, invalid features at coordinates {4,3}, and invalid features at coordinates {4,4}. Afterwards, the alignment sub-feature maps of the above two granularities are compressed to obtain an alignment feature map. Figure 3 As shown in FIG, it is a flow chart of aligning the compressed sparse feature data stream to obtain the aligned feature map corresponding to the compressed sparse feature data stream. It should be noted that the aligned feature map of the compressed arrangement is not Figure 3 The form displayed.

[0117] Afterwards, the first sparse feature map of the compressed arrangement output by the sparse convolutional neural network can be determined based on the compressed coordinate index model corresponding to each resolution information and each aligned feature map.

[0118] In specific implementation, the effective coordinate positions indicated by the compressed coordinate index model can be used to restore the aligned feature maps of each compressed arrangement to the feature maps of the sparse arrangement. Afterwards, the sparse convolutional neural network is used to perform inference acceleration processing on the feature maps of the sparse arrangement to obtain the first sparse feature map of the compressed arrangement.

[0119] Taking any compressed coordinate index model as an example, you can refer to Figure 4 As shown, it is a schematic diagram of using a compressed coordinate index model to restore a compressed aligned feature map to a sparsely arranged feature map. Wherein, Figure (a) represents the first aligned sub-feature map 41 and the second aligned sub-feature map 42 containing non-zero features in the aligned feature map, and the valid coordinate position in Figure (b) is the position of the black circle in the figure, and the invalid coordinate position is the position of the white circle in the figure. The valid coordinate position indicated by the black circle 43 is used to restore the position of the first aligned sub-feature map 41 in the sparsely arranged feature map, and the valid coordinate position indicated by the black circle 44 is used to restore the position of the second aligned sub-feature map 42 in the sparsely arranged feature map, and the restored sparsely arranged feature map shown in Figure (c) is obtained.

[0120] Afterwards, the sparsely arranged feature map can be input into the sparse convolutional neural network, and processed in sequence using each network layer in the sparse convolutional neural network. For example, it includes N convolutional layers, where N is a positive integer. The first convolutional layer is used to process the input sparsely arranged feature map to obtain the output result. After that, the second convolutional layer is used to process the output result of the previous convolutional layer to obtain the output result, ..., until the last convolutional layer in the sparse convolutional neural network is processed to output the compressed arranged first sparse feature map.

[0121] According to the above-mentioned processing form of feature data by the sparse convolutional neural network, it is necessary to align the compressed sparse feature data stream into an aligned feature map, and use the compressed sparse coordinate index and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to perform alignment processing on each compressed sparse feature data stream respectively. It is possible to reduce the influence of invalid feature data in the aligned feature map while retaining non-zero feature data. Processing the aligned feature map can not only improve the model operation efficiency, but also improve the accuracy of the output compressed arranged first sparse feature map.

[0122] In some embodiments, the alignment feature map corresponding to each compressed sparse feature data stream is determined, and processing can also be performed based on the following method. Specifically, the compressed sparse coordinate index is reorganized using the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to obtain at least one coordinate index block; for each compressed sparse feature data stream, the compressed sparse feature data stream is aligned based on the coordinate position indicated by each coordinate index block to obtain the alignment feature map corresponding to the compressed sparse feature data stream.

[0123] Continuing with the above example, when it is determined that the compressed sparse coordinate index includes {1,1}, {2,2}, and {3,3}, the compressed sparse coordinate index is reorganized using the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, such as a concurrent granularity of 2×2, to obtain coordinate positions indicated by coordinate index block A including {1,1}, {1,2}, {2,1}, and {2,2}; and coordinate positions indicated by coordinate index block B including {3,3}, {3,4}, {4,3}, and {4,4}. For each compressed sparse feature data stream, based on the coordinate positions indicated by each coordinate index block, the compressed sparse feature data stream is aligned to obtain an aligned feature map corresponding to the compressed sparse feature data stream. The aligned feature map includes a first aligned sub-feature map and a second aligned sub-feature map with a granularity of 2×2.

[0124] In some embodiments, based on the compressed coordinate index model corresponding to each resolution information and each aligned feature map, determining the compressed arrangement of the first sparse feature map output by the sparse convolutional neural network can also be processed in the following manner, including steps 1 to 5:

[0125] Step 1: For the first network layer in the sparse convolutional neural network, based on the resolution information corresponding to the first network layer, a first index model corresponding to the first network layer is selected from the determined multiple compressed coordinate index models.

[0126] Since the resolution information corresponding to the network layer is determined based on the network architecture of the network layer, when it is known that the current network layer is the first network layer in the sparse convolutional neural network, the first index model that matches the resolution information can be directly screened out from multiple compressed coordinate index models according to the resolution information corresponding to the first network layer.

[0127] Step 2: Based on the first index model and each aligned feature map, determine the compressed and arranged second sparse feature map output by the first network layer.

[0128] In some embodiments, for each aligned feature map, the aligned feature map can be restored based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network and the first index model to obtain the corresponding sparsely arranged third sparse feature map; then, based on the sparsely arranged third sparse feature map, the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network and the parameter information corresponding to the first network layer, the compressed arranged second sparse feature map output by the first network layer is determined.

[0129] Specifically, the effective coordinate position indicated by the first index model can be used to restore the compressed aligned feature map to a sparsely arranged third sparse feature map according to the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network. Thereafter, multiple partial region feature maps are obtained from the sparsely arranged third sparse feature map using the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network. Then, the first network layer is configured using the parameter information corresponding to the first network layer, and the multiple partial region feature maps are processed using the configured first network layer to obtain the compressed second sparse feature map output by the first network layer.

[0130] Alternatively, a feature padding operation may be performed on the sparsely arranged third sparse feature map to obtain a padded sparsely arranged third sparse feature map. Then, the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network (that is, the data range indicated by the compressed sparse coordinate index) may be used, for example, according to a granularity of 2×2, to obtain multiple partial area feature maps from the padded sparsely arranged third sparse feature map. Then, a padding operation may be performed on the multiple partial area feature maps, and the configured first network layer may be used to process the padded multiple partial area feature maps to obtain the compressed arranged second sparse feature map output by the first network layer.

[0131] Here, the network layer may be a convolutional layer, and the parameter information may be parameters such as convolution weights for configuring the convolutional layer.

[0132] In other embodiments, the first network layer can be configured based on the parameter information corresponding to the first network layer, and the sparsely arranged third sparse feature map can be processed using the configured first network layer to obtain the compressed arranged second sparse feature map output by the first network layer.

[0133] Step 3: For any network layer except the first network layer in the sparse convolutional neural network, based on the resolution information corresponding to the network layer, select the second index model corresponding to the network layer from the determined multiple compressed coordinate index models.

[0134] Specifically, when the sparse convolutional neural network includes multiple network layers, for any network layer other than the first network layer in the sparse convolutional neural network, for example, the second network layer, based on the resolution information corresponding to the second network layer, a second index model that matches the resolution information of the second network layer is screened out from the determined multiple compressed coordinate index models.

[0135] Step 4: Based on the second sparse feature map output by the previous network layer of the network layer and the second index model corresponding to the network layer, determine the compressed and arranged second sparse feature map output by the network layer.

[0136] Taking the network layer in this step as the second network layer as an example, for the previous network layer of the second network layer, that is, the second sparse feature map output by the first network layer, the second sparse feature map output by the first network layer is restored using the second index model corresponding to the second network layer to obtain a sparsely arranged third sparse feature map corresponding to the second sparse feature map output by the first network layer.

[0137] The second sparse feature map output by the first network layer is restored using the second index model corresponding to the second network layer. For example, when the resolution information corresponding to the second network layer does not include the first resolution ratio (1:1), see Figure 5 As shown, it is a flow chart of restoring the second sparse feature map output by the previous network layer and compressed and arranged using the second index model. Figure (a) shows the second sparse feature map output by the previous network layer and compressed and arranged; Figure (b) shows the second index model; Figure (c) shows the restored sparsely arranged third sparse feature map. Specifically, the second sparse feature map can be restored according to the valid coordinate positions (i.e., black circles) indicated by the second index model, wherein the second sparse feature map includes sparse feature map 511 and sparse feature map 512, and it is known that sparse feature map 521 and sparse feature map 522 correspond to valid coordinate positions 521 and valid coordinate positions 522 in the second index model, respectively. The second sparse feature map is restored using the valid coordinate positions indicated by the second index model. In addition to restoring sparse feature map 511 and sparse feature map 512 according to valid coordinate positions 521 and 522, it is also necessary to restore sparse feature map 511 according to valid coordinate positions 523, 514, and 525. 15. The valid coordinate positions 516, 517 and 518 restore the sparse feature maps 513, sparse feature maps 514, sparse feature maps 515, sparse feature maps 516, sparse feature maps 517 and sparse feature maps 518, wherein the process of restoring the sparse feature maps 513, sparse feature maps 514, sparse feature maps 515, sparse feature maps 516, sparse feature maps 517 and sparse feature maps 518 is the process of filling invalid features, that is, there are only invalid features in the sparse feature maps 513, sparse feature maps 514, sparse feature maps 515, sparse feature maps 516, sparse feature maps 517 and sparse feature maps 518.

[0138] As mentioned above, the second index model can be obtained based on the corrosion and expansion processing of the compressed sparse coordinate index. Therefore, the valid coordinate position indicated by the second index model includes not only the compressed arranged second sparse feature map 52, but also the feature map with only invalid features, namely, sparse feature map 524, sparse feature map 525, and sparse feature map 525.

[0139] Afterwards, the parameter information corresponding to the second network layer can be used to configure the second network layer, and the configured second network layer can be used to process the sparsely arranged third sparse feature map corresponding to the second sparse feature map output by the first network layer to obtain the compressed arranged second sparse feature map output by the second network layer.

[0140] Alternatively, you can first use the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to obtain multiple partial area feature maps from the sparsely arranged third sparse feature map corresponding to the second sparse feature map output by the first network layer, and then input the multiple partial area feature maps into the configured second network layer for processing to obtain the compressed arranged second sparse feature map output by the second network layer.

[0141] The network layer in steps 2 and 4 above accelerates the reasoning process of the third sparse feature map or multiple partial region feature maps. Figure 6 As shown in the figure, it is a flow chart for inference acceleration processing of partial area feature maps. For partial area feature maps, feature padding processing is performed, and the processing results are input into the process element PE (Process Element) architecture in the sparse convolutional neural network. At the same time, the convolution weights are also input into the PE architecture. The PE architecture is used to process multiple partial area feature maps after feature padding, and the processing results are input into the neurons (Neuron) in the convolution layer for operations such as multi-input channel accumulation, bias, precision adjustment Crop, nonlinear activation function Relu, pooling (downsampling), convolution kernel sliding stride, alignment zero padding (according to the PE processing granularity, such as 2×2 or 4×4). Afterwards, the processing results of the above operations, that is, the sparse feature map, are compressed to obtain a compressed and arranged second sparse feature map.

[0142] Step 5: The compressed and arranged second sparse feature map output by the last network layer in the sparse convolutional neural network is used as the compressed and arranged first sparse feature map output by the sparse convolutional neural network.

[0143] For each network layer in the sparse convolutional neural network, the above steps 3 and 4 can be repeated until the second sparse feature map of the compressed arrangement output by the last network layer in the sparse convolutional neural network is determined, and it is used as the first sparse feature map of the compressed arrangement output by the sparse convolutional neural network.

[0144] See also Figure 7As shown, it is a specific flow chart of the accelerated processing of a neural network. It includes a compressed sparse feature data stream preprocessing stage, a compressed sparse coordinate index preprocessing stage, and an inference acceleration stage of a sparse feature map (i.e., the inference acceleration processing of the third sparse feature map or multiple partial area feature maps by the network layer in the above-mentioned sparse convolutional neural network). For the compressed sparse feature data stream preprocessing stage, each compressed sparse feature data stream can be input into an acceleration device for sparse feature data stream preprocessing, and the sparse feature data stream is aligned using the acceleration device for sparse feature data stream preprocessing to obtain an aligned feature map corresponding to each compressed sparse feature data stream. For the compressed sparse coordinate index preprocessing stage, the compressed sparse coordinate index can be input into a sparse coordinate index accelerator, and the sparse coordinate index accelerator is used to process the compressed sparse coordinate index based on at least one resolution information corresponding to the network layer in the sparse convolutional neural network, and obtain compressed coordinate index models for each resolution information, such as compressed coordinate index models for resolution information such as 1:1, 2:1, 4:1 and 8:1. For the inference acceleration stage of the sparse feature map, the aligned feature maps corresponding to each compressed sparse feature data stream can be input into the sparse convolutional neural network accelerator. Using the sparse convolutional neural network accelerator, starting from the first network layer, the first index model is screened out from multiple compressed coordinate index models, and the aligned feature map is restored using the first index model. The restored aligned feature map, that is, the sparsely arranged third sparse feature map, is processed using the first network layer to output the compressed arranged second sparse feature map. For the next network layer, the above processing is repeated for the output result of the previous network layer until the compressed arranged second sparse feature map output by the last network layer in the sparse convolutional neural network is determined, and it is used as the compressed arranged first sparse feature map output by the sparse convolutional neural network.

[0145] The above-mentioned processing of each aligned feature map, as well as the subsequent processing of the second sparse feature map output by any network layer, all utilize their respective corresponding index models, namely the first index model and the second index model. Here, the first index model can more accurately locate the position of the non-zero features in the aligned feature map in the corresponding channel feature map. Therefore, by using the first network layer to process the non-zero features in each aligned feature map, it is possible to more quickly output an accurate compressed and arranged second sparse feature map. Similarly, the second index model can more accurately locate the position of the non-zero features in the corresponding channel feature map output by the previous network layer. Therefore, by using the current network layer to process the non-zero features in the second sparse feature map output by the previous network layer, it is possible to more quickly output an accurate compressed and arranged second sparse feature map. Each network layer in the entire sparse convolutional neural network is processed using the above-mentioned inference process, and the sparse convolutional neural network can more quickly output a compressed and arranged first sparse feature map.

[0146] For S102, the resolution information may include a first resolution ratio, where the first resolution ratio is used to indicate that the ratio between the resolution of the input feature map of the network layer and the resolution of the output feature map is 1. The input feature map is the feature map input to the network layer, and the output feature map is the feature map output after the network layer processes the input feature map.

[0147] For the first resolution ratio, a compressed coordinate index model corresponding to the first resolution ratio is determined. In some embodiments, the compressed sparse coordinate index can be aligned based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to obtain a first aligned coordinate index corresponding to the compressed sparse coordinate index; and the first aligned coordinate index can be pooled based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to determine the compressed coordinate index model corresponding to the first resolution ratio.

[0148] Generate a compressed coordinate index model corresponding to a first resolution ratio of 1:1. For example, align the compressed sparse coordinate index according to the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, such as 2×2 or 4×4 granularity, to obtain a first aligned coordinate index corresponding to the compressed sparse coordinate index. In the first aligned coordinate index, if there is no sparse coordinate index at a certain granularity, fill the point without a sparse coordinate index at that granularity with 0, and fill the point with a sparse coordinate index with 1, where "0" indicates that the current coordinate position is an invalid feature and "1" indicates that the current coordinate position is a non-zero feature. Afterwards, the first aligned coordinate index is pooled according to the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, such as max-pooling. The maximum value indicated by the minimum two-dimensional concurrent granularity can be used as the valid coordinate position (the position represented by the black circle) in the compressed coordinate index model at that granularity, thereby determining the compressed coordinate index model corresponding to the first resolution ratio. See. Figure 8 As shown in FIG, it is a schematic diagram of the process of converting a compressed sparse coordinate index into a compressed coordinate index model. The compressed coordinate index model obtained by this process is based on the compressed sparse coordinate index aligned at a 2×2 granularity and pooled.

[0149] Since there are no invalid coordinates in the compressed sparse coordinate index, the valid coordinate index, i.e., the compressed sparse coordinate index, is aligned using the minimum two-dimensional concurrent granularity to obtain a first aligned coordinate index with valid coordinates. Pooling the first aligned coordinate index using the minimum two-dimensional concurrent granularity yields a relatively accurate compressed coordinate index model that matches the first resolution ratio.

[0150] For S102, the resolution information also includes a second resolution ratio; the second resolution ratio is used to indicate that the ratio between the resolution of the input feature map of the network layer and the resolution of the output feature map is not 1.

[0151] For each second resolution ratio, the compressed sparse coordinate index is downsampled based on the second resolution ratio; based on the compressed sparse coordinate index after downsampling and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the compressed coordinate index model corresponding to the second resolution ratio is determined.

[0152] Here, the second resolution ratio may include 2:1, 4:1, 8:1, etc.

[0153] Generate a compressed coordinate index model corresponding to the second resolution ratio of 2:1. Specifically, the compressed sparse coordinate index can be downsampled according to 2:1. The specific downsampling method can be downsampling according to the max-pooling method. The specific processing process refers to the max-pooling processing process above. The repeated part will not be repeated here. For the structure of the compressed sparse coordinate index after downsampling, please refer to Figure 8 The structural form of the compressed coordinate index model shown. Afterwards, the compressed sparse coordinate index after downsampling processing can be aligned according to the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to obtain a third aligned coordinate index corresponding to the compressed sparse coordinate index after downsampling processing. Afterwards, the third aligned coordinate index is pooled according to the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, such as max-pooling processing, and the maximum value indicated by the minimum two-dimensional concurrent granularity is used as the valid coordinate position in the compressed coordinate index model at this granularity, thereby determining the compressed coordinate index model corresponding to the second resolution ratio.

[0154] The above-described downsampling of the compressed sparse coordinate index can obtain a downsampled compressed sparse coordinate index that matches the second resolution ratio. Processing the downsampled compressed sparse coordinate index at the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network can obtain a relatively accurate compressed coordinate index model that matches the second resolution ratio.

[0155] In some embodiments, the compressed sparse coordinate index after downsampling processing can also be corroded and expanded to obtain a corroded and expanded coordinate index; based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the corroded and expanded coordinate index is aligned to obtain a second aligned coordinate index corresponding to the corroded and expanded coordinate index; based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the second aligned coordinate index is pooled to determine the compressed coordinate index model corresponding to the second resolution ratio.

[0156] The expansion processing can be to expand the valid coordinate position in the compressed sparse coordinate index after downsampling processing according to the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network. The corrosion processing can be to reduce the compressed sparse coordinate index after expansion processing according to the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network. This cycle is repeated until the number of iterations meets the algorithm accuracy requirements of the sparse convolutional neural network, for example, 3 times of expansion and 1 time of corrosion.

[0157] See also Figure 9 As shown, it is a flow chart of aligning the eroded and expanded coordinate index to obtain a second aligned coordinate index corresponding to the eroded and expanded coordinate index, and pooling the second aligned coordinate index to obtain a compressed coordinate index model corresponding to the second resolution ratio.

[0158] Repeat the above embodiment process, and process in the order of 2:1, 4:1, and 8:1 until the resolution ratio traversal corresponding to the resolution information of each network layer in the sparse convolutional neural network is completed.

[0159] Afterwards, the alignment and pooling processing of the eroded and expanded coordinate indexes can refer to the alignment and pooling processing of the compressed sparse coordinate indexes after downsampling processing. That is, for the eroded and expanded coordinate indexes, the alignment processing can be performed according to the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to obtain the second aligned coordinate index corresponding to the eroded and expanded coordinate index. Afterwards, the second aligned coordinate index is pooled according to the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, such as max-pooling processing, and the maximum value indicated by the minimum two-dimensional concurrent granularity is used as the valid coordinate position in the compressed coordinate index model at this granularity, thereby determining the compressed coordinate index model corresponding to the second resolution ratio.

[0160] Performing erosion and dilation on the compressed sparse coordinate index after downsampling expands the valid coordinates in the compressed sparse coordinate index to obtain an eroded and dilated coordinate index. The compressed coordinate index model determined by the second aligned coordinate index corresponding to the eroded and dilated coordinate index can indicate the valid coordinates of more feature data in the sparse feature map during subsequent calculations, thereby improving the accuracy of the sparse feature map processing results.

[0161] Performing erosion and dilation on the compressed sparse coordinate index after downsampling expands the valid coordinates in the compressed sparse coordinate index to obtain an eroded and dilated coordinate index. The compressed coordinate index model determined by the second aligned coordinate index corresponding to the eroded and dilated coordinate index can indicate the valid coordinates of more feature data in the sparse feature map during subsequent calculations, thereby improving the accuracy of the sparse feature map processing results.

[0162] See also Figure 10 Figure 1 shows the preprocessing flow for compressed sparse coordinate indexing. For the first resolution ratio, alignment and pooling are performed directly. For the second resolution ratio, downsampling is performed first, followed by erosion and dilation, and finally alignment and pooling. Finally, a compressed coordinate index model is obtained for each resolution.

[0163] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0164] Based on the same inventive concept, the embodiments of the present disclosure also provide an accelerated processing device for a neural network corresponding to the accelerated processing method for a neural network. Since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to the accelerated processing method for a neural network in the embodiments of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0165] Reference Figure 11 FIG. 1 is a schematic diagram of a neural network acceleration processing device provided by an embodiment of the present disclosure, wherein the device includes: an acquisition module 111, a first determination module 112, and a second determination module 113; wherein,

[0166] An acquisition module 111 is configured to acquire compressed sparse feature data streams corresponding to a plurality of channel feature maps, and compressed sparse coordinate indexes; the compressed sparse coordinate indexes are used to indicate the positions of non-zero features in the compressed sparse feature data streams in the channel feature maps;

[0167] A first determining module 112 is configured to determine a compressed coordinate index model for each resolution information based on the compressed sparse coordinate index and at least one resolution information corresponding to a network layer in the sparse convolutional neural network;

[0168] The second determination module 113 is used to determine the compressed and arranged first sparse feature map output by the sparse convolutional neural network based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each of the channel feature maps.

[0169] In an optional embodiment, the second determining module 113 is configured to perform alignment processing on each of the compressed sparse feature data streams based on the compressed sparse coordinate index and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, to obtain an alignment feature map corresponding to each of the compressed sparse feature data streams;

[0170] Based on the compressed coordinate index model corresponding to each resolution information and each of the aligned feature maps, a first sparse feature map of the compressed arrangement output by the sparse convolutional neural network is determined.

[0171] In an optional embodiment, the second determining module 113 is configured to, for a first network layer in the sparse convolutional neural network, filter out a first index model corresponding to the first network layer from the determined multiple compressed coordinate index models based on resolution information corresponding to the first network layer;

[0172] Determining a compressed and permuted second sparse feature map output by the first network layer based on the first index model and each of the aligned feature maps;

[0173] For any network layer other than the first network layer in the sparse convolutional neural network, based on resolution information corresponding to the network layer, screening out a second index model corresponding to the network layer from the determined multiple compressed coordinate index models;

[0174] Determine a compressed and arranged second sparse feature map output by the network layer based on a second sparse feature map output by a previous network layer of the network layer and a second index model corresponding to the network layer;

[0175] The compressed and arranged second sparse feature map output by the last network layer in the sparse convolutional neural network is used as the compressed and arranged first sparse feature map output by the sparse convolutional neural network.

[0176] In an optional embodiment, the second determining module 113 is configured to perform a restoration process on each aligned feature map based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network and the first index model to obtain a corresponding sparsely arranged third sparse feature map;

[0177] Based on the sparsely arranged third sparse feature map, the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network and the parameter information corresponding to the first network layer, the compressed arranged second sparse feature map output by the first network layer is determined.

[0178] In an optional implementation, the second determining module 113 is configured to reorganize the compressed sparse coordinate index using the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to obtain at least one coordinate index block;

[0179] For each of the compressed sparse feature data streams, alignment processing is performed on the compressed sparse feature data stream based on the coordinate position indicated by each coordinate index block to obtain an alignment feature map corresponding to the compressed sparse feature data stream.

[0180] In an optional embodiment, the resolution information includes a first resolution ratio; the first resolution ratio is used to indicate that the ratio between the resolution of the input feature map of the network layer and the resolution of the output feature map is 1;

[0181] The first determining module 112 is configured to perform alignment processing on the compressed sparse coordinate index based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network for the first resolution ratio, to obtain a first aligned coordinate index corresponding to the compressed sparse coordinate index;

[0182] Based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the first alignment coordinate index is pooled to determine a compressed coordinate index model corresponding to the first resolution ratio.

[0183] In an optional implementation, the resolution information further includes a second resolution ratio; the second resolution ratio is used to indicate that the ratio between the resolution of the input feature map and the resolution of the output feature map of the network layer is not 1;

[0184] The first determining module 112 is configured to perform downsampling processing on the compressed sparse coordinate index based on each second resolution ratio;

[0185] Based on the compressed sparse coordinate index after downsampling processing and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, a compressed coordinate index model corresponding to the second resolution ratio is determined.

[0186] In an optional embodiment, the first determining module 112 is configured to perform erosion and dilation processing on the compressed sparse coordinate index after downsampling processing to obtain an erosion and dilation coordinate index;

[0187] Based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the erosion and expansion coordinate index is aligned to obtain a second alignment coordinate index corresponding to the erosion and expansion coordinate index;

[0188] Based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the second alignment coordinate index is pooled to determine a compressed coordinate index model corresponding to the second resolution ratio.

[0189] For the description of the processing flow of each module in the neural network acceleration processing device and the interaction flow between each module, please refer to the relevant description in the above-mentioned neural network acceleration processing method embodiment, which will not be described in detail here.

[0190] Based on the same technical concept, the embodiment of the present application also provides a computer device. Figure 12 FIG. 1 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, including:

[0191] Processor 121, memory 122 and bus 123. The memory 122 stores machine-readable instructions executable by the processor 121, and the processor 121 is used to execute the machine-readable instructions stored in the memory 122. When the machine-readable instructions are executed by the processor 121, the processor 121 performs the following steps: S101: obtaining compressed sparse feature data streams and compressed sparse coordinate indexes corresponding to multiple channel feature maps respectively; the sparse coordinate index is used to indicate the compressed position of the non-zero features in the compressed sparse feature data stream in the channel feature map; S102: based on the compressed sparse coordinate index and at least one resolution information corresponding to the network layer in the sparse convolutional neural network, respectively determine the compressed coordinate index model of each resolution information; S103: based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each channel feature map, determine the first sparse feature map of the compressed arrangement output by the sparse convolutional neural network.

[0192] The above-mentioned memory 122 includes internal memory 1221 and external memory 1222; the memory 1221 here is also called internal memory, which is used to temporarily store the calculation data in the processor 121, as well as the data exchanged with the external memory 1222 such as the hard disk. The processor 121 exchanges data with the external memory 1222 through the memory 1221. When the computer device is running, the processor 121 and the memory 122 communicate through the bus 123, so that the processor 121 executes the execution instructions mentioned in the above method embodiment.

[0193] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the neural network acceleration processing method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0194] The present disclosure also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the above-mentioned method for accelerating the processing of a neural network. The computer program product can be any product that can implement the above-mentioned method for accelerating the processing of a neural network. Part or all of the solutions that contribute to the prior art in the computer program product can be embodied in the form of a software product (e.g., a software development kit (SDK)). The software product can be stored in a storage medium, and the computer instructions contained therein enable a related device or processor to execute part or all of the steps of the above-mentioned method for accelerating the processing of a neural network.

[0195] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0196] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0197] In addition, each functional module in each embodiment of the present disclosure may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0198] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0199] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A method for accelerating processing of a neural network, characterized in that: include: Obtaining compressed sparse feature data streams and compressed sparse coordinate indexes corresponding to a plurality of channel feature maps, respectively, and storing the compressed sparse feature data streams and the compressed sparse coordinate indexes; the compressed sparse coordinate indexes are used to indicate the positions of non-zero features in the compressed sparse feature data streams in the channel feature maps; Determine a compressed coordinate index model for each resolution information based on the compressed sparse coordinate index and at least one resolution information corresponding to a network layer in the sparse convolutional neural network; Based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each of the channel feature maps, a first sparse feature map of the compressed arrangement output by the sparse convolutional neural network is determined.

2. The method according to claim 1, characterized in that The method of determining a compressed and arranged first sparse feature map output by the sparse convolutional neural network based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each of the channel feature maps comprises: Based on the compressed sparse coordinate index and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, each of the compressed sparse feature data streams is aligned to obtain an alignment feature map corresponding to each of the compressed sparse feature data streams; the minimum two-dimensional concurrent granularity is the data range indicated by the compressed sparse coordinate index; Based on the compressed coordinate index model corresponding to each resolution information and each of the aligned feature maps, a first sparse feature map of the compressed arrangement output by the sparse convolutional neural network is determined.

3. The method according to claim 2, characterized in that The step of determining a first sparse feature map of the compressed arrangement output by the sparse convolutional neural network based on the compressed coordinate index model corresponding to each resolution information and each of the aligned feature maps includes: For a first network layer in the sparse convolutional neural network, based on resolution information corresponding to the first network layer, screening out a first index model corresponding to the first network layer from the determined multiple compressed coordinate index models; Determining a compressed and permuted second sparse feature map output by the first network layer based on the first index model and each of the aligned feature maps; For any network layer other than the first network layer in the sparse convolutional neural network, based on resolution information corresponding to the network layer, screening out a second index model corresponding to the network layer from the determined multiple compressed coordinate index models; Determine a compressed and arranged second sparse feature map output by the network layer based on a second sparse feature map output by a previous network layer of the network layer and a second index model corresponding to the network layer; The compressed and arranged second sparse feature map output by the last network layer in the sparse convolutional neural network is used as the compressed and arranged first sparse feature map output by the sparse convolutional neural network.

4. The method according to claim 3, characterized in that The step of determining the compressed and arranged second sparse feature map output by the first network layer based on the first index model and each of the aligned feature maps includes: For each aligned feature map, based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network and the first index model, restore the aligned feature map to obtain a corresponding sparsely arranged third sparse feature map; Based on the sparsely arranged third sparse feature map, the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network and the parameter information corresponding to the first network layer, the compressed arranged second sparse feature map output by the first network layer is determined.

5. The method according to claim 2, characterized in that The method further comprises: performing alignment processing on each of the compressed sparse feature data streams based on the compressed sparse coordinate index and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to obtain an alignment feature map corresponding to each of the compressed sparse feature data streams, including: Reorganize the compressed sparse coordinate index using the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network to obtain at least one coordinate index block; For each of the compressed sparse feature data streams, alignment processing is performed on the compressed sparse feature data stream based on the coordinate position indicated by each coordinate index block to obtain an alignment feature map corresponding to the compressed sparse feature data stream.

6. The method according to claim 1, characterized in that The resolution information includes a first resolution ratio; the first resolution ratio is used to indicate that the ratio between the resolution of the input feature map of the network layer and the resolution of the output feature map is 1; The step of determining a compressed coordinate index model for each resolution information based on the compressed sparse coordinate index and at least one resolution information corresponding to a network layer in the sparse convolutional neural network comprises: For the first resolution ratio, based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, align the compressed sparse coordinate index to obtain a first aligned coordinate index corresponding to the compressed sparse coordinate index; Based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the first alignment coordinate index is pooled to determine a compressed coordinate index model corresponding to the first resolution ratio.

7. The method according to claim 1, characterized in that The resolution information includes a second resolution ratio; the second resolution ratio is used to indicate that the ratio between the resolution of the input feature map and the resolution of the output feature map of the network layer is not 1; The step of determining a compressed coordinate index model for each resolution information based on the compressed sparse coordinate index and at least one resolution information corresponding to a network layer in the sparse convolutional neural network comprises: For each second resolution ratio, performing downsampling processing on the compressed sparse coordinate index based on the second resolution ratio; Based on the compressed sparse coordinate index after downsampling processing and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, a compressed coordinate index model corresponding to the second resolution ratio is determined.

8. The method according to claim 7, characterized in that The step of determining a compressed coordinate index model corresponding to the second resolution ratio based on the compressed sparse coordinate index after downsampling processing and the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network includes: Performing corrosion and expansion processing on the compressed sparse coordinate index after downsampling processing to obtain the corrosion and expansion coordinate index; Based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the erosion and expansion coordinate index is aligned to obtain a second alignment coordinate index corresponding to the erosion and expansion coordinate index; Based on the minimum two-dimensional concurrent granularity corresponding to the sparse convolutional neural network, the second alignment coordinate index is pooled to determine a compressed coordinate index model corresponding to the second resolution ratio.

9. A neural network acceleration processing device, characterized in that: include: Obtaining compressed sparse feature data streams and compressed sparse coordinate indexes corresponding to a plurality of channel feature maps, respectively, and storing the compressed sparse feature data streams and the compressed sparse coordinate indexes; the compressed sparse coordinate indexes are used to indicate the positions of non-zero features in the compressed sparse feature data streams in the channel feature maps; Determine a compressed coordinate index model for each resolution information based on the compressed sparse coordinate index and at least one resolution information corresponding to a network layer in the sparse convolutional neural network; Based on the determined multiple compressed coordinate index models and the compressed sparse feature data stream corresponding to each of the channel feature maps, a first sparse feature map of the compressed arrangement output by the sparse convolutional neural network is determined.

10. A computer device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the method for accelerating processing of a neural network as described in any one of claims 1 to 8 are performed.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for accelerating processing of a neural network as claimed in any one of claims 1 to 8.

Images