Sparse convolutional neural network calculation acceleration method, system and device, storage medium and program product
By extracting the coordinate information and validity indicators of sparse feature maps, dynamically scheduling the weights of convolutional kernels, generating a sparse data stream with a fixed clock rate, and performing computation only on valid data, the problem of resource waste and power consumption loss in traditional sparse data processing is solved, and efficient computation of sparse convolutional neural networks is achieved.
Patent Information
- Application Number
- CN202512018264.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-10
AI Technical Summary
Traditional pulsating array architectures suffer from wasted computing resources and power consumption when processing sparse data, especially when processing 3D point cloud data with a sparsity of over 90%, where the computing units still perform meaningless multiply-accumulate operations.
The coordinate information and validity indication of the sparse feature map are extracted by the prefetching unit to generate a sparse labeled stream. The dynamic weight scheduler identifies the target processing unit and distributes the convolution kernel weights. The data stream generator transforms the sparse data into a fixed clock rate stream. The target processing unit performs valid calculations. The result aggregation unit reorganizes the calculation results.
It achieves efficient processing of sparse data, avoids a large number of meaningless zero-value calculations, improves computing efficiency and hardware utilization, and reduces energy consumption.
Smart Images
Figure CN121835761A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, system, computer device, computer-readable storage medium, and computer program product for accelerating sparse convolutional neural network computation. Background Technology
[0002] With the widespread application of deep learning in fields such as 3D perception, autonomous driving, and robot vision, sparse convolutional neural networks have attracted much attention due to their ability to efficiently process non-uniformly distributed data. Traditional systolic array architectures (such as Google TPU) perform well when processing dense data, but they have significant drawbacks when processing sparse data (such as 3D point clouds, where sparsity often exceeds 90%): even if the input data is zero, the computing units still perform meaningless multiplication and addition operations, resulting in a large waste of computing resources and power consumption. Summary of the Invention
[0003] Therefore, it is necessary to provide a method, system, computer device, computer-readable storage medium, and computer program product that can improve computational efficiency for accelerating sparse convolutional neural network computation, in response to the above-mentioned technical problems.
[0004] In a first aspect, this application provides a method for accelerating the computation of sparse convolutional neural networks, the method comprising:
[0005] The coordinate information and validity indication of each element are extracted from the sparse feature map by the prefetching unit to generate the current sparse label stream;
[0006] The dynamic weight scheduler identifies the target processing unit in the target gated computing array based on the validity indication in the current sparse labeled stream, determines the corresponding convolution kernel weights based on the coordinate information, and distributes the corresponding convolution kernel weights to the target processing unit.
[0007] The sparse data values are extracted from the sparse feature map by a data stream generator, and the sparse data values are converted into a sparse data stream with a fixed clock rate based on the current sparse label stream; the sparse data stream is then sent to the target processing unit; wherein the time taken to extract the sparse data values from the sparse feature map is greater than the time taken to generate the current sparse label stream.
[0008] When the validity indication is valid and the convolution kernel weights corresponding to the coordinate information of the elements in the sparse data stream have been obtained, each target processing unit performs convolution calculation on the sparse data values of the coordinate information of the elements in the sparse data stream and the convolution kernel weights corresponding to the same coordinate information to obtain the calculation result.
[0009] The result aggregation unit reorganizes the calculation results into an output sparse feature map based on the corresponding coordinate information.
[0010] In one embodiment, the step of converting the sparse data values into a sparse data stream with a fixed clock rate based on the current sparse tag stream includes:
[0011] The coordinate information in the current sparse marker stream is traversed according to the preset flow order.
[0012] For each coordinate information, if the validity indication in the current sparse label stream is valid, the data value corresponding to each coordinate information is determined to be the sparse data value;
[0013] If the validity indication in the current sparse marker stream is invalid, the data value corresponding to each coordinate information is determined to be a padding value;
[0014] The sparse data stream is determined by outputting the sparse data values and the padding values at a fixed clock rate according to the flow sequence.
[0015] In one embodiment, the method further includes:
[0016] If the validity indication is valid but the convolution kernel weights corresponding to the coordinate information of the elements in the sparse data stream are not obtained, or if the validity indication is invalid, the target processing unit in the target gated computing array is prohibited from executing computing logic.
[0017] In one embodiment, the target gated computation array further includes multiple regions, each region including multiple processing units; the step of identifying target processing units in the target gated computation array based on the validity indication in the current sparse labeled stream, and determining corresponding convolutional kernel weights based on the coordinate information; and distributing the corresponding convolutional kernel weights to the target processing units, including:
[0018] Based on the validity indication in the current sparse label stream, a region-level sparse label is determined for each region; wherein, the region-level sparse label indicates whether the region contains valid data;
[0019] When the region-level sparse tag indicates that there is no valid data in the region, the control gate controller prohibits the execution of computational logic in the region; wherein, one region corresponds to one gate controller;
[0020] When the region-level sparse label indicates that there is valid data in the region, the target processing unit in the region is identified, and the corresponding convolution kernel weight is determined based on the coordinate information; the convolution kernel weight corresponding to the target processing unit in the region is sent to the gating controller.
[0021] In one embodiment, the method further includes:
[0022] The sparse prediction module predicts the next predicted sparse labeling stream based on the current sparse labeling stream.
[0023] The dynamic weight scheduler predicts the predicted convolutional kernel weights of the target processing unit based on the next predicted sparse label;
[0024] When the actual next sparse label is different from the predicted next sparse label, the predicted convolution kernel weights of the target processing unit are corrected by the correction module, and the corrected predicted convolution kernel weights are distributed to the corresponding target processing units.
[0025] If the actual next sparse label is the same as the predicted next sparse label, the predicted convolutional kernel weights are distributed to the corresponding target processing units by the dynamic weight scheduler.
[0026] In one embodiment, prior to identifying the target processing unit in the target gated computing array based on the validity indication in the current sparse labeled stream via a dynamic weight scheduler, the method further includes:
[0027] The current sparsity is obtained through the prefetching unit; the target gated computing array is determined based on the current sparsity and the granularity mode of the pre-configured pulsating array.
[0028] Secondly, this application also provides a sparse convolutional neural network computation acceleration system, the system comprising:
[0029] The prefetch unit is used to extract the coordinate information and validity indication of each element from the sparse feature map and generate the current sparse label stream;
[0030] A dynamic weight scheduler is used to identify target processing units in the target gated computing array based on the validity indication in the current sparse labeled stream, determine the corresponding convolutional kernel weights based on the coordinate information, and distribute the corresponding convolutional kernel weights to the target processing units.
[0031] A data stream generator is used to extract sparse data values from the sparse feature map, convert the sparse data values into a sparse data stream with a fixed clock rate based on the current sparse label stream, and send the sparse data stream to the target processing unit; wherein the time taken to extract sparse data values from the sparse feature map is greater than the time taken to generate the current sparse label stream;
[0032] The target processing unit is configured to perform convolution calculation on the sparse data values of the coordinate information of the elements in the sparse data stream and the convolution kernel weights corresponding to the same coordinate information, when the validity indication is valid and the convolution kernel weights corresponding to the coordinate information of the elements in the sparse data stream have been obtained, and obtain the calculation result.
[0033] The result aggregation unit is used to reorganize the calculation results into an output sparse feature map according to the corresponding coordinate information.
[0034] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.
[0035] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.
[0036] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.
[0037] The aforementioned sparse convolutional neural network computation acceleration method, system, computer device, computer-readable storage medium, and computer program product first extract the coordinate information and validity indication of each element from the sparse feature map through a prefetching unit to generate a current sparse label stream; then, based on the validity indication in the current sparse label stream, a dynamic weight scheduler identifies the target processing unit in the target systolic array, determines the corresponding convolution kernel weight based on the coordinate information, and distributes the corresponding convolution kernel weight to the target processing unit; finally, the cosine of the current sparse label stream is used to identify data validity, and dynamic weight scheduling is implemented to achieve efficient processing of sparse data. Secondly, a data stream generator extracts sparse data values from the sparse feature map. Based on the current sparse label stream, the sparse data values are transformed into a sparse data stream with a fixed clock rate. This sparse data stream is then sent to the target processing unit. The time spent extracting sparse data values from the sparse feature map is greater than the time spent generating the current sparse label stream. When the validity indicator is valid and the convolution kernel weights corresponding to the coordinate information of the elements in the sparse data stream have been obtained, each target processing unit performs convolution calculations on the sparse data values of the coordinate information of the elements in the sparse data stream and the convolution kernel weights corresponding to the same coordinate information, obtaining the calculation results. Convolution calculations are only performed on valid values, avoiding a large number of meaningless zero-value calculations. Finally, the result aggregation unit reorganizes the calculation results into an output sparse feature map based on the corresponding coordinate information. This allows for pre-detection of data validity, guiding weight scheduling and avoiding meaningless calculations, thus achieving efficient processing of sparse data. Attached Figure Description
[0038] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 This is a flowchart illustrating a method for accelerating sparse convolutional neural network computation in one embodiment;
[0040] Figure 2 This is a sparse label prefetch timing diagram in one embodiment;
[0041] Figure 3 Here is a block diagram of a dynamic weight scheduler in one embodiment;
[0042] Figure 4 This is a schematic diagram of a process for converting sparse data values into a sparse data stream with a fixed clock rate in one embodiment.
[0043] Figure 5Here is a flowchart of data flow generation in one embodiment;
[0044] Figure 6 Here is a structural diagram of the gated computing unit in one embodiment;
[0045] Figure 7 This is a schematic diagram illustrating the process of determining the convolution kernel weights of multiple regions within a target gated computation array in one embodiment.
[0046] Figure 8 This is a flowchart illustrating the prediction mechanism for enhanced sparse labeling in one embodiment;
[0047] Figure 9 This is a block diagram of a sparse convolutional neural network computation acceleration system in one embodiment;
[0048] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0050] In one embodiment, such as Figure 1 As shown, a method for accelerating the computation of sparse convolutional neural networks is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and to a system including both a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps S102 to S110:
[0051] Step S102: Extract the coordinate information and validity indication of each element from the sparse feature map through the prefetching unit to generate the current sparse label stream.
[0052] Sparse feature maps are a special data structure processed by convolutional neural networks, characterized by a majority of elements being zero. Validity indicators include valid and invalid; if an element's value is 0, the validity indicator is invalid; if the element's value is not 0, the validity indicator is valid. The coordinate information can be two-dimensional or three-dimensional.
[0053] Optionally, the terminal receives the input sparse feature map. The terminal extracts the coordinate information and validity indication of each element in the sparse feature map according to the element coordinate order through the prefetching unit. For example, for a sparse feature map of size 4×4 with a total of 9 elements, the terminal records the coordinate information and validity indication of each element according to the element coordinate order: (0,0):1; (0,1):0; (0,2):0; (0,3):0; (1,0):0; (1,1):0; (1,2):1; (1,3):0; (2,0):0; (2,1):0; (2,2):0; (2,3):0; (3,0):0; (3,1):0; (3,2):0; (3,3):1; and generates the current sparse label stream.
[0054] Step S104: Based on the validity indication in the current sparse label stream, the dynamic weight scheduler identifies the target processing unit in the target gated computing array, determines the corresponding convolution kernel weight based on the coordinate information, and distributes the corresponding convolution kernel weight to the target processing unit.
[0055] Optionally, the terminal sends the current sparse labeled stream to the dynamic weight scheduler via the prefetching unit. The dynamic weight scheduler identifies the target processing unit in the target gated computing array based on the validity indication in the current sparse labeled stream. For example, if the target gated computing array is a 4×4 PE array (16 processing units), the dynamic weight scheduler identifies the target processing unit from the target gated computing array (i.e., the 16 processing units) based on the validity indication in the current sparse labeled stream, such as the target processing unit PE that processes (0,0). 00 ; Target processing unit PE for processing (1,2) 12 ; Processing (3,3 target processing unit PE) 33 Other processing units process zero-value data. The terminal determines the corresponding convolutional kernel weights based on coordinate information through a dynamic weight scheduler and distributes the corresponding convolutional kernel weights only to the target processing unit.
[0056] Step S106: Extract sparse data values from the sparse feature map using a data stream generator, convert the sparse data values into a sparse data stream with a fixed clock rate based on the current sparse label stream, and send the sparse data stream to the target processing unit.
[0057] Among these, extracting sparse data values from the sparse feature map takes longer than generating the current sparse label stream. Extracting the current sparse label stream only requires knowing a few bits for each element (coordinates + 1 valid label bit), which is faster, while extracting data values requires knowing the full precision of each element, making it more time-consuming.
[0058] like Figure 2The diagram illustrates the time relationship between validity indicators and data values arriving at the processing unit. PE(0,0) receives a valid sparse flag at time T1, PE(1,2) receives a sparse flag at time T2, and PE(3,3) receives a sparse flag at time T4. No data value has been received by time T4.
[0059] Optionally, the terminal extracts sparse data values from the sparse feature map using a data stream generator, transforms these sparse data values into a fixed-clock-rate sparse data stream based on the current sparse label stream, and sends the sparse data stream to the target processing unit. A fixed-clock-rate sparse data stream represents the same number of data packets in each cycle. Different target processing units can also be configured with different computational precisions, such as using FP16 for critical regions and INT8 for non-critical regions.
[0060] Step S108: If the validity indicator is valid and the convolution kernel weights corresponding to the coordinate information of the elements in the sparse data stream have been obtained, each target processing unit performs convolution calculation on the sparse data values of the coordinate information of the elements in the sparse data stream and the convolution kernel weights corresponding to the same coordinate information to obtain the calculation result.
[0061] like Figure 3 As shown, target processing unit PE(0,0) receives convolution kernel weight W1, and the validity indicator is valid; target processing unit PE(1,2) receives convolution kernel weight W2, and the validity indicator is valid; target processing unit PE(3,3) receives convolution kernel weight W3, and the validity indicator is valid. At this time, target processing unit PE(0,0) performs the operation of multiplying data value 1 and weight W1; similarly, target processing unit PE(1,2) performs the operation of multiplying data value 2 and weight W2; target processing unit PE(3,3) performs the operation of multiplying data value 3 and weight W3; the convolution calculation results of each target processing unit are obtained, and the convolution calculation results are accumulated into the result register.
[0062] Step S110: The calculation results are reorganized into an output sparse feature map based on the corresponding coordinate information through the result aggregation unit.
[0063] Optionally, the terminal receives each calculation result through the result aggregation unit and reorganizes each calculation result into an output sparse feature map according to the corresponding coordinate information.
[0064] In traditional methods, the terminal directly reads data values from a sparse feature map (4*4), which is slower than reading the labeled stream in this embodiment. The terminal calculates the convolutional kernel weights corresponding to each element based on the sparse feature map, and sends the data value and corresponding kernel weights of each element to a gated computation array (4*4). Each processing unit in the gated computation array calculates the multiplication of the kernel weights and the data value, for a total of 16 calculations.
[0065] In this application, by pre-reading the tag stream and adjusting the weights in a timely manner according to the tag stream, the weights corresponding to the elements whose validity indication is valid are distributed to the target computing units in the target gated computing array. The target computing units calculate the product of the kernel weight and the data value, and calculate it three times in total according to the above embodiment, which greatly reduces the meaningless calculation of non-zero values and improves the computing efficiency.
[0066] In the above-mentioned method for accelerating sparse convolutional neural network computation, firstly, the coordinate information and validity indication of each element are extracted from the sparse feature map by the prefetching unit to generate the current sparse label stream; then, the target processing unit in the target systolic array is identified by the dynamic weight scheduler based on the validity indication in the current sparse label stream, and the corresponding convolution kernel weight is determined based on the coordinate information, and the corresponding convolution kernel weight is distributed to the target processing unit; finally, the data validity is identified by the cosine of the current sparse label stream, and the dynamic weight scheduling is implemented to achieve efficient processing of sparse data. Secondly, a data stream generator extracts sparse data values from the sparse feature map. Based on the current sparse label stream, the sparse data values are transformed into a sparse data stream with a fixed clock rate. This sparse data stream is then sent to the target processing unit. The time spent extracting sparse data values from the sparse feature map is greater than the time spent generating the current sparse label stream. When the validity indicator is valid and the convolution kernel weights corresponding to the coordinate information of the elements in the sparse data stream have been obtained, each target processing unit performs convolution calculations on the sparse data values of the coordinate information of the elements in the sparse data stream and the convolution kernel weights corresponding to the same coordinate information, obtaining the calculation results. Convolution calculations are only performed on valid values, avoiding a large number of meaningless zero-value calculations. Finally, the result aggregation unit reorganizes the calculation results into an output sparse feature map based on the corresponding coordinate information. This allows for pre-detection of data validity, guiding weight scheduling and avoiding meaningless calculations, thus achieving efficient processing of sparse data.
[0067] In one exemplary embodiment, such as Figure 4 As shown, based on the current sparse marker stream, the sparse data values are transformed into a sparse data stream with a fixed clock rate, including the following steps S402 to S408. Wherein:
[0068] Step S402: Traverse each coordinate information in the current sparse marker stream according to the preset flow order.
[0069] like Figure 5 The diagram illustrates the process of converting a sparse feature map into a fixed-rate data stream. The input and output coordinates and the way PEs are labeled on the diagram are for illustrative purposes only. Figure 5 The dataset contains four non-zero data points arranged compactly to save storage. The sparse feature map has many gaps, while the compact arrangement eliminates these gaps. However, this leads to a problem: the labeled data stream generator addresses the fundamental conflict between sparse data and the rhythm requirements of the gated computation array. Traditional sparse processing schemes typically attempt to skip zero values, i.e., compact arrangement, but this disrupts the fixed rhythm of the pulsating array, resulting in decreased pipeline efficiency.
[0070] Optionally, the terminal uses a data stream generator to generate data in a preset flow order, such as the sparse feature map in the order of each row, or the sparse feature map in the order of each column.
[0071] Step S404: For each coordinate information, if the validity indication in the current sparse label stream is valid, determine that the data value corresponding to each coordinate information is a sparse data value.
[0072] Optionally, for each coordinate information, if the validity indication in the current sparse label stream is valid, the data value corresponding to each coordinate information of the terminal is a sparse data value, that is, a data value that is not zero.
[0073] Step S406: If the validity indication in the current sparse marker stream is invalid, determine that the data value corresponding to each coordinate information is a padding value.
[0074] Optionally, if the validity indicator in the current sparse marker stream is invalid, the data value corresponding to each coordinate information is determined to be a padding value, in which case the padding value is 0.
[0075] Step S408: Output sparse data values and padding values at a fixed clock rate according to the flow sequence to determine the sparse data stream.
[0076] Optionally, the terminal outputs sparse data values and padding values of 0 at a fixed clock rate according to the flow sequence to determine the sparse data stream, such as period 1: [0.0]; period 2: [5.2]; period 3: [0.0]; period 4: [0.0]; period 5: [3.1]; period 6: [0.0]; period 7: [0.0]; period 8: [0.0]; period 9: [7.8].
[0077] If we follow the traditional method, the cycle would be: Cycle 1: [5.2]; Cycle 2: [3.1]; Cycle 3: [7.8].
[0078] In this embodiment, by outputting sparse data values and padding values at a fixed clock rate, the generated data stream format is regular, making it easy for hardware to process. The bus is always fully loaded with no idle cycles.
[0079] In an exemplary embodiment, the sparse convolutional neural network computation acceleration method further includes: prohibiting the target processing unit in the target gating computation array from executing computation logic when the validity indication is valid but the coordinate information of the elements in the sparse data stream corresponding to the convolutional kernel weights is not obtained, or when the validity indication is invalid.
[0080] like Figure 6 As shown, if the validity indicator is valid and the convolution kernel weights corresponding to the coordinate information of the elements in the sparse data stream have been obtained, it is recorded as 1, and the data flows to the circuit corresponding to 1 for processing through the non-zero value detection circuit. If the validity indicator is invalid, it is recorded as 0, and the data flows to the circuit corresponding to 0 for processing through the non-zero value detection circuit; that is, the target processing unit in the target gated computing array is prohibited from executing the computation logic. If the validity indicator is valid but the convolution kernel weights corresponding to the coordinate information of the elements in the sparse data stream have not been obtained, it indicates an abnormal error, and an error message is displayed while the target processing unit in the target gated computing array is prohibited from executing the computation logic.
[0081] It should be noted that when the circuit corresponding to flow 0 is processed, the accumulator holds the current value and is not updated, and the data is directly passed (keeping the pipeline flowing).
[0082] In this embodiment, by using a non-zero value detection circuit to dynamically enable the computing function based on valid markers, the meaningless calculations for zero values can be greatly reduced, thus improving computing efficiency. Traditional pulsating arrays, when processing sparse data, must reserve resources for all data even if most of the data is zero, resulting in low hardware utilization. In contrast, this invention uses a gating mechanism to concentrate computing resources on non-zero data, significantly improving hardware utilization. Furthermore, by dynamically adjusting the operating state of the computing unit (including logic gating, clock gating, and power gating) based on the validity of the data, energy consumption is significantly reduced.
[0083] In one exemplary embodiment, such as Figure 7 As shown, the target gated computation array also includes multiple regions, each region comprising multiple processing units; based on the validity indication in the current sparse labeled stream, the target processing unit in the target gated computation array is identified, and the corresponding convolutional kernel weights are determined based on coordinate information; the corresponding convolutional kernel weights are distributed to the target processing unit, including steps S702 to S706. Wherein:
[0084] Step S702: Based on the validity indication in the current sparse label stream, determine the region-level sparse label for each region.
[0085] Among them, the regional sparse marker indicates whether the region contains valid data.
[0086] The PE gated computing array is divided into multiple regions, each sharing a gate controller. Gating is performed collectively based on the overall sparsity of the regions. Taking a 3x3 gated computing array divided by column as an example: Region 0: PE 00 PE 10 PE 20 Area 1: PE 01 PE 11 PE 21 Area 2: PE 02 PE 12 PE 22 .
[0087] In addition to fine-grained tags for each PE, region-level coarse-grained tags are also generated:
[0088] Area 0: PE 00 =0, PE 10 =0, PE 20 =0, region flag =0 (represents all invalid);
[0089] Area 1: PE 01 =1, PE 11 =1, PE 21 =0, region tag=1 (representing at least one valid);
[0090] Area 2: PE 02 =0, PE 12 =0, PE 22 =1, region tag =1 (meaning one less valid one).
[0091] Optionally, the terminal determines the region-level sparse label for each region based on the validity indication in the current sparse label stream using a dynamic weight scheduler. For example, if the region-level sparse label for region 0 is 0, it means that all regions in region 0 are invalid. If the region-level sparse label for region 1 is 1, it means that at least one region in region 1 is valid.
[0092] Step S704: When the region-level sparse marker indicates that there is no valid data in the region, the control gate controller is prohibited from executing computation logic in the region.
[0093] One area corresponds to one gate controller.
[0094] Optionally, when a region-level sparse flag indicates that there is no valid data within the region, the control gating controller is prohibited from executing computational logic within that region. This means the weight bus for the entire region 0 is shut down; no weights are sent to region 0, and all PEs in region 0 enter a deep low-power mode.
[0095] Step S706: When the region-level sparse label indicates that there is valid data in the region, identify the target processing unit in the region and determine the corresponding convolution kernel weight based on the coordinate information; send the convolution kernel weight corresponding to the target processing unit in the region to the gating controller.
[0096] Optionally, when the region-level sparse label indicates the presence of valid data within the region, the target processing unit within the region is identified, and the corresponding convolutional kernel weights are determined based on the coordinate information; the convolutional kernel weights corresponding to the target processing unit within the region are then sent to the gating controller. For example, if the region-level sparse label for region 1 is 1, the weights are sent to the PE... 01 PE 11 Send weights. (Through methods such as...) Figure 6 The weight bus of region 1 is activated as shown, but the internal PE is still labeled with fine granularity. 01 : Valid, receiving weight w 01 ;PE 11 : Valid, receiving weight w 11 ;PE 21 Invalid, weight not accepted.
[0097] In this embodiment, by dividing the area into zones, the number of controllers can be reduced, and the power supply can be completely shut off in invalid areas, further reducing power consumption; control signals can also be shared between zones, reducing wiring.
[0098] In one exemplary embodiment, such as Figure 8 As shown, the prediction-enhanced sparse labeling mechanism in the computational acceleration of sparse convolutional neural networks includes steps S802 to S808. Wherein:
[0099] Step S802: The sparse prediction module predicts the next predicted sparse labeling stream based on the current sparse labeling stream.
[0100] Optionally, the terminal can also add prediction logic to the basic sparse label prefetching, using spatial / temporal locality to predict subsequent sparse patterns and optimize gating decisions in advance. The terminal uses the sparse prediction module to predict the next predicted sparse label stream based on the current sparse label stream.
[0101] Step S804: The predicted convolutional kernel weights of the target processing unit are predicted based on the next predicted sparse label by the dynamic weight scheduler.
[0102] Optionally, the terminal uses a dynamic weight scheduler to predict the predicted convolutional kernel weights of the target processing unit based on the next predicted sparse label that arrives earlier than the actual next sparse label.
[0103] In step S806, if the actual next sparse label is different from the predicted next sparse label, the predicted convolutional kernel weights of the target processing unit are corrected by the correction module, and the corrected predicted convolutional kernel weights are distributed to the corresponding target processing units.
[0104] Optionally, if the actual next sparse label is different from the predicted next sparse label, it indicates a prediction error. The terminal corrects the predicted convolutional kernel weights of the target processing unit through the correction module and distributes the corrected predicted convolutional kernel weights to the corresponding target processing unit.
[0105] Step S808: If the actual next sparse label is the same as the predicted next sparse label, the predicted convolutional kernel weights are distributed to the corresponding target processing units through the dynamic weight scheduler.
[0106] Optionally, if the actual next sparse label is the same as the predicted next sparse label, it indicates that the prediction is correct. In this case, the dynamic weight scheduler does not need to configure the convolution kernel weights anymore, but only needs to distribute the predicted convolution kernel weights to the corresponding target processing units.
[0107] In this embodiment, if the prediction is correct, the weights can be configured in advance, reducing the weight configuration time. If the prediction is correct but incorrect, only a small amount of overhead is needed to correct the incorrect part of the prediction, while the correct part can be reused, which can reduce the weight configuration time to some extent.
[0108] In an exemplary embodiment, before identifying the target processing unit in the target gated computing array based on the validity indication in the current sparse label stream by a dynamic weight scheduler, the method further includes: obtaining the current sparsity through a prefetching unit; and determining the target gated computing array based on the current sparsity and the pre-configured granularity mode of the gated computing array.
[0109] Optionally, the current sparsity is obtained through a prefetching unit; the current sparsity is determined by the ratio of the number of valid indicators to the total number of valid indicators. The terminal determines the target gated computation array based on the current sparsity and the pre-configured granularity mode of the systolic array. For example, a 3×3 array can be configured with different modes: Mode 1 (fine-grained): 9 independent PEs (1×1 granularity); Mode 2 (medium-grained): 4 subarrays (2×2 granularity, the last one being 1×1); Mode 3 (coarse-grained): 1 large array (3×3 granularity). When the current sparsity is greater than a first threshold, it indicates very sparseness, and a coarse-grained target gated computation array is used; when the current sparsity is greater than a second threshold but less than or equal to the first threshold, it indicates medium sparsity, and a medium-grained target gated computation array is used; other cases indicate relatively denseness, and a fine-grained target gated computation array is used.
[0110] In this embodiment, it can adapt to data streams with different sparsity. When the sparsity is high, coarse-grained granularity is used to achieve energy saving; when the sparsity is low, fine-grained granularity is used to achieve efficient and accurate calculation.
[0111] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0112] Based on the same inventive concept, this application also provides a sparse convolutional neural network (SCNN) computation acceleration system for implementing the aforementioned sparse convolutional neural network computation acceleration method. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more embodiments of the sparse convolutional neural network computation acceleration system provided below can be found in the limitations of the sparse convolutional neural network computation acceleration method described above, and will not be repeated here.
[0113] In one exemplary embodiment, such as Figure 9 As shown, a sparse convolutional neural network computation acceleration system is provided, comprising: a prefetching unit connected to a data stream generator; a data stream generator connected to a gated computation array; a dynamic weight scheduler also connected to the gated computation array; and the gated computation array connected to a result aggregation unit, including:
[0114] The prefetching unit is used to extract the coordinate information and validity indication of each element from the sparse feature map to generate the current sparse label stream.
[0115] The dynamic weight scheduler is used to identify target processing units in the target gated computing array based on the validity indication in the current sparse labeled stream, determine the corresponding convolutional kernel weights based on coordinate information, and distribute the corresponding convolutional kernel weights to the target processing units.
[0116] A data stream generator is used to extract sparse data values from a sparse feature map, convert the sparse data values into a sparse data stream with a fixed clock rate based on the current sparse label stream, and send the sparse data stream to the target processing unit. The time taken to extract sparse data values from the sparse feature map is greater than the time taken to generate the current sparse label stream.
[0117] The target processing unit is used to perform convolution calculations on the sparse data values of the coordinate information of the elements in the sparse data stream and the convolution kernel weights corresponding to the same coordinate information, when the validity indication is valid and the convolution kernel weights corresponding to the coordinate information of the elements in the sparse data stream have been obtained, to obtain the calculation result.
[0118] The result aggregation unit is used to reorganize the calculation results into an output sparse feature map based on the corresponding coordinate information.
[0119] In an exemplary embodiment, the data stream generator is further configured to traverse each coordinate information in the current sparse marker stream according to a preset flow order; for each coordinate information, if the validity indication in the current sparse marker stream is valid, determine that the data value corresponding to each coordinate information is a sparse data value; if the validity indication in the current sparse marker stream is invalid, determine that the data value corresponding to each coordinate information is a padding value; and output the sparse data value and the padding value at a fixed clock rate according to the flow order to determine the sparse data stream.
[0120] In an exemplary embodiment, a gated computing array is configured to prevent the target processing unit in the target gated computing array from executing computational logic when the validity indication is valid but the convolutional kernel weights corresponding to the coordinate information of the elements in the sparse data stream are not obtained, or when the validity indication is invalid.
[0121] In an exemplary embodiment, the target gated computing array further includes multiple regions, each region including multiple processing units; based on the validity indication in the current sparse label stream, the target processing unit in the target gated computing array is identified, and the corresponding convolutional kernel weight is determined based on coordinate information; the corresponding convolutional kernel weight is distributed to the target processing unit, and the dynamic weight scheduler is also used to determine the region-level sparse label for each region based on the validity indication in the current sparse label stream; wherein, the region-level sparse label indicates whether the region contains valid data; when the region-level sparse label indicates that there is no valid data in the region, the gate controller is controlled to prohibit the execution of computing logic in the region; wherein, one region corresponds to one gate controller; when the region-level sparse label indicates that there is valid data in the region, the target processing unit in the region is identified, and the corresponding convolutional kernel weight is determined based on coordinate information; the convolutional kernel weight corresponding to the target processing unit in the region is sent to the gate controller.
[0122] In one exemplary embodiment, the sparse convolutional neural network computation acceleration system further includes: a sparse prediction module, configured to predict the next predicted sparse labeling stream based on the current sparse labeling stream.
[0123] A dynamic weight scheduler is used to predict the kernel weights of the target processing unit based on the next predicted sparse label.
[0124] The correction module is used to correct the predicted convolutional kernel weights of the target processing unit when the actual next sparse label is different from the predicted next sparse label, and then distribute the corrected predicted convolutional kernel weights to the corresponding target processing unit.
[0125] A dynamic weight scheduler is used to distribute the predicted convolutional kernel weights to the corresponding target processing units when the actual next sparse label is the same as the predicted next sparse label.
[0126] In an exemplary embodiment, the sparse convolutional neural network computation acceleration system further includes: a matching module; wherein, a prefetching unit is used to obtain the current sparsity; and the matching module is used to determine the target gated computation array based on the current sparsity and the granularity mode of a pre-configured gated computation array.
[0127] The modules in the aforementioned sparse convolutional neural network computation acceleration system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0128] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores sparse feature map data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a method for accelerating sparse convolutional neural network computation.
[0129] Those skilled in the art will understand that Figure 10The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0130] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0131] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0132] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0133] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0134] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0135] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A sparse convolutional neural network computation acceleration method, characterized in that, The method comprises: extracting coordinate information and validity indication of each element from the sparse feature map by a prefetch unit to generate a current sparse marker stream; identifying target processing units in a target GCA based on the validity indication in the current sparse marker stream and determining corresponding convolution kernel weights based on the coordinate information by a dynamic weight scheduler, and distributing the corresponding convolution kernel weights to the target processing units; extracting sparse data values from the sparse feature map by a data stream generator, converting the sparse data values into a sparse data stream with a fixed clock rate based on the current sparse marker stream, and sending the sparse data stream to the target processing units; wherein the time consumption of extracting sparse data values from the sparse feature map is greater than the time consumption of generating the current sparse marker stream; in the case that the validity indication is valid and the convolution kernel weight corresponding to the coordinate information of the element in the sparse data stream has been obtained, each target processing unit performs convolution calculation on the sparse data value of the coordinate information of the element in the sparse data stream and the convolution kernel weight corresponding to the same coordinate information to obtain a calculation result; reorganizing each calculation result into an output sparse feature map according to the corresponding coordinate information by a result aggregation unit.
2. The method of claim 1, wherein, The method further comprises: in the case that the validity indication is valid but the convolution kernel weight corresponding to the coordinate information of the element in the sparse data stream has not been obtained, or the validity indication is invalid, prohibiting the target processing units in the target GCA from executing the calculation logic. The target GCA further comprises a plurality of regions, each region comprising a plurality of processing units; the target processing units in the target GCA are identified based on the validity indication in the current sparse marker stream and the corresponding convolution kernel weights are determined based on the coordinate information; distributing the corresponding convolution kernel weights to the target processing units comprises: determining a region-level sparse marker of each region based on the validity indication in the current sparse marker stream; wherein the region-level sparse marker indicates whether the region contains valid data; 3. The method of claim 1, wherein, when the region-level sparse marker indicates that there is no valid data in the region, controlling a gate controller to prohibit the execution of the calculation logic in the region; wherein one region corresponds to one gate controller. 4. The method of claim 1, wherein, When the region-level sparse label indicates that there is valid data in the region, the target processing unit in the region is identified, and the corresponding convolution kernel weight is determined based on the coordinate information; the convolution kernel weight corresponding to the target processing unit in the region is sent to the gating controller.
5. The method of claim 1, wherein, The method further includes: The sparse prediction module predicts the next predicted sparse labeling stream based on the current sparse labeling stream. The dynamic weight scheduler predicts the predicted convolutional kernel weights of the target processing unit based on the next predicted sparse label; When the actual next sparse label is different from the predicted next sparse label, the predicted convolution kernel weights of the target processing unit are corrected by the correction module, and the corrected predicted convolution kernel weights are distributed to the corresponding target processing units. If the actual next sparse label is the same as the predicted next sparse label, the predicted convolutional kernel weights are distributed to the corresponding target processing units by the dynamic weight scheduler.
6. The method of claim 1, wherein, Before identifying the target processing unit in the target gated computing array based on the validity indication in the current sparse labeled stream using a dynamic weight scheduler, the method further includes: The current sparsity is obtained through the prefetching unit; the target gated computation array is determined based on the current sparsity and the pre-configured granularity mode of the gated computation.
7. A sparse convolutional neural network computing acceleration system, comprising: The system includes: The prefetch unit is used to extract the coordinate information and validity indication of each element from the sparse feature map and generate the current sparse label stream; A dynamic weight scheduler is used to identify target processing units in the target gated computing array based on the validity indication in the current sparse labeled stream, determine the corresponding convolutional kernel weights based on the coordinate information, and distribute the corresponding convolutional kernel weights to the target processing units. A data stream generator is used to extract sparse data values from the sparse feature map, convert the sparse data values into a sparse data stream with a fixed clock rate based on the current sparse label stream, and send the sparse data stream to the target processing unit; wherein the time taken to extract sparse data values from the sparse feature map is greater than the time taken to generate the current sparse label stream; The target processing unit is configured to perform convolution calculation on the sparse data values of the coordinate information of the elements in the sparse data stream and the convolution kernel weights corresponding to the same coordinate information, when the validity indication is valid and the convolution kernel weights corresponding to the coordinate information of the elements in the sparse data stream have been obtained, and obtain the calculation result. The result aggregation unit is used to reorganize the calculation results into an output sparse feature map according to the corresponding coordinate information.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.