Data processing method and apparatus for neural network

Abstract

The present disclosure provides a data processing method and device for a neural network. The method comprises: sparsifying one or more dense tensors fetched from a shared data buffer using a tensor sparsity system to obtain corresponding sparse tensors, the sparse tensors being cached in the shared data buffer; sparsifying one or more dense weights fetched from the shared data buffer using a weight sparsity system to obtain corresponding sparse weights, the sparse weights being cached in the weight sparsity system, wherein a data processing rate of the tensor sparsity system is lower than a data processing rate of the weight sparsity system; and sending the sparse weights from the weight sparsity system and the sparse tensors from the shared data buffer to a multiply-accumulate array for computation to obtain an output result of the neural network.

Description

Technical Field

[0001] This disclosure relates to the field of neural networks, and more particularly to a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product for neural networks. Background Technology

[0002] With the rapid iteration of artificial intelligence technology, deep neural networks have evolved from basic shallow structures to complex deep and ultra-deep architectures, and are widely used in fields such as image recognition, natural language processing, and intelligent decision-making. However, behind the improvement in model performance is an exponential increase in computational load. Today, the number of parameters in neural networks can reach billions or even tens of billions. The massive weight calculations and data processing not only consume a large amount of hardware storage resources, but also lead to slower inference speeds and high energy consumption.

[0003] Sparse computing techniques in neural networks alleviate the computational burden on neural networks. The core idea is to identify and eliminate redundant parameters and invalid data in the model, retaining only the core data and connections that play a crucial role in the model's output. In this way, sparse computing can significantly reduce the amount of data involved in the computation, effectively reducing the computational load without altering the core model structure: on the one hand, it simplifies redundant parameters and compresses the model's storage volume; on the other hand, it skips invalid zero-value operations, reducing unnecessary computational steps, thereby significantly reducing the overall computational load while ensuring that the model's accuracy remains largely unaffected. Summary of the Invention

[0004] This disclosure provides a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product for neural networks.

[0005] According to one aspect of this disclosure, a data processing method for a neural network is provided, the method comprising: sparsifying one or more dense tensors retrieved from a shared data buffer using a tensor sparse system to obtain corresponding sparse tensors, the sparse tensors being cached in the shared data buffer; sparsifying one or more dense weights retrieved from the shared data buffer using a weight sparse system to obtain corresponding sparse weights, the sparse weights being cached in a weight sparse system, wherein the data processing rate of the tensor sparse system is lower than that of the weight sparse system; and feeding the sparse weights from the weight sparse system and the sparse tensors from the shared data buffer into a multiply-accumulate array for computation to obtain the output of the neural network.

[0006] In some embodiments, the tensor sparse system includes one or more first inverse butterfly networks to sparse the dense tensors through the first inverse butterfly networks; the weight sparse system includes one or more second inverse butterfly networks to sparse the dense weights through the second inverse butterfly networks.

[0007] In some embodiments, sparsifying the dense tensor using the first inverse butterfly network includes: removing zero data from the dense tensor using the first inverse butterfly network to aggregate the non-zero tensor data in the dense tensor into continuous data for output; and sparsifying the dense weights using the second inverse butterfly network includes: removing zero data from the dense weights using the second inverse butterfly network to aggregate the non-zero weight data in the dense weights into continuous data for output.

[0008] In some embodiments, the data processing rate of the tensor sparse system is lower than that of the weighted sparse system by at least one of the following: the number of the first inverse butterfly network in the tensor sparse system is less than the number of the second inverse butterfly network in the weighted sparse system; the first clock frequency of the tensor sparse system is lower than the second clock frequency of the weighted sparse system; and the data path width of the tensor sparse system is narrower than that of the weighted sparse system.

[0009] In some embodiments, the cache capacity of the tensor sparse system is smaller than that of the weighted sparse system.

[0010] In some embodiments, the second clock frequency of the weighted sparse system is the same as the third clock frequency of the multiply-accumulate array.

[0011] In some embodiments, the calculation of the multiply-accumulate array in the third clock cycle corresponding to the third clock frequency is based on the sparse tensor of the first data quantity and the sparse weight of the second data quantity; the weighted sparse system obtains the sparse weight of the second data quantity in the second clock cycle corresponding to the second clock frequency, wherein the second clock cycle is the same as the third clock cycle; before the multiply-accumulate array performs calculations, the sparse tensor of the first data quantity is sent from the shared data buffer to the multiply-accumulate array.

[0012] According to another aspect of this disclosure, a data processing apparatus for a neural network is provided, the apparatus comprising: a tensor sparsity unit configured to sparsify one or more dense tensors retrieved from a shared data buffer using a tensor sparsity system to obtain corresponding sparse tensors, the sparse tensors being cached in the shared data buffer; a weight sparsity unit configured to sparsify one or more dense weights retrieved from the shared data buffer using a weight sparsity system to obtain corresponding sparse weights, the sparse weights being cached in the weight sparsity system, wherein the data processing rate of the tensor sparsity system is lower than that of the weight sparsity system; and a result generation unit configured to feed the sparse weights from the weight sparsity system and the sparse tensors from the shared data buffer to a multiply-accumulate array for computation to obtain the output result of the neural network.

[0013] According to another aspect of this disclosure, a computer device is provided, the computer device comprising: at least one processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the aforementioned method.

[0014] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, causes the processor to perform the aforementioned method.

[0015] According to another aspect of this disclosure, a computer program product is provided, which includes a computer program that, when executed by a processor, causes the processor to perform the aforementioned method.

[0016] According to the method proposed in this disclosure, tensor sparsity processing and weight sparsity processing can be performed separately in two systems with different data processing rates, thereby improving the utilization of hardware resources and optimizing the allocation of hardware and computing resources.

[0017] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0018] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 An exemplary flowchart illustrating a data processing method for a neural network according to an embodiment of the present disclosure is shown; Figure 2 An example inverse butterfly network according to an embodiment of the present disclosure is illustrated; Figure 3 An exemplary architecture diagram of a data processing method for a neural network according to embodiments of the present disclosure is illustrated; Figure 4 An exemplary block diagram of a data processing apparatus for a neural network according to embodiments of the present disclosure is illustrated; Figure 5 An exemplary block diagram of a computer device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0022] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0023] As mentioned earlier, sparse computing techniques in neural networks can alleviate the computational burden on neural networks by removing invalid data from the model. Specifically, during the training and inference process of deep neural networks, weights and tensors (e.g., activation tensors) are the core data carriers supporting model operations, and their data scale and computational efficiency directly determine the overall performance of the model. Therefore, sparsifying weights and tensors, and accurately identifying and removing invalid zero elements, becomes a key means to reduce computational load and memory access overhead.

[0024] To achieve efficient sparsification of weights and tensors, related technologies employ an integrated design approach, combining weight sparsification and tensor sparsification within the same high-speed computing system. Since both sparsification operations need to be performed in the same pipeline, the weight and tensor sparsification operations utilize the same processing rate, and hardware resources (including computing units and storage bandwidth) are allocated uniformly. This approach ensures the synchronization of weights and tensors during processing, avoids pipeline interruptions and data delays caused by inconsistent processing rhythms, and simplifies hardware scheduling logic.

[0025] However, this integrated design scheme has significant drawbacks in resource utilization. From a data characteristics perspective, weights typically have a much larger data size than tensors, and their update frequency is extremely low. They are mostly used permanently after model training, with computational complexity primarily concentrated in intensive operations such as large-scale matrix multiplications. Tensors, on the other hand, have a relatively smaller data size, more flexible access patterns, are mostly dynamically generated and frequently updated, and have lower computational complexity. According to the integrated design scheme, placing both in the same high-speed computing system and allocating high-bandwidth, high-computing-power resources based on the processing needs of the weights leads to the tensor processing component consuming the same high-performance hardware resources as the weights, resulting in a severe waste of hardware resources, even though tensors can be processed without high-performance resources. This results in the over-allocated resources not being fully utilized, reducing hardware utilization and significantly increasing device power consumption, thus hindering the practical application of sparse neural network computing technology.

[0026] To address the aforementioned problems, this disclosure proposes a data processing method for neural networks. This method separates tensor sparsity processing and weight sparsity processing into two systems with different data processing rates. Specifically, tensor sparsity processing is performed in a tensor sparse system, while weight sparsity processing is performed in a weight sparse system, with the tensor sparse system having a lower data processing rate than the weight sparse system. In this way, weight sparsity processing can fully utilize the high-performance weight sparse system, while the less computationally demanding tensor sparsity processing is distributed across the tensor sparse system, thereby improving hardware resource utilization and optimizing the allocation of hardware and computational resources.

[0027] Below, refer to Figure 1 An exemplary flowchart 100 is provided to describe a data processing method for a neural network according to embodiments of the present disclosure.

[0028] In step 101, a tensor sparsity system is used to sparsify one or more dense tensors retrieved from the shared data buffer to obtain corresponding sparse tensors, which are then cached in the shared data buffer.

[0029] In step 102, a weighted sparse system is used to sparse one or more dense weights taken from the shared data cache to obtain corresponding sparse weights. The sparse weights are cached in the weighted sparse system. The data processing rate of the tensor sparse system is lower than that of the weighted sparse system.

[0030] In step 103, the sparse weights from the sparse weight system and the sparse tensors from the shared data buffer are sent to the multiplication-accumulation array for calculation to obtain the output of the neural network.

[0031] By utilizing the method provided in the embodiments of this disclosure, the sparse processing of weights and tensors can be distributed across two systems with different data processing rates. Appropriate processing resources can be allocated according to the characteristics of the weight data and tensor data, avoiding the sparse processing of tensors from occupying the computational resources of the sparse processing of weights, thereby improving the utilization rate of hardware resources and achieving a reasonable allocation of hardware resources.

[0032] The above steps will be explained in detail below.

[0033] In step 101, a tensor sparsity system is used to sparsify one or more dense tensors retrieved from the shared data buffer to obtain corresponding sparse tensors, which are then cached in the shared data buffer.

[0034] In this paper, dense tensors refer to raw tensor data that has not undergone sparse processing, such as activation tensor data used in neural networks, KV tensor data used in large models, and other tensor data.

[0035] Additionally, the shared data cache can be used to cache dense tensors and dense weights (mentioned below). Both the tensor sparse system and the weight sparse system can retrieve dense tensors and dense weights from the shared data cache for sparsity processing. This shared data cache can be, for example, a level 2 or level 3 on-chip cache, or a large-capacity on-chip cache such as shared on-chip SRAM.

[0036] In some embodiments, before caching sparse tensors in the shared data buffer, the generated sparse tensors can be temporarily cached in the buffer of the tensor sparse system. Once the sparse tensors reach a certain data volume, they are then sent to the shared data buffer for caching. For example, if the tensor sparse system can generate one set of sparse tensors per unit clock cycle, this one set of sparse tensors can be cached in the tensor sparse system's buffer first. After the tensor sparse system generates eight sets of sparse tensors, all eight sets are sent to the shared data buffer for caching. This reduces the number of accesses to the shared data buffer and improves communication efficiency.

[0037] In this step, dense tensors can be sparsified using a tensor sparsity system to obtain corresponding sparse tensors.

[0038] In some examples, the tensor sparse system can be designed based on one or more inverse butterfly networks: the tensor sparse system may include one or more first inverse butterfly networks to sparsify dense tensors. The first inverse butterfly network removes zero data from the dense tensor and aggregates the non-zero tensor data into continuous data for output.

[0039] An inverse butterfly network is a network structure that removes zero data from dense tensors and aggregates scattered non-zero tensor data. The structural characteristics of an inverse butterfly network include L levels of switching units and N inputs and N outputs (N = 2^3). L The network consists of a crossbar switch at each level, which can be used to cross or cut through data. Figure 2 An example of an inverse butterfly network 200 is illustrated. The inverse butterfly network 200 is a network with a three-stage hierarchical design, eight inputs and eight outputs. Each stage contains eight nodes, each uniquely identified by a number. The input layer of the inverse butterfly network 200 receives dense tensors. The intermediate multi-level interconnection layer is the core of the inverse butterfly network 200, responsible for data shifting, exchanging, and integrating data, gathering non-zero tensor data to higher-order ports and discarding zero data by pushing it to lower-order ports. The output layer outputs the recombined sparse tensors. In tensor sparse systems comprising multiple first inverse butterfly networks, these networks can operate in parallel to improve data processing efficiency.

[0040] In other examples, the tensor sparse system can also be designed using sparse methods such as structured pruning sparsity, unstructured fine-grained sparsity, and matrix diagonal sparsity.

[0041] In step 102, a weighted sparse system is used to sparse one or more dense weights retrieved from the shared data cache to obtain corresponding sparse weights, which are cached in the weighted sparse system; wherein, the data processing rate of the tensor sparse system is lower than that of the weighted sparse system.

[0042] In this paper, dense weights refer to the raw weight data without sparsity processing, such as the weight data of fully connected components used in neural networks, and the weight data of the Q / K / V projection matrices used in large models. Dense weights can be cached in the aforementioned shared data cache.

[0043] Furthermore, unlike sparse tensors which are cached in a shared data cache, sparse weights are cached in a weight-sparse system. This is to facilitate the direct feeding of the acquired sparse weights into the multiply-accumulate array for computation, thereby improving the computational efficiency of the multiply-accumulate array.

[0044] In this step, dense weights can be thinned using a weight sparsity system to obtain corresponding sparse weights.

[0045] In some examples, the weight sparse system can be designed based on one or more inverse butterfly networks: the weight sparse system may include one or more second inverse butterfly networks to sparse the dense weights. The second inverse butterfly network removes zero data from the dense weights and aggregates the non-zero weight data into continuous data for output. Similar to the first inverse butterfly network described above, the second inverse butterfly network uses a similar inverse butterfly network to remove zero data from the dense weights and aggregate the dispersed non-zero weight data. Multiple second inverse butterfly networks in the weight sparse system can also run in parallel, which will not be elaborated here. By including one or more inverse butterfly networks in the tensor sparse system or weight sparse system, efficient sparse processing of dense tensors or dense weights can be achieved, and the architecture of the inverse butterfly network is well-defined, making the implementation of sparse processing relatively easy.

[0046] Tensor sparse systems have a lower data processing rate than weighted sparse systems. In other words, in the same amount of time, a tensor sparse system generates fewer sparse tensors than a weighted sparse system generates fewer sparse weights. As mentioned earlier, the size of tensor data in a neural network system is much smaller than the size of weight data. Because the size of the tensors requiring sparse processing is small, tensor sparse systems do not need high data processing speeds or capabilities. Therefore, by designing tensor sparse systems and weighted sparse systems separately, reasonable computational resources can be allocated to the sparse processing of tensor data and weight data respectively.

[0047] The lower data processing rate of a tensor sparse system compared to a weighted sparse system can be achieved by at least one of the following: the number of first inverse butterfly networks in the tensor sparse system is less than the number of second inverse butterfly networks in the weighted sparse system; the first clock frequency of the tensor sparse system is lower than the second clock frequency of the weighted sparse system; the data path width of the tensor sparse system is narrower than that of the weighted sparse system; or one or more other parameters related to the data processing rate.

[0048] Specifically, the data processing rate of tensor sparse systems is lower than that of weighted sparse systems, which can be reflected in one or more aspects of the performance design of sparse systems.

[0049] The number of first inverse butterfly networks in a tensor sparse system can be less than the number of second inverse butterfly networks in a weighted sparse system. In other words, because the number of first inverse butterfly networks in a tensor sparse system is smaller, the amount of sparse tensor data that a tensor sparse system can generate in the same amount of time is less than the amount of sparse weight data that a weighted sparse system can generate. This is also related to the characteristic mentioned earlier that the tensor data size in neural network systems is much smaller than the weight data size. Because the tensor data size is smaller, fewer inverse butterfly networks are needed to perform sparse processing on dense tensors; conversely, more inverse butterfly networks are needed to perform sparse processing on dense weights. In one embodiment, if a first inverse butterfly network can generate a set of sparse tensors and a second inverse butterfly network can obtain a set of sparse weights, the tensor sparse system includes two first inverse butterfly networks and the weight sparse system includes 16 second inverse butterfly networks. Under this condition, if the clock frequency or other parameters related to the data processing rate of the tensor sparse system and the weight sparse system are the same, then in the same time period, only two sets of sparse tensors can be obtained through the tensor sparse system and 16 sets of sparse weights can be obtained through the weight sparse system.

[0050] Alternatively or additionally, the first clock frequency of the tensor sparse system can be lower than the second clock frequency of the weighted sparse system. That is, assuming other parameters related to data processing speed, such as the number of inverse butterfly networks, remain constant, the tensor sparse system will perform sparse processing much slower than the weighted sparse system due to its lower clock frequency. Therefore, in the same amount of time, the tensor sparse system generates fewer sparse tensors than the weighted sparse system generates sparse weights. This is also related to the characteristic mentioned earlier that the tensor data size in neural network systems is much smaller than the weight data size. Because the tensor data size is small, the tensor sparse system does not require a high clock frequency; conversely, the weighted sparse system requires a high clock frequency to quickly process large-scale dense weights. For example, if the first clock frequency of the tensor sparse system is 800MHz and the second clock frequency of the weighted sparse system is 1600MHz, and both have a data path width of 512 bits, then the tensor sparse system can generate a total of 800 × 10^6 bits of sparse tensors per second. 6 ×512 = 409600000000 bits. The total number of bits that a weighted sparse system can generate for sparse weights per second is 1600 × 10^512. 6 ×512=819200000000 bits. It can be seen that, since the second clock frequency of the weighted sparse system is twice the first clock frequency of the tensor sparse system, the weighted sparse system processes twice the amount of data per second as the tensor sparse system.

[0051] Additionally or alternatively, the data path width of a tensor sparse system is narrower than that of a weighted sparse system. Here, data path width refers to the total number of binary data bits that an internal chip's processing unit can process in parallel within a single clock cycle. In other words, assuming the number of inverse butterfly networks or clock frequency and other parameters related to data processing rate remain constant, the tensor sparse system will process less data in the same amount of time compared to a weighted sparse system due to its narrower data path width. This is also related to the characteristic mentioned earlier that the tensor data size in neural network systems is much smaller than the weight data size. Because the tensor data size is small, tensor sparse systems do not require excessively wide data path widths; conversely, weighted sparse systems require wide data path widths to process more weights in parallel. For example, if the first clock frequency of the tensor sparse system and the second clock frequency of the weighted sparse system are both 1 GHz, the data path width of the tensor sparse system is 256 bits, and the data path width of the weighted sparse system is 1024 bits, then the total number of bits that the tensor sparse system can generate per second for sparse tensors is: 1000 × 10⁻⁶. 6 ×256 = 256000000000 bits. The total number of bits that a weighted sparse system can generate for sparse weights per second is 1000 × 10. 6 ×1024 = 1024000000000 bits. It can be seen that, at the same clock frequency, the amount of data generated by a tensor sparse system is only 1 / 4 of the amount of data generated by a weighted sparse system.

[0052] The design of tensor sparse systems and weighted sparse systems can consider one or more of the above parameters. Those skilled in the art can also consider other parameters related to the data processing rate of sparse systems, such as coding ratio, number of transmit / receive channels, etc. These parameters can all replace or be combined with the foregoing in the design of tensor sparse systems and weighted sparse systems, and this disclosure does not impose any limitations. The aforementioned design for the performance of tensor sparse systems and weighted sparse systems effectively utilizes the data scale and computational characteristics of combining weights and tensors in neural networks, allowing for reasonable allocation of hardware resources, efficient use of computing power, and reduced chip power consumption.

[0053] In some respects, the cache capacity of a tensor sparse system is smaller than that of a weighted sparse system. As mentioned earlier, since a tensor sparse system generates fewer sparse tensors, it requires only a smaller cache capacity to cache the generated sparse tensors. Conversely, a weighted sparse system requires a larger cache capacity to cache the generated sparse weights. Furthermore, in some embodiments of subsequent step 103, the sparse weights from the weighted sparse system can be directly sent to the multiply-accumulate array for computation, thus requiring a larger cache capacity to store all generated sparse weights. However, a tensor sparse system can cache the generated sparse tensors in a shared data cache, therefore a smaller cache capacity is sufficient for its use.

[0054] In step 103, the sparse weights from the sparse weight system and the sparse tensors from the shared data buffer are sent to the multiplication-accumulation array for calculation to obtain the output of the neural network.

[0055] The multiply-accumulate array here can perform multiply-accumulate operations on sparse weights and sparse tensors, i.e., MAC (Multiply-accumulate) operations. For example, first sparse weight × first sparse tensor + second sparse weight × second sparse tensor, and so on. For example, if sparse weight = [1,2,3,4,5,6,7,8] and sparse tensor = [2,5,7,9], the multiply-accumulate array can perform 4-way parallel computation, resulting in the multiply-accumulate calculation 1×2 + 2×5 + 3×7 + 4×9 + 5×2 + 6×5 + 7×7 + 8×9. The above is only an example of multiply-accumulate array operation and is not intended to limit the embodiments of this disclosure. Those skilled in the art can configure sparse weights and sparse tensors to appropriate dimensions according to actual conditions and adaptively adjust the specific computation method of the multiply-accumulate array.

[0056] In some embodiments, the second clock frequency of the weighted sparse system and the third clock frequency of the multiply-accumulate array can be the same. That is, the clock cycles of the weighted sparse system and the multiply-accumulate array are exactly the same. At this clock frequency, the sparse weights generated by the weighted sparse system in each clock cycle can be received by the multiply-accumulate array and multiplied and accumulated in the corresponding clock cycles of the same length. Using the above method, sparse processing of weights and multiply-accumulate array calculations based on sparse weights can be implemented in the same pipeline control. Therefore, there will be no situation where sparse weights are generated too quickly, while the multiply-accumulate array has not yet finished calculating all the data, which would cause a large number of sparse weights to queue in the buffer of the weighted sparse system, or even overflow the buffer of the weighted sparse system; nor will there be a situation where sparse weights are generated too slowly, while the multiply-accumulate array has already finished calculating all the data, which would lead to a waste of the computing power of the multiply-accumulate array.

[0057] In some examples, the computation of the multiply-accumulate array in the third clock cycle corresponding to the third clock frequency is based on a sparse tensor of the first data volume and sparse weights of the second data volume; the weighted sparse system obtains the sparse weights of the second data volume in the second clock cycle corresponding to the second clock frequency; the sparse tensor of the first data volume is sent from the shared data buffer to the multiply-accumulate array before computation. When the second clock frequency of the weighted sparse system is the same as the third clock frequency of the multiply-accumulate array, the second clock cycle is the same as the third clock cycle.

[0058] In this scenario, the amount of sparse weights sent from the weighted sparse system to the multiply-accumulate array and the amount of sparse tensors sent from the shared data buffer to the multiply-accumulate array match the computational power of the multiply-accumulate array in one clock cycle. For example, the multiply-accumulate array's computation in the third clock cycle is based on 4 sets of sparse tensors and 16 sets of sparse weights. Correspondingly, the weighted sparse system can generate 16 sets of sparse weights in the same clock cycle. Before this computation, the tensor sparse system has already generated 4 sets of sparse tensors and stored them in the tensor sparse system's buffer or shared data buffer. Therefore, when the multiply-accumulate array receives 16 sets of weights, it can retrieve 4 sets of sparse tensors from the tensor sparse system's buffer or shared data buffer for multiply-accumulate computation. In this way, the computational power of the multiply-accumulate array can be fully utilized, improving computational efficiency.

[0059] In other examples, before each computation of the multiply-accumulate array, the weighted sparse system may have already generated the sparse weights of the second data volume required for this computation. The generated sparse weights of the second data volume can be stored in the weighted sparse system's cache or a shared data cache. Using this method, the second clock frequency of the weighted sparse system and the third clock frequency of the multiply-accumulate array can also be different. Therefore, those skilled in the art can flexibly configure the clock frequencies within each system according to actual needs.

[0060] Figure 3 An exemplary architecture 300 for a data processing method for a neural network according to embodiments of the present disclosure is illustrated. Figure 3 The architecture 300 can execute the data processing methods for neural networks described above, which will not be repeated here. Additionally, from... Figure 3As can be seen, the density of the tensor and the density of the weights are separated into tensor sparse system 301 and weight sparse system 303. The sparse tensors obtained using tensor sparse system 301 are cached in shared data cache 302, and the sparse weights obtained using weight sparse system 303 are cached in weight sparse system 303. The sparse weights and sparse tensors will undergo multiplication and accumulation calculations in multiplication and accumulation array 304 to further obtain the output of the neural network.

[0061] Below, for reference Figure 4 To describe a data processing apparatus 400 for a neural network according to an embodiment of the present disclosure.

[0062] The apparatus 400 includes a tensor sparse unit 401, a weighted sparse unit 402, and a result generation unit 403. In addition to these units, the apparatus 400 may also include other components; however, since these components are not relevant to the content of this disclosure embodiment, their illustrations and descriptions are omitted herein. Furthermore, the specific details of the operations performed by the apparatus 400 according to this disclosure embodiment are consistent with those referenced above. Figure 1 The details described are the same, so repeated descriptions of the same details are omitted here to avoid repetition.

[0063] The apparatus 400 includes a tensor sparsity unit 401 configured to sparsify one or more dense tensors retrieved from a shared data buffer using a tensor sparsity system to obtain corresponding sparse tensors, which are cached in the shared data buffer.

[0064] The apparatus 400 includes a weighted sparse unit 402 configured to use a weighted sparse system to sparse one or more dense weights retrieved from a shared data cache to obtain corresponding sparse weights, wherein the sparse weights are cached in the weighted sparse system, wherein the data processing rate of the tensor sparse system is lower than that of the weighted sparse system.

[0065] The apparatus 400 includes a result generation unit 403, configured to feed sparse weights from a weighted sparse system and sparse tensors from a shared data buffer to a multiply-accumulate array for computation to obtain the output of the neural network.

[0066] Figure 5 An example computer device 500 is shown in which any of the embodiments described herein may be implemented. The computer device 500 may be used to implement one or more components of the systems and methods described above. The computer device 500 may include a bus 502 or other communication mechanism for communicating information, and one or more processors 504 coupled to the bus 502 for processing information. The processor 504 may be, for example, one or more general-purpose microprocessors.

[0067] Computer device 500 may also include main memory 506, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 502, for storing information and instructions to be executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 504. Such instructions, when stored in a storage medium accessible to processor 504, can make computer device 500 a special-purpose machine customized to perform the operations specified in the instructions. Main memory 506 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common media formats may include, for example, floppy disks, collapsible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs (read-only optical disc drives), any other optical data storage media, any physical media with a perforated arrangement, RAM (random access memory), DRAM (dynamic random access memory), PROM (programmable read-only memory) and EPROM (erasable programmable read-only memory), FLASH-EPROM (fast erase programmable read-only memory), NVRAM (non-volatile random access memory), any other memory chips or tape cartridges, or network versions of the above.

[0068] Computer device 500 may implement the techniques described herein using custom hardwired logic, one or more ASICs (Application-Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays), firmware, and / or program logic, which, when combined with computer device 500, enable computer device 500 to become a special-purpose machine or to be programmed therein. According to one embodiment, the techniques described herein are executed by computer device 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 508. Executing the sequence of instructions contained in main memory 506 causes processor 504 to perform the processing steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in main memory 506. When these instructions are executed by processor 504, they may perform the steps shown in the corresponding figures and as described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0069] Computer device 500 also includes a network interface 510 coupled to bus 502. Network interface 510 can provide bidirectional data communication coupled to one or more network links connected to one or more networks. As another example, network interface 510 can be a local area network (LAN) card to provide data communication connectivity with a compatible LAN (or a WAN component communicating with a WAN (wide area network)). Wireless links can also be implemented.

[0070] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0071] Each process, method, and algorithm described in the preceding sections can be embodied in a code module executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by them. These processes and algorithms can be implemented, in part or in whole, in a specific application circuit.

[0072] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, can be embodied in the form of a software product. This software product can be stored in a storage medium and includes instructions to cause a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods described in the embodiments of this application. The storage medium may include a flash drive, a portable hard drive, ROM, RAM, a magnetic disk, an optical disk, another medium suitable for storing program code, or any combination thereof.

[0073] The embodiments disclosed herein can be implemented via a cloud platform, server, or group of servers that interact with a client. The client can be a terminal device or a client registered by a user on the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can install platform applications.

[0074] The various features and processes described above can be used independently or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in a non-specifically disclosed order, or multiple blocks or states may be combined in a single block or state. Exemplary blocks or states may be executed serially, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed exemplary embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed exemplary embodiments.

[0075] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. An algorithm may consist of program code or instructions stored in memory (such as the non-transitory computer-readable storage medium described above). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not be explicitly programmed into the computer to perform the function, but may learn from training data to obtain a predictive model for performing that function.

[0076] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute the engine of a processor implementation whose operation is to perform one or more of the operations or functions described herein.

[0077] Similarly, the methods described herein can be implemented at least partially by a processor, where a specific processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also run in a “cloud computing” environment or as “Software as a Service” (SaaS) to support the execution of the relevant operations. For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), which can be accessed via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0078] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0079] The following describes a computer-readable storage medium according to the present disclosure, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the aforementioned data processing method for a neural network.

[0080] The following describes a computer program product according to the present disclosure, which includes a computer program that, when executed by a processor, causes the processor to perform the aforementioned data processing method for a neural network.

[0081] In this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are described and illustrated as independent operations, one or more individual operations may be performed concurrently, and these operations are not required to be performed in the order shown. Structures and functionalities presented as independent components in the example configuration may be implemented as combined structures or components. Similarly, structures and functionalities presented as individual components may be implemented as independent components. These and other variations, modifications, additions, and improvements are all within the scope of this document.

[0082] As used herein, “or” is inclusive rather than exclusive unless explicitly stated or indicated by context. Furthermore, “and” is both common and individual unless explicitly stated or indicated by context. Moreover, multiple instances may be provided for the resources, operations, or structures described herein as a single example. Furthermore, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and specific operations are illustrated within the context of a particular illustrative configuration. The allocation of other functionalities is conceivable and may fall within the scope of various embodiments of this disclosure. Generally, structures and functionalities presented as independent resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functionalities presented as individual resources may be implemented as independent resources. These and other variations, modifications, additions, and improvements are all within the scope of embodiments of this disclosure. Therefore, this specification and accompanying drawings should be viewed in an illustrative rather than restrictive sense.

[0083] The terms “comprising” or “including” are used to indicate the presence of a subsequently stated feature, but do not preclude the addition of other features. Conditional language, in particular, such as “may,” “can,” or “may,” unless specifically stated or otherwise understood in the context of use, is generally intended to express that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Therefore, such conditional language generally does not imply that a feature, element, and / or step is necessary in any way for one or more embodiments, or that one or more embodiments must include logic that, with or without user input or prompting, determines whether such features, elements, and / or steps are included in any particular embodiment, or whether they are to be performed in any particular embodiment.

Claims

1. A data processing method for neural networks, characterized in that, The method includes: A tensor sparsity system is used to sparsify one or more dense tensors retrieved from a shared data buffer to obtain corresponding sparse tensors, which are cached in the shared data buffer. A weighted sparse system is used to sparse one or more dense weights retrieved from the shared data cache to obtain corresponding sparse weights. The sparse weights are cached in the weighted sparse system. The data processing rate of the tensor sparse system is lower than that of the weighted sparse system. The sparse weights from the weighted sparse system and the sparse tensors from the shared data buffer are fed into a multiplication-accumulation array for calculation to obtain the output of the neural network.

2. The method according to claim 1, characterized in that, The tensor sparse system includes one or more first inverse butterfly networks to sparse the dense tensors through the first inverse butterfly networks; The weighted sparse system includes one or more second inverse butterfly networks to sparse the dense weights through the second inverse butterfly networks.

3. The method according to claim 2, characterized in that, The process of sparsifying the dense tensor using the first inverse butterfly network includes: The first inverse butterfly network is used to remove zero data from the dense tensor, thereby aggregating the non-zero tensor data in the dense tensor into continuous data for output; and The dense weights are sparsified using the second inverse butterfly network, including: The second inverse butterfly network removes zero data from the dense weights, thereby aggregating the non-zero weight data in the dense weights into continuous data for output.

4. The method according to claim 2, characterized in that, The data processing rate of the tensor sparse system is lower than that of the weighted sparse system by at least one of the following: The number of the first inverse butterfly network in the tensor sparse system is less than the number of the second inverse butterfly network in the weighted sparse system; The first clock frequency of the tensor sparse system is lower than the second clock frequency of the weighted sparse system. and The data path width of the tensor sparse system is narrower than that of the weighted sparse system.

5. The method according to claim 1, characterized in that: The cache capacity of the tensor sparse system is smaller than that of the weighted sparse system.

6. The method according to claim 1, characterized in that: The second clock frequency of the weighted sparse system is the same as the third clock frequency of the multiply-accumulate array.

7. The method according to claim 6, characterized in that: The calculation of the multiply-accumulate array in the third clock cycle corresponding to the third clock frequency is based on the sparse tensor of the first data quantity and the sparse weight of the second data quantity. The weighted sparse system obtains the sparse weight of the second data in the second clock cycle corresponding to the second clock frequency, wherein the second clock cycle is the same as the third clock cycle. Before the multiply-accumulate array performs calculations, the sparse tensor of the first data volume is sent from the shared data buffer to the multiply-accumulate array.

8. A data processing apparatus for neural networks, characterized in that, The device includes: Tensor sparsity unit is configured to use a tensor sparsity system to sparsify one or more dense tensors retrieved from a shared data buffer to obtain corresponding sparse tensors, which are cached in the shared data buffer. A weighted sparse unit is configured to use a weighted sparse system to sparse one or more dense weights retrieved from the shared data cache to obtain corresponding sparse weights, wherein the sparse weights are cached in the weighted sparse system, wherein the data processing rate of the tensor sparse system is lower than that of the weighted sparse system. The result generation unit is configured to send the sparse weights from the weighted sparse system and the sparse tensors from the shared data buffer to a multiplication-accumulation array for calculation to obtain the output result of the neural network.

9. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-7.

Images