Neural network acceleration device, control method and electronic equipment

Abstract

The invention discloses a neural network acceleration device, a control method and electronic equipment, and the device comprises a memory module which is used for storing activation data, calculation result data and weight data; and the control module is used for dynamically adjusting the storage format of the weight data in the memory module according to the matching condition of the number of the input channels and the parallelism degree of the input channels, and controlling the reading mode of the activation data and the rearrangement mode of the activation data and the weight data based on the adjusted storage format. And / or dynamically adjusting the access bit width of the activation data according to the matching condition of the output channel number and the output channel parallelism degree; and the processing unit array is used for carrying out multiply-accumulate calculation on the activation data and the weight data output by the control module, and writing calculation result data obtained by calculation into the memory module. Therefore, through cooperative configuration of hardware modules, the problem that the MAC utilization rate is low due to the fact that the data dimension and the hardware parallelism degree are not matched is effectively solved, and the convolution calculation and matrix multiplication energy efficiency is remarkably improved.

Description

Technical Field

[0001] This application relates to the field of neural network computing acceleration technology, and more particularly to a neural network acceleration device, control method, and electronic device. Background Technology

[0002] In deep learning and high-performance computing, convolution and matrix multiplication are the core computational operations, and their execution efficiency directly impacts the overall system performance. These operations are typically implemented in hardware using parallel computing unit arrays (such as MAC arrays), and their efficiency depends on the utilization rate of these computing units. In practical applications, the number of input / output channels for the data to be processed is often related to the fixed parallelism of the hardware design (e.g., input channel parallelism P). c Output channel parallelism P f The mismatch causes some computing units to be idle in each cycle, resulting in a significant reduction in utilization.

[0003] Currently, the industry has proposed several technical solutions to alleviate the above problems. For example: Applechip Technology's patent (CN202311575161.9) discloses a general convolution implementation method. Based on a defined MAC array structure, it improves utilization by marking the position of the effective convolution kernel within the maximum supported convolution kernel, starting from the kernel dimension. However, this method does not optimize from the channel dimension, and the optimization effect is limited by the preset convolution kernel size, resulting in limited applicability.

[0004] The patent solution proposed by Xi'an Jiaotong University (CN202210924138.5) improves MAC utilization by partitioning the block memory used to store feature maps to store feature maps from different layers in a multi-layer neural network, thereby reducing the number of accesses to external memory. However, this solution optimizes from a macroscopic level of network structure and does not address the microscopic problem of the mismatch between the number of channels and parallelism in specific convolutional layers; therefore, its effect on improving MAC utilization is also limited.

[0005] Baidu's patent (CN202011287840.2) involves a data format transformation method to improve the utilization of the MAC (Macroscopic Convolutional Layer) in an AI engine. This method transforms the data format of the input data (such as RGB images) of the first layer of a neural network to better match the channel orientation of the MAC array structure. However, this solution primarily targets the first layer input, optimizing only the channel dimension and failing to address adjustments to the convolutional kernel dimension. It lacks effective optimization methods for the more common mid-channel convolutional layers in networks.

[0006] In summary, most existing technical solutions only perform local optimization from a single dimension (such as convolution kernel, memory architecture, or data format), lacking a systematic solution that can simultaneously and adaptively handle the mismatch between input and output channels and hardware parallelism, making it difficult to continuously achieve high computing unit utilization in various network layers and computational structures. Summary of the Invention

[0007] The purpose of this application is to provide a neural network acceleration device, control method, and electronic device to solve the problem of decreased data processing utilization caused by the mismatch between the number of input / output channels and the hardware parallelism. To achieve the above objective, the neural network acceleration device provided in this application includes: a memory module for storing activation data, calculation result data, and weight data; a control module for controlling the reading method of the activation data according to the matching of the number of input channels and the parallelism of the input channels; and / or dynamically adjusting the storage format of the weight data in the memory module, and controlling the reading method of the activation data and the rearrangement method of the activation data and weight data based on the adjusted storage format; and a processing unit array for performing multiplication and accumulation calculations on the activation data and weight data output by the control module, and writing the calculated result data into the memory module.

[0008] In one embodiment of this application, the memory module includes an activation memory and a weight memory; the activation memory is used to store activation data and calculation result data; the weight memory is used to store weight data.

[0009] In one embodiment of this application, the control module includes a reading unit and a buffer unit; the reading unit includes an activation reading unit and a weight reading unit, which are respectively used to read the activation data and the weight data from the activation memory and the weight memory; the buffer unit includes an activation buffer and a weight buffer, which are respectively connected to the activation reading unit and the weight reading unit, and are used to cache and rearrange the activation data and the weight data.

[0010] In one embodiment of this application, when the number of input channels is an integer multiple of the parallelism of the input channels, the control module controls the activation reading unit to read the activation data in a multi-step manner to match the amount of weight data read by the weight reading unit.

[0011] In one embodiment of this application, when the number of input channels is less than the parallelism of the input channels, the control module selects different storage format optimization strategies according to the size of the number of input channels.

[0012] In one embodiment of this application, when the number of input channels is half of the parallelism of the input channels, the control module controls the activation memory to store half of the channel data of two points in the width direction at one address, and controls the weight memory to store weight data eight times that of half of the channels at one address.

[0013] In one embodiment of this application, when the number of input channels is less than half of the input channel parallelism and the product of the number of input channels and the width and height of the convolution kernel does not exceed the input channel parallelism, the control module controls the weight memory to store the weight data in a way that concatenates the input channels with the width and height dimensions of the convolution kernel.

[0014] In one embodiment of this application, when the number of input channels is less than half of the input channel parallelism and the product of the number of input channels and the convolution kernel width does not exceed the input channel parallelism, the control module controls the weight memory to store the weight data in a way that concatenates the input channels and the convolution kernel width dimensions.

[0015] In one embodiment of this application, when the number of input channels is greater than the parallelism of the input channels but is not an integer multiple thereof, the control module splits the input channels into a regular part and a remaining part for separate processing.

[0016] In one embodiment of this application, the activation memory adopts a multi-memory bank structure, with each memory bank divided into multiple memory spaces. Each memory space has an independent read / write enable signal and is organized using a low-order interleaved addressing method.

[0017] In one embodiment of this application, the control module dynamically adjusts the access bit width of the activation data by controlling the read / write enable of multiple storage spaces of the activation memory according to the matching of the number of output channels and the parallelism of the output channels.

[0018] In one embodiment of this application, when the number of output channels is less than the parallelism of the output channels, the control module increases the amount of active data read in a single cycle by simultaneously enabling read operations of multiple storage spaces.

[0019] In one embodiment of this application, when the number of output channels is greater than the parallelism of the output channels but is not an integer multiple thereof, the control module splits the output channels into a regular part and the remaining part for separate processing.

[0020] In one embodiment of this application, the activation buffer performs a corresponding rearrangement operation on the activation data according to the storage format of the weight data and the convolution calculation method.

[0021] In one embodiment of this application, the ratio of the activation data bit width to the weight data bit width set by the control module is one to four.

[0022] This application also provides a control method for a neural network acceleration device, comprising: controlling the reading mode of the activation data in response to a mismatch between the number of input channels and the parallelism of the input channels of the processing unit; and / or dynamically adjusting the storage format of the weight data and the corresponding reading mode of the activation data, as well as the rearrangement mode of the activation data and the weight data; and / or dynamically adjusting the access bit width of the activation data in response to a mismatch between the number of output channels and the parallelism of the output channels of the processing unit.

[0023] In one embodiment of this application, the input data of the neural network acceleration device includes the input feature map and weight data for convolution operation, or the input left matrix and input right matrix for matrix multiplication operation.

[0024] This application also provides a chip that includes the aforementioned neural network acceleration device.

[0025] This application also provides a board card including the aforementioned chip.

[0026] This application also provides an electronic device having the aforementioned board or chip deployed thereon.

[0027] The beneficial technical effects of this application are as follows: by dynamically adjusting the data storage format and access bit width in a responsive manner, the processing unit array can still achieve a utilization rate close to or reaching 100% even when the data dimension and hardware parallelism are not regularly matched, greatly reducing idle computing resources; from the perspective of input and output channels, it can effectively cope with various irregularly configured operation layers (such as convolution channels and matrix dimensions), and has strong versatility; by adopting a low-bit interleaved addressing storage structure and corresponding data rearrangement strategies, it improves computing efficiency while reducing the number of memory accesses and power consumption, achieving a balance between performance and energy efficiency.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the present application. Other features and aspects of the present application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating the storage format of dynamically adjusted weight data and the corresponding activation data reading method provided in an embodiment of this application. Figure 2 This is a schematic diagram of the application process of a data processing utilization improvement method provided in an embodiment of this application. Figure 3 This is a schematic diagram of the structure of a neural network acceleration device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the architecture of a convolutional module provided in an embodiment of this application; Figure 5 This is a schematic diagram of the SPM-A and SPM-W storage formats under regular convolution provided in an embodiment of this application; Figure 6 A schematic diagram of a closely packed active storage format in SPM-A provided in an embodiment of this application; Figure 7 This is a schematic diagram of the weight storage format for concatenating C, Kx, and Ky in SPM-W provided in an embodiment of this application; Figure 8 This is a schematic diagram of the weight storage format for concatenating C and Kx in SPM-W provided in an embodiment of this application; Figure 9 A schematic diagram of the low-order cross-addressing storage structure adopted in the SPM-A storage cell bank provided in an embodiment of this application; Figure 10 This is a schematic diagram of the SPM-A and SPM-W storage formats provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0031] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0032] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0033] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed description. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0034] This application is mainly applied in neural network accelerators. When the input or output channels of the convolutional layers or matrix operations in the neural network are not equal to the parallelism of the input or output channels of the accelerator, this method can significantly improve the utilization rate of the MAC or matrix operation units of the accelerator, thereby improving the computing speed.

[0035] When neural network accelerators are used to process AI-ISP algorithm networks, convolution operations account for more than 95% of the total computation in the AI-ISP algorithm network, and the number of input and output channels of the convolutional layers is mostly less than P. c and P f Using this application, the MAC utilization rate of the convolutional layer, which was originally 50% or even 25%, can be increased to 100%, significantly improving the performance of the AI-ISP algorithm network.

[0036] Please refer to Figure 3 As shown, the neural network acceleration device provided in this application includes: a memory module for storing activation data, calculation result data, and weight data; a control module for controlling the reading method of the activation data according to the matching of the number of input channels and the parallelism of the input channels; and / or dynamically adjusting the storage format of the weight data in the memory module, and controlling the reading method of the activation data and the rearrangement method of the activation data and weight data based on the adjusted storage format; and a processing unit array for performing multiplication and accumulation calculations on the activation data and weight data output by the control module, and writing the calculated result data into the memory module.

[0037] The reading method refers to the number of read cycles for the activation data, such as reading the activation data in four cycles, eight cycles, etc. The specific number of cycles to read the activation data will be explained in detail in subsequent embodiments. The matching conditions include: the number of input channels is the same as the number of input channels, the number of input channels is an integer multiple of the number of input channels, and the number of input channels is greater than or less than the number of input channels. Different processing logics are adopted for different matching conditions, and the specific implementation methods will be described in detail in subsequent embodiments, and will not be described in detail here.

[0038] In the above embodiments, the memory module includes an activation memory and a weight memory; the activation memory is used to store activation data and calculation result data; the weight memory is used to store weight data. Further, the control module includes a reading unit and a buffer unit; the reading unit includes an activation reading unit and a weight reading unit, respectively used to read the activation data and the weight data from the activation memory and the weight memory; the buffer unit includes an activation buffer and a weight buffer, the activation buffer and the weight buffer being connected to the activation reading unit and the weight reading unit respectively, used to cache and rearrange the activation data and the weight data. To facilitate a clearer understanding of the implementation structure and logic of the neural network acceleration device provided in this application, the following uses a convolutional neural network accelerator as an example to illustrate the neural network acceleration device of this application. Those skilled in the art will understand that this example is only one possible implementation of this application and does not limit the neural network acceleration device provided in this application in any way.

[0039] For details, please refer to Figure 4 As shown, the neural network acceleration device provided in this application includes: The Activation Memory (SPM-A) is used to store activation data and calculation results. The weight memory (SPM-W) is used to store weight data; The control module (Ctrl) includes an activation read unit (act_read) and a weight read unit (wgt_read), which are used to read data from the activation memory and the weight memory, respectively. The activation buffer (act_buffer) and weight buffer (wgt_buffer) are used to cache and rearrange the data received from the control module; The processing unit array (PE computing array) is used to perform multiplication and accumulation operations on the rearranged activation data and weight data; The control module dynamically adjusts the storage format of the weight data and the reading method of the activation data based on the matching of the number of input channels and the parallelism of the input channels of the data to be processed.

[0040] In practical applications, when neural network acceleration devices are used to accelerate convolutional operations, the convolutional module is mainly used to perform convolutional operations on activations and weights. A classic convolutional module architecture is as follows: Figure 4As shown, SPM-A stores activation data and calculation result data, while SPM-W stores weight data. The `act_read` and `wgt_read` functions in the Ctrl module are responsible for reading activation and weight data from SPM-A and SPM-W, respectively, and then sending the data to `act_buffer` and `wgt_buffer`. The `act_buffer` in the buffer rearranges the activation data according to the requirements of the PE calculation array and sends it to the PE calculation array for calculation. The `wgt_buffer` also performs corresponding operations. The PE calculation array performs multiplication and accumulation operations on the input activation and weight data, and finally writes the calculation result back to SPM-A.

[0041] Taking convolution as an example, let the dimensions of input activation be HI, WI, and C; the dimensions of input weights be Ky, Kx, F, and C; the dimensions of output activation be HO, WO, and F; and the input channel parallelism of convolution computation be P. c The parallelism of the output channels is P. f P c = P f The number of MACs is P f ×P c , where P c P f Generally, it is a power of 2. The least-multiply accumulation unit for the convolution operation between activations and weights in the PE computation array is P in the C direction. c / 2 numbers. When the number of input channels C of the convolution is less than the input channel parallelism P. c At that time, without special processing, the convolution will follow P c The MAC utilization of convolution will also become C / P. c < 100%, when C is smaller or P c As the number of input channels C increases, the MAC utilization will decrease; and when the number of input channels C of the convolution is greater than the input channel parallelism P... c If C is not P c When the value is an integer multiple of , the MAC utilization of convolution still does not reach 100%, and the MAC utilization is C / ( C / P c ×P c < 100%.

[0042] Similarly, without special processing, when the number of output channels F of the convolution is less than the parallelism of the output channels P... f Or even if the number of output channels F is greater than the parallelism of output channels P f And F is not P fEven when the number of input channels C is an integer multiple of the input channel parallelism P, the MAC utilization rate of convolution still does not reach 100%. A MAC utilization rate below 100% increases the inference latency of the algorithm network, thus reducing its performance. This application addresses the pain point of the convolution MAC utilization rate not reaching 100%, explaining how to improve the convolution MAC utilization rate from both input and output channel perspectives, thereby improving the algorithm network performance. Specifically, when the number of input channels C is an integer multiple of the input channel parallelism P... c The number of output channels F is an integer multiple of the number of output channels P. f When the input channel number C and the output channel number F are not integer multiples of P, this application adopts a scheme of reading multi-phase activation data to match the weighted data with a larger data volume. c and P f When the number of input channels to be processed is an integer multiple of the number of input channels to be processed, this application starts from two dimensions: input channels and output channels. When the number of input channels to be processed does not match the parallelism of the input channels of the processing unit, the storage format of the weight data and the corresponding activation data reading method are dynamically adjusted. When the number of output channels to be processed does not match the parallelism of the output channels of the processing unit, the access bit width of the activation data is dynamically adjusted, thereby improving the performance of the algorithm network. The specific implementation logic will be described in detail in subsequent embodiments, and will not be described in detail here.

[0043] In one embodiment of this application, when the number of input channels is an integer multiple of the parallelism of the input channels, the control module controls the activation reading unit to read the activation data in a multi-step manner to match the amount of weight data read by the weight reading unit.

[0044] In practical applications, when the number of input channels C of a convolutional layer is equal to the input channel parallelism P... c The number of output channels F is an integer multiple of the number of output channels P. f When the value is an integer multiple of the integer value, this convolution is called a regular convolution. Under regular convolution, one address of SPM-A stores P. c The data is stored at one address of SPM-W, which holds 4×P. c For data storage format, see [link / reference]. Figure 5 In P f Within 4 cycles, the convolutional module can read P f ×P c The weight data can be read in the W direction P f / 4 points P c Activation data is fed into the PE array for convolution calculations, ultimately resulting in P... f Within 4 cycles, the PE computing array completes P f / 4 points P f ×P c The calculation of the data is equivalent to each cycle completing P.f ×P c Data calculation, P f ×P c All MAC units participate in the operation. Then the convolution traverses Kx, Ky, C, F, WO, and HO according to the same rules. All MAC units participate in the operation during the entire calculation process, so the MAC utilization rate of this convolution is 100%.

[0045] In one embodiment of this application, when the number of input channels is less than the parallelism of the input channels, the control module selects different storage format optimization strategies according to the size of the number of input channels.

[0046] Furthermore, this storage format optimization strategy may include the following strategies, such as: When the number of input channels is half of the parallelism of the input channels, the control module controls the activation memory to store half of the channel data of two points in the width direction at one address, and controls the weight memory to store eight times the weight data of half of the channels at one address.

[0047] When the number of input channels is less than half of the input channel parallelism and the product of the number of input channels and the width and height of the convolution kernel does not exceed the input channel parallelism, the control module controls the weight memory to store the weight data in a way that concatenates the input channels with the width and height dimensions of the convolution kernel.

[0048] When the number of input channels is less than half of the input channel parallelism and the product of the number of input channels and the convolution kernel width does not exceed the input channel parallelism, the control module controls the weight memory to store the weight data in a way that concatenates the input channels and the convolution kernel width dimensions.

[0049] Specifically, when the number of input channels C of the convolutional layer is only equal to the input channel parallelism P... c When it is half of the value, that is, C=P c / 2. This application improves MAC utilization by modifying the SPM data storage format and reducing the number of data read cycles. This scheme eliminates bubbles in the SPM, ensuring that weight data and activation data are closely arranged. At this point, one address of SPM-A stores P for two points in the W direction. c / 2 of the data, SPM-W stores 8×P at one address. c / 2 of the data, in P f Within 8 cycles, the convolutional module can read P f ×P c The weight data of / 2 allows reading the P direction in the W direction. f / 4 points P c / 2 activation data, which is then fed into the PE array for convolution computation. Since the minimum multiplication and accumulation unit for convolution operations between activations and weights in the PE computation array is P in the C direction... c / 2 numbers, so the final value in P f Within 8 cycles, the PE computing array can complete P f / 4 points P f ×P c The calculation of / 2 data is equivalent to each cycle completing P. f ×P c Data calculation, P f ×P c All MAC units participate in the operation, so the measures taken in this application can also achieve 100% MAC utilization in this case.

[0050] When the number of input channels C of the convolutional layer is less than P c When / 2, since the smallest multiplication and accumulation unit in the PE calculation array is P c / 2, therefore, the MAC utilization rate will definitely be less than 100% according to the above method. Since the data in the C direction and the data in the Kx and Ky directions all participate in multiplication and accumulation operations during the convolution calculation process, this application adopts a scheme to improve the MAC utilization rate by modifying the weight storage format and the convolution calculation method (the convolution calculation method is to concatenate C with Kx or C with Kx, Ky, etc.). At the same time, the activation data is also closely arranged; please refer to the storage format of the activation data. Figure 6 As shown.

[0051] When the number of input channels C and the kernel size of the convolutional layer are both small, that is, when C×Kx×Ky <= P c In this application, the weight data in SPM-W is modified to a storage method that concatenates C with Kx and Ky, such as... Figure 7 As shown, the weight data of Kx1C, Kx2C, ..., KxmC, ..., KynC are stored in the original P. c This storage method eliminates bubbles in SPM-W, saving storage space, and reduces the number of cycles read by `wgt_read`, thus saving power. Since the activation data used for convolution with weights from different kernels is different, and there is reuse of activation data, the activation data read by `act_read` needs to be rearranged according to the storage format of the weight data. `act_buffer` caches the activation data and rearranges it according to the current weight storage format and convolution calculation method. With this optimization, the MAC utilization rate of the convolution calculation is (C×Kx×Ky) / P. c As C×Kx×Ky gets closer to P cAt that time, the higher the MAC utilization, the better compared to unoptimized C / P. c The optimization scheme improved MAC utilization by Kx×Ky times.

[0052] When the number of input channels C of the convolutional layer and the kernel size do not satisfy C×Kx×Ky <= P c But it satisfies C×Kx<= P c In this application, the weight data in SPM-W is modified to a storage method that concatenates C and Kx, such as... Figure 8 As shown, the weight data of Kx1C, Kx2C, ..., KxmC are stored in the original P. c At the same position, the `act_buffer` caches the activation data and rearranges it according to the current weight storage format and convolution calculation method. In this example, the `act_buffer` concatenates W1C, W2C, ..., WmC from the activation data, occupying W1P. c The storage space, and according to the original W1P c The data stream is convolved with F1(Kx1~Kxm)C, F2(Kx1~Kxm)C, ..., P respectively. f The convolution operation is performed using weights of (Kx1~Kxm)C. Following this optimization method, the MAC utilization rate of the convolution computation is (C×Kx) / P. c As C×Kx gets closer to P c At that time, the higher the MAC utilization, the better compared to unoptimized C / P. c The optimization scheme improved MAC utilization by Kx times.

[0053] Overall, when the number of input channels is less than the parallelism, different optimization methods are adopted based on the number of channels. If the number of channels is small, a dimension concatenation and data rearrangement strategy is used, concatenating the input channel dimensions with the convolution kernel space dimensions for storage, while rearranging the activation data accordingly. If the number of channels is large but still less than the parallelism, a strategy of optimizing the storage format to reduce the number of data read cycles is adopted. Please refer to [reference needed] for specific storage formats. Figure 6 and Figure 7 As shown.

[0054] In another embodiment of this application, when the number of input channels is greater than the parallelism of the input channels but is not an integer multiple thereof, the control module splits the input channels into a regular part and a remaining part for separate processing.

[0055] Specifically, in practice, the normalized portion can be processed using normalized data processing methods, while the remaining portion can be optimized using the aforementioned strategies based on its size. For example, when the number of input channels C of the convolutional layer is greater than the input channel parallelism P... c But it's not P.c When the value is an integer multiple of P, this application splits C into two parts and executes them using two instructions: one part is C1, and C1 is P. c The first part is an integer multiple of P, and the MAC utilization rate during this part's execution is 100%; the other part is C2, C2 = C % P. c The instruction is executed using the above scheme, and the MAC utilization rate during execution can reach 100% or approach 100%. At the same time, C2 is added to the calculation result of C1 during execution to obtain the final calculation result of the convolution layer. Finally, the MAC utilization rate of the convolution layer during execution can reach 100% or approach 100%.

[0056] The above content focuses on improving the MAC utilization of convolutional operations from the input channel dimension of the convolutional layer. Next, we will optimize the MAC utilization from the output channel dimension of the convolutional layer. In one embodiment of this application, the activation memory adopts a multi-bank structure, with each bank divided into multiple storage spaces. Each storage space has an independent read / write enable signal and is organized using low-order interleaving addressing. Specifically, the SPM-A in this application mainly adopts a multi-bank storage structure, with banks using high-order interleaving addressing. Each bank is divided into four storage spaces, each with a separate read / write enable signal. The four storage spaces are linked by low-order interleaving addressing. Figure 9 As shown, this storage structure allows the convolution module to access one or more storage spaces (2 to 4) simultaneously within a single bank in one cycle by controlling the read / write enable signal of the storage space, thereby solving the problem of insufficient activation data bandwidth.

[0057] Based on the above structure, this application also provides corresponding optimization schemes from the output channel dimension. When the number of output channels is less than the parallelism, the amount of active data read in a single cycle is increased by increasing the access width of the active data. When the number of output channels is greater than the parallelism but not an integer multiple, the output channels are split into a regular part and a remaining part, and the strategy of increasing the access width of the active data is adopted for the remaining part. These optimizations are achieved through the multi-bank storage structure unique to this application, which adopts a high-low bit interleaving addressing method, allowing parallel access to multiple storage spaces in one cycle.

[0058] In one embodiment of this application, the control module dynamically adjusts the access bit width of the activation data by controlling the read / write enable of multiple storage spaces of the activation memory according to the matching of the number of output channels and the parallelism of the output channels.

[0059] The matching conditions are similar to those described above and may include: the number of output channels is the same as the number of output channels, the number of output channels is an integer multiple of the number of output channels, and the number of output channels is greater than or less than the number of output channels. Different processing logics are adopted for different matching conditions. The specific implementation methods will be described in detail in subsequent embodiments and will not be described in detail here.

[0060] Specifically, when the number of output channels is less than the parallelism of the output channels, the control module increases the amount of active data read in a single cycle by enabling read operations of multiple storage spaces simultaneously.

[0061] When the number of output channels is greater than the parallelism of the output channels but is not an integer multiple thereof, the control module will split the output channels into a regular part and the remaining part for separate processing.

[0062] In practical applications, when the number of output channels F of a convolutional layer is less than the parallelism P of the output channels... f In this application, the approach of increasing the activation data bit width and reading more activation data is adopted to improve MAC utilization. For example, when the number of output channels F of the convolutional layer is only equal to the output channel parallelism P... f When it is half of the value, that is, F=P f / 2, see storage format. Figure 10 At this point, the act_read module reads SPM-A activation data from two addresses within one cycle, at P f Within 8 cycles, the convolutional module can read P f / 2×P c The weight data can be read in the W direction P f / 4 points P c Activation data is fed into the PE array for convolution calculations, ultimately resulting in P... f Within 8 cycles, the PE computing array can complete P f / 4 points P f / 2×P c The calculation of the data is equivalent to each cycle completing P. f ×P c Data calculation, P f ×P c All MAC units participate in the computation, resulting in 100% MAC utilization. When the number of output channels F of the convolutional layer equals the output channel parallelism P... f When it is one-quarter of the time, i.e., F=P f / 4, at this point, the act_read module can read SPM-A activation data from four addresses within one cycle, enabling the MAC utilization of convolution computation to reach 100%. When the number of output channels F of the convolutional layer is smaller, dividing a single bank into more storage spaces can also enable the MAC utilization of convolution computation to reach 100%.

[0063] When the number of output channels F of the convolutional layer is greater than the parallelism of the output channels P f But it's not P. f When F is an integer multiple of P, it is split into two parts and executed using two instructions: one part is F1, and F1 is P. f The instruction's MAC utilization is 100% when it is executed, and the other part is F2, where F2 = F % P. f When the instruction is executed using the above scheme, the MAC utilization rate can reach 100% or approach 100%.

[0064] In one embodiment of this application, the activation buffer performs a corresponding rearrangement operation on the activation data according to the storage format of the weight data and the convolution calculation method. Furthermore, the control module sets the ratio of the activation data bit width to the weight data bit width to be one to four.

[0065] Specifically, in practical work, the demand for weight data during convolution calculation is much greater than that for activation data. Therefore, the bit width of the weight data read by wgt_read in SPM-W is also much greater than the bit width of the activation data read by act_read in SPM-A. However, an excessively large read data bit width in SPM will lead to a sharp increase in power consumption, while an excessively small data bit width will reduce performance. This invention, by balancing performance and power consumption, sets the ratio of activation data bit width to weight data bit width to act_width:wgt_width = 1:4, and matches a larger amount of weight data by reading multiple phases of activation data.

[0066] Based on the neural network acceleration device provided in this application, a control method is also provided in one embodiment, specifically including: In response to a mismatch between the number of input channels and the parallelism of the input channels of the processing unit, the method of reading the activation data is controlled; and / or the storage format of the weight data and the corresponding method of reading the activation data and the method of rearranging the activation data and the weight data are dynamically adjusted. And / or, In response to a mismatch between the number of output channels and the parallelism of the output channels of the processing unit, the access bit width for active data is dynamically adjusted.

[0067] The input data of the neural network acceleration device includes the input feature map and weight data for convolution operation, or the input left matrix and input right matrix for matrix multiplication operation.

[0068] The input left matrix and the input right matrix are two matrices referenced in the matrix multiplication operation, respectively. This application does not impose any restrictions on the input left matrix and the input right matrix, and can refer to the prior art for understanding. They will not be described in detail here.

[0069] Specifically, in one embodiment of this application, dynamically adjusting the storage format of the weight data and the corresponding activation data reading method includes: When the number of input channels is less than the parallelism of the input channels, the corresponding storage format and the reordering strategy of the activation data are selected based on the comparison result between the number of input channels and the preset conditions.

[0070] Furthermore, the step of selecting the appropriate storage format and the reordering strategy for activating data based on the comparison results includes: When the number of input channels is less than or equal to the first threshold, the weight data is dimensionally concatenated and the storage format is rearranged, and the activation data is rearranged accordingly. When the number of input channels is greater than the first threshold and less than the parallelism of the input channels, the storage format is rearranged to reduce the number of data read cycles.

[0071] In this embodiment, there are two different cases, as follows: Scenario 1: When the number of input channels C is very small, i.e., less than or equal to the first threshold, a strategy is adopted to perform dimensional concatenation and storage format rearrangement on the weight data, and to rearrange the activation data accordingly. In an optional embodiment, the dimensional concatenation includes concatenating the input channel dimension (C) with the convolution kernel width dimension (Kx), or concatenating the input channel dimension (C) with the convolution kernel height dimension (Ky) and width dimension (Kx) simultaneously, so that the size of the concatenated data matches the parallelism of the input channels of the processing unit. Correspondingly, the activation data also needs to be rearranged according to the format of the rearranged weight data to ensure data alignment and calculation correctness. In this way, the compactness of weight data storage and data reading efficiency can be significantly improved, thereby improving the utilization rate of the computing unit.

[0072] Scenario 2: When the number of input channels C is large (greater than the first threshold but less than the input channel parallelism), a strategy is adopted to rearrange the storage format of the weight data to reduce the number of data read cycles. In an optional implementation, by adjusting the arrangement of weights in memory, "bubbles" in the original storage structure are eliminated, achieving a compact data arrangement, thereby reducing the number of data access cycles required to read the complete weights. This method is particularly suitable for cases where the number of channels is large but not yet an integer multiple of the parallelism, and can further improve the computational throughput without significantly changing the data organization pattern.

[0073] It should be understood that the aforementioned first threshold can be set according to the characteristics of the hardware architecture and common network structures, for example, it can be determined based on the requirements of the smallest computational granularity in the processing unit, and its specific value should not be construed as a limitation on the scope of protection of this application. Through the above two different optimization strategies, this application can dynamically select the most suitable data recombination method according to the actual number of input channels, thereby effectively improving the utilization rate of the computing unit and reducing resource idleness under various channel configurations.

[0074] Conversely, in another embodiment of this application, when the number of input channels is greater than the parallelism of the input channels, a similar processing method can also be adopted. Please refer to [link / reference needed] for details. Figure 1 As shown, in one embodiment of this application, dynamically adjusting the storage format of the weight data and the corresponding activation data reading method includes: S101: When the number of input channels is greater than the parallelism of the input channels and is not an integer multiple, the input channels are split into a regular part and the remaining part for separate processing; S102: Based on the comparison result between the remaining part and the preset conditions, select the corresponding storage format rearrangement strategy and the corresponding activation data reading method.

[0075] Specifically, if the remaining portion is less than or equal to the first threshold, a strategy of dimension concatenation and storage format rearrangement is adopted for the weight data of that portion, and the corresponding activation data is rearranged; if the remaining portion is greater than the first threshold, the storage format of the weight data of that portion is rearranged to reduce the number of data read cycles.

[0076] Using the above method, for input channels that are not integer multiples, they can be decomposed into multiple regularized parts and a residual part. The regularized parts can be processed in a conventional manner to achieve 100% computational unit utilization; the residual part, depending on its size, can be processed using one of the two optimization strategies described in the above embodiments, thereby achieving a high overall computational unit utilization. Finally, the calculation results of the regularized parts and the residual part are summed to obtain the complete calculation result. In this way, through a hierarchical processing strategy, the computational efficiency of the regularized parts is guaranteed, while the optimization processing of the residual part maximizes the utilization of overall computing resources, which is particularly suitable for convolution calculations or matrix operation scenarios with a large number of input channels.

[0077] In one embodiment of this application, the mismatch between the number of output channels of the data to be processed and the parallelism of the output channels of the processing unit includes: the number of output channels is less than the parallelism of the output channels, or the number of output channels is greater than the parallelism of the output channels but is not an integer multiple.

[0078] Furthermore, dynamically adjusting the access bit width for activation data may include: When the number of output channels is less than the parallelism of the output channels, the amount of active data read in a single cycle is increased; When the number of output channels is greater than the parallelism of the output channels but is not an integer multiple thereof, the output channels are split into a regular part and a remaining part for processing separately, and the amount of active data read in a single cycle is increased for the remaining part.

[0079] In the above embodiments, the increase in the amount of active data read in a single cycle is achieved by controlling the access enable of multiple memory banks in a memory employing low-order interleaving addressing. Specifically, the memory module includes multiple memory banks organized using low-order interleaving addressing, each memory bank having independent access control logic. By coordinating the read / write enable signals of these memory banks, data can be read from multiple memory banks simultaneously within one clock cycle, thereby combining multiple narrow-bit-width data accesses into a wider effective data access bit width, achieving the goal of reading more active data in a single cycle. This hardware architecture support enables efficient dynamic adjustment of the access bit width.

[0080] Similar to the optimization approach for input channels, this embodiment also provides corresponding optimization strategies for different numbers of output channels, as detailed below: Case 1: When the number of output channels F is less than the parallelism of the output channels P f hour: In this situation, the core challenge lies in whether the weighted data can achieve a high degree of parallelism (P). fThe supply of data is sufficient, but the corresponding activation data read volume is insufficient, causing the computing unit to be unable to operate at full capacity. To solve this problem, a strategy of dynamically increasing the activation data read volume per cycle is adopted. For example, when F = P... f When F = P / 2, by enabling control, two memory banks can be accessed simultaneously in a single cycle, doubling the effective read bit width of the active data. This ensures that a sufficient amount of active data is read to match and calculate with the weight data, guaranteeing that the computing unit array can operate at full capacity. Similarly, when F = P f When the bit width is 4, bit width adaptation can be achieved by accessing four memory banks simultaneously within a single cycle.

[0081] Case 2: When the number of output channels F is greater than the parallelism of the output channels P f And when it is not an integer multiple: In this case, a hierarchical processing strategy is adopted to balance computational efficiency and resource utilization. First, the total number of output channels F is split into a normalized portion F1 and a remaining portion F2. The normalized portion F1 is P f Integer multiples of (i.e., F1 = F / P f ×P f This part can be processed in the conventional way, and the utilization rate of the computing unit can reach 100%. For the remaining part F2 (F2 = F % P) f Since its number of channels is less than P f If the problem persists, the strategy described in Case 1 is adopted instead. This involves controlling the access enable of the memory bank to increase the amount of active data read in a single cycle, thus meeting the computational demands and ensuring high utilization of the computing units when processing the remaining part. Finally, the computational results of the regularized part and the remaining part are merged.

[0082] Through the two strategies described above, this application can flexibly handle various irregular scenarios regarding the number of output channels. The core lies in dynamically adapting to different data requirements through a hardware-supported parallel access mechanism for memory, thereby achieving extremely high overall computing unit utilization. This optimization strategy is also applicable to scenarios such as convolution operations and matrix multiplication.

[0083] It should be understood that the above optimization strategies for output channels can be used independently or in combination with the aforementioned optimization strategies for input channels. When the number of input and output channels is mismatched, the overall computational efficiency can be further improved by combining the two optimization strategies.

[0084] To facilitate a clearer understanding of the data processing utilization improvement method provided in this application, the following is combined with... Figure 2This document provides a complete description of a method for improving data processing utilization. As will be known from those skilled in the art, this embodiment is merely an example of one application of this application and does not impose any limitations on it.

[0085] like Figure 2 As shown, the control method provided in this application begins with the configuration of the initial convolution or matrix multiplication operation. First, it determines whether the number of input channels matches the parallelism of the input channels of the processing unit. If they match, the operation is performed according to the regularized data flow, at which point the utilization rate of the computing unit can reach 100%; that is, regularized convolution is performed on the basis of dimension matching, and multiple activations are read to match the weight data of a larger data volume during the convolution calculation process; if they do not match, the input channel dimension optimization processing flow is entered.

[0086] In the input channel dimension optimization process, the first step is to determine whether the number of input channels is less than the input channel parallelism. If it is less, an appropriate optimization strategy is selected based on the comparison between the number of input channels and preset conditions: when the number of input channels is less than or equal to the first threshold, a dimension concatenation and data rearrangement strategy is adopted; when the number of input channels is greater than the first threshold, a strategy to reduce the number of data read cycles is adopted. If the number of input channels is greater than the input channel parallelism but is not an integer multiple, the input channels are split into a regularized part and a remaining part for separate processing: the regularized part is processed according to the regularized data flow, and the remaining part is optimized according to its size.

[0087] In parallel processing, during the input channel dimension processing and judgment, the output channel matching judgment stage can also be entered to determine whether the number of output channels matches the parallelism of the output channels of the processing unit. If they match, the operation is performed according to the regularized data flow; if they do not match, the output channel dimension optimization processing flow is entered.

[0088] In the output channel dimension optimization process, the first step is to determine whether the number of output channels is less than the output channel parallelism. If it is less, the amount of active data read in a single cycle is increased by controlling the access enable of multiple memory banks in the memory. If it is greater than but not an integer multiple, the output channel is split into a regular part and a remaining part for separate processing: the regular part is processed according to the regular data stream, and the remaining part is processed by increasing the amount of data read in a single cycle.

[0089] Finally, the calculation results from all parts are integrated to complete the operation and end the entire processing flow. Through this complete judgment and processing flow, this application can systematically address various situations where the number of input and output channels does not match the hardware parallelism. By dynamically selecting the optimal data processing strategy and combining the three factors, the MAC utilization rate of the final convolution calculation reaches 100% or close to 100%, thereby significantly improving the utilization rate of the computing unit.

[0090] The beneficial technical effects of this application are as follows: by dynamically adjusting the data storage format and access bit width in a responsive manner, the computing unit can still achieve or approach 100% utilization even when the data dimension and hardware parallelism are not regularly matched, significantly reducing idle computing resources; starting from multiple data dimensions such as input and output, it can effectively cope with various irregularly configured operation layers (such as convolution channels and matrix dimensions), and has strong versatility; by adopting a low-bit interleaved addressing storage structure and corresponding data rearrangement strategies, it improves computing efficiency while reducing the number of memory accesses and power consumption, achieving a balance between performance and energy efficiency.

[0091] This application also provides a chip that includes the aforementioned neural network acceleration device.

[0092] This application also provides a board card including the aforementioned chip.

[0093] This application also provides an electronic device having the aforementioned board or chip deployed thereon.

[0094] Figure 11 This is a block diagram illustrating an electronic device 900 for implementing the above-described data processing method, according to an exemplary embodiment. For example, the electronic device 900 may be an AI server, a training and promotion integrated machine, etc.

[0095] Reference Figure 11 The electronic device 900 may include one or more of the following components: an AI-accelerated computing module, a CPU module, a power supply module, a hard drive module, and a fan module. Each module works in conjunction with the bus system through a standardized hardware interface, with the specific architecture as follows: The AI-accelerated computing module comprises multiple AI accelerator cards deployed in parallel. Each AI accelerator card integrates at least one AI accelerator chip (such as an RPU chip, GPU chip, or CGRA chip). Data communication between the AI ​​accelerator cards is achieved through a high-speed card-to-card (C2C) interconnect structure, supporting low-latency, high-bandwidth horizontal scaling. The AI ​​accelerator chip is dedicated to performing AI computing tasks such as high-density matrix operations, neural network model training, and / or inference, providing the main computing power support.

[0096] The CPU module includes at least one CPU board, which houses a central processing unit (CPU) chip and associated CPU memory (such as DDR4 / DDR5, RAM). The CPU chip serves as the system control center, responsible for task scheduling, resource allocation, I / O management, and coordinating the parallel computing of the AI ​​acceleration computing module, while also handling non-accelerated general-purpose computing tasks.

[0097] The power module is equipped with redundant power supply units to provide stable power distribution and management for the AI ​​acceleration computing module, CPU module and other modules.

[0098] The hard drive module integrates a high-speed solid-state drive (SSD) and / or a large-capacity hard disk drive (HDD), connected to the system bus via a backplane. The hard drive stores the operating system, AI training datasets, model parameters, and calculation results, providing high-throughput data read / write channels and supporting data preprocessing and persistence.

[0099] The fan module uses a multi-zone independent speed-controlled fan array, which is configured in key heat source areas (such as AI accelerator cards and CPU heat dissipation areas) to achieve system heat dissipation through forced air cooling and ensure the stable operation of high-efficiency computing components.

[0100] The CPU module is connected to the AI ​​acceleration computing module via the PCIe bus to enable task distribution, result collection, and memory coordination.

[0101] The CPU module manages the data access of the hard drive module through SATA / SAS / NVMe interfaces.

[0102] The power module provides tiered power to all functional modules through the power distribution backplane.

[0103] The fan module adjusts the fan speed based on temperature monitoring signals from the CPU board and AI accelerator card.

[0104] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0105] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0106] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A neural network acceleration device, characterized in that, include: The memory module is used to store activation data, calculation result data, and weight data; The control module is used to control the reading method of the activation data according to the matching of the number of input channels and the parallelism of the input channels; and / or dynamically adjust the storage format of the weight data in the memory module, and control the reading method of the activation data and the rearrangement method of the activation data and weight data based on the adjusted storage format; The processing unit array is used to perform multiplication and accumulation calculations on the activation data and weight data output by the control module, and write the calculated result data into the memory module.

2. The neural network acceleration device according to claim 1, characterized in that, The memory module includes an activation memory and a weight memory; The activation memory is used to store activation data and calculation result data; The weight memory is used to store weight data; The control module includes a reading unit and a buffer unit; The reading unit includes an activation reading unit and a weight reading unit, which are respectively used to read the activation data and the weight data from the activation memory and the weight memory; The buffer unit includes an activation buffer and a weight buffer, which are respectively connected to the activation reading unit and the weight reading unit, and are used to cache and rearrange the activation data and weight data.

3. The neural network acceleration device according to claim 2, characterized in that, When the number of input channels is an integer multiple of the parallelism of the input channels, the control module controls the activation reading unit to read the activation data in a multi-step manner to match the amount of weight data read by the weight reading unit.

4. The neural network acceleration device according to claim 2, characterized in that, When the number of input channels is half of the parallelism of the input channels, the control module controls the activation memory to store half of the channel data of two points in the width direction at one address, and controls the weight memory to store eight times the weight data of half of the channels at one address. When the number of input channels is less than half of the input channel parallelism and the product of the number of input channels and the width and height of the convolution kernel does not exceed the input channel parallelism, the control module controls the weight memory to store the weight data in a way that concatenates the input channels with the width and height of the convolution kernel. When the number of input channels is less than half of the input channel parallelism and the product of the number of input channels and the convolution kernel width does not exceed the input channel parallelism, the control module controls the weight memory to store the weight data in a way that concatenates the input channels and the convolution kernel width dimensions.

5. The neural network acceleration device according to claim 2, characterized in that, The activation memory adopts a multi-memory bank structure, with each memory bank divided into multiple storage spaces. Each storage space has an independent read / write enable signal and is organized using a low-order interleaved addressing method.

6. The neural network acceleration device according to claim 5, characterized in that, The control module dynamically adjusts the access bit width of the activation data by controlling the read / write enable of multiple storage spaces in the activation memory, based on the matching between the number of output channels and the parallelism of the output channels.

7. The neural network acceleration device according to claim 6, characterized in that, When the number of output channels is less than the parallelism of the output channels, the control module increases the amount of active data read in a single cycle by enabling read operations of multiple storage spaces simultaneously.

8. The neural network acceleration device according to claim 2, characterized in that, The activation buffer performs corresponding rearrangement operations on the activation data according to the storage format of the weight data and the convolution calculation method.

9. The neural network acceleration device according to claim 1, characterized in that, The control module sets the ratio of the activation data bit width to the weight data bit width to be 1:

4.

10. A control method applicable to the neural network acceleration device according to any one of claims 1 to 9, characterized in that, The method includes: In response to a mismatch between the number of input channels and the parallelism of the input channels of the processing unit, the method of reading the activation data is controlled; and / or the storage format of the weight data and the corresponding method of reading the activation data and the method of rearranging the activation data and the weight data are dynamically adjusted. And / or, In response to a mismatch between the number of output channels and the parallelism of the output channels of the processing unit, the access bit width for active data is dynamically adjusted.

11. A chip, characterized in that, It includes the neural network acceleration device according to any one of claims 1 to 9.

12. A circuit board, characterized in that, Including the chip as described in claim 11.

13. An electronic device, characterized in that, It is equipped with the board as described in claim 12 or the chip as described in claim 11.

Images