A neural network inference chip, a neural network inference method and a terminal

Abstract

This application provides a neural network inference chip, a neural network inference method, and a terminal. The chip includes a control unit, an array of arithmetic units, and an array of partial summation and accumulation units. The arithmetic unit array includes a first arithmetic unit subarray and a second arithmetic unit subarray located in the same column. The control unit is used to input first feature data and second feature data into the first arithmetic unit subarray and the second arithmetic unit subarray, respectively; determine and activate the target selection branch of the second arithmetic unit subarray; the first arithmetic unit subarray is used to obtain the first subarray operation result based on the first feature data and the first weight data; the second arithmetic unit subarray is used to obtain the second subarray intermediate operation result based on the second feature data and the second weight data; and the target selection branch is used to perform operations on the second subarray intermediate operation result and the first subarray operation result to obtain the second subarray operation result, which is then output to the partial summation and accumulation unit subarray.

Description

Technical Field

[0001] This application relates to the field of chip design, and more particularly to a neural network inference chip, a neural network inference method, and a terminal. Background Technology

[0002] In recent years, with the continuous development of computer science and chip architecture, neural network algorithms, leveraging the powerful computing capabilities of modern computers, have demonstrated advantages far exceeding those of traditional algorithms in fields such as computer vision, audio and video processing, and autonomous driving. Neural networks are characterized by their computational and storage-intensive nature, which poses greater challenges to their hardware deployment.

[0003] Current neural network inference chip architectures mainly include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs).

[0004] For neural network inference chip architectures in related technologies, the computational parallelism of the architecture is determined by an array of computational units connected in a systolic array (SA) configuration. This includes the parallelism of input channels (ICs) and output channels (OCs), allowing convolution operations to be performed using the computational unit array with this computational parallelism. However, when the number of input channel ICs is not an integer multiple of the IC parallelism of the architecture, there is a problem of low utilization of the computational unit array. Summary of the Invention

[0005] This application provides a neural network inference chip, a neural network inference method, and a terminal. It proposes a chip architecture that can realize application scenarios with multiple input channels and different quantization accuracy requirements of the network layer.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a neural network inference chip, comprising: a control unit, an array of arithmetic units, and an array of partial summation and accumulation units. The array of arithmetic units includes a first subarray of arithmetic units and a second subarray of arithmetic units located in the same column.

[0008] The control unit is used to control the input of the first feature data and the second feature data corresponding to the input channel into the first computing unit subarray and the second computing unit subarray, respectively; to determine the target selection branch of the second computing unit subarray and to activate the target selection branch;

[0009] The first computing unit subarray is used to obtain the computing result of the first subarray based on the first feature data and the first weight data, and output it to the second computing unit subarray;

[0010] The second subarray of computing units is used to obtain intermediate computing results of the second subarray based on the second feature data and the second weight data; to perform calculations on the intermediate computing results of the second subarray and the computing results of the first subarray through the target selection branch to obtain the computing results of the second subarray; and to output the computing results of the second subarray to the part and accumulation unit subarray located in the same column as the second computing unit subarray in the part and accumulation unit array for accumulation calculation.

[0011] Secondly, embodiments of this application provide a neural network inference method, which is applied to a neural network inference chip. The neural network inference chip includes: a control unit, an array of arithmetic units, and an array of partial summation and accumulation units. The array of arithmetic units includes a first subarray of arithmetic units and a second subarray of arithmetic units located in the same column.

[0012] The control unit controls the input of the first feature data and the second feature data corresponding to the input channel into the first computing unit subarray and the second computing unit subarray, respectively; determines the target selection branch of the second computing unit subarray and activates the target selection branch;

[0013] The first subarray of computing units obtains the computing result of the first subarray based on the first feature data and the first weight data, and outputs it to the second subarray of computing units;

[0014] The second subarray of computing units obtains intermediate computing results based on the second feature data and the second weight data; through the target selection branch, the intermediate computing results of the second subarray and the computing results of the first subarray are processed to obtain the computing results of the second subarray; the computing results of the second subarray are output to the part and accumulation unit subarray located in the same column as the second subarray of computing units in the part and accumulation unit array for accumulation operation.

[0015] Thirdly, embodiments of this application provide a terminal including the neural network inference chip as described in the first aspect.

[0016] Fourthly, embodiments of this application provide a terminal, which includes a neural network inference chip, a processor, and a memory storing executable instructions. When the instructions are executed, the neural network inference method as described in the second aspect is implemented.

[0017] This application provides a neural network inference chip, a neural network inference method, and a terminal. The neural network inference chip includes a control unit, a computation unit array, and a partial sum-accumulation unit array. The computation unit array includes a first computation unit subarray and a second computation unit subarray located in the same column. The control unit is used to control the input of first feature data and second feature data corresponding to the input channel into the first computation unit subarray and the second computation unit subarray, respectively; determine the target selection branch of the second computation unit subarray and activate the target selection branch; the first computation unit subarray is used to obtain the first subarray operation result based on the first feature data and the first weight data, and output it to the second computation unit subarray; the second computation unit subarray is used to obtain the second subarray intermediate operation result based on the second feature data and the second weight data; perform operations on the second subarray intermediate operation result and the first subarray operation result through the target selection branch to obtain the second subarray operation result; and output the second subarray operation result to the partial sum-accumulation unit subarray located in the same column as the second computation unit subarray in the partial sum-accumulation unit array for accumulation operation. In other words, in this embodiment, based on the feature data of the input channels to the first and second operation unit subarrays, a target selection branch is determined from multiple selection branches of the second operation unit subarray. Through the target selection branch, the intermediate operation result of the second subarray obtained by the second operation unit subarray is compared with the operation result of the first subarray obtained by the first operation unit subarray to obtain the corresponding second subarray operation result. Thus, this embodiment provides a chip architecture capable of supporting application scenarios with multiple input channels and different quantization precision requirements in the network layer. Different selection branches are selected for operation processing in different application scenarios, improving the chip's utilization and computational efficiency in different application scenarios, increasing the utilization of the array operation units, reducing network inference latency, and accelerating network layer inference. Attached Figure Description

[0018] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0019] Figure 1 This is a schematic diagram of the structure of an optional neural network inference chip provided in an embodiment of this application;

[0020] Figure 2This is a calculation diagram of an optional convolution operation provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the structure of an optional neural network inference chip provided in an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of an optional neural network inference chip architecture provided in an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the structure of an optional neural network inference chip provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of an optional first selected branch provided in an embodiment of this application;

[0025] Figure 7 This is a schematic diagram of an optional second branch provided in an embodiment of this application;

[0026] Figure 8 This is a circuit diagram corresponding to an optional selected branch provided in an embodiment of this application;

[0027] Figure 9 This is a schematic diagram of an optional neural network inference chip architecture provided in an embodiment of this application;

[0028] Figure 10 This is a schematic diagram of the circuit structure of the arithmetic units located in the same column in the 2×2 second arithmetic unit subarray provided in the embodiments of this application;

[0029] Figure 11 A flowchart illustrating an optional neural network inference method provided in an embodiment of this application;

[0030] Figure 12 A schematic diagram of an optional terminal provided in an embodiment of this application;

[0031] Figure 13 This is a schematic diagram of an optional terminal provided in an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0034] It should be noted that in this application example, terms such as "first" and "second" are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0035] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0036] In recent years, with the continuous development of computer science and chip architecture, neural network algorithms, leveraging the powerful computing capabilities of modern computers, have demonstrated advantages far exceeding those of traditional algorithms in fields such as computer vision, audio and video processing, and autonomous driving. Neural networks are characterized by their computational and storage-intensive nature, which poses greater challenges to their hardware deployment.

[0037] As an important method in computer vision, neural networks take image data as input and, through a trained neural network, calculate the required semantic information, such as the category of objects. This process is called neural network inference.

[0038] Neural network inference chips have become a hot research topic in the field of artificial intelligence hardware in recent years. Neural networks are characterized by computational complexity and large data volumes, requiring extensive parallel computing and memory access, thus placing extremely high demands on computing power. The deployment of neural network algorithms in terminal devices (such as mobile phones, smart homes, and autonomous vehicles) is subject to strict limitations in terms of power consumption and cost. Furthermore, different neural networks exhibit significant differences in structure, data size, and operator types, thus also requiring a certain degree of flexibility in the chip design.

[0039] Current neural network inference chip architectures mainly include Graphics Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs), and Application Specific Integrated Circuits (ASICs). GPUs employ a massively parallel general-purpose computing architecture, significantly shortening algorithm training time and accelerating algorithm iteration. However, GPUs cannot efficiently support flexible and varied neural network algorithms, failing to fully leverage their computing power advantages, and their network inference chips have relatively high power consumption. FPGAs offer high reconfigurability but have limited computing resources, resulting in higher inference speed and cost. ASICs allow for customized hardware architecture design tailored to the characteristics of neural network algorithms, optimizing chip performance, power consumption, and area. Therefore, based on the design philosophy of ASICs, neural network processing unit (NPU) architecture has become one of the hottest research topics in the field of artificial intelligence hardware architecture.

[0040] Convolution operations account for a significant proportion of computation in current deep learning algorithms. Based on the operator characteristics of computational regularity and data reusability, the NPU computing architecture of SA (Simplified Analytic Array) has become a key research focus in the industry. SA has a simple, regular, and highly modular structure, and the short data communication distance between processing elements (PEs) makes the design of data flow and control flow, as well as synchronous control, simple and regular. Furthermore, all PEs in the array can operate simultaneously, exhibiting high computational parallelism and achieving high computational efficiency and throughput.

[0041] For neural network inference chip architectures in related technologies, the computational parallelism of the architecture is determined by using a systolic array of computational units (ICs), including the parallelism of input channels (ICs) and output channels (OCs), so that convolution operations can be performed using the IC array with this computational parallelism. However, when the number of input channel ICs is not an integer multiple of the IC parallelism of the architecture or IC parallelism / 2, there is a problem of low utilization of the computational unit array.

[0042] For example, the computational parallelism of this architecture is determined by using an 8×8 array of computational units connected in a systolic array manner; that is, the IC parallelism is 8 and the OC parallelism is also 8. If the number of input channel ICs in the current network layer is 60, the number of iterations required for iterative computation using this architecture is... Furthermore, in the last iteration, the computational units (PEs) in the last four rows stopped working, causing the utilization rate of the current network layer to decrease from 100% to 60 / 64 × 100% = 93.75%. Additionally, it's understandable that the smaller the number of integrated circuits (ICs) in a network layer, the greater the impact on computational unit utilization from non-integer multiples of IC parallelism. Moreover, most current networks have a first-layer IC with three input channels: red (r), green (g), and blue (b), resulting in a current network layer utilization rate of only 3 / 8 × 100% = 37.5%.

[0043] It should also be noted that while most neural network algorithms currently achieve good inference performance with 8-bit quantization, some complex neural network algorithms still require 16-bit or even 24-bit quantization to obtain high-precision inference results. For example, the computation unit, such as the MMP core, uses an 8-bit × 8-bit multiplier, requiring two cycles to obtain the product in a single 16-bit × 8-bit calculation. As the data is scaled up to the entire network, the hardware inference speed decreases.

[0044] To address the aforementioned problems, embodiments of this application provide a neural network inference chip. Figure 1 A schematic diagram of an optional neural network inference chip provided in an embodiment of this application, as shown below. Figure 1 As shown, the neural network inference chip 100 includes: a control unit 10, a computation unit array 20, and a partial summation unit array 30. The computation unit array 20 includes a first computation unit subarray 21 and a second computation unit subarray 22 located in the same column.

[0045] Control unit 10 is used to control the input of the first feature data and the second feature data corresponding to the input channel into the first arithmetic unit subarray 21 and the second arithmetic unit subarray 22 respectively; determine the target selection branch 221 of the second arithmetic unit subarray 22 and enable the target selection branch 221;

[0046] The first operation unit subarray 21 is used to obtain the operation result of the first subarray based on the first feature data and the first weight data, and output it to the second operation unit subarray 22.

[0047] The second operation unit subarray 22 is used to obtain the intermediate operation result of the second subarray based on the second feature data and the second weight data; to perform operations on the intermediate operation result of the second subarray and the operation result of the first subarray through the target selection branch to obtain the operation result of the second subarray; and to output the operation result of the second subarray to the partial sum accumulation unit subarray 31 in the partial sum accumulation unit subarray 30, which is located in the same column as the second operation unit subarray, for accumulation operation.

[0048] In this embodiment, the computational unit array is used to store the weight data of each layer of the neural network input and to perform the calculation of the convolution operator. The computational unit array includes N×N computational units, and the N×N computational units are connected in a spatial array, such as a systolic array. It should be noted that the number of rows in the computational unit array corresponds to the integrated parallelism (IC) of the convolution operation, and the number of columns in the computational unit array corresponds to the open parallelism (OC) of the convolution operation. The array-based computational structure of the computational unit array utilizes the data reusability of the convolution operation, that is, each row of computational units shares the same feature data, thus reducing data movement and further reducing circuit area and power consumption. Here, the computational unit array is an array composed of matrix multiply processors (MMPs), and the computational unit is an mmp_core. The computational unit array includes N×N mmp_cores, that is, the N×N mmp_cores are arranged in N rows and N columns. It can be understood that in this application, N can be an integer greater than or equal to 2.

[0049] In this embodiment of the application, the arithmetic unit subarray includes K×K arithmetic units, where K is an integer greater than or equal to 1 and less than or equal to N / 2.

[0050] In this embodiment, the partial sum accumulation unit array is used to perform the accumulation of partial sums. The partial sum accumulation unit array includes 1×N partial sum accumulation units. Here, the partial sum accumulation unit array is an array composed of partial sum accumulators (PSUM), and the partial sum accumulation unit is a psum_core. The partial sum accumulation unit array includes 1×N psum_cores, and the N psum_cores can work in parallel.

[0051] In this embodiment of the application, the partial and accumulator subarray includes 1×K partial and accumulator units, where K is an integer greater than or equal to 1 and less than or equal to N / 2.

[0052] In this embodiment, the first feature data corresponding to an input channel can be the feature data of complete bytes corresponding to a portion of the multiple input channels, and the second feature data can be the feature data of complete bytes corresponding to the remaining input channels; that is, the first feature data and the second feature data are feature data of complete bytes corresponding to two different input channels, respectively. Of course, the first feature data corresponding to an input channel can also be the low-byte feature data of each of the multiple input channels, and the second feature data can be the high-byte feature data of each of the multiple input channels; that is, the first feature data and the second feature data are feature data of different byte positions corresponding to the same input channel, and this application does not impose specific limitations on this.

[0053] In this embodiment of the application, the target selection branch includes a first selection branch and a second selection branch. The first selection branch is used to perform data operations on the feature data of complete bytes corresponding to different input channels, and the second selection branch is used to perform data operations on the feature data of different byte bits of the same input channel.

[0054] In this embodiment, the control unit can select and activate a target selection branch in the second operation unit subarray from at least two selection branches based on the correlation between the feature data input to the first and second operation unit subarrays, so as to perform operation processing on the data through the target selection branch. The correlation between the feature data includes: correlations where feature data of complete bytes correspond to different input channels, and correlations where feature data of different byte bits correspond to the same input channel.

[0055] In this embodiment, the weight data is the weight corresponding to the feature data, and the weight data is pre-loaded into the internal memory of the computing unit. That is, the first weight data is the weight corresponding to the first feature data, and the first weight data is pre-loaded into the internal memory of the computing unit in the first computing unit subarray; the second weight data is the weight corresponding to the second feature data, and the second weight data is pre-loaded into the internal memory of the computing unit in the second computing unit subarray.

[0056] In this embodiment, the first subarray operation result represents the calculation result obtained after the first operation unit subarray performs a convolution operation. The first subarray operation result can be the convolution calculation result transmitted from the first operation unit subarray 21 to the second operation unit subarray 22, or it can be a partial sum of the convolution calculation results transmitted from the first operation unit subarray 21 to the second operation unit subarray 22. This application does not impose specific limitations on this.

[0057] In this embodiment, the intermediate computation result of the second subarray represents the intermediate computation result obtained after the second computation unit subarray performs convolution operation. The computation result of the second subarray is the computation result obtained after performing operations on the intermediate computation result of the second subarray and the computation result of the first subarray.

[0058] In this embodiment, firstly, the control unit inputs the first feature data and second feature data corresponding to multiple input channels into the first and second operation unit subarrays, respectively; and determines the target selection branch of the second operation unit subarray based on the correlation between the input first and second feature data, and controls the target selection branch to be activated. Then, the first operation unit subarray, based on the first feature data and the first weight data corresponding to the first feature data, obtains the first subarray operation result and outputs the obtained first subarray operation result to the second operation unit subarray. Finally, the second operation unit subarray, based on the second feature data and the second weight data corresponding to the second feature data, obtains the second subarray intermediate operation result; and through the activated target selection branch, performs operations on the second subarray intermediate operation result and the first subarray operation result to obtain the second subarray operation result; the second subarray operation result is output to the partial sum accumulation unit subarray located in the same column as the second operation unit subarray in the partial sum accumulation unit subarray, where the partial sum accumulation unit subarray performs a partial sum accumulation operation on the second subarray operation result.

[0059] Here, to facilitate understanding of the algorithmic principle behind SA's acceleration of inference through convolution operations, we will combine... Figure 2 Further explanation of convolution operations follows. (See reference...) Figure 2 As shown, the leftmost layer represents the input feature data corresponding to the three RGB input channels, IC0, IC1, and IC2. These input feature data are multiplied and added using four convolutional kernels of different sizes (kernel0, kernel1, kernel2, and kernel3), resulting in the output feature data corresponding to the four output channels on the right: OC0, OC1, OC2, and OC3. It can be observed that all convolutional kernels share all the input feature data, and the computation of input feature data within a single convolutional kernel is independent of data dependencies. Therefore, it is highly suitable for parallel computation acceleration using Scalar Algorithm (SA).

[0060] Here, the SA (Self-Solving Array) consists of arrayed computational units, and convolution operations are achieved through the periodic input of feature data in different rows. There are two data flows for convolution operations in the SA: weight stationary (WS) and output stationary (OS). WS can be understood as weight data being pre-loaded into the internal storage of the computational unit PE (Performance Array). Feature data is shared within the horizontal computational units, and partial products propagate and accumulate along the vertical direction. Furthermore, the input feature data and local sums must pass through the spatial array and global cache; this reduces the cost of moving weight data. OS can be understood as weight data propagating along the vertical computational unit PE, with partial products accumulating within the computational unit without needing to propagate vertically. Again, the input feature data and weight data must pass through the spatial array and global cache; this reduces the cost of moving local sums. It should be noted that this embodiment uses a weight stationary data flow to implement convolution operations in the SA.

[0061] In this application embodiment, the neural network inference chip can take any form, such as a terminal, chip, integrated circuit (IC), or application-specific integrated circuit. For example, the neural network inference chip proposed in this application embodiment can be an embedded neural network processor.

[0062] The NPU employs a "data-driven parallel computing" architecture, making it particularly adept at processing massive amounts of multimedia data, such as video and images. The NPU works by simulating human neurons and synapses at the circuit layer and directly processing large-scale neurons and synapses using a deep learning instruction set; a single instruction completes the processing of a group of neurons. Compared to the Central Processing Unit (CPU) and GPU, the NPU improves operational efficiency by emphasizing weights to integrate storage and computation.

[0063] NPUs are built to mimic biological neural networks. While CPUs and GPUs require thousands of instructions to process neurons, NPUs can accomplish the same task with just one or a few instructions, giving them a significant advantage in processing efficiency for deep learning. Like GPUs, NPUs also require the assistance of the CPU to complete specific tasks.

[0064] Typically, a dedicated system-on-chip (SoC) for mobile devices integrates a CPU, GPU, and NPU. The CPU handles smooth switching between mobile applications, the GPU supports fast loading of game graphics, and the NPU is specifically responsible for artificial intelligence (AI) computations and the implementation of AI applications. In other words, the CPU is responsible for computation and overall coordination, the GPU handles image-related tasks, and the NPU handles AI-related tasks. The workflow is as follows: any task must first pass through the CPU, which then determines which component to assign it based on the nature of the task. Graphics-related calculations are assigned to the GPU, while AI-related calculations are assigned to the NPU.

[0065] Unlike GPU acceleration, NPU acceleration is mainly characterized by the fact that the calculation results of each neuron do not need to be output to main memory. Instead, they are passed to the next neuron for further calculation according to the connections of the neural network. Therefore, it has a significant improvement in both computing performance and power consumption.

[0066] NPUs simulate neurons at the circuit level, integrating storage and computation through synaptic weights. A single instruction completes the processing of a group of neurons, improving operational efficiency. They are primarily used in communications, big data, and image processing. As a type of dedicated ASIC (Application-Specific Integrated Circuit), the NPU is a chip customized to meet specific requirements. Aside from its lack of scalability, it offers advantages in power consumption, reliability, and size, especially in high-performance, low-power mobile devices.

[0067] As described above, in this embodiment, based on the feature data of the input channels to the first and second operational unit subarrays, a target selection branch is determined from multiple selection branches of the second operational unit subarray. Through the target selection branch, the intermediate operation result of the second subarray obtained by the second operational unit subarray is compared with the operation result of the first subarray obtained by the first operational unit subarray to obtain the corresponding second subarray operation result. Thus, this embodiment provides a chip architecture capable of supporting various application scenarios with multiple input channels and different quantization precision requirements in the network layer. Different selection branches are selected for operation processing in different application scenarios, improving the chip's utilization and computational efficiency in different application scenarios, increasing the utilization of the array operational units, reducing network inference latency, and accelerating network layer inference.

[0068] In other embodiments of this application, refer to Figure 3As shown, the neural network inference chip 100 further includes a vector operation unit array 40. The vector operation unit array 40 includes a first vector operation unit subarray 41 located in the same row as the first operation unit subarray 21, and a second vector operation unit subarray 42 located in the same row as the second operation unit subarray 22. The vector operation unit array 40 is used to store the feature data of each layer input of the neural network and to perform operator calculations other than convolution.

[0069] In this embodiment, the vector operation unit array includes N×1 vector operation units. Here, the vector operation unit array is an array composed of vector scalar processors (VSPs), and the vector operation unit is a vsp_core. The vector operation unit array includes N×1 vsp_cores, and the N vsp_cores can work in parallel.

[0070] In this embodiment of the application, the vector operation unit subarray includes K×1 vector operation units, where K is an integer greater than or equal to 1 and less than or equal to N / 2.

[0071] In this embodiment, the first vector operation unit subarray 41, located in the same row as the first operation unit subarray 21, stores the first feature data of the input channel, and the second vector operation unit subarray 42, located in the same row as the second operation unit subarray 22, stores the second feature data of the input channel. Further, the control unit 10 obtains the first feature data from the first vector operation unit subarray 41 and inputs the first feature data into the first operation unit subarray 21 for operation; similarly, the control unit 10 obtains the second feature data from the second vector operation unit subarray 42 and inputs the second feature data into the second operation unit subarray 22 for operation.

[0072] In a feasible application scenario, taking N=4 and K=2 as an example, the following explanation is provided. Figure 4 , Figure 4This is a schematic diagram of an optional neural network inference chip architecture provided in an embodiment of this application. The vector operation unit array 40 includes 4×1 vsp_cores, grouped into K=2 vsp_cores. The vector operation unit subarray 40 includes 2×1 vsp_cores, namely Group00 and Group10. The operation unit array 20 includes 4×4 mmp_cores, grouped into K=2 mmp_cores. Each first operation unit subarray 21 includes 2×2 mmp_cores, each first operation unit subarray being Harden00 and Harden01. Each second operation unit subarray 22 includes 2×2 mmp_cores, each second operation unit subarray being Harden10 and Harden11. Thus, Harden is used as the smallest unit for data transmission and feature data is transmitted. Here, for the operation unit array 20, the number of rows corresponds to the IC parallelism in convolution operations, and the number of columns corresponds to the OC parallelism in convolution operations. The partial sum accumulation cell array 30 includes 1×4 psum_cores, grouped into K=2 psum_cores. The partial sum accumulation cell subarray 30 includes 1×2 psum_cores, namely Set00 and Set01.

[0073] Continue to refer to Figure 4 It includes feature data from four input channels, namely IC0, IC1, IC2, and IC3. Each row of feature data is transmitted as a data stream. Figure 4 As shown by the solid arrows, each row of MMP cores shares the same feature data. After two beat cycles, the feature data can be transferred to the second harden01 in the first row and the first harden10 in the second row. The convolution operation and psum are accumulated and transferred in the column direction, and the psum data flow is as follows: Figure 2 As shown by the dashed arrow. Here, psum1 of harden00 is added to psum2 of harden10 and then propagated downwards in a staccato pattern. Each part and accumulation unit accumulates the column-wise portion and the final output. This systolic array data stream can perform convolution and fully connected operations very well.

[0074] Continue to refer to Figure 4Taking the first column of the operation unit subarray as an example, firstly, after the first tapping cycle, the control unit (not shown in the figure) obtains the first feature data from the first vector operation unit subarray 41 and inputs the first feature data into the first operation unit subarray Harden00. Based on the first feature data and the stored first weight data corresponding to the first feature data, the first operation unit subarray Harden00 obtains the first subarray operation result psum00 corresponding to the array and outputs the obtained first subarray operation result psum00 to the second operation unit subarray Harden10.

[0075] Secondly, after the second tapping cycle, the control unit controls the transmission of the first feature data to the first computing unit subarray Harden01 in a data flow manner. Based on the first feature data and the stored first weight data corresponding to the first feature data, the first computing unit subarray Harden01 obtains the first subarray operation result psum01 corresponding to the subarray and outputs the obtained first subarray operation result psum01 to the second computing unit subarray Harden11.

[0076] Meanwhile, during the second tapping cycle, the control unit also obtains the second feature data from the second vector operation unit subarray 42, inputs the second feature data into the second operation unit subarray Harden10, and determines the target selection branch in the second operation unit subarray Harden10 and the second operation unit subarray Harden11 based on the input first feature data and second feature data, and activates the target selection branch.

[0077] Then, the second subarray Harden10 obtains the intermediate operation result of the second subarray based on the second feature data and the stored second weight data corresponding to the second feature data; and through the activated target selection branch, it performs operations on the intermediate operation result of the second subarray and the operation result of the first subarray to obtain the operation result psum10 of the second subarray; the operation result psum10 of the second subarray is output to the partial sum accumulation unit subarray Set00, and the partial sum accumulation unit subarray Set00 performs partial sum accumulation operation on the operation result psum10 of the second subarray. In this way, the data flow of the systolic array can effectively complete the convolution and fully connected operations.

[0078] In some embodiments, refer to Figure 5 As shown, Figure 5 This is a schematic diagram of an optional neural network inference chip provided in an embodiment of this application, wherein the target selection branch 221 includes a first selection branch 222, wherein...

[0079] The control unit 10 is used to control the input of first feature data and second feature data of different byte bits corresponding to the same input channel into the first arithmetic unit subarray 21 and the second arithmetic unit subarray 22 respectively, and to determine the first selection branch 222 and enable the first selection branch 222.

[0080] The second operation unit subarray 22 is used to obtain the intermediate operation result of the second subarray based on the second feature data and the second weight data; to perform shift processing on the intermediate operation result of the second subarray through the first selection branch 222; to sum the shifted intermediate operation result of the second subarray with the operation result of the first subarray to obtain the operation result of the second subarray; and to output the operation result of the second subarray to the partial sum accumulation unit subarray 31 located in the same column for accumulation operation.

[0081] In this embodiment, firstly, the control unit inputs feature data of different byte positions from the same input channel (the feature data of different byte positions includes first feature data and second feature data) into the first and second arithmetic unit subarrays respectively. For example, the feature data of a complete byte corresponding to the same input channel is divided into high-byte feature data and low-byte feature data, wherein the first feature data is low-byte feature data and the second feature data is high-byte feature data. Secondly, the control unit determines the first selection branch of the second arithmetic unit subarray and controls the activation of the first selection branch. Then, the first arithmetic unit subarray obtains the first subarray operation result based on the low-byte feature data and the stored first weight data corresponding to the low-byte feature data, and outputs the obtained first subarray operation result to the second arithmetic unit subarray. Furthermore, the second operation unit subarray obtains the intermediate operation result of the second subarray based on the high-byte feature data and the stored second weight data corresponding to the high-byte feature data; and shifts the intermediate operation result of the second subarray through the activated first selection branch to obtain the shifted intermediate operation result of the second subarray; and sums the shifted intermediate operation result of the second subarray with the operation result of the first subarray to obtain the operation result of the second subarray; and outputs the operation result of the second subarray to the partial sum accumulation unit subarray located in the same column, and the partial sum accumulation unit subarray performs the partial sum accumulation operation on the operation result of the second subarray.

[0082] As described above, in this embodiment, when the feature data input to the first and second arithmetic unit subarrays are feature data of different byte positions (i.e., high byte feature data and low byte feature data) of the same input channel, a first selection branch is determined from multiple selection branches of the second arithmetic unit subarray. Through the first selection branch, the intermediate operation result of the second subarray obtained by the second arithmetic unit subarray is shifted. The shifted intermediate operation result of the second subarray is then operated with the operation result of the first subarray obtained by the first arithmetic unit subarray to obtain the second subarray corresponding to the second arithmetic unit subarray. The results of the column operations are then used. Thus, this application provides a chip architecture capable of handling application scenarios where the number of input channels (IC) of the network layer is not an integer multiple of the parallelism of the chip IC / 2, and high bit data quantization precision is required. In this application scenario, the operation result obtained based on high-byte feature data is shifted through a first selected branch and summed with the operation result obtained based on low-byte feature data to obtain the operation result of the subarray. This improves the chip's utilization and computational efficiency in this application scenario, while also increasing the utilization of the array operation units, reducing network inference latency, and accelerating network layer inference.

[0083] In other embodiments of this application, refer to Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of the first selection branch provided in an embodiment of this application. The first selection branch 222 includes a selector 2221, a shifter 2222, and an accumulator 2223.

[0084] Control unit 10 is used to control selector 2221 to select the first selection branch 222;

[0085] Shifter 2222 is used to shift the intermediate operation result of the second subarray to the left by the target shift number when the target shift number of the operation result obtained after the operation on the second feature data is determined based on the number of bytes of the first feature data of the same input channel, so as to obtain the shifted intermediate operation result of the second subarray.

[0086] Accumulator 2223 is used to sum the intermediate operation result of the shifted second subarray with the operation result of the first subarray to obtain the operation result of the second subarray.

[0087] In this embodiment, the target shift number is determined based on the number of bytes of the first feature data of the same input channel. For example, if the feature data of the complete byte corresponding to the same input channel is 16 bits, the 16-bit complete byte feature data is divided into 8 bits of high-byte feature data (corresponding to the second feature data) and 8 bits of low-byte feature data (corresponding to the first feature data). Based on the 8 bits of low-byte feature data, the target shift number of the operation result obtained after operating on the 8 bits of high-byte feature data is determined to be 8 bits. Here, the operation result obtained after operating on the 8 bits of high-byte feature data can be understood as: operating on the 8 bits of high-byte feature data to obtain the intermediate operation result of the second subarray corresponding to the 8 bits of high-byte feature data.

[0088] In this embodiment, after the control unit controls the selector to select the first selection branch, firstly, the shifter determines the target shift number of the operation result obtained after operating on the second feature data based on the number of bytes of the first feature data of the same input channel. Then, the shifter shifts the intermediate operation result of the second subarray to the left by the target shift number to obtain the shifted intermediate operation result of the second subarray. Then, the accumulator sums the shifted intermediate operation result of the second subarray with the operation result of the first subarray to obtain the operation result of the second subarray. The operation result of the second subarray is then output to the partial sum accumulator subarray located in the same column. The partial sum accumulator subarray performs the partial sum accumulator operation on the operation result of the second subarray.

[0089] As described above, in this embodiment of the application, a chip architecture is provided that can realize application scenarios where the number of input channels IC of the network layer is not an integer multiple of the parallelism of the chip IC / 2, and high bit data quantization accuracy is required. In this application scenario, the shifter selected by the selector performs shift processing on the operation result obtained based on the high byte feature data, and the accumulator sums the operation result obtained based on the low byte feature data to obtain the operation result of the subarray. In this way, by adding selectors and shifters, that is, by adding a very small circuit area, the utilization rate of the array operation unit is greatly improved, the network inference latency is reduced, and the inference of the network layer is accelerated.

[0090] In some embodiments, continue to refer to Figure 5 Target selection branch 221 includes second selection branch 223, wherein,

[0091] The control unit 10 is used to control the input of the first feature data and the second feature data corresponding to different input channels into the first arithmetic unit subarray 21 and the second arithmetic unit subarray 22, respectively, and to determine the second selection branch 223 and enable the second selection branch 223.

[0092] The second subarray of the second operation unit is used to obtain the intermediate operation result of the second subarray based on the second feature data and the second weight data; the intermediate operation result of the second subarray is summed with the operation result of the first subarray through the second selection branch 223 to obtain the operation result of the second subarray; the operation result of the second subarray is output to the partial sum accumulation unit subarray 31 located in the same column for accumulation operation.

[0093] In this embodiment, firstly, the control unit inputs the first feature data and the second feature data corresponding to different input channels into the first and second operation unit subarrays, respectively. For example, the feature data of the complete bytes corresponding to different input channels are used as the first feature data and the second feature data, respectively. Secondly, the control unit determines the second selection branch of the second operation unit subarray and controls the second selection branch to be enabled. Then, the first operation unit subarray obtains the first subarray operation result based on the first feature data and the stored first weight data corresponding to the first feature data, and outputs the obtained first subarray operation result to the second operation unit subarray. Further, the second operation unit subarray obtains the second subarray intermediate operation result based on the second feature data and the stored second weight data corresponding to the second feature data; and sums the second subarray intermediate operation result with the first subarray operation result through the enabled second selection branch to obtain the second subarray operation result; the second subarray operation result is output to the partial sum accumulation unit subarray located in the same column, and the partial sum accumulation unit subarray performs a partial sum accumulation operation on the second subarray operation result.

[0094] As described above, in this embodiment, when the feature data input to the first and second subarrays of the computing unit are complete bytes of feature data from different input channels, a second selection branch is determined from multiple selection branches of the second subarray. Through the second selection branch, the intermediate operation result of the second subarray obtained by the second subarray is summed with the operation result of the first subarray obtained by the first subarray, thereby obtaining the operation result of the second subarray corresponding to the second subarray. Thus, this embodiment provides a chip architecture capable of realizing that the number of input channels IC in the network layer is an integer multiple of the parallelism of the chip IC, and supporting application scenarios with different quantization precision requirements. In this application scenario, the operation results obtained based on different feature data are summed through the second selection branch to obtain the operation result of the subarray. This improves the utilization and computational efficiency of the chip in this application scenario, while also increasing the utilization of the array computing units, reducing network inference latency, and accelerating network layer inference.

[0095] In other embodiments of this application, refer to Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of the second selection branch provided in an embodiment of this application. The second selection branch 223 includes a selector 2221 and an accumulator 2223.

[0096] Control unit 10 is used to control selector 2221 to select the second selection branch 223;

[0097] Accumulator 2223 is used to sum the intermediate operation result of the second subarray with the operation result of the first subarray to obtain the operation result of the second subarray.

[0098] In this embodiment, after the control unit controls the selector to select the second selection branch, it sums the intermediate calculation result of the second subarray with the calculation result of the first subarray to obtain the calculation result of the second subarray. This second subarray calculation result is then output to the partial sum accumulation unit subarray located in the same column. The partial sum accumulation unit subarray performs a partial sum accumulation operation on the calculation result of the second subarray. It should be noted that the first and second selection branches share a single selector, and each branch has its own accumulator. Of course, the accumulator can be reused, meaning the first and second selection branches share the same selector and accumulator to reduce chip area.

[0099] In a feasible application scenario, refer to Figure 8 As shown, Figure 8 This application provides a circuit diagram of an optional selection branch. Selector MUX2221 selects the first selection branch 222, shifts the intermediate operation result of the second subarray through shifter 2222, and uses an accumulator to combine the shifted intermediate operation result of the second subarray with the operation result psum_in of the first subarray to obtain the operation result PSUM_OUT of the second subarray; and selector MUX2221 selects the second selection branch 223, and uses an accumulator to combine the intermediate operation result of the second subarray with the operation result psum_in of the first subarray to obtain the operation result PSUM_OUT of the second subarray.

[0100] To achieve the functionality of the aforementioned subarray of computational units, in other embodiments of this application, the subarray of computational units includes at least two merging computational units located in different columns, the merging computational units include at least one first computational unit and one second computational unit located in the same column, and the feature data includes at least third feature data and fourth feature data, wherein...

[0101] The control unit is used to control the input of the third feature data and the fourth feature data corresponding to different input channels into the first arithmetic unit and the second arithmetic unit, respectively.

[0102] The first operation unit is used to perform positional multiplication calculations on the third feature data and the third weight data to obtain the first product, and then output it to the second operation unit.

[0103] The second operation unit is used to perform positional multiplication calculations on the fourth feature data and the fourth weight data to obtain the second product; to sum the first product and the second product to obtain the partial sum accumulation result and store it in the tap register; and to perform accumulation processing on the partial sum accumulation result through the addition tree to obtain the initial subarray operation result.

[0104] Understandably, the merging operation unit includes a first operation unit and a second operation unit located in the same column. It should be noted that there is at least one first operation unit, and the number of first operation units corresponds to the number of feature data received from the input channels; that is, one first operation unit corresponds to the feature data of one input channel, two first operation units correspond to the feature data of two input channels, ..., N first operation units correspond to the feature data of N input channels.

[0105] In this embodiment of the application, the adder_tree consists of multiple accumulation units (MACs), and the adder_tree is used to perform accumulation operations on partial sums and accumulation results.

[0106] Here, the control unit inputs third feature data to the first arithmetic unit and second feature data to the second arithmetic unit. Upon receiving the third feature data, the first arithmetic unit multiplies the third feature data with the corresponding stored third weight data element-wise at the appropriate positions to obtain a first product, which is then output to the second arithmetic unit. The second arithmetic unit multiplies the fourth feature data with the corresponding stored fourth weight data element-wise at the appropriate positions to obtain a second product. Then, the second product is summed with the first product from the first arithmetic unit using an adder in the second arithmetic unit, and the sum is stored in a tap register. Finally, the partial sums are accumulated using an addition tree to obtain the initial subarray operation result.

[0107] In other words, the first arithmetic unit performs product processing on the third feature data and the third weight data of the input channel at corresponding positions, without performing an addition transformation. Instead, the first product is output to the second arithmetic unit, where the adder sums the second product (which is the product of the fourth feature data and the fourth weight data of the input channel at corresponding positions) with the first product to obtain the summation result, which is then stored in the tap register. This allows for the processing of feature data from different input channels in N / K tap cycles, where the feature data flows in units of the arithmetic unit subarrays. In each tap cycle, only one set of adders is inserted into the arithmetic unit subarrays for summation, and one set of registers is used for data storage. This reduces the number of adders and tap registers required, decreases chip area and power consumption, and improves the performance of the neural network inference chip.

[0108] In other embodiments of this application, the arithmetic unit array includes P first arithmetic unit subarrays located in the same column, the first first arithmetic unit subarray includes the aforementioned arithmetic unit subarray; the p-th first arithmetic unit subarray includes the aforementioned arithmetic unit subarray and an addition unit, where p is an integer greater than 1 and less than or equal to P, P = N / 2K, and the first feature data includes at least the fifth feature data, the sixth feature data, and the seventh feature data, wherein...

[0109] The first first operation unit subarray is used to obtain the initial subarray operation result based on the fifth feature data and the stored fifth weight data corresponding to the fifth feature data through the operation unit subarray. This initial subarray operation result is the first first subarray operation result corresponding to the first first operation unit subarray, and the first first subarray operation result is transmitted to the next first operation unit subarray.

[0110] The p-th first operation unit subarray is used to obtain the initial subarray operation result based on the sixth feature data and the stored sixth weight data corresponding to the sixth feature data. This initial subarray operation result is the intermediate operation result of the p-th first subarray corresponding to the p-th first operation unit subarray. The intermediate operation result of the p-th first subarray is summed with the operation results of the p-1 first subarrays through the addition unit to obtain the operation result of the p-th first subarray, and the operation result of the p-th first subarray is transmitted to the p+1-th first operation unit subarray.

[0111] The Pth first operation unit subarray is used to obtain the initial subarray operation result based on the seventh feature data and the stored seventh weight data corresponding to the seventh feature data. This initial subarray operation result is the intermediate operation result of the Pth first subarray corresponding to the Pth first operation unit subarray. The intermediate operation result of the Pth first subarray and the operation results of P-1 first subarrays are summed by the addition unit to obtain the operation result of the Pth first subarray. The operation result of the Pth first subarray is then transmitted to the second operation unit subarray.

[0112] In other embodiments of this application, the arithmetic unit array includes P second arithmetic unit subarrays located in the same column, the p-th second arithmetic unit subarray includes the aforementioned arithmetic unit subarray and a target selection branch, p is an integer greater than or equal to 1 and less than or equal to P, P = N / 2K, and the second feature data includes at least the eighth feature data, the ninth feature data, and the tenth feature data, wherein...

[0113] The first second operation unit subarray is used to perform an initial subarray operation based on the eighth feature data and the stored eighth weight data corresponding to the eighth feature data. The initial subarray operation result obtained through the operation unit subarray is the intermediate operation result of the first second subarray corresponding to the first second operation unit subarray. Through the target selection branch, the intermediate operation result of the first second subarray is operated on with the first subarray operation result transmitted to the first second operation unit subarray to obtain the first second subarray operation result. The first second subarray operation result is then transmitted to the next second operation unit subarray.

[0114] The p-th second operation unit subarray is used to obtain the initial subarray operation result based on the ninth feature data and the stored ninth weight data corresponding to the ninth feature data. This initial subarray operation result is the intermediate operation result of the p-th second subarray corresponding to the p-th second operation unit subarray. The intermediate operation result of the p-th second subarray and the operation result of the (p-1)-th second subarray are operated on through the target selection branch to obtain the operation result of the p-th second subarray. The operation result of the p-th second subarray is then transmitted to the (p+1)-th second operation unit subarray.

[0115] The Pth second operation unit subarray is used to obtain the initial subarray operation result based on the tenth feature data and the stored tenth weight data corresponding to the tenth feature data. This initial subarray operation result is the intermediate operation result of the Pth second subarray corresponding to the Pth second operation unit subarray. The intermediate operation result of the Pth second subarray and the operation result of the (P-1)th second subarray are operated on through the target selection branch to obtain the operation result of the Pth second subarray. The operation result of the Pth second subarray is then transmitted to the partial sum accumulation unit subarray located in the same column for accumulation operation.

[0116] In a feasible application scenario, taking N=8 and K=2 as an example, the following explanation is provided. Figure 9 , Figure 9 This is a schematic diagram of an optional neural network inference chip architecture provided for an embodiment of this application.

[0117] Here, the vector operation unit array includes 8×1 vsp_cores, grouped into K=2 vsp_cores. The vector operation unit subarray includes 2×1 vsp_cores, namely Group00, Group10, Group20 and Group30. In other words, every two vsp_cores constitute a vector operation unit subarray Group to provide operation data to the corresponding MMP row.

[0118] Here, the computational unit array comprises 8×8 mmp_cores, grouped into K=2 mmp_cores. Each first computational unit subarray comprises 2×2 mmp_cores, namely Harden00, Harden01, Harden02, Harden03, Harden10, Harden11, Harden12, and Harden13. Each second computational unit subarray comprises 2×2 mmp_cores, namely Harden20, Harden21, Harden22, Harden23, Harden30, Harden31, Harden32, and Harden33. Each second computational unit subarray includes a selection branch. It should be noted that the MMP is composed of 8×8 arrays of mmp_cores. The number of rows corresponds to the inter-integral (IC) parallelism in convolution computation, and the number of columns corresponds to the open-ended (OC) parallelism in convolution computation. This array-based computational structure utilizes the data reusability of convolution computation, reducing data movement. Meanwhile, in order to further reduce circuit area and power consumption, every four mmp_cores constitute a harden. The harden serves as the smallest unit for data transmission timing, reducing the number of timing registers required by 50%.

[0119] Here, the partial and cumulative cell array includes 1×4 psum_cores, grouped into K=2 psum_cores. Each partial and cumulative cell subarray includes 1×2 psum_cores, namely Set00, Set01, Set02 and Set03.

[0120] Here, the vector processor (VSP) and the multiplication matrix processor (MMP) are the computational units of the neural network processor (NPU). The VSP is responsible for the computation of non-convolutional operators and the storage of features, while the MMP is responsible for the computation of convolutional and fully connected operators and the storage of weight parameters.

[0121] Continue to refer to Figure 9 Each row of feature data transmission data stream is as follows Figure 9 As shown by the solid arrows, each row of `mmp_core` shares the same feature data. After four beat cycles (Cycle1, Cycle2, Cycle3, and Cycle4), the feature data is transmitted to the last harden03 in the first row, the third harden12 in the second row, the second harden21 in the third row, and the first harden30 in the fourth row. The convolutional operations and `psum` are accumulated and transmitted in the column direction, and the `psum` data stream is as follows: Figure 9 As shown by the dashed arrow, the psum of harden00 is exactly added to the psum of harden10 and transmitted downwards in a staccato pattern, along with the accumulated portion in the column direction of the 4th cycle and the final output. This systolic array data stream can perform convolution and fully connected operations very well.

[0122] In one implementation, continue to refer to Figure 9If the current network layer has 4 input channels, that is, 4 input channels with complete byte feature data, and the complete byte feature data to be quantized is 16 bits, where the 4 input channels are IC0, IC1, IC2 and IC3 respectively. First, the complete byte feature data corresponding to each input channel is divided into high byte feature data data_in[7:0] and low byte feature data data_in[15:8], and stored in the corresponding vector operation unit. Since each second operation unit subarray structure includes a first selection branch, at this time, the control unit controls each second operation unit subarray structure to open the first selection branch for data operation on feature data of different byte bits of the same input channel. At this time, it can be seen that in this structure, the first operation unit subarray of the first two rows calculates the low byte feature data data_in[7:0] of IC0-IC3 respectively, and the second operation unit subarray of the last two rows calculates the high byte feature data data_in[15:8] of IC0-IC3 respectively. Partial summation is performed in the column direction. The first selected branch performs multiplication of the high and low bytes of the feature data and the weight data through shifting and addition. The calculation process is as follows: Formula (1).

[0123] mul=(fm H ×w)<<HB+fm L ×w (1)

[0124] Among them, fm H For high-byte feature data, fm L is the low-byte feature data, w is the weight data corresponding to the feature data, and HB is the left shift bit. In this embodiment, HB is 8.

[0125] As shown above, when the parallelism of the neural network inference chip is 8, if the number of input channels in the current network layer is 4, the utilization rate of the current network layer is 8 / 8 × 100% = 100%. Similarly, if the number of input channels in the current network layer is 60, the number of iterations required for iterative calculation using this architecture is... In the first 7 iterations, the number of input channels was 8, and the feature data was processed by selecting the second selection branch. The utilization rate of the network layer was 100% during the first 7 iterations. In the last iteration, the number of input channels was 4. In the last iteration calculation, the feature data of each channel was divided into high-byte feature data and low-byte feature data and input into different rows. In the last iteration, the utilization rate of the network layer was 100%. The total utilization rate of the current network layer is 64 / 64 = 100%. In this way, the utilization rate of the network layer is improved by providing the neural network inference chip.

[0126] Here, we will use the circuit structure of arithmetic units located in the same column in the 2×2 second arithmetic unit subarray as an example for explanation, referring to... Figure 10 , Figure 10 This is a schematic diagram of the circuit structure of the arithmetic units located in the same column of the 2×2 second arithmetic unit subarray. In this diagram, `mmp_core0` in the first row receives the high-byte feature data from `IC0`. This high-byte feature data from `IC0` undergoes feature rearrangement via the threshold setting module `fm` and the preprocessing module `pre`. Since the weight data has been pre-loaded into the internal storage unit `memory` in `mmp_core`, the weight data can be rearranged via the weight control module `wgt_ctrl`. The rearranged weight data is then multiplied by the input feature data in the multiplier module `pu`. It should be noted that the `wgt_ctrl` module is used to concatenate and rearrange the weight data in `memory`. The `fm` module is used to constrain the data according to the stride length of different convolution operators, setting the connections directly to 0 to prevent connection flipping and reduce power consumption. The `pre` module is used to rearrange the feature data according to the stride length of different convolution operators and the size of the convolution kernel. Additionally, Figure 10 The ext_in instruction is used to preload the weight data into the computation unit mmp_core, and to retrieve the weight data. mmp_cmd can be understood as a command to give the weight address to the internal storage unit memory (mem) of the computation unit mmp_core.

[0127] The second line, `mmp_core1`, receives the high-byte feature data from IC1. This high-byte feature data from IC1 undergoes feature rearrangement via the `fm` module and the `pre` module. Since the weight data has been pre-loaded into the internal memory of `mmp_core`, it can be rearranged using the weight control module `wgt_ctrl`. The rearranged weight data is then multiplied and accumulated with the input feature data in the `pu` module. The partial sums of different ICs are accumulated to obtain the partial sum accumulation result, which is the MAC data. The MAC data is then input into the addition tree for further accumulation. Finally, it is accumulated with the `psum_in` data transmitted from the previous line `harden`, and `psum_out` is output to the next line `harden` until all four lines of `harden` have been processed.

[0128] This application provides a neural network inference chip, comprising: a control unit, an array of arithmetic units, and an array of partial sum-accumulation units. The array of arithmetic units includes a first subarray of arithmetic units and a second subarray of arithmetic units located in the same column. The control unit is used to control the input of first feature data and second feature data corresponding to the input channel into the first subarray of arithmetic units and the second subarray of arithmetic units, respectively; to determine and activate the target selection branch of the second subarray of arithmetic units; the first subarray of arithmetic units is used to obtain the operation result of the first subarray based on the first feature data and the first weight data, and output it to the second subarray of arithmetic units; the second subarray of arithmetic units is used to obtain the intermediate operation result of the second subarray based on the second feature data and the second weight data; to perform operations on the intermediate operation result of the second subarray and the operation result of the first subarray through the target selection branch to obtain the operation result of the second subarray; and to output the operation result of the second subarray to the partial sum-accumulation unit subarray located in the same column as the second subarray of arithmetic units for accumulation operation. In other words, in this embodiment, based on the feature data of the input channels to the first and second operation unit subarrays, a target selection branch is determined from multiple selection branches of the second operation unit subarray. Through the target selection branch, the intermediate operation result of the second subarray obtained by the second operation unit subarray is compared with the operation result of the first subarray obtained by the first operation unit subarray to obtain the corresponding second subarray operation result. Thus, this embodiment provides a chip architecture capable of supporting application scenarios with multiple input channels and different quantization precision requirements in the network layer. Different selection branches are selected for operation processing in different application scenarios, improving the chip's utilization and computational efficiency in different application scenarios, increasing the utilization of the array operation units, reducing network inference latency, and accelerating network layer inference.

[0129] Based on the same inventive concept, this application provides a neural network inference method that can be applied to a neural network inference chip. The neural network inference chip includes a control unit, an array of arithmetic units, and an array of partial summation and accumulation units. The arithmetic unit array includes a first arithmetic unit subarray and a second arithmetic unit subarray located in the same column. Figure 11 A flowchart illustrating an optional neural network inference method provided in an embodiment of this application is shown below. Figure 11 As shown, the neural network inference method may include:

[0130] Step 501: The control unit controls the input of the first feature data and the second feature data corresponding to the input channel into the first arithmetic unit subarray and the second arithmetic unit subarray respectively; determines the target selection branch of the second arithmetic unit subarray and activates the target selection branch;

[0131] Step 502: The first subarray of the first computing unit obtains the computing result of the first subarray based on the first feature data and the first weight data, and outputs it to the second subarray of the computing unit.

[0132] Step 503: The second subarray of the second operation unit obtains the intermediate operation result of the second subarray based on the second feature data and the second weight data; the intermediate operation result of the second subarray and the operation result of the first subarray are operated on through the target selection branch to obtain the operation result of the second subarray; the operation result of the second subarray is output to the partial sum accumulation subarray located in the same column as the second operation unit subarray in the partial sum accumulation subarray for accumulation operation.

[0133] In one alternative embodiment, the target selection branch includes a first selection branch, wherein,

[0134] The control unit controls the input of first feature data and second feature data of different byte bits corresponding to the same input channel into the first arithmetic unit subarray and the second arithmetic unit subarray respectively, and determines the first selection branch and activates the first selection branch;

[0135] The second subarray of the second operation unit obtains the intermediate operation result of the second subarray based on the second feature data and the second weight data; the intermediate operation result of the second subarray is shifted through the first selection branch; the shifted intermediate operation result of the second subarray is summed with the operation result of the first subarray to obtain the operation result of the second subarray; the operation result of the second subarray is output to the partial sum accumulation unit subarray located in the same column for accumulation operation.

[0136] In one optional embodiment, the first selection branch includes a selector, a shifter, and an accumulator, wherein,

[0137] The control unit controls the selector to select the first selection branch;

[0138] Based on the first feature data and the second feature data, the shifter determines the target number of shifts, and then shifts the intermediate operation result of the second subarray to the left by the target number of shifts to obtain the shifted intermediate operation result of the second subarray.

[0139] The accumulator sums the intermediate operation result of the shifted second subarray with the operation result of the first subarray to obtain the operation result of the second subarray.

[0140] In one alternative embodiment, the target selection branch includes a second selection branch, wherein,

[0141] The control unit controls the input of the first feature data and the second feature data corresponding to different input channels into the first arithmetic unit subarray and the second arithmetic unit subarray, respectively, and determines the second selection branch and activates the second selection branch.

[0142] The second subarray of the operation unit obtains the intermediate operation result of the second subarray based on the second feature data and the second weight data; through the second selection branch, the intermediate operation result of the second subarray is summed with the operation result of the first subarray to obtain the operation result of the second subarray; the operation result of the second subarray is output to the partial sum accumulation unit subarray located in the same column for accumulation operation.

[0143] In one alternative embodiment, the second selection branch includes a selector and an accumulator, wherein,

[0144] The control unit controls the selector to select the second selection branch;

[0145] The accumulator sums the intermediate operation results of the second subarray with the operation results of the first subarray to obtain the operation result of the second subarray.

[0146] In one optional embodiment, the subarray of arithmetic units includes at least two merge arithmetic units located in different columns, the merge arithmetic units include at least one first arithmetic unit and one second arithmetic unit located in the same column, and the feature data includes at least third feature data and fourth feature data, wherein,

[0147] The control unit controls the input of the third and fourth feature data corresponding to different input channels into the first and second arithmetic units, respectively.

[0148] The first processing unit performs a multiplication calculation on the third feature data and the third weight data at the corresponding positions to obtain the first product, and outputs it to the second processing unit;

[0149] The second operation unit performs positional multiplication on the fourth feature data and the fourth weight data to obtain the second product; it sums the first product and the second product to obtain the partial sum accumulation result and stores it in the tap register; it then performs accumulation processing on the partial sum accumulation result through the addition tree to obtain the initial subarray operation result.

[0150] Based on the same inventive concept, embodiments of this application provide a terminal. Figure 12 A schematic diagram of an optional terminal provided in an embodiment of this application, such as... Figure 12 As shown, the terminal 200 includes the neural network inference chip 100 described in one or more of the above embodiments.

[0151] Based on the same inventive concept, embodiments of this application provide a terminal. Figure 13A schematic diagram of an optional terminal provided in an embodiment of this application, such as... Figure 13 As shown, the terminal 200 may also include a memory 25 storing executable instructions of the processor 24. Furthermore, the terminal 200 may also include a communication interface 26, a bus 27 for connecting the processor 24, the memory 25 and the communication interface 26, and a neural network inference chip 100.

[0152] In the embodiments of this application, the processor 24 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and this application embodiment does not specifically limit it. The terminal 200 may also include a memory 25, which can be connected to the processor 24. The memory 25 is used to store executable program code, which includes computer operation instructions. The memory 25 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.

[0153] In embodiments of this application, bus 27 is used to connect communication interface 26, processor 24 and memory 25, neural network inference chip 100 and the mutual communication between these devices.

[0154] In embodiments of this application, memory 25 is used to store instructions and data.

[0155] In practical applications, the aforementioned memory 25 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 24.

[0156] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0157] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage media include: ferromagnetic random access memory (FRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or compact disk-read-only memory (CD-ROM), and other media capable of storing program code; or various devices including one or any combination of the above-mentioned memories.

[0158] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0159] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file containing other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTL) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0160] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A neural network inference chip, characterized in that, The neural network inference chip includes: a control unit, an array of arithmetic units, and an array of partial summation and accumulation units. The arithmetic unit array includes a first arithmetic unit subarray and a second arithmetic unit subarray located in the same column. The control unit is used to control the input of the first feature data and the second feature data corresponding to the input channel into the first computing unit subarray and the second computing unit subarray, respectively; to determine the target selection branch of the second computing unit subarray and to activate the target selection branch; The first computing unit subarray is used to obtain the computing result of the first subarray based on the first feature data and the first weight data, and output it to the second computing unit subarray; The second subarray of computing units is used to obtain intermediate computing results of the second subarray based on the second feature data and the second weight data; to perform calculations on the intermediate computing results of the second subarray and the computing results of the first subarray through the target selection branch to obtain the computing result of the second subarray; and to output the computing result of the second subarray to the part and accumulation unit subarray located in the same column as the second computing unit subarray in the part and accumulation unit array for accumulation calculation. The target selection branch includes a first selection branch, wherein... The control unit is used to control the input of the first feature data and the second feature data of different byte bits corresponding to the same input channel into the first operation unit subarray and the second operation unit subarray, respectively, and to determine the first selection branch and activate the first selection branch. The second subarray of computing units is used to obtain intermediate computing results of the second subarray based on the second feature data and the second weight data; to shift the intermediate computing results of the second subarray through the first selection branch; to sum the shifted intermediate computing results of the second subarray with the computing results of the first subarray to obtain the computing results of the second subarray; and to output the computing results of the second subarray to the partial sum accumulation unit subarray located in the same column for accumulation operation.

2. The neural network inference chip according to claim 1, characterized in that, The first selection branch includes a selector, a shifter, and an accumulator, wherein, The control unit is used to control the selector to select the first selection branch; The shifter is used to shift the intermediate operation result of the second subarray to the left by the target shift number when the target shift number of the operation result obtained after operating on the second feature data is determined based on the number of bytes of the first feature data in the same input channel, so as to obtain the shifted intermediate operation result of the second subarray. The accumulator is used to sum the intermediate operation result of the shifted second subarray with the operation result of the first subarray to obtain the operation result of the second subarray.

3. The neural network inference chip according to claim 1, characterized in that, The target selection branch includes a second selection branch, wherein... The control unit is used to control the input of the first feature data and the second feature data corresponding to different input channels into the first operation unit subarray and the second operation unit subarray, respectively, and to determine the second selection branch and activate the second selection branch. The second subarray of computing units is used to obtain intermediate computing results of the second subarray based on the second feature data and the second weight data; to sum the intermediate computing results of the second subarray with the computing results of the first subarray through the second selected branch to obtain the computing results of the second subarray; and to output the computing results of the second subarray to the partial sum accumulation unit subarray located in the same column for accumulation operation.

4. The neural network inference chip according to claim 3, characterized in that, The second selection branch includes a selector and an accumulator, wherein, The control unit is used to control the selector to select the second selection branch; The accumulator is used to sum the intermediate operation result of the second subarray with the operation result of the first subarray to obtain the operation result of the second subarray.

5. The neural network inference chip according to any one of claims 1 to 4, characterized in that, The subarray of operational units includes at least two merge operational units located in different columns. Each merge operational unit includes at least one first operational unit and one second operational unit located in the same column. The feature data includes at least third and fourth feature data. The control unit is used to control the input of the third feature data and the fourth feature data corresponding to different input channels into the first arithmetic unit and the second arithmetic unit respectively; The first processing unit is used to perform positional multiplication calculations on the third feature data and the third weight data to obtain a first product, and output it to the second processing unit; The second operation unit is used to perform positional multiplication calculations on the fourth feature data and the fourth weight data to obtain a second product; to sum the first product and the second product to obtain a partial sum accumulation result, and store it in the beat register; and to perform accumulation processing on the partial sum accumulation result through an addition tree to obtain the initial subarray operation result.

6. A neural network inference method, characterized in that, The neural network inference method is applied to a neural network inference chip, which includes: a control unit, an array of arithmetic units, and an array of partial summation and accumulation units. The arithmetic unit array includes a first arithmetic unit subarray and a second arithmetic unit subarray located in the same column. The control unit controls the input of the first feature data and the second feature data corresponding to the input channel into the first computing unit subarray and the second computing unit subarray, respectively; determines the target selection branch of the second computing unit subarray and activates the target selection branch; The first subarray of computing units obtains the computing result of the first subarray based on the first feature data and the first weight data, and outputs it to the second subarray of computing units; The second subarray of the computing unit obtains the intermediate computing result of the second subarray based on the second feature data and the second weight data; through the target selection branch, the intermediate computing result of the second subarray and the computing result of the first subarray are processed to obtain the computing result of the second subarray; the computing result of the second subarray is output to the part and accumulation unit subarray located in the same column as the second computing unit subarray in the part and accumulation unit array for accumulation operation. The target selection branch includes a first selection branch, wherein... The control unit controls the input of the first feature data and the second feature data of different byte bits corresponding to the same input channel into the first operation unit subarray and the second operation unit subarray, respectively, and determines the first selection branch and activates the first selection branch. The second subarray of the operation unit obtains the intermediate operation result of the second subarray based on the second feature data and the second weight data; the intermediate operation result of the second subarray is shifted through the first selection branch; the shifted intermediate operation result of the second subarray is summed with the operation result of the first subarray to obtain the operation result of the second subarray; the operation result of the second subarray is output to the partial sum accumulation unit subarray located in the same column for accumulation operation.

7. The neural network inference method according to claim 6, characterized in that, The first selection branch includes a selector, a shifter, and an accumulator, wherein, The control unit controls the selector to select the first selected branch; Based on the first feature data and the second feature data, the shifter determines the target shift number and then shifts the intermediate operation result of the second subarray to the left by the target shift number to obtain the shifted intermediate operation result of the second subarray. The accumulator sums the intermediate operation result of the shifted second subarray with the operation result of the first subarray to obtain the operation result of the second subarray.

8. The neural network inference method according to claim 6, characterized in that, The target selection branch includes a second selection branch, wherein... The control unit controls the input of the first feature data and the second feature data corresponding to different input channels into the first computing unit subarray and the second computing unit subarray, respectively, and determines the second selection branch and activates the second selection branch; The second subarray of the operation unit obtains the intermediate operation result of the second subarray based on the second feature data and the second weight data; the intermediate operation result of the second subarray is summed with the operation result of the first subarray through the second selection branch to obtain the operation result of the second subarray; the operation result of the second subarray is output to the partial sum accumulation unit subarray located in the same column for accumulation operation.

9. The neural network inference method according to claim 8, characterized in that, The second selection branch includes a selector and an accumulator, wherein, The control unit controls the selector to select the second selection branch; The accumulator sums the intermediate operation result of the second subarray with the operation result of the first subarray to obtain the operation result of the second subarray.

10. The neural network inference method according to any one of claims 6 to 9, characterized in that, The subarray of operational units includes at least two merge operational units located in different columns. Each merge operational unit includes at least one first operational unit and one second operational unit located in the same column. The feature data includes at least third and fourth feature data. The control unit controls the input of the third feature data and the fourth feature data corresponding to different input channels into the first processing unit and the second processing unit, respectively; The first processing unit performs a multiplication calculation on the third feature data and the third weight data at corresponding positions to obtain a first product, and outputs it to the second processing unit; The second operation unit performs positional multiplication on the fourth feature data and the fourth weight data to obtain a second product; it sums the first product and the second product to obtain a partial sum accumulation result, which is stored in the beat register; and it performs accumulation processing on the partial sum accumulation result through an addition tree to obtain the initial subarray operation result.

11. A terminal, characterized in that, The terminal includes the neural network inference chip according to any one of claims 1 to 5.

12. A terminal, characterized in that, The terminal includes a neural network inference chip, a processor, and a memory storing executable instructions, which, when executed, implement the method described in any one of claims 6 to 10.

Images