Vector processing unit, neural-network processing unit, and depth camera

By designing vector processing units in neural network processors, the integer data is improved to floating-point data for processing, which solves the problem of algorithm accuracy loss in the existing technology, and improves the accuracy and performance of neural network processors .

WO2024239619A1PCT designated stage expired Publication Date: 2024-11-28SHENZHEN ORBBEC CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2023/139024
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2023-12-15
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

In the process of reducing area and power consumption, existing neural network processors use integer data or custom floating-point data, resulting in loss of algorithm accuracy.

Method used

A vector processing unit is designed, including an accuracy increase module, an inverse quantization module, an operation module, a quantization module and a reduction precision module. By increasing the accuracy of the integer data into floating-point data for calculation and processing, it ensures that the internal internal network processor is All data is processed as floating point data, thus maintaining algorithm accuracy.

Benefits of technology

It effectively improves the accuracy of the neural network processor, avoids information loss and algorithm accuracy loss caused by data type conversion, and improves the performance and accuracy of the processor.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023139024_28112024_PF_FP_ABST
    Figure CN2023139024_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application is applicable to the technical field of chips, and relates to a vector processing unit for a neural-network processing unit, a neural-network processing unit, and a depth camera. The vector processing unit comprises a precision increasing module, an inverse quantization module, an operation module, a quantization module, and a precision reduction module. The precision increasing module is configured to increase the precision of input first integer data to first floating-point data. The inverse quantization module is configured to perform inverse quantization on the first floating-point data to obtain inverse quantized data, wherein the inverse quantized data is floating-point data. The operation module is configured to perform operation on the inverse quantized data to obtain second floating-point data. The quantization module is configured to quantize the second floating-point data to obtain quantized data, wherein the quantized data is floating-point data. The precision reduction module is configured to reduce the precision of the quantized data to obtain second integer data. According to the present application, operations in the vector processing unit are performed on the basis of the floating-point data, so that the algorithm precision of the neural-network processing unit is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

A vector processing unit, neural network processor and depth camera

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 24, 2023, with application number 2023105981671, and invention name “A vector processing unit, neural network processor and depth camera”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application belongs to the field of chip technology, and in particular relates to a vector processing unit (VPU), a neural network processor, and a depth camera. Background Art

[0003] Convolutional neural networks are computing systems based on biological neural networks. As a joint product of biological and computer science, they have widespread applications in fields such as machine learning and pattern recognition. To solve increasingly complex and abstract learning problems, convolutional neural networks are becoming increasingly large. Some large neural networks, such as the Google Cat system network, have approximately one billion connected neurons, dramatically increasing the amount of data and computational complexity. Therefore, designing high-performance, low-power hardware acceleration architectures for convolutional neural network forward inference, such as neural network processing units (NPUs), within a limited footprint has become a research hotspot.

[0004] At present, in order to significantly reduce area and power consumption, the data used in the internal computing units of many neural network processors are all integer data or custom floating-point data, which will greatly lose the accuracy of the algorithm.

[0005] Summary of the Invention

[0006] In view of this, the embodiments of the present application provide a vector processing unit, a neural network processor and a depth camera, which can solve the technical problem of loss of algorithm accuracy of the neural network processor in the related art.

[0007] In a first aspect, an embodiment of the present application provides a vector processing unit for a neural network processor, the vector processing unit comprising a precision increase module, a dequantization module, an operation module, a quantization module, and a precision reduction module. The precision increase module is used to increase the precision of input first integer data to first floating-point data; the dequantization module is used to dequantize the first floating-point data to obtain dequantized data, which is floating-point data; the operation module is used to operate on the dequantized data to obtain second floating-point data; the quantization module is used to quantize the second floating-point data to obtain quantized data; and the precision reduction module is used to reduce the precision of the quantized data to obtain second integer data.

[0008] In the second aspect, an embodiment of the present application provides a neural network processor, comprising a vector processing unit, a first input buffer unit, a convolution unit and a second input buffer unit as described in the embodiment of the first aspect, the convolution unit being used to perform convolution processing on the input data; the first input buffer unit transmitting the input data to the vector processing unit through the first path or the second path; the first path is for the first input buffer unit to transmit the input data to the convolution unit, which is convolved by the convolution unit and then output to the second input buffer unit, and then input into the vector processing unit by the second input buffer unit, wherein the first integer data is the data obtained after convolution by the convolution unit; the second path is for the first input buffer unit to directly transmit the input data to the vector processing unit, and the first integer data is the input data.

[0009] In a third aspect, an embodiment of the present application provides a depth camera, comprising a neural network processor as described in the embodiment of the second aspect.

[0010] The vector processing unit of the embodiment of the present application increases the precision of the first integer data to first floating-point data, and then sequentially performs dequantization, calculation, quantization, and precision reduction on the first floating-point data to obtain second integer data. In this way, floating-point data calculations are all used within the vector processing unit, thereby ensuring the accuracy of the algorithm and, in turn, the accuracy of the neural network processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] FIG1 is a schematic diagram of a neural network processor architecture provided by an embodiment of the present application;

[0013] FIG2 is a schematic diagram of the overall architecture of a vector processing unit provided in an embodiment of the present application;

[0014] FIG3 is a schematic diagram of a data path of a first path entering a vector processing unit in a neural network processor provided by an embodiment of the present application;

[0015] FIG4 is a schematic diagram of a sigmoid function curve and its second-order derivative curve provided in an embodiment of the present application;

[0016] FIG5A is a schematic diagram of a sigmoid function curve provided in an embodiment of the present application;

[0017] FIG5B is a schematic diagram of a fitting curve of a sigmoid function provided in an embodiment of the present application;

[0018] FIG5C is a schematic diagram comparing a sigmoid function curve and a sigmoid function fitting curve provided in an embodiment of the present application;

[0019] FIG6 is a schematic diagram of the working principle of the on-chip storage module provided in an embodiment of the present application;

[0020] FIG7A is a schematic diagram of address space allocation of an on-chip storage module provided in some embodiments of the present application;

[0021] FIG7B is a schematic diagram of address space allocation of an on-chip storage module provided in some other embodiments of the present application;

[0022] FIG7C is a schematic diagram of address space allocation of an on-chip storage module provided in some further embodiments of the present application;

[0023] FIG8A is a schematic diagram of address allocation within each group of data in full OCH mode provided by some embodiments of the present application;

[0024] FIG8B is a schematic diagram of address allocation within each data set in a 2x resolution mode provided in other embodiments of the present application;

[0025] FIG8C is a schematic diagram of address allocation within each data set in a 4x resolution mode provided in some further embodiments of the present application;

[0026] FIG9A is a schematic diagram of address allocation of an on-chip storage module provided in some embodiments of the present application;

[0027] FIG9B is a schematic diagram of address allocation of on-chip storage modules provided in other embodiments of the present application;

[0028] FIG9C is a schematic diagram of address allocation of an on-chip storage module provided in some further embodiments of the present application;

[0029] FIG10 is a schematic diagram of OCH data storage for each piece of data in a 4x resolution mode according to an embodiment of the present application;

[0030] FIG11 is a schematic diagram of splicing slice data in a 4x resolution mode according to an embodiment of the present application;

[0031] FIG12 is a schematic diagram of a data path of the second path entering the vector processing unit in the neural network processor provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0033] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0034] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of the present application include a particular feature, structure, or characteristic described in conjunction with that embodiment. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0035] In addition, in the description of this application, "a plurality of" means two or more. The terms "first" and "second" are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0036] It should also be understood that, unless otherwise expressly specified or limited, the term "connection" should be understood broadly. For example, it can mean fixed connection, detachable connection, or integration; it can mean direct connection, indirect connection through an intermediate medium, internal communication between two elements, or interaction between two elements. Those skilled in the art will understand the specific meanings of the above terms in this application based on specific circumstances.

[0037] In order to better illustrate the technical solution of the present application, the technical solution of the present application is described and explained in detail with reference to some specific parameters. It should be understood that these parameters are only preferred parameters for the convenience of explanation and cannot be interpreted as specific limitations of the present application.

[0038] This application provides a neural network processor, which can be applied to electronic devices such as depth cameras, mobile phones, tablets, smart door locks, payment devices, etc. The neural network processor can be mounted on a central processing unit as a coprocessor, and the central processing unit assigns tasks to it.

[0039] Figure 1 is a schematic diagram of the architecture of a neural network processor provided by the present application. The neural network processor includes a bus, a first input buffer unit 101, a convolution unit 102, a second input buffer unit 103, a vector processing unit 104, and an output buffer unit 105. The neural network processor exchanges data with devices such as the central processing unit through the bus, and realizes data interaction between various internal units. The first input buffer unit 101 is used to store data input by the bus (for example, data output by the previous processing unit). The output end of the first input buffer unit 101 is connected to the convolution unit 102 and the vector processing unit 104. The data output by the first input buffer unit 101 can enter the vector processing unit 104 through two paths: a first path (the path represented by a dotted line in Figure 1) and a second path (the path represented by a solid line in Figure 2). Under the first path, the data output by the first input buffer unit 101 first enters the convolution unit 102. The convolution module 102 performs convolution processing on the input data and then outputs the convolution result to the second input buffer unit 103, which is then output to the vector processing unit 104. In the second path, the data output by the first input buffer unit 101 is directly input to the vector processing unit 104. For the data input to the vector processing unit 104 through the two different paths, the vector processing unit 104 processes the input data differently.

[0040] It should be noted that FIG1 only shows the part of the neural network processor that is relevant to the present application. The neural network processor may also include other processing units and other components, which are not limited here.

[0041] The vector processing unit 104 is used in the neural network processor to process operators other than convolution and store and transmit data. Specifically, the vector processing unit 104 is used to perform precision upscaling, inverse quantization, activation function (ACT) operator operations, pooling operator operations, global pooling operator operations, Eltwise operator operations, resize operator operations, binocular operator (SGM) operations, quantization, precision downscaling, and other processing on the input data, and writes the operation results to the output buffer unit 105 according to the data arrangement format required by the next processing unit connected to the vector processing unit in the neural network processor. The output buffer unit 105 can be a double data rate synchronous dynamic random access memory (DDR SDRAM), which is an off-chip memory located outside the vector processing unit 104 and is used to buffer the data output by the vector processing unit 104 for input to the next processing unit.

[0042] Figure 2 shows an architecture diagram of a vector processing unit in some embodiments of the present application. The vector processing unit includes a precision increase module, a precision decrease module, a quantization module, a dequantization module, and an operation module. The input data of the vector processing unit can be input by the second input buffer unit under the first path, or directly input by the first input buffer unit under the second path. The data entering the vector processing unit under the first path and the second path are both first integer data. The difference is that the data under the first path is processed by the convolution unit before entering the vector processing unit. The precision increase module is used to increase the precision of the first integer data to first floating-point data, the dequantization module is used to dequantize the first floating-point data to obtain dequantized data, the operation module is used to operate on the dequantized data to obtain second floating-point data, the quantization module is used to quantize the second floating-point data to obtain quantized data, and the precision decrease module is used to reduce the precision of the quantized data to obtain second integer data. Therefore, all data processing within the vector processing unit is based on floating-point data, ensuring calculation accuracy.

[0043] In some embodiments, the data type output by the first input buffer unit is int8 or int16. Since the result calculated by the convolution unit may exceed the data range of int8 or int16, data overflow may occur, resulting in information loss. To this end, the convolution unit may perform data expansion on the data input by the first input buffer unit to avoid data overflow. The representation range after expansion is larger than the representation range of the original data type. For example, int8 data will be expanded to int32 after convolution, and int16 data will be expanded to int48 after convolution. Therefore, the first integer data type input to the vector processing unit under the first path is int32 or int48, and the first integer data type input to the vector processing unit under the second path is int8 or int16. In addition, the data after precision processing may exceed the range that can be represented by the original type of data, resulting in data overflow and information loss. In this embodiment, the precision increase module is also used to determine the data type of the first integer data. Data of non-preset types needs to be expanded to accommodate more data. Assuming int48 is the default type, when the first integer data input to the vector processing unit is of int8, int16, and int32 type, the precision increase module is further configured to perform bit expansion on the first integer data, expanding the int8, int16, and int32 type data to int48 type.

[0044] Specifically, when the first integer data is of type int8, int16, or int32, the precision increase module is used to add a sign bit to the highest bit of the first integer data until it is expanded to 48 bits, and then increase its precision to convert the int48 type to the fp32 type. More specifically, the sign bit (highest bit) of int48 is the sign bit of fp32, and the complement of int48 is calculated to convert the signed number int48 into the unsigned number uint48; starting from the high bit, the position of the first 1 of uint48 is searched to obtain the exponent bit and the decimal place; rounding is performed, for example, using the round-to-even mode to obtain the first floating-point data; and the round-to-even mode is used to eliminate the average error caused by rounding up or rounding down, thereby improving the calculation accuracy of the vector processing unit.

[0045] After the first integer data is precision-upgraded, first floating-point data is obtained, which needs to be dequantized by the dequantization module before the next calculation and processing can be performed. In some embodiments, the dequantization module includes a floating-point multiplier for multiplying the first floating-point data by a given dequantization parameter to obtain dequantized data, and the dequantized data is still floating-point data. The first integer data in the first path is the result of the first input buffer unit directly transmitting the input data to the convolution unit and convolving the convolution unit. Therefore, the dequantization parameter (scale) in the first path is the product of the dequantization parameter (scale_feature) required for the input data and the dequantization parameter (scale_kernel) of the convolution unit weight, that is, scale = scale_feature × scale_kernel. In the second path, the first integer data is the input data directly input by the first input buffer unit, and the dequantization parameter in the second path is the dequantization parameter scale_feature required for the input data.

[0046] As shown in Figure 2, the operation module includes an adder, a multiplier and a comparison operator. The adder is used to perform addition operations on data, the multiplier is used to perform multiplication operations on data, and the comparison operator is used to compare data to obtain the maximum value of multiple data. In the embodiment shown in Figure 3, the multiplier, the adder and the comparison operator are all described using single-precision floating-point numbers fp32 as an example. In other embodiments, half-precision floating-point numbers (fp16 or bfp16) multipliers, adders and comparison operators can also be used, or multipliers, adders and comparison operators of some custom floating-point numbers can be used. The operation module can be combined with at least one operator in the adder, the multiplier and the comparison operator to implement at least one operation in the activation function operator operation, the Eltwise operator operation, the Resize operator operation, the pooling operator operation, the global pooling operator operation, and the binocular operator operation.

[0047] In some embodiments, the vector processing unit further includes an on-chip random access memory (RAM) module configured to store the first integer data. Furthermore, the on-chip memory module is configured to write the second integer data into an output buffer unit according to a data type and arrangement format required by a next processing unit connected to the vector processing unit, so as to facilitate access by the next processing unit.

[0048] The embodiments shown in FIG3 to FIG11 illustrate the working principle of the vector processing unit in the first path. The working principle of the vector processing unit in the first path is described below using the embodiments shown in FIG3 to FIG11 .

[0049] FIG3 is a schematic diagram of a data path of a vector processing unit in a first path, wherein the operation processing of the operation module in the first path is illustrated by taking an example of an activation function operator operation.

[0050] The dequantized data is operated on by the activation function operator. The vector processing unit can support two major types of activation functions: one is the RELU-related activation function, and the other is the activation function with an exponential function (such as sigmoid).

[0051] The first type of activation function: RELU-related activation function, which supports four types: ReLU, ReLU6, LReLU and PReLU. When implementing the RELU-related activation function, the data range of the input data x is first determined. Different ranges have different slopes k. Then x and the corresponding k are used as the two inputs of the floating-point multiplier for multiplication to achieve activation.

[0052] The second type of activation function: an activation function with an exponential shape. There are four main methods for hardware implementation of exponential functions: the first is a lookup table. This method requires significantly more storage resources as computational accuracy increases, resulting in high resource consumption; the second is a Taylor series expansion. This method significantly increases the number of multipliers and adders when higher precision is required; the third is the CORDIC algorithm (i.e., coordinate rotation digital calculation method). This method requires more iterations as precision increases, slowing down the computational speed; and the fourth is fitting using the least squares method.

[0053] In addition to supporting the above two types of activation functions, the vector processing unit also improves the sigmoid function in the second type of activation function. In some embodiments, the activation function operation is realized by using the sigmoid function obtained by the method of non-uniform piecewise polynomial fitting. The function can be realized by a small number of multipliers and adders. In one embodiment, the first-order polynomial fitting of sigmoid is performed in this embodiment, as shown in the following formula:

[0054] Specifically, the sigmoid function can be approximately expressed as a linear function plus an error term of the second order or higher about x, where a represents the slope of the sigmoid function, b represents the intercept of the sigmoid function at x = 0, and O(x 2 ) is the error term, which can usually be ignored when x approaches 0. The above approximation only works when x approaches 0, but the error becomes very large when x is large.

[0055] To further improve the sigmoid function To improve the accuracy of first-order polynomial fitting, this embodiment performs non-uniform segmentation calculation on the first-order polynomial. Specifically, Taylor expansion is performed on the sigmoid function, and it can be seen that the second-order derivative of the sigmoid function determines its accuracy. As shown in Figure 4, the upper curve in Figure 4 is the sigmoid function curve, and the lower curve diff2 is the second-order derivative curve of the sigmoid function. The second-order derivative of the sigmoid function determines its fitting accuracy, that is, the larger the absolute value of the second-order derivative, the greater the error. Therefore, the density of the segmented intervals of the segmented fitting needs to be proportional to the absolute value of the second-order derivative, that is, the larger the absolute value of the second-order derivative, the finer the segmented interval; the smaller the absolute value, the sparser the segmented interval. As can be seen from Figure 4, the areas with larger absolute values ​​of the second-order derivative are concentrated between [-2, -0.5] or [0.5, 2], so the segmentation of this area needs to be more dense.

[0056] Since the sigmoid function is symmetric about the point (0, 0.5), in order to reduce the segmentation interval, only the points x>0 are fitted. The points x<0 can be calculated based on the symmetry. The fitted sigmoid expression is as follows:

[0057] To speed up the calculation, a lookup table can be constructed by combining a, b, and 1-b with different x values. Subsequently, the sigmoid function can be calculated by simply looking up the table for the three values ​​of a, b, and 1-b based on the x values ​​in the segmented interval. Because the sigmoid function input is floating-point data, the segmented interval for fitting the sigmoid function must also be segmented according to floating-point data to facilitate table lookup.

[0058] This embodiment adopts the method of non-uniform piecewise first-order polynomial fitting to obtain the sigmoid activation function. On the one hand, it solves the shortcomings of traditional hardware in implementing exponential functions, such as many iterations, slow calculation speed, low precision, and large resource overhead. On the other hand, since the segmentation interval is adjustable, the precision is controllable. Further, as shown in FIG5 , the actual sigmoid function curve is compared with the fitted sigmoid function curve. FIG5A is the sigmoid function curve sigmoid(x), FIG5B is the fitting curve polyfit(x) of the sigmoid function, and FIG5C is the curve after the sigmoid function is fitted. It can be seen that the sigmoid function is implemented by the method of non-uniform piecewise polynomial fitting, and the fitting error is small. In one embodiment, the fitting error is less than 5×10 -5 .

[0059] In some other embodiments of the present application, the sigmoid fitting function can also reduce the segmentation interval through high-order fitting, and the number of multipliers and adders required is more than that of first-order fitting, and the specific selection can be made according to the number of multipliers and adders.

[0060] In other embodiments, there may not be an activation function after convolution in some networks, and the activation function will be skipped (bypassACT). In this case, a bias (bais) is generally added. If no bias is added, a 0 is added and then output to the quantization module.

[0061] After being activated by the activation function operator or after being biased, the data is still floating-point data. It must be quantized by the quantization module according to the quantization parameters of the next processing unit in the neural network processor. It must then be reduced in precision by the precision reduction module according to the data type required by the next processing unit to achieve data type conversion. The data is then cached by the on-chip storage module. In one embodiment, the quantization module includes a floating-point multiplier.

[0062] Assume that the data type required by the next processing unit connected to the vector processing unit in a neural network processor is int8 or int16. The precision reduction module is specifically used to separate the sign bit, exponent bit, and fractional bit; shift the fractional bit according to the size of the exponent bit to obtain an unsigned integer; round the unsigned number to obtain uint8 or uint16; and calculate the two's complement of uint8 or uint16 based on the sign bit to obtain a signed number int8 or int16.

[0063] The on-chip memory module in the first path is primarily used to cache computation results and write them back to the output buffer according to the data type and data layout required by the next processing unit. Different input and output data types have different address allocation methods in the on-chip memory module. The on-chip memory module supports both int8 and int16 input and output data. To meet the needs of the next processing unit, the on-chip memory module can output data in NC4HW, NC8HW, or NC16HW layouts, and can output in full OCH (Output Channel) mode, 2x resolution parallel mode, or 4x resolution parallel mode.

[0064] Specifically, the full OCH mode means that all cached calculation results are transferred to the next processing unit for processing; the 2x resolution parallel mode means that the cached calculation results are divided into two parts and transferred to different processing units for processing; the 4x resolution parallel mode means that the calculation results are divided into four parts and transferred to different processing units for processing.

[0065] In some embodiments, the on-chip storage module may include multiple storage sub-units, and a storage unit is obtained by merging multiple storage sub-units, so that the on-chip storage module can store as many calculation results as possible and improve the read and write bandwidth. Take the on-chip storage module including 32 storage sub-units, and the storage space of each storage sub-unit is 32 bits * 1024 depth as an example. Under the first path, the vector processing unit processes 1 column and 16 rows of data per cycle. As shown in Figure 6, the on-chip storage module merges 32 32-bit storage sub-units in pairs and uses them as 16 64-bit * 1024 depth storage units, and each storage unit is used to cache 1 row of data. The address space allocation method of the 16 storage units is exactly the same, and one storage unit is used as an example.

[0066] Specifically, assuming that a storage unit with a depth of 1024 performs ping-pong reading and writing at a depth of 512, the input and output data types determine how many output channel data can be stored in a storage space with a depth of 64 bits (bit) * 512. In some embodiments, in order to improve computing efficiency, reduce memory usage, and reduce the demand for data transmission, the data stored in the storage unit is divided into smaller blocks for processing. Different input and output data types mean different block sizes, and the storage unit can store a different number of OCHs at most. For example, when the input data type is int8, assuming that the size of each block is 16 (rows) * 16 (width), the data is divided into small blocks of 16 rows and 16 columns for processing. When the input data type is int16, the size of each block is 16 (rows) * 8 (width), that is, the input data is divided into small blocks of 16 rows and 8 columns for processing.

[0067] The maximum number of OCHs that a storage unit can store is determined by the size of the storage space, the width of the input data, and the number of bits of the output data. In some embodiments, assuming that the maximum number of OCHs that can be stored is och_num, the combination of input data type and output data type has the following four cases: the input data type is int8, the output data type is int8, then The input data type is int8 and the output data type is int16, then The input data type is int16 and the output data type is int8, then The input data type is int16 and the output data type is int16, then

[0068] In one embodiment, assuming that a convolution unit can only include 128 output channel data at most, 128 OCH data are defined as a group of data. Therefore, the maximum number of data groups that can be stored in each storage unit with a depth of 512 and the address depth occupied by each data group in the storage unit are shown in Table 1 below.

[0069] Table 1

[0070] Figures 7A, 7B, and 7C illustrate the address space allocation for storage units corresponding to the four combinations of input and output data types. Figure 7A corresponds to int8(in) and int8(out), and int16(in) and int16(out); Figure 7B corresponds to int8(in) and int16(out); and Figure 7C corresponds to int16(in) and int8(out). In and out represent input and output, respectively.

[0071] Each data set has three different resolution operating modes: full OCH mode, 2x resolution mode, and 4x resolution mode. The number of slices each data set is divided into and the number of OCHs contained in each slice in different modes are shown in Table 2 below.

[0072] Table 2

[0073] The address allocation within each data group in full OCH mode, 2x resolution mode, and 4x resolution mode is shown in Figure 8A, Figure 8B, and Figure 8C, respectively. In full OCH mode, OCH0-OCH127 are allocated to one piece of data; in 2x resolution mode, OCH0-OCH63 are stored in one piece of data, and OCH64-OCH127 are allocated to another piece of data; in 4x resolution mode, OCH0-OCH31 are stored in the first piece of data, OCH32-OCH63 are allocated in the second piece of data, OCH64-OCH95 are stored in the third piece of data, and OCH96-OCH127 are allocated in the fourth piece of data.

[0074] The input data type, output data type, and output data arrangement format determine the address allocation within each piece of data. In some embodiments, the output data arrangement format can be NC4HW, NC8HW, or NC16HW. NC4HW, NC8HW, or NC16HW is an optimization of the NCHW format, which means that the data of 4, 8, or 16 consecutive output channels are grouped together, and each group of data is stored in the NCHW format, where N represents the number of output data, C represents the number of output channels; H represents the height of the output data; and W represents the width of the output data. For example, when the data arrangement format output by the on-chip storage module is NC4HW. In the NC4HW format, the data is divided into several groups of C4W, each group of C4W represents a data block containing 4 output channels and a width of W. Therefore, when the output data type is int8, one group of C4W needs to occupy 32 bits (ie 4 bytes) of space, two groups of C4W (C4W2) constitute 64 bits, and when the output data type is int16, one group of C4W needs to occupy 64 bits (ie 8 bytes) of space.

[0075] Taking the distribution of 64-bit data as an example, assuming that the number of OCHs in each piece of data is 4n, and 4 OCHs form a group to form a C4, then the number of C4s in each piece of data is n, and these n C4s are numbered in sequence as C4(0), C4(1), C4(2), ..., C4(n). Figure 9A shows the on-chip address allocation when the output data type is int8 and the output data arrangement format is NC4HW. Figure 9B shows the on-chip address allocation when the output data type is int8 and the output data arrangement format is NC8HW or NC16HW; Figure 9C shows the on-chip address allocation when the output data type is int16 and the output data arrangement format is NC4HW, NC8HW or NC16HW. Each C4W (or C4W(even), C4W(odd)) is arranged in the order of w, where C4W(even) and C4W(odd) represent the two modes of rounding to even and rounding to odd, respectively.

[0076] The memory cell read address calculation is determined by the input data type, output data type, output data layout, and operating mode. The input data type determines the output data width W. The output data type and layout determine whether the read data is C4W, C4W2, or C8W. Different modes also determine the order in which OCH data is read. Internally, slice data is read in ascending OCH order. In full OCH mode, regardless of the output data layout (NC4HW, NC8HW, or NC16HW), it is output in ascending OCH order. Increasing OCH does not necessarily mean increasing read addresses; the read address is calculated based on how the OCH data is stored in the on-chip memory modules.

[0077] In one embodiment, the order in which OCH data is read between slices of data is illustrated using the 4x resolution mode as an example. The same applies to the 2x resolution mode. In the 4x resolution mode, each group of data is divided into four slices: slice 0, slice 1, slice 2, and slice 3, and the four slices are output sequentially. As shown in Figure 10, rp represents slice. For ease of understanding, the OCH data within each slice is grouped and numbered according to C4, C8, and C16. The order in which each slice is read is shown in Figure 11, where x can be 4, 8, or 16. To ensure the continuity of the convolution resolution of the next layer's processing unit, groups with the same order in different slices are concatenated according to the slice order, and groups with different orders are concatenated according to the group order. A group of data can also be understood as a slice of data, and ping-pong reading and writing can also be understood as supporting one group of data in parallel, two groups of data in parallel, and four groups of data in parallel. The addressing method within each group can refer to the addressing method within a slice of data described above, and the addressing method between groups of data can refer to the addressing method between slices described above. This will not be further described here.

[0078] The embodiment shown in Figure 12 illustrates the working principle of the vector processing unit under the second path. The first integer data of the input vector processing unit under the second path is input by the first input buffer unit. The vector processing unit under the second path can use a variety of operators to operate on the input data, including but not limited to Eltwise operators, pooling operators, global pooling operators, Resize operators, activation function operators, binocular operators, etc. Among them, the Eltwise operator, pooling operator, global pooling operator, Resize operator, activation function operator, binocular operator can reuse or share the same computing resources (adder, multiplier and comparison operator) and storage resources, that is, the adder, multiplier and comparison operator used by the Eltwise operator, pooling operator, global pooling operator, Resize operator, activation function operator, binocular operator are at least partially the same, thereby reducing the occupied area of ​​the vector processing unit. The vector processing unit can also add other operators while reasonably utilizing reusable resources, and the vector processing unit has good functional scalability. In some embodiments, if the data entering the Eltwise operator is preceded by a coefficient, the coefficient is taken into account in the inverse quantization parameter.

[0079] In one embodiment, an Eltwise operator is used to perform dot multiplication, addition, subtraction, and maximum calculations on data corresponding to input data from different channels. The vector processing unit splits the input data according to the input data type and parallelizes operations such as multiplication, addition, and maximum calculations on multiple pixel data from channel a and multiple pixel data from another channel b. The output data is then written to the output buffer unit based on the result of the operation and the output data type.

[0080] The global pooling operator uses pooling methods including maximum pooling and average pooling. The pooling window size varies and is used to transform the input data from w*h*n to 1*1*n. The principle of the global pooling operator is explained using the example of inputting several columns of data in parallel per channel. The first step is to take the maximum value or sum of these column-wise data. To improve acceleration performance, the vector processing unit also includes a register to write the maximum or average result to a register and define an initial value for the global pooling operation. In maximum mode, the initial value is an infinitesimal represented by a floating-point number; in average mode, the initial value is 0. When the register is empty, the output of the first step is accumulated or compared with the initial value, and the accumulated or compared result is written to the register. When the register is not empty, the output of the first step is accumulated or compared with the data in the register, and the accumulated or compared result is written back to the register. This cycle continues until the last input data is received. Due to the characteristic of floating-point adders or comparators that they calculate a result over multiple cycles, some intermediate results will be cached in the register after the last input data is accumulated or compared. Therefore, the remaining data in the register must be accumulated or compared one by one until the register is empty. In maximum mode, the output result after the register is empty is the result of the global pooling operation. In average mode, the output result after the register is empty must be divided by the size of the input data, that is, multiplied by the inverse of the input data size (floating point number) to obtain the final result.

[0081] The pooling operator, also known as downsampling, is used to reduce the dimensionality of data. It can be viewed as a fuzzy filter, preventing overfitting, reducing the effects of distortion, and enhancing the neural network's ability to resist distortion. Pooling methods include max pooling and average pooling. The pooling window size is 2*2 or 3*3, with a step size of 1 or 2. It supports three modes: general mode, ceiling mode, and floor mode. The pooling mode should be determined based on the input and output data size and padding.

[0082] In one embodiment, the specific mode can be determined based on the input and output width (Y), direction, and padding. Specifically, the width Y1 of the source image in the general mode can be calculated based on the width Y0 of the target image. Then, the actual input source image width Y2 is compared with the calculated source image width Y1 to determine the pooling mode. Y1 is calculated as follows: Y1 = Y0 × pool_stride + (pool_size - pool_stride) - pad_left - pad_right.

[0083] Where pool_stride represents the stride, pool_size represents the pooling window size, pad_left and pad_right represent the number of paddings on the left and right sides, respectively. In general mode, Y1 = Y2; in round-up mode, Y1 > Y2; in round-down mode, Y1 < Y2.

[0084] The Resize operator is used to perform interpolation operations on input data, for example, the Resize operator interpolates the input feature map. In some embodiments, the Resize operator uses a bilinear interpolation algorithm, such as interpolating in the horizontal direction first and then in the vertical direction, or interpolating in the vertical direction first and then in the horizontal direction.

[0085] In summary, the vector processing unit of the embodiment of the present application has high bandwidth utilization, high data parallelism, and pipeline processing of vector data, which can fully utilize the computing power of the neural network processor and improve acceleration performance.

[0086] This application also provides a depth camera, which includes the neural network processor described in the above embodiment, and is used to process image data collected by the depth camera. The depth camera may also include other components, such as a laser emission module and an infrared imaging module, which are not listed here.

[0087] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0088] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A vector processing unit for a neural network processor, characterized in that: include: A precision increasing module, used for increasing the precision of the input first integer data to first floating point data; A dequantization module, used for dequantizing the first floating-point data to obtain dequantized data, wherein the dequantized data is floating-point data; A calculation module, used for calculating the dequantized data to obtain second floating point data; A quantization module, used for quantizing the second floating-point data to obtain quantized data, wherein the quantized data is floating-point data; The precision reduction module is used to reduce the precision of the quantized data to obtain second integer data.

2. The vector processing unit according to claim 1, wherein: The operation module includes a comparison operator, a multiplier and an adder. The operation module implements at least one of an activation function operator operation, an Eltwise operator operation, a Resize operator operation, a pooling operator operation, a global pooling operator operation, and a binocular operator operation through the comparison operator, the multiplier and the adder.

3. The vector processing unit according to claim 2, characterized in that: The activation function in the activation function operator is obtained by piecewise polynomial fitting.

4. The vector processing unit according to claim 3, characterized in that: The activation function has an exponential function shape, and the expression of the activation function is: Among them, x is the input data of the activation function sigmoid(x), and a and b are obtained by looking up the table of the segmented interval where x is located.

5. The vector processing unit according to claim 2, characterized in that: The comparison operator, the multiplication operator and the addition operator used by the activation function operator, the Eltwise operator, the Resize operator, the pooling operator, the global pooling operator and the binocular operator for calculation are at least partially identical.

6. The vector processing unit according to claim 1, wherein: When the first integer data is not of a preset type, the precision increasing module is used to perform bit expansion on the first integer data and then increase the precision to the first floating point data.

7. The vector processing unit according to any one of claims 1 to 6, characterized in that: The vector processing unit also includes an on-chip storage module, which is used to store the second integer data and write the second integer data into an off-chip memory according to the data arrangement format required by the next processing unit connected to the vector processing unit in the neural network processor.

8. The vector processing unit according to claim 7, characterized in that: The on-chip storage module is used to allocate read address and write address space according to the data type of the second integer data, the data arrangement format and data type required by the next processing unit, and the corresponding selected working mode, so as to write the second integer data into the off-chip memory; wherein the data arrangement format includes NC4HW, NC8HW and NC16HW; the working mode includes full output channel mode, double resolution parallel mode and quadruple resolution parallel mode.

9. A neural network processor, characterized in that: The neural network processor comprises the vector processing unit according to any one of claims 1 to 8, a first input buffer unit, a convolution unit and a second input buffer unit, wherein the convolution unit is used to perform convolution processing on input data; The first input buffer unit inputs the input data into the vector processing unit through the first path or the second path; The first path is to transmit the input data cached in the first input buffer unit to the convolution unit, and output the data to the second input buffer unit after convolution by the convolution unit, and input the data to the vector processing unit from the second input buffer unit, wherein the first integer data is the data obtained after convolution by the convolution unit; The second path is that the first input buffer unit directly transmits the input data to the vector processing unit, and the first integer data is the input data.

10. The neural network processor according to claim 9, characterized in that In the first path, the inverse quantization parameter of the inverse quantization module is the product of the inverse quantization parameter required for the input data and the inverse quantization parameter of the convolution unit weight; In the second path, the inverse quantization parameter of the inverse quantization module is the inverse quantization parameter required by the input data.

11. A depth camera, characterized in that: Includes the neural network processor described in any one of claims 9-10.

Citation Information

Patent Citations

  • Vector processing unit, neural network processor and depth camera

    CN116611488A

  • Training neural network accelerators using mixed precision data formats

    US20200193274A1

  • Neural network activation compression with narrow block floating-point

    US20200210838A1

  • Adjusting activation compression for neural network training

    US20200264876A1

  • Neural network processing unit, neural network processing method and device

    US20220292337A1