Convolution operation system and convolution operation method
By converting the data format to DFP16 before convolution operations, the hardware for format conversion and two's complement operations of the processing unit are reduced, solving the problems of increased power consumption and area in the prior art and achieving more efficient convolution operations.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAWEI TECH CO LTD
- Filing Date
- 2020-03-25
- Publication Date
- 2026-07-10
AI Technical Summary
In existing convolutional neural network systems, the processing units experience increased power consumption and area due to frequent format conversions, especially in the multiplication and accumulation operations of convolutional kernels and feature map data, which require a large amount of two's complement arithmetic hardware.
By converting the data format to a new floating-point format (DFP16) before convolution operations, the hardware for format conversion in the processing unit is reduced, two's complement operations are avoided, and a forward carry propagation adder is used for operations, thereby reducing power consumption and area.
It effectively reduces the hardware required for format conversion in the processing unit, lowers power consumption and area, and improves computing efficiency, especially in data reuse scenarios.
Smart Images
Figure CN115244504B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing, and in particular to convolution operation systems and convolution operation methods. Background Technology
[0002] Convolutional neural networks (CNNs) are a class of feedforward neural networks that include convolution operations and have a deep structure. They have broad application prospects in fields such as image recognition, speech recognition, and language processing.
[0003] The convolution operation in a CNN refers to the multiply-accumulate (MAC) operation using kernel data and feature map data. A convolution operation system includes one or more processing elements (PEs), and the MAC operation can be performed by one or more processing elements. For example... Figure 1 As shown, Figure 1 This is a schematic diagram of the processing unit 101. The processing unit obtains K convolutional kernel data from registers A1-AK and K feature map data from registers B1-BK. Each convolutional kernel data is multiplied by its corresponding feature map data to obtain K products. Before adding the K products, the processing unit needs to obtain the two's complement of each product. The processing unit adds the two's complements of the K products to obtain a sum in two's complement format. After accumulation, the processing unit converts the format of this sum to be the same as the format of the feature map data or the convolutional kernel data. For example, if the feature map data is in IEEE 754 half-precision FP16 format, the processing unit converts the format of the sum to IEEE 754 half-precision FP16.
[0004] The processing unit needs to convert the format of K products to the 2-complement format, and then convert the sum of the 2-complement format back to the original format. This results in a large amount of format conversion hardware on the processing unit, which increases the power consumption and area of the processing unit. Summary of the Invention
[0005] This application provides a convolution operation system and a convolution operation method. Applied to the field of data processing, it can reduce the hardware requirements for format conversion in processing units, and lower the power consumption and area of the processing units.
[0006] The first aspect of this application provides a convolution operation system.
[0007] The convolution operation system includes a first conversion module and a convolution processing module. The first conversion module is used to convert a first data group with a first format into a second data group with a second format. The convolution processing module includes a first processing unit, which is used to process the convolution operation of the second data group to obtain first convolution data with a second format. The convolution operation does not include two's complement operation.
[0008] By converting the format of the first data group before the convolution operation, the first processing unit does not need to perform two's complement operations when performing convolution. This reduces the hardware required for format conversion in the first processing unit, thereby lowering its power consumption and area.
[0009] Based on the first aspect of this application, in a first embodiment of the first aspect of this application, the convolution operation system further includes a second conversion module, which is used to convert the first convolution data into second convolution data with a first format. By converting the first convolution data into second convolution data with the first format, the second format data is used only within the convolution operation system, ensuring data robustness.
[0010] Based on the first aspect of this application, or the first embodiment of the first aspect of this application, in a second implementation of the first aspect of this application, the second data group includes convolution kernel data and feature map data, and the first processing unit is specifically used to perform multiplication and accumulation operations on the convolution kernel data and feature map data. Here, the environment in which the convolution operation system is used is limited to CNN, and the extensive data reuse scenarios in CNN can further highlight the beneficial effects of the embodiments of this application.
[0011] Based on the first aspect of this application, or any one of the first to second embodiments of the first aspect of this application, in the third implementation of the first aspect of this application, the data in the first format consists of a sign bit, an exponent bit, a hidden bit, and a mantissa bit from the high bit to the low bit, and / or the data in the second format consists of an exponent bit, a sign bit, a hidden bit, and a mantissa bit from the high bit to the low bit.
[0012] Based on the third implementation of the first aspect of this application, in the fourth implementation of the first aspect of this application, the data in the first format and the data in the second format satisfy at least one of the following relationships:
[0013] The sign bits of the first and second formats are equal;
[0014] The exponents in the first and second formats are equal;
[0015] When the exponent bit of the first format is equal to 0, the mantissa bit of the second format is equal to the complement of the mantissa bit of the first format, and the hidden bit of the second format is equal to the sign bit of the first format.
[0016] When the exponent bit of the first format is greater than 0 and less than 31, the mantissa bit of the second format is equal to the complement of the mantissa bit of the first format, and the hidden bit of the second format is equal to the sign bit of the first format XORed with 1.
[0017] When the exponent of the first format is 31 and the mantissa of the first format is 0, the data in the second format is infinite.
[0018] When the exponent in the first format is 31 and the mantissa in the first format is not 0, the data in the second format is NOT a number.
[0019] Based on the fourth implementation of the first aspect of this application, in the fifth implementation of the first aspect of this application, when the data in the second format is infinite, it includes:
[0020] When the sign bit of the first format is 0, the data in the second format is positive infinity;
[0021] When the sign bit of the first format is 1, the data in the second format is negative infinity.
[0022] Based on any one of the first to fifth embodiments of the first aspect of this application, in the sixth implementation of the first aspect of this application, the convolution processing module further includes a second processing unit and a secondary accumulation unit. The second processing unit is used to process the convolution operation of a third data group having a second format to obtain third convolution data having a second format. The secondary accumulation unit is used to obtain the sum of the first convolution data and the third convolution data. The second conversion module is specifically used to convert the sum of the first convolution data and the third convolution data into second convolution data. Since the first conversion module converts the data with the first format into data with the second format, no two's complement operation is required in the secondary accumulation unit. Therefore, the number of format conversion hardware components in the secondary accumulation unit can be reduced, thereby reducing the power consumption and area of the secondary accumulation unit.
[0023] Based on any one of the second to sixth embodiments of the first aspect of this application, in the seventh implementation of the first aspect of this application, the first processing unit includes a carry propagate adder (CPA). Instead of directly obtaining the first convolutional data from the second data set, the first processing unit obtains the first convolutional data by using the carry propagate adder to sum the result of the current multiply-accumulate operation and the result of the previous multiply-accumulate operation. Since it is not necessary to convert data with a second format to data with a first format in the processing unit, a negative carry propagate adder is not required, thus reducing the power consumption and area of the first processing unit.
[0024] Based on the first aspect of this application, or any one of the first to seventh embodiments of the first aspect of this application, in the eighth implementation of the first aspect of this application, the first format is IEEE754 half-precision FP16, and / or the second format is the new floating-point format DFP16.
[0025] The second aspect of this application provides a method for convolution operations.
[0026] The device acquires a first data group with a first format and converts the first data group into a second data group with a second format. The device can also perform a convolution operation based on the second data group to obtain first convolutional data with a second format, wherein the convolution operation does not include two's complement operations.
[0027] Based on the second aspect of this application, in the first embodiment of the second aspect of this application, after obtaining the first convolutional data in the second format, the device can further convert the first convolutional data into second convolutional data having the first format.
[0028] Based on the second aspect of this application, or the first embodiment of the second aspect of this application, in the second implementation of the second aspect of this application, the second data group includes convolution kernel data and feature map data, and the device performs multiplication and accumulation operations on the convolution kernel data and feature map data.
[0029] Based on the first aspect of this application, or any one of the first to second embodiments of the second aspect of this application, in the third implementation of the second aspect of this application, the data in the first format consists of a sign bit, an exponent bit, a hidden bit, and a mantissa bit from the high bit to the low bit, and / or the data in the second format consists of an exponent bit, a sign bit, a hidden bit, and a mantissa bit from the high bit to the low bit.
[0030] Based on the third implementation of the second aspect of this application, in the fourth implementation of the second aspect of this application, the data in the first format and the data in the second format satisfy at least one of the following relationships:
[0031] The sign bits of the first and second formats are equal;
[0032] The exponents in the first and second formats are equal;
[0033] When the exponent bit of the first format is equal to 0, the mantissa bit of the second format is equal to the complement of the mantissa bit of the first format, and the hidden bit of the second format is equal to the sign bit of the first format.
[0034] When the exponent bit of the first format is greater than 0 and less than 31, the mantissa bit of the second format is equal to the complement of the mantissa bit of the first format, and the hidden bit of the second format is equal to the sign bit of the first format XORed with 1.
[0035] When the exponent of the first format is 31 and the mantissa of the first format is 0, the data in the second format is infinite.
[0036] When the exponent in the first format is 31 and the mantissa in the first format is not 0, the data in the second format is NOT a number.
[0037] Based on the fourth implementation of the second aspect of this application, in the fifth implementation of the second aspect of this application, when the data in the second format is infinite, it includes:
[0038] When the sign bit of the first format is 0, the data in the second format is positive infinity;
[0039] When the sign bit of the first format is 1, the data in the second format is negative infinity.
[0040] Based on any one of the first to fifth embodiments of the second aspect of this application, in the sixth implementation of the second aspect of this application, the device may further perform a convolution operation based on a third data group having a second format to obtain third convolution data having a second format. After obtaining the third convolution data, the device obtains the sum of the first convolution data and the third convolution data, and converts the sum of the first convolution data and the third convolution data into second convolution data.
[0041] Based on any one of the second to sixth embodiments of the second aspect of this application, in the seventh implementation of the second aspect of this application, the device does not directly obtain the first convolution data through the second data group, but obtains the first convolution data by obtaining the sum of the result of the current multiplication-accumulation operation and the result of the previous multiplication-accumulation operation.
[0042] Based on the second aspect of this application, or any one of the first to seventh embodiments of the second aspect of this application, in the eighth implementation of the second aspect of this application, the first format is IEEE754 half-precision FP16, and / or the second format is the new floating-point format DFP16.
[0043] The description of the second aspect of this application, and any embodiment thereof, can be referred to the description of the first aspect or any embodiment thereof, which will not be repeated here. Attached Figure Description
[0044] Figure 1 This is a schematic diagram of the processing unit.
[0045] Figure 2 This is a flowchart illustrating the convolution operation method in the embodiments of this application;
[0046] Figure 3 This is a schematic diagram illustrating the conversion between FP16 format and DFP16 format when the exponent bit is equal to 0 in an embodiment of this application.
[0047] Figure 4 This is a schematic diagram illustrating the conversion between FP16 format and DFP16 format when the exponent bit is greater than 0 and less than 31 in an embodiment of this application.
[0048] Figure 5 This is a schematic diagram of the convolution operation system in an embodiment of this application;
[0049] Figure 6 This is a schematic diagram of the structure of the first processing unit in the embodiments of this application;
[0050] Figure 7 This is a schematic diagram of the structure of the first processing unit including a forward propagation carry adder in an embodiment of this application. Detailed Implementation
[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Exemplarily, the features or contents marked with dashed lines in the accompanying drawings related to the embodiments of this application can be understood as optional operations or optional structures of the embodiments.
[0052] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0053] Before describing the technical solutions of the embodiments of this application, the technical scenarios of the embodiments of this application will be described first.
[0054] The technical solution of this application embodiment can be applied to the field of data processing that requires convolution operations, especially convolution operation scenarios involving large amounts of data reuse, such as the convolution operation of CNN. Assume that a first feature map containing N×N data points and a first convolution kernel containing N×N data points are used to obtain an N×N second feature map. Each data point in the second feature map is equal to the convolution operation of the N×N data points in the first convolution kernel with the N×N data points in the first feature map. To obtain the second feature map with N×N data points, padding needs to be applied to the first feature map. Each data point in the second feature map is obtained from a PE (Preprocessor Entity), and the convolution processing module requires a total of N×N PEs. The number of complement operations required in each PE is N+1, so the number of complement operations required by the convolution processing module is (N+1)×N^2. The number of complement operations required by the convolution processing module is on the order of N to the power of 3, therefore, as N increases, the number of complement operations required by the convolution processing module increases dramatically. These complement operations require a large amount of format conversion hardware, which increases the power consumption and area of the convolution processing module. Reducing the number of complement operations can help lower power consumption and reduce area. By using the convolution operation method in this embodiment, format conversion can be performed before the data is input to the convolution processing module, so that the total number of complement operations required is equal to the number of input data, i.e., 2 × N^2 + 1. Here, +1 represents the padding data, assuming that the padding data is all the same. In particular, even if the data is format converted after the output convolution processing module, the increased number of complement operations required is equal to the number of output data, i.e., N^2. Therefore, the total number of complement operations required is 3 × N^2 + 1.
[0055] As described above, CNN convolution operations can involve data reuse within convolution kernels or feature maps. Data reuse within a convolution kernel refers to the same data appearing in multiple feature maps (PEs), for example, the N×N PEs mentioned above all use N×N data points from the first convolution kernel. Data reuse within a feature map refers to the same data appearing in multiple PEs. Specifically, in CNN convolution operations, kernel reuse or feature map reuse also occurs. Kernel reuse refers to a convolution operation between one convolution kernel and multiple feature maps; for example, the first convolution kernel mentioned above can also be convolutionally processed with a third feature map, or vice versa. Feature map reuse refers to a convolution operation between one feature map and multiple convolution kernels; for example, the first feature map mentioned above can also be convolutionally processed with a second convolution kernel. In these data reuse scenarios, the technical solutions provided in this application have more significant effects.
[0056] The convolution operation method in the embodiments of this application is described below. This convolution operation method is applied in the field of data processing and can reduce the hardware required for format conversion in devices, thereby reducing the power consumption and area of the devices. For clarity, data will be used in the description below. It should be noted that the amount or format of the data used should not limit the scope of protection of the embodiments of this application.
[0057] Assume the first feature map is a 5×5 matrix. The first feature map includes 25 data points (excluding padding data), which are stored in IEEE 754 half-precision FP16 format. The data of the first feature map is shown in Table 1.
[0058] t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16 t17 t18 t19 t20 t21 t22 t23 t24 t25
[0059] Table 1
[0060] Assume the first convolution kernel is a 4×4 matrix. The first convolution kernel contains 16 data points, which are stored in IEEE 754 half-precision FP16 format. The data of the first convolution kernel is shown in Table 2.
[0061] a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 a13 a14 a15 a16
[0062] Table 2
[0063] This application embodiment will illustrate the process of performing a convolution operation between the first convolution kernel and the data of a receptive field. The data of the receptive field is a portion of the data in the first feature map. The data of the receptive field is shown in Table 3.
[0064] t1 t2 t3 t4 t6 t7 t8 t9 t11 t12 t13 t14 t16 t17 t18 t19
[0065] Table 3
[0066] Please see Figure 2 , Figure 2 This is a flowchart illustrating the convolution operation method in the embodiments of this application.
[0067] In step 201, the device acquires a first data group in a first format.
[0068] The format of a data set refers to the format of the data within it. Data can be expressed in different ways, and therefore can have different formats. Examples include floating-point numbers and fixed-point numbers. Floating-point numbers can be further divided into half-precision, single-precision, and double-precision floating-point numbers. Half-precision floating-point numbers can also have different representations, such as sign-magnitude, one's complement, and two's complement. Assume the first data set includes the data t3, t4, t8, and t9 from Table 1 above, and also includes the data a3, a4, a7, and a8 from Table 2 above. In CNN convolution operations, a single processing unit does not necessarily need to complete all convolution operations. For example, if the first convolution kernel includes 4×4 data points, then in one convolution operation, 16+16 data points are needed, resulting in 16 multiplication operations. These 16 multiplication operations can be distributed among different processing units. This embodiment uses two processing units as an example, with each unit allocated 8 multiplication operations. The eight multiplication operations are further divided into two processes by a single processing unit, with four multiplication operations performed each time. The device refers to a device capable of performing data format conversion and convolution operations; convolution operations can be performed by the processing unit.
[0069] In step 202, the device converts the format of the first data group to obtain a second data group in a second format.
[0070] The data in Tables 1 and 2 above are in IEEE 754 half-precision FP16 format. The device converts the format of the first data group to obtain the second data group in DFP16 format. That is, the first format is IEEE 754 half-precision FP16 format, and the second format is DFP16 format. The second data group includes T3, T4, T8, T9, A3, A4, A7, and A8. Among them, T3 is obtained by converting the format based on t3, and A3 is obtained by converting the format based on a3. The conversion method between DFP16 format and IEEE 754 half-precision FP16 format is described below.
[0071] Please see Figure 3 , Figure 3 This diagram illustrates the conversion between FP16 and DFP16 formats when the exponent bit is equal to 0 in an embodiment of this application. The data in the IEEE 754 half-precision FP16 format includes, from most significant bit to least significant bit, a sign bit 301, an exponent bit 302, a hidden bit 303, and a mantissa bit 304. The data in the DFP16 format includes, from most significant bit to least significant bit, an exponent bit 305, a sign bit 306, a hidden bit 307, and a mantissa bit 308. For simplicity, the term "first format" will be used to refer to the IEEE 754 half-precision FP16 format, and "second format" will be used to refer to the DFP16 format.
[0072] The sign bit 301 in the first format is equal to the sign bit 306 in the second format.
[0073] The exponent 302 in the first format is equal to the exponent 305 in the second format.
[0074] When the exponent 302 of the first format is equal to 0, the mantissa 308 of the second format is equal to the complement of the mantissa 304 of the first format, and the hidden bit 307 of the second format is equal to the sign bit 301 of the first format.
[0075] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the conversion between FP16 format and DFP16 format when the exponent bit is greater than 0 and less than 31 in an embodiment of this application. When the exponent bit 302 of the first format is greater than 0 and less than 31, the mantissa bit 308 of the second format is equal to the complement of the mantissa bit 304 of the first format, and the hidden bit 307 of the second format is equal to the XOR operation between the sign bit 301 of the first format and 1.
[0076] When the exponent 302 of the first format equals 31 and the mantissa 304 of the first format equals 0, the data in the second format is infinite.
[0077] When the exponent 302 of the first format is equal to 31 and the mantissa 304 of the first format is not equal to 0, the data in the second format is NOT a number.
[0078] In this context, hidden bit 303 in the first format and hidden bit 307 in the second format are hidden bits. Since the mantissa of a normalized binary floating-point number is always 1, it is unnecessary to store a bit whose value is always 1. Hidden bits are generally not explicitly represented in the data format, but they are reflected in specific operations, hence the name "hidden bit." The fact that the mantissa 307 in the second format is equal to the complement of the mantissa 304 in the first format means: if the sign bit 301 in the first format is 0, then the mantissa 307 in the second format is equal to the mantissa 304 in the first format; if the sign bit 301 in the first format is 1, then the mantissa 307 in the second format is equal to the complement of the mantissa 304 in the first format. The complement of the mantissa 304 in the first format is equal to the one's complement of the mantissa 304 in the first format plus 1. The hidden bit 307 of the second format is XORed with the sign bit 301 of the first format and 1. This means that if the sign bit 301 of the first format is 0, then the hidden bit 307 of the second format is 1; if the sign bit 301 of the first format is 1, then the hidden bit 307 of the second format is 0.
[0079] In step 203, the device performs a convolution operation using the second data set to obtain the result of the current multiplication-accumulation operation.
[0080] The second set of data includes T3, T4, T8, T9, A3, A4, A7, and A8. Let the result of the current multiplication-accumulation operation equal P1, then P1 = T3 × A3 + T4 × A4 + T8 × A7 + T9 × A8. Let T3 × A3 = TA3 and T4 × A4 = TA4. During the convolution operation, there is no need to convert the format of TA3 or TA4; that is, there is no need to perform two's complement operations on TA3 and TA4. Two's complement operations can include one's complement operations and +1 operations. One's complement operations are used to obtain the one's complement of the data, specifically by flipping each bit of the data. +1 operations are used to add 1 to the data after the one's complement operation. Specifically, two's complement operations can include one's complement operations and +K operations. +K operations refer to accumulating the data after K one's complement operations and then adding K. Specifically, after performing one's complement operations on the above four data points (TA3, TA4, etc.) and accumulating them to obtain P1, P1 is then subjected to a +K operation, i.e., P1 + 4.
[0081] In step 204, the device uses the first PE to obtain the sum of the result of the current multiply-accumulate operation and the result of the previous multiply-accumulate operation to obtain the first convolutional data with the second format.
[0082] As described above, the eight multiplication operations are divided into two operations by a single processing unit. The result of the previous multiply-accumulate operation belongs to the convolution data preceding the result of the current multiply-accumulate operation. The result of the previous multiply-accumulate operation is obtained by convolving the fourth data group in the second format. Assume that the fourth data group includes T1, T2, T6, T7, A1, A2, A5, and A6. Among them, T1 is obtained by converting the format according to t1, and A1 is obtained by converting the format according to a1. Before obtaining the result of the previous multiply-accumulate operation, the device also needs to perform format conversion on the data groups t1, t2, t6, t7, a1, a2, a5, and a6. The conversion method between DFP16 format and IEEE 754 half-precision FP16 format has been described above and will not be repeated here.
[0083] By increasing the number of multiplication-accumulation operations in the PE (Preprocessor Entity), the number of multipliers and adders in the PE can be reduced. However, when obtaining the sum of results from different numbers of operations, positive CPA and negative CPA are required. For example, let the result of the previous multiplication-accumulation operation be equal to P2, P = P1 + P2. If the sign bit of P is 0, the result of positive CPA is output; if the sign bit of P is 1, the result of negative CPA is output. The convolution operation method of this application uses data in a second format for convolution operation, and then converts the data format after PE, so negative CPA is not required in PE. Negative CPA can be achieved by inverting the data after accumulation. To distinguish it from negative CPA, CPA that does not invert the data after accumulation is called positive CPA. Carry propagate adders (CPA) include ripple carry adders (RCA), carry skip adders (CSKA), carry select adders (CSA), carry increment adders (CIA), carry lookahead adders (CLA), parallel prefix adders (PPA), and conditional sum adders (COSA), among others.
[0084] Optionally, when the first processing unit does not acquire the sum of results from different counts, the first processing unit may use the result of the current multiplication-accumulation operation as the first convolution data.
[0085] In step 205, the device acquires the sum of the first convolutional data and the third convolutional data.
[0086] The third convolutional data is obtained by the second processing unit performing convolution operations on the third data group in the second format. The process by which the second processing unit obtains the third convolutional data can refer to the process by which the first processing unit obtains the first convolutional data. The third data group includes feature map data T11, T12, T13, T14, T16, T17, T18, and T19, and convolutional kernel data A9, A10, A11, A12, A13, A14, A15, and A16. T11 is obtained based on t11 in the first format, and A9 is obtained based on a9 in the first format.
[0087] When each processing unit completes part of the multiplication operation of the convolution operation, for example, each processing unit completes 8 multiplications in this embodiment, the device still needs to use a secondary accumulation unit to perform secondary accumulation of the convolution data of each processing unit. Before the secondary accumulation, the data also needs to be subjected to a two-complement operation, which increases the power consumption and area of the device. The convolution operation method in this embodiment uses data in a second format for convolution operation, and converts the data format after the secondary accumulation. No two-complement operation is performed during the secondary accumulation process, so the number of format conversion hardware in the secondary accumulation unit can be reduced, thereby reducing the power consumption and area of the secondary accumulation unit. The convolution operation method in this embodiment can also reduce the power consumption and area of the processing unit. Therefore, the reduced power consumption and area can be accumulated to further reduce the power consumption and area of the device. In particular, the device may include multiple processing units, and the number of processing units is large. Therefore, even if only the power consumption and area of the processing units can be reduced, it can have a significant impact on the power consumption and area of the device.
[0088] In step 206, the device converts the sum of the first convolutional data and the third convolutional data into second convolutional data in a first format.
[0089] When the first format is IEEE754 half-precision FP16 and the second format is DFP16, the conversion method from the second format to the first format is as follows.
[0090] The sign bit of the first format is equal to the sign bit of the second format.
[0091] The exponent digits in the first format are equal to those in the second format.
[0092] When the exponent bit of the second format is equal to 0, the mantissa bit of the first format is equal to the complement of the mantissa bit of the second format, and the hidden bit of the first format is equal to the sign bit of the second format.
[0093] When the exponent bit of the second format is greater than 0 and less than 31, the mantissa bit of the first format is equal to the complement of the mantissa bit of the second format, and the hidden bit of the first format is equal to the sign bit of the second format XORed with 1.
[0094] When the exponent of the second format is 31 and the mantissa of the second format is 0, the data in the first format is infinite.
[0095] When the exponent of the second format is equal to 31 and the mantissa of the second format is not equal to 0, the data in the first format is NOT a number.
[0096] Optionally, step 205 may be omitted. When step 205 is omitted, the convolution operation is performed by the first PE. In step 206, the device converts the first convolution data in the second format into the second convolution data in the first format.
[0097] The convolution operation method in the embodiments of this application has been described above. The convolution operation system in the embodiments of this application is described below.
[0098] Please see Figure 5 , Figure 5 This is a schematic diagram of the convolution operation system in an embodiment of this application. The convolution operation system includes a first conversion module 501 and a convolution processing module 502. The first conversion module 501 is used to acquire a first data group in a first format and convert the first data group in the first format into a second data group in a second format. The convolution processing module 502 includes a first processing unit, which is used to acquire the second data group and perform convolution operations on the second data group to obtain first convolution data in the second format.
[0099] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of the first processing unit in an embodiment of this application. Convolution operation includes multiplication and accumulation operations. Figure 6 The system includes a first processing unit 601, which comprises K multiplication units 602 and an accumulation unit 603. The multiplication units 602 retrieve a second data group in a second format from registers A1-AK and registers B1-BK. After retrieving the second data group, the multiplication units 602 perform multiplication operations using the second data group to obtain K products. The accumulation unit 603 then performs an accumulation operation using the K products. By converting the format of the first data group before the convolution operation by the first conversion module 501, the first processing unit 601 does not need to perform two's complement operations during the convolution operation. Therefore, the format conversion hardware of the first processing unit 601 can be reduced, thus lowering its power consumption and area.
[0100] Optionally, the convolution operation system further includes a second conversion module 503. After obtaining the first convolution data in a second format, the second conversion module 503 is used to convert the first convolution data into second convolution data with a first format.
[0101] Optionally, the second data group includes convolution kernel data and feature map data, and the first processing unit is used to perform multiplication and accumulation operations on the convolution kernel data and feature map data.
[0102] Optionally, the data in the first format consists of the sign bit, exponent bit, hidden bit, and mantissa bit from the most significant bit to the least significant bit, and / or the data in the second format consists of the exponent bit, sign bit, hidden bit, and mantissa bit from the most significant bit to the least significant bit.
[0103] Optionally, the data in the first format and the data in the second format satisfy at least one of the following relationships:
[0104] The sign bits of the first and second formats are equal;
[0105] The exponents in the first and second formats are equal;
[0106] When the exponent bit of the first format is equal to 0, the mantissa bit of the second format is equal to the complement of the mantissa bit of the first format, and the hidden bit of the second format is equal to the sign bit of the first format.
[0107] When the exponent bit of the first format is greater than 0 and less than 31, the mantissa bit of the second format is equal to the complement of the mantissa bit of the first format, and the hidden bit of the second format is equal to the sign bit of the first format XORed with 1.
[0108] When the exponent of the first format is 31 and the mantissa of the first format is 0, the data in the second format is infinite.
[0109] When the exponent in the first format is 31 and the mantissa in the first format is not 0, the data in the second format is NOT a number.
[0110] Optionally, when the data in the second format is infinite, it includes:
[0111] When the sign bit of the first format is 0, the data in the second format is positive infinity;
[0112] When the sign bit of the first format is 1, the data in the second format is negative infinity.
[0113] Optionally, the convolution processing module 502 further includes a second processing unit and a secondary accumulation unit;
[0114] The second processing unit is used to process the convolution operation of the third data group with the second format to obtain the third convolution data with the second format.
[0115] The second-order accumulation unit is used to obtain the sum of the first and third convolutional data;
[0116] The second conversion module is specifically used to convert the sum of the first and third convolutional data into the second convolutional data.
[0117] Optionally, the first processing unit includes a forward carry propagation adder; such as Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of the first processing unit including a forward carry propagation adder in an embodiment of this application. Figure 7The system includes a first processing unit 701, which includes an accumulation unit 702 and the result 703 of the previous multiplication-accumulation operation. The accumulation unit 702 can obtain the result of the current multiplication-accumulation operation. The first processing unit 701 also includes a forward carry-propagation adder 704, which is used to obtain the sum of the result of the current multiplication-accumulation operation and the result 703 of the previous multiplication-accumulation operation to obtain the first convolution data.
[0118] Optionally, the first format is IEEE 754 half-precision FP16, and / or the second format is the new floating-point format DFP16.
[0119] The convolution operation system in this application embodiment can refer to the foregoing. Figure 2 The description of the convolution operation method described in [the document].
[0120] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.
Claims
1. A convolution operation system, characterized in that, Applications in the field of data processing include: The first conversion module and the convolution processing module; The first conversion module is used to convert a first data group with a first format into a second data group with a second format. The convolution processing module includes a first processing unit, which includes K multiplication units and an accumulation unit. The first processing unit is used to process the convolution operation of the second data group, including: the k multiplication units are used to perform multiplication operations on the second data group to obtain k products, and the accumulation unit is used to perform accumulation operations on the K products to obtain first convolution data with the second format. The convolution operation does not include two's complement operations. The first processing unit is implemented through a hardware structure. The data in the first format and the data in the second format satisfy at least one of the following relationships: The sign bits of the first format and the second format are equal; The exponent bits of the first format and the second format are equal; When the exponent bit of the first format is equal to 0, the mantissa bit of the second format is equal to the complement of the mantissa bit of the first format, and the hidden bit of the second format is equal to the sign bit of the first format. When the exponent bit of the first format is greater than 0 and less than 31, the mantissa bit of the second format is equal to the complement of the mantissa bit of the first format, and the hidden bit of the second format is equal to the sign bit of the first format XORed with 1. When the exponent of the first format is equal to 31 and the mantissa of the first format is equal to 0, the data in the second format is infinite. When the exponent of the first format is equal to 31 and the mantissa of the first format is not equal to 0, the data in the second format is NOT a number.
2. The system according to claim 1, characterized in that, Also includes: The second conversion module is used to convert the first convolutional data into second convolutional data having the first format.
3. The system according to claim 1, characterized in that, The second data group includes convolution kernel data and feature map data. The first processing unit is used to process the convolution operation of the second data group, specifically including: The first processing unit performs multiplication and accumulation operations on the convolution kernel data and the feature map data.
4. The system according to any one of claims 1 to 3, characterized in that, The first format data consists of, from most significant bit to least significant bit, the sign bit, the hidden bit, and the mantissa bit, and / or the second format data consists of, from most significant bit to least significant bit, the sign bit, the hidden bit, and the mantissa bit.
5. The system according to claim 1, characterized in that, When the data in the second format is infinite, it includes: When the sign bit of the first format is 0, the data in the second format is positive infinity; When the sign bit of the first format is 1, the data in the second format is negative infinity.
6. The system according to claim 2, characterized in that, The convolution processing module further includes a second processing unit and a secondary accumulation unit; The second processing unit is used to process the convolution operation of the third data group having the second format to obtain the third convolution data having the second format; The secondary accumulation unit is used to obtain the sum of the first convolutional data and the third convolutional data; The second conversion module is specifically used to convert the sum of the first convolutional data and the third convolutional data into the second convolutional data.
7. The system according to claim 3, characterized in that, The first processing unit includes a forward carry propagation adder; The forward carry propagation adder is used to obtain the sum of the result of the current multiply-accumulate operation and the result of the previous multiply-accumulate operation to obtain the first convolutional data.
8. The system according to any one of claims 1 to 3, characterized in that, The first format is IEEE 754 half-precision FP16, and / or the second format is the new floating-point format DFP16.