Data processing device and method and neural network

Abstract

The invention discloses a data processing device and method and a neural network. The device comprises an accumulation unit suitable for receiving input data and performing exponential function transformation on each input data to obtain a first exponential function corresponding to each input data; decomposing the index of each first exponential function to obtain an integer part and a decimal part of the index in each first exponential function; extracting the maximum value of the integer part of the index in each first exponential function; calculating an exponential function cumulative sum of a decimal part of an index in each first exponential function; the power function operation unit is suitable for calculating a reciprocal of a first exponential function cumulative sum of a decimal part of an index in the first exponential function corresponding to each input data to obtain a first normalization coefficient; and the normalization unit is suitable for utilizing the first normalization coefficient to obtain a Softmax operation result corresponding to each piece of input data. By adopting the scheme, the memory access requirement of the Softmax operator can be reduced, so that the operation efficiency of the Softmax operator is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing device, method and neural network. Background Art

[0002] Neural network is a network model in the field of deep learning. Neural networks are usually composed of many operators. The operator in a neural network is the basic mathematical operation unit used in the neural network and is also a node in the neural network calculation graph. Each type of operator has a specific operation function, which can perform certain mathematical operations on its input data and pass the output data to the next operator.

[0003] The Softmax operator is a common nonlinear operator in neural networks, which is used to perform Softmax operations on input data. The Softmax operation is used to convert a set of arbitrary real numbers into real numbers representing probability distribution. It is essentially a normalization function that can convert a set of arbitrary real values ​​into probability values ​​between [0,1].

[0004] Currently, when the Softmax operator is implemented, there is a problem of large memory access requirements, which leads to low operating efficiency of the Softmax operator. Summary of the invention

[0005] The problem to be solved by the present invention is to reduce the memory access requirement of the Softmax operator so as to improve the operation efficiency of the Softmax operator.

[0006] To solve the above problems, an embodiment of the present invention provides a data processing device, which is suitable for performing Softmax operation; the data processing device includes: an accumulation unit, which is suitable for receiving input data, performing exponential function transformation on each input data, and obtaining a first exponential function corresponding to each input data; decomposing the exponent of each first exponential function to obtain the integer part and the decimal part of the exponent in each first exponential function; extracting the maximum value of the integer part of the exponent in each first exponential function; and calculating the exponential function accumulation sum of the decimal part of the exponent in each first exponential function; a power function operation unit, which is suitable for calculating the reciprocal of the first exponential function accumulation sum of the decimal part of the exponent in the first exponential function corresponding to each input data to obtain a first normalization coefficient; a normalization unit, which is suitable for using the first normalization coefficient to obtain the Softmax operation result corresponding to each input data.

[0007] Optionally, the accumulation unit is suitable for calculating a first difference between the integer part of the exponent in the first exponential function of the j-th input data and the maximum value of the integer part of the exponent in the first exponential function of the previous j-1 input data, and using the first difference calculation result to calculate the first exponential function cumulative sum of the decimal part of the exponent in the first exponential function of the previous j-1 input data or the number of shifts of the first exponential function value of the decimal part of the first exponential function of the j-th input data, and performing shift and sum operations to obtain the first exponential function cumulative sum of the decimal part of the exponent in the first exponential function of the previous j input data.

[0008] The embodiment of the present invention further provides a data processing method, which is suitable for performing a Softmax operation; the method comprises:

[0009] Receive input data, perform exponential function transformation on each input data, and obtain a first exponential function corresponding to each input data; decompose the exponent of each first exponential function to obtain the integer part and the decimal part of the exponent in each first exponential function; extract the maximum value of the integer part of the exponent in each first exponential function; calculate the exponential function cumulative sum of the decimal part of the exponent in each first exponential function;

[0010] Calculate the reciprocal of the cumulative sum of the first exponential function of the decimal part of the exponent in the first exponential function corresponding to each input data to obtain a first normalization coefficient;

[0011] The first normalization coefficient is used to obtain the Softmax operation result corresponding to each input data.

[0012] An embodiment of the present invention further provides a neural network, which includes any one of the above-mentioned data processing devices.

[0013] Compared with the prior art, the technical solution of the embodiment of the present invention has the following advantages:

[0014] By applying the scheme of the present invention, the first exponential function corresponding to each input data is obtained by performing an exponential function transformation on each input data, then the exponent of each first exponential function is decomposed to obtain the integer part and the decimal part of the exponent in each first exponential function, then the maximum value of the integer part of the exponent in each first exponential function is extracted, and then the exponential function cumulative sum of the decimal part of the exponent in each first exponential function is calculated, and then the reciprocal of the first exponential function cumulative sum of the decimal part of the exponent in the first exponential function corresponding to each input data is calculated to obtain the first normalization coefficient, and finally the Softmax operation result corresponding to each input data is obtained. In the process of calculating the first exponential function cumulative sum of the decimal part of the exponent in the first exponential function corresponding to the input data, only one traversal of the input data is required to complete the calculation, thereby reducing the memory access requirement of the Softmax operator and improving the operating efficiency of the Softmax operator. In addition, the exponent in the first exponential function of the input data is divided into the integer part and the decimal part, and the subsequent operation is continued, which can avoid the result overflow that may be caused by directly executing the exponential function operation.

[0015] Furthermore, the data processing device is also suitable for performing normalization operations. The normalization operation and the Softmax operation can reuse at least one of the accumulation unit, the power function operation unit and the normalization unit, thereby improving the utilization rate of hardware resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a structural schematic diagram of a data processing device in an embodiment of the present invention;

[0017] Figure 2 is a schematic structural diagram of an accumulation unit in an embodiment of the present invention;

[0018] Figure 3 is a schematic diagram of the structure of a power function operation unit in an embodiment of the present invention;

[0019] Figure 4 is a structural schematic diagram of a normalization unit in an embodiment of the present invention;

[0020] Figure 5 is a schematic diagram of hardware implementation of a data processing device in an embodiment of the present invention;

[0021] Figure 6 It is a flow chart of a data processing method in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The Softmax operator is a common operator in neural networks. The calculation formula of the Softmax operator can be expressed as:

[0023]

[0024] In formula (1), exp() represents an exponential function. i ) represents the input data x i The exponential function value of ∑ j exp(x j ) represents the cumulative sum of the exponential functions of all j input data, x j is the jth data among all input data.

[0025] In order to avoid overflow of the calculation result of the exponential function, it is usually necessary to offset the input data. At this time, the calculation formula of the modified Softmax operator can be expressed as:

[0026]

[0027] In formula (2), x max It represents the maximum value of the input data.

[0028] At present, the Softmax operator can be implemented using a general-purpose processor, such as a central processing unit. The central processing unit has powerful computing functions and program processing capabilities, and can be implemented by decomposing complex operators into multiple specific calculation steps. Specifically, the calculation of the Softmax operator usually requires the following four steps in sequence:

[0029] 1) Traverse the input data once to get the maximum value x of the input data max ;

[0030] 2) Traverse the input data again, calculate the exponential function value after subtracting the maximum value, and accumulate it to obtain the exponential function cumulative sum ∑ j exp(x j -x max );

[0031] 3) Calculate the reciprocal of the sum of exponential functions 1 / ∑ j exp(x j -x max );

[0032] 4) Traverse the input data at least once again, calculate the exponential function value after subtracting the maximum value, and multiply it by the reciprocal of the accumulated sum of the exponential function.

[0033] When implementing the Softmax operator using traditional methods, the exponential function accumulation process requires at least two passes through the input data to complete the calculation task, which results in high memory access requirements and ultimately low computational efficiency.

[0034] In addition, in neural networks, the normalization operator is also a common nonlinear operator. The normalization operator and the Softmax operator are usually implemented using different circuits, which have problems such as low circuit resource utilization.

[0035] Common normalization operators include batch normalization, group normalization, instance normalization, and layer normalization. Different normalization operators have slightly different specific calculations and are mostly used in different computing tasks. But in general, their calculation formulas can be uniformly expressed as:

[0036]

[0037] In the above formula (3), x represents input data and y represents output data. and Respectively represent the mean and variance calculated from the input data, and ∈ is a small constant whose function is to avoid the denominator being 0. Usually, the operation of subtracting the mean from the input data and multiplying it by the reciprocal square root of the variance is called the standard operation, which is different from the normalization operation. γ and β are trainable parameters in the neural network, which remain unchanged after the network training is completed and are used to complete an affine transformation. Usually, the calculation of the normalization operator requires the following three steps:

[0038] (1) Traverse the input data at least once to calculate the mean of the input data and variance

[0039] (2) Calculate the reciprocal square root of the variance

[0040] (3) Traverse the input data at least once again, standardize the input data, and perform an affine transformation operation as needed.

[0041] In the normalization operator, the algorithmic step that is expensive to implement in hardware is the calculation of the reciprocal square root of the variance.

[0042] For the normalization operator and the Softmax operator, common hardware implementation solutions can be divided into the following three types:

[0043] (1) Using a general-purpose processor, such as a central processing unit, for implementation. The central processing unit of a modern computer has powerful computing functions and program processing capabilities, and can be implemented by decomposing complex operators into multiple specific computing steps.

[0044] (2) Design a dedicated circuit scheme for each operator based on its computational function. At this time, each circuit scheme can only support one type of operator.

[0045] (3) Design a computing unit with general computing functions so that it does not have program processing capabilities but can effectively support multiple different types of operators.

[0046] The above solution 1 can theoretically support the operation of any operator, but it has problems such as low computational parallelism, high number of memory accesses, and low computational efficiency.

[0047] The above-mentioned solution 2 designs a dedicated hardware circuit for each operator, which can efficiently support the operation of different operators, but it has problems such as low hardware resource utilization and large design workload.

[0048] The above solution 3 is a compromise between solution 1 and solution 2. It can fully reuse hardware resources and effectively improve the computing efficiency of operators. However, in scenarios with high real-time requirements, solution 3 will face problems such as insufficient computing efficiency of operators; in embedded devices with low production costs, solution 3 will face problems such as limited hardware resources.

[0049] In view of this problem, the present invention provides a data processing device. By using the data processing device, in the process of calculating the cumulative sum of the exponential function, the input data only needs to be traversed once to complete the calculation, thereby reducing the memory access requirements of the Softmax operator and improving the calculation efficiency of the Softmax operator. In addition, the exponential value in the first exponential function of the input data is divided into an integer part and a decimal part, and subsequent operations are continued, which can avoid the result overflow that may be caused by directly executing the exponential function operation.

[0050] Furthermore, the data processing device in the present invention can efficiently reuse hardware resources and is compatible with hardware acceleration of two different operations, the normalization operator and the Softmax operator, thereby saving hardware resources while ensuring the operational efficiency of the normalization operator and the Softmax operator.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0052] Reference Figure 1 The embodiment of the present invention provides a data processing device, which is suitable for performing Softmax operation. Specifically, the data processing device may include: an accumulation unit 11, a power function operation unit 12 and a normalization unit 13. Among them:

[0053] The accumulating unit 11 is adapted to receive input data, perform exponential function transformation on each input data to obtain a first exponential function corresponding to each input data; decompose the exponent of each first exponential function to obtain the integer part and the decimal part of the exponent in each first exponential function; extract the maximum value of the integer part of the exponent in each first exponential function; and calculate the exponential function cumulative sum of the decimal part of the exponent in each first exponential function;

[0054] The power function operation unit 12 is adapted to calculate the reciprocal of the first exponential function accumulation sum of the decimal part of the exponent in the first exponential function corresponding to each input data to obtain a first normalization coefficient;

[0055] The normalization unit 13 is adapted to obtain a Softmax operation result corresponding to each input data using the first normalization coefficient.

[0056] In a specific implementation, the accumulating unit 11 may use a variety of methods to calculate the first exponential function value of all input data, for example, it may be implemented by a lookup table method or a polynomial fitting method.

[0057] In one embodiment of the present invention, the first exponential function is an exponential function with e as the base. Specifically, for any input data x i , there should be x i =(-ln2)u i +v i ,u i is an integer and has -ln2 <v i ≤0. For exponential function By performing exponential function transformation, we can get the input data x i The corresponding first exponential function can be expressed as follows:

[0058]

[0059] Among them, u i is the integer part of the exponent of the first exponential function, v i is the fractional part of the exponent of the first exponential function.

[0060] Since u in formula (4) i is an integer value, and the integer power of two is equivalent to the shift operation, so we can get:

[0061]

[0062] In another embodiment of the present invention, the first exponential function is an exponential function with base 2. Specifically, for the exponential function Applying the base-changing formula to transform the exponential function, we can get:

[0063]

[0064] The exponent x in formula (6) i log2e is divided into the integer part u i and the fractional part v i , that is, x i log2e=u i +v i , log2e is called the exponential transformation coefficient, then the exponential function The calculation can be converted into the following formula:

[0065]

[0066] From formula (7), we can see that by adjusting the exponential function value The exponential function can be obtained by shifting the calculated result of The calculation result of .

[0067] At this point, the exponential function value can be Fitting to v i +1. Thanks v i Satisfy 0≤v i <1, so by i The integer position of is filled with 1 to get v i +1 value, thus obtaining the exponential function value An approximation of v is not required, without the need for an adder i +1 for the calculation.

[0068] For any input data, the exponent of its first exponential function can be decomposed into an integer part and a decimal part. In the process of calculating the cumulative sum of the first exponential functions of the decimal parts of the exponents in the first exponential functions of all input data, the cumulative sum of the exponential functions of the decimal parts of the exponents in the first exponential function of the received input data can be calculated while continuously updating the maximum value of the integer part of the exponent in the first exponential function corresponding to the received input data.

[0069] In one embodiment, the accumulation unit 11 can calculate the first difference between the integer part of the exponent in the first exponential function of the j-th input data and the maximum value of the integer part of the exponent in the first exponential function of the previous j-1 input data, and use the difference calculation result to accumulate the exponential function of the decimal part of the exponent in the first exponential function of the previous j-1 input data or the number of shifts of the first exponential function value of the decimal part of the exponent in the first exponential function of the j-th input data, and perform shift and sum operations to obtain the exponential function of the decimal part of the exponent in the first exponential function of the previous j input data.

[0070] Specifically, when the first difference is greater than 0, a first shift operation is performed on the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j-1 input data using the first difference; and a summation operation is performed on the result of the first shift operation and the first exponential function value of the decimal part of the exponent in the first exponential function of the j-th input data to obtain the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j input data.

[0071] When the first difference is less than 0, a second shift operation is performed on the first exponential function value of the decimal part of the exponent in the first exponential function of the j-th input data using the first difference; and a summation operation is performed on the result of the second shift operation and the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j-1 input data to obtain the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j input data.

[0072] Assume that the exponential function of the input data can be decomposed into two parts, p and u, as shown in the following formula:

[0073]

[0074] Among them, p i Represents the input data x i The first exponential function value of the fractional part of the exponent in the first exponential function of u; i Represents the input data x i The integer part of the exponent in the corresponding first exponential function.

[0075] By extracting u i It can avoid the overflow of the exponential function calculation result and be used to obtain the maximum value u of the integer part of the exponent in the first exponential function of the received input data. max , for ease of description, the maximum value u of the integer part of the exponent in the first exponential function of the received input data max It is called the greatest common factor.

[0076] In the specific implementation, p i The value of can be realized by using a lookup table method or a polynomial fitting method. For example, in formula (5), In formula (7), It can be fitted as v i +1.

[0077] In one embodiment, the accumulating unit 11 may use the following formula to calculate the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j input data:

[0078]

[0079] Where sum(j) represents the cumulative sum of the exponential functions of the decimal parts of the exponents in the first exponential function of the first j input data, sum(j-1) represents the cumulative sum of the exponential functions of the decimal parts of the exponents in the first exponential function of the first j-1 input data, and u j represents the integer part of the exponent in the first exponential function of the jth input data, p j The exponential function value representing the fractional part of the exponent in the first exponential function value of the jth input data, u max(j-1) Represents the maximum value of the integer part of the exponent in the first exponential function of the first j-1 input data.

[0080] In the specific implementation, when u j >u max(j-1) When , the maximum value of the integer part of the exponential in the first exponential function of the first j input data is u j . When u j ≤u max(j-1) When , the maximum value of the exponential integer part of the first exponential function of the first j input data is u max , as shown in formula (10):

[0081]

[0082] After obtaining the exponential function accumulation sum (j) of the decimal part of the exponent in the first exponential function of the first j input data, the power function operation unit 12 can calculate the reciprocal of the exponential function accumulation sum (j) of the decimal part of the exponent in the first exponential function of the first j input data, that is, calculate 1 / sum (j), and the normalization unit 13 obtains the Softmax operation result of each input data according to the following formula (11):

[0083]

[0084] In the present invention, in order to improve the efficiency of Softmax operation, the maximum value x of the input data is not directly calculated. max Instead, in the process of calculating the cumulative sum of the first exponential function of the decimal part of the exponent of the first exponential function of the input data, the maximum value u of the integer part of the exponent of the first exponential function of the input data currently received is extracted. max As a common factor, u is continuously updated during the accumulation process max .

[0085] For example, for the input data {x i , x2,…,x j}, with the exponential of the first exponential function {x1log2e, x2log2e, …, x j The integer part and fractional part corresponding to log2e} are recorded as {u1, u2, …, uj} and {v1, v2, …, v j}, that is, x j log2e=u j +v j . From {u1, u2, ..., u j} to extract the largest integer part u max As a common factor, the sum of the exponential functions of the fractional part of the exponent in the first exponential function calculated by the first j input data is recorded as sum(j), and there is ∑ j exp(x j )=sum(j)<<u max .

[0086] More specifically, after receiving the input data x1, the exponential function accumulates and Greatest common factor u max =u1. After receiving the input data x2, we should have:

[0087]

[0088] After receiving the first j input data, sum(j) and the common factor u max The calculation method can be expressed by the following formula:

[0089]

[0090] Therefore, in the process of calculating the sum of the exponential functions of the fractional part of the exponent in the first exponential function, sum(j), the greatest common factor u is continuously updated. max , so as to achieve the purpose of calculating the cumulative sum of the decimal part of the exponent in the first exponential function while finding the greatest common factor. At this time, the cumulative sum of the exponential function of the first j input data should be ∑ j exp(x j )=sum(j)<<u max However, in the specific implementation of this method, it is not necessary to calculate the exponential function sum ∑ j exp(x j ), but only the cumulative sum (j) of the fractional parts of the exponents in the first exponential function is obtained.

[0091] The Softmax operation result of the final input data can be expressed as:

[0092]

[0093] in,

[0094] The key points of the method proposed by the present invention are: on the one hand, the complex exponential operation ex It is decomposed into two parts, p value and shift number u, which can be obtained by approximate fitting, to avoid overflow of the result caused by direct exponential operation. On the other hand, in the process of calculating the accumulated sum of exponential function, the exponential value e is not directly accumulated. x , but records the maximum number of shifts of all current input data as the common factor u max , and calculate the cumulative sum of the shifted p values. By comparing the u corresponding to the jth input data j u corresponding to the first j-1 input data max The difference between j The number of shifts of the current cumulative sum sum(j-1) is used to obtain the new cumulative sum sum(j) and the new greatest common factor u max , realizing u max The exponential function accumulation and sum(j) of the decimal part of the exponent in the first exponential function are updated synchronously.

[0095] The method proposed in the present invention can achieve the purpose of calculating the cumulative sum of the exponential functions of the decimal part of the exponent in the first exponential function while finding the greatest common factor by extracting the integer part of the exponent in the first exponential function as a common factor. Therefore, the method proposed in the present invention can calculate the cumulative sum of the exponential functions of the decimal part of the exponent in the first exponential function by traversing the input data only once, and can obtain the first normalization coefficient, which effectively improves the operation efficiency of the Softmax operation and better unifies the operation process of the normalization operator and the Softmax operator.

[0096] In one embodiment of the present invention, in order to improve the utilization rate of hardware resources, the data processing device is also suitable for performing a normalization operation, and the normalization operation and the Softmax operation reuse at least one of the accumulation unit, the power function operation unit and the normalization unit. For example, the normalization operation and the Softmax operation can reuse only the accumulation unit, the power function operation unit or the normalization unit. The normalization operation and the Softmax operation can also reuse any two or even all of the accumulation unit, the power function operation unit and the normalization unit.

[0097] When the normalization operation and the Softmax operation reuse the accumulation unit, the accumulation unit is also suitable for calculating the square sum of each input data and the total sum of the input data.

[0098] In one embodiment of the present invention, referring to Figure 2 , the accumulation unit 11 may include: a first selector 111, a first multiplier 112, a second selector 113, a maximum value calculator 114, a first shifter 115, a second shifter 116, a first accumulator 117 and a second accumulator 118, wherein:

[0099] The output end of the first selector 111 is connected to the first multiplier 112, the output end of the first multiplier 112 is connected to the second selector 113, the output end of the second selector 113 is connected to the maximum value calculator 114 and the second shifter 116, and the output end of the maximum value calculator 114 is connected to the first shifter 115 and the second shifter 116;

[0100] The output end of the first shifter 115 and the output end of the second shifter 116 are connected to the first accumulator 117; the output end of the first accumulator 117 is connected to the first shifter 115;

[0101] The output terminal of the second accumulator 118 is connected to the input terminal of the second accumulator 118 .

[0102] When performing the normalization operation, the first selector 111 passes the input data to the first multiplier 112, and the first multiplier 112 is used to calculate the square of the input data x 2 The second selector 113 can convert the square of the input data x 2 The second shifter 116 transfers the square of the input data x to the first shifter 115 and the second shifter 116 does not perform a shift operation. 2 The first accumulator 117 is passed to the first accumulator 117. The output of the first accumulator 117 is connected to the first shifter 115. Through the first shifter 115, the first accumulator 117 can perform square x of each input data. 2 Perform the accumulation operation to obtain the square sum of the input data ∑x 2 , as output data 1. In addition, the input data can also be directly input to the second accumulator 118, and the second accumulator 118 accumulates the received input data to obtain the sum ∑x of the input data as output data 2.

[0103] In one embodiment, after the first accumulator 117 and the second accumulator 118, by adding a small amount of hardware resources, the accumulator can calculate the mean of the input data. and variance Of course, the mean of the input data can also be calculated by other units that perform normalization operations and variance For example, the power function unit can calculate the mean of the input data and variance No restrictions here.

[0104] Among them, for the mean of the input data It can be calculated by the following formula:

[0105]

[0106] The variance of the input data It can be calculated by the following formula:

[0107]

[0108] When performing the Softmax operation, the first selector 111 may pass the exponential function transformation coefficient (eg, log2e or -1 / ln2) to the first multiplier 112. In one embodiment, when the exponential transformation coefficient is log2e, the first multiplier 112 may calculate x i The specific value of log2e is separated by the second selector 113 to obtain x i The integer part u of log2e i and the fractional part v i , and u i The exponential value is passed to the maximum value calculator 114. The approximate result (for example, v i +1) is transmitted to the second shifter 116. The second shifter 116 is based on u i -u max The first shifter 115 performs a shift operation on the exponential function value of the fractional part of the exponent in the first exponential function of the i-th input data. i -u max The first accumulator 117 performs an addition operation on the outputs of the first shifter 115 and the second shifter 116 to obtain the exponential function accumulation sum of the fractional part of the exponent in the first exponential function of the first i-1 input data.

[0109] On the one hand, the maximum value calculator 114 needs to record the largest integer part of the value received from the second selector 113, and record its value as u max , and calculate u i -u max On the other hand, the accumulated sum of the fractional parts in the first exponential function of the currently received input data can be calculated through the first shifter 115, the second shifter 116 and the first accumulator 117.

[0110] In another embodiment of the present invention, the normalization operation and the Softmax operation reuse the power function operation unit. In this case, the power function operation unit is also suitable for calculating the mean and variance of the input data, and calculating the reciprocal square root to obtain the second normalization coefficient.

[0111] In a specific implementation, when the normalization operation and the Softmax operation reuse the accumulation unit and the power function operation unit, there are multiple implementation schemes. In one embodiment, Figure 2 The output data 1 of the accumulator unit is the square sum of the input data Σx 2 , the output data 2 is the sum of the input data Σx; in another embodiment, the output data 1 of the accumulation unit is the variance σ of the input data n 2 , output data 2 is the mean of the input data In a specific implementation, when the accumulation unit is used to implement the Softmax operation, the output data 1 of the accumulation unit is the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the input data, and the output data 2 is kept at 0.

[0112] The power function operation involved in the normalization operation is the reciprocal square root function, and the power function operation involved in the Softmax operation is the reciprocal function, and there are certain differences between the two. In order to be compatible with these two power function operations in the same circuit solution, in one embodiment, the carmack algorithm can be used to obtain the first normalization coefficient or the second normalization coefficient.

[0113] In the prior art, the Carmack algorithm is used to implement fast calculation of the reciprocal square root function. The pseudo code of the original Carmack algorithm is as follows:

[0114] Input:float x / / Input floating point data x

[0115] Output:float y / / Output floating point data y

[0116] function carmack(x) / / Calculate the Carmack algorithm result for x

[0117] float x half =0.5x / / Take half of the floating point data x as x half

[0118] int q x =*(int*)&x / / treat the floating point number x as the integer number q x

[0119] q x =0x5F3759DF-(q x >>1) / / Change q x Shift right 1 bit and then subtract from 0x5F3759DF

[0120] float x ini =*(float*)&q x / / Convert the integer q x x treated as a floating point number ini

[0121] y=x ini (1.5-x half × ini × ini ) / / Find the reciprocal square root

[0122] return y / / return y

[0123] end function / / end

[0124] In the embodiment of the present invention, the carmack algorithm is optimized and a Boolean variable M is added to control the power function operation unit to select a calculation mode. Specifically, when M is logically true, the power function operation unit can realize the calculation of the reciprocal square root function. When M is logically false, the power function operation unit can realize the calculation of the reciprocal function. By adding configuration options, the constants and Newton iteration formulas in the carmack algorithm are adjusted to meet the needs of different power function operations. Therefore, the power function operation unit is backward compatible with the calculation of the reciprocal function.

[0125] The pseudo code of the optimized Carmack algorithm is as follows:

[0126] Input:floatx / / Input floating point data x

[0127] Mode: bool M / / input mode variable M

[0128] Output:floaty / / Output floating point data y

[0129] function carmack(x) / / Calculate the Carmack algorithm result for x

[0130] if M / / If M is true

[0131] floatx half =0.5x / / Take half of the floating point data x as x half

[0132] else / / otherwise

[0133] floatx half =x / / Use floating point data x as x half

[0134] intq x =*(int*)&x / / treat the floating point number x as the integer number q x

[0135] if M / / If M is true

[0136] q x =0x5F3759DF-(q x >>1) / / Change q x Shift right 1 bit and then subtract from 0x5F3759DF

[0137] q x =0x7EF477DA-q x / / 0x7EF477DA and q x Difference

[0138] floatx ini =*(float*)&q x / / Convert the integer q x x treated as a floating point number ini

[0139] if M / / If M is true

[0140] y=x ini ×(1.5-x half × ini × ini ) / / Find the reciprocal square root

[0141] else

[0142] y=x ini ×(2-x half × ini ×1) / / Find the inverse

[0143] returny / / return y

[0144] end function / / end

[0145] In one embodiment of the present invention, referring to Figure 3 The power function operation unit may include: a fixed-point subtractor 121, a first floating-point multiplier 122, a second floating-point multiplier 123, a floating-point subtractor 124 and a third floating-point multiplier 125; wherein: the output end of the fixed-point subtractor 121 is connected to the first floating-point multiplier 122; the output end of the first floating-point multiplier 122 is connected to the second floating-point multiplier 123; the output end of the second floating-point multiplier 123 is connected to the floating-point subtractor 124; the output end of the floating-point subtractor 124 is connected to the third floating-point multiplier 125.

[0146] Specifically, the fixed-point subtractor 121 can be used to perform the fixed-point subtraction operation in the optimized carmack algorithm, for example, to calculate 0x5F3759DF-(qx >>1) Or calculate 0x7EF477DA-q x The first floating-point multiplier 122 is used to perform the floating-point multiplication operation in the optimized Carmack algorithm, for example, to calculate x half x ini The second floating-point multiplier 123 is used to perform another floating-point multiplication operation in the optimized Carmack algorithm, for example, to calculate (x half × ini )×x ini or (x half × ini )×1. The floating-point subtractor 124 is used to perform the floating-point subtraction operation in the optimized Carmack algorithm, for example, to calculate 1.5-x half × ini × ini or 2-x half × ini ×1. The third floating-point multiplier 125 is used to perform the floating-point multiplication operation in the optimized Carmack algorithm, for example, to calculate x ini ×(1.5-x half × ini × ini ), or x ini ×(2-x half × ini ×1).

[0147] In the specific implementation, it should be noted that when the Softmax operation is performed, the input of the power function operation unit is the exponential function accumulation sum (j) of the decimal part of the exponent in the first exponential function, and the output of the power function operation unit is the first normalization coefficient. When the normalization operator is executed, the input of the power function operation unit can be the mean and variance of the input data, or the sum of the input data and the sum of the squares of the input data. When the input of the power function operation unit is the sum of the input data and the sum of the squares of the input data, the mean and variance of the input data can be calculated first, and then the carmack algorithm is executed. At this time, the output of the power function operation unit is the second normalization coefficient.

[0148] When the normalization operation and the Softmax operation reuse the normalization unit, the normalization unit is also suitable for using the first or second normalization coefficient to obtain a normalized Softmax operation result.

[0149] In a specific implementation, when the normalization operation and the Softmax operation reuse the normalization unit, when performing the normalization operation, the role of the normalization unit is to implement the standardization operation and the affine transformation. Specifically, it can be expressed by the following formula:

[0150]

[0151] y=γ×x'+β (18)

[0152] In formula (17), coef represents the second normalization coefficient.

[0153] It should be noted that affine transformation is not necessary in normalization operation. In the computational graph of the neural network, the affine coefficients of the normalization operator can be fused into other operators by means such as operator fusion to simplify the computational graph of the neural network, reduce the amount of computation and parameters of the network, and improve the operation efficiency of the network.

[0154] When performing the Softmax operation, the role of the normalization unit is to implement the standardization operation. The specific formula can be expressed as follows:

[0155] y i =coef×(p i >>(u max -u i )) (19)

[0156] In formula (19), coef represents the first normalization coefficient.

[0157] Figure 4 FIG. 1 is a schematic diagram of a normalization unit in an embodiment of the present invention. Figure 4 The normalization unit may include: a third shifter 131, a second multiplier 132 and a first adder 133, wherein: the output end of the third shifter 131 is connected to the second multiplier 132, and the output end of the second multiplier 132 is connected to the first adder 133.

[0158] Specifically, when the normalization operation is performed, the third shifter 131 does not work, and the normalization operation is completed by the second multiplier 132 and the first adder 133. When an affine transformation needs to be performed, the output of the normalization unit can be temporarily stored in the on-chip cache, and re-transmitted to the normalization unit, and a multiplication and addition operation is completed again by the second multiplier 132 and the first adder 133 to obtain the result after the affine transformation.

[0159] When performing the Softmax operation, the normalization unit shifts the input data through the third shifter 131 and performs a multiplication operation through the second multiplier 132 to obtain the output result. At this time, the addend of the first adder 133 needs to be set to zero to achieve direct pass.

[0160] It should be noted that, in a specific implementation, for the normalization operation and the Softmax operation, only one functional unit may be reused according to the above embodiment, or all functional units may be reused, which is not limited here.

[0161] It should be noted that when the data processing device reuses resources to perform normalization operations and Softmax operations, the data processing device may further include a control unit, which may control the reused units to switch operation modes. For example, when the data processing device reuses an accumulation unit, a power function operation unit, and a normalization unit, the control unit may be connected to the accumulation unit, the power function operation unit, and the normalization unit, and send corresponding operation mode indication information to the accumulation unit, the power function operation unit, and the normalization unit, respectively, so that the accumulation unit, the power function operation unit, and the normalization unit can know the operation operation that needs to be performed currently, thereby controlling the internal devices to switch the operation mode for matching.

[0162] It can be seen from the above content that for normalization operation and Softmax operation, the data processing device in the embodiment of the present invention can be implemented through a hardware circuit with high hardware resource reuse and high computational efficiency, and can be compatible with hardware acceleration of two different operations, normalization operation and Softmax operation. In view of the problem of large memory access requirements in the Softmax operation, the data processing device can calculate the exponential function accumulation sum of the decimal part of the exponent in the first exponential function while finding the maximum value of the integer part of the exponent in the first exponential function of the input data, thereby reducing the memory access requirements of the Softmax operation and effectively improving the operation efficiency of the operator.

[0163] It should be noted that the data processing device in the embodiment of the present invention does not require the division of circuit modules. For example, by reusing hardware resources, the accumulation unit and the normalization unit can be further integrated into a hardware module 51. Figure 5 shown.

[0164] The key to the data processing device in the embodiment of the present invention is to unify and match the normalization operation and the Softmax operation to keep the same calculation process to meet the rigid requirements brought by the reuse of hardware resources, which specifically includes the following three calculation steps:

[0165] (1) The first multiplier and the first accumulator of the accumulation unit are reused to obtain the mean value of the input data by traversing the input data once. and variance Or get the sum of the exponential functions of the fractional part of the exponent in the first exponential function sum(n) and the greatest common factor u maxIn order to obtain the exponential function cumulative sum sum(n) by traversing the input data only once, the present invention proposes a method for calculating the exponential function cumulative sum of the decimal part of the exponent in the first exponential function of the input data while finding the maximum value of the integer part of the exponent in the first exponential function.

[0166] (2) Reuse the power function operation unit to obtain the reciprocal square root of the variance Or the reciprocal of the sum of the exponential functions of the fractional part of the exponent in the first exponential function 1 / sum(n). In order to be compatible with these two different power function operations, the present invention proposes a method compatible with the calculation of the reciprocal function and the reciprocal square root function by optimizing the Carmack algorithm.

[0167] (3) The second multiplier and the first adder resources of the normalization unit are reused to traverse the input data again, perform normalization operations and affine transformations on it, and then the output data can be obtained.

[0168] The data processing device in the embodiment of the present invention has higher operating efficiency for the implementation of normalization operation and Softmax operation than a general-purpose processor, and can better meet the real-time requirements of neural network deployment. Compared with circuit schemes designed for normalization operation and Softmax operation respectively, and a computing unit with general computing function, the data processing device has higher hardware resource utilization and higher operator operation efficiency.

[0169] Furthermore, the data processing device in the embodiment of the present invention can be applied to a general-purpose processor, or can be used as a separate normalization operation and Softmax operation processing circuit, and can also perform normalization operation or Softmax operation under the control of an external unit.

[0170] In order to enable those skilled in the art to better understand and implement the present invention, the method and neural network corresponding to the above-mentioned device are described in detail below.

[0171] Reference Figure 6 The embodiment of the present invention further provides a data processing method, which is suitable for performing Softmax operation. Specifically, the method includes:

[0172] Step 61, receiving input data, performing exponential function transformation on each input data, and obtaining the first exponential function corresponding to each input data; decomposing the exponent of each first exponential function to obtain the integer part and the decimal part of the exponent in each first exponential function; extracting the maximum value of the integer part of the exponent in each first exponential function; calculating the exponential function cumulative sum of the decimal part of the exponent in each first exponential function.

[0173] In a specific implementation, by decomposing the exponent of the first exponential function of the input data, the integer part and the decimal part of the exponent in the first exponential function can be obtained. The input data is traversed, the maximum value of the integer part of the exponent in the first exponential function in the received input data is updated, and the exponential function cumulative sum of the decimal part of the exponent in the first exponential function of the input data is calculated using formula (9) until the exponential function cumulative sum of the decimal part of the exponent in the first exponential function of all input data is obtained.

[0174] Specifically, when calculating the exponential function cumulative sum of the decimal parts of the exponents in the first exponential functions of all input data based on the maximum value of the integer part of the exponents in the first exponential functions of all input data, the first difference between the integer part of the exponent in the first exponential function of the j-th input data and the maximum value of the integer part of the exponent in the first exponential function of the first j-1 input data can be calculated first, and the first difference calculation result is used to shift the exponential function cumulative sum of the decimal parts of the exponents in the first exponential function of the first j-1 input data or the first exponential function value of the decimal part of the exponent in the first exponential function of the j-th input data, and perform shift and sum operations to obtain the exponential function cumulative sum of the decimal parts of the exponents in the first exponential functions of the first j input data.

[0175] When the first difference is greater than 0, a first shift operation is performed on the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j-1 input data using the first difference; and a sum operation is performed on the result of the first shift operation and the first exponential function value of the decimal part of the exponent in the first exponential function of the j-th input data to obtain the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j input data.

[0176] When the first difference is less than 0, a second shift operation is performed on the first exponential function value of the decimal part of the exponent in the first exponential function of the j-th input data using the first difference; and a summation operation is performed on the result of the second shift operation and the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j-1 input data to obtain the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j input data.

[0177] On the one hand, the complex exponential operation e x It is decomposed into two parts: the p value obtained by the approximate fitting calculation and the number of shifts u, so as to avoid overflow of the result caused by directly performing the exponential operation. On the other hand, in the process of calculating the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the input data, the exponential result e is not directly accumulated. x , but records the maximum number of shifts of all current input data as the common factor u max , and calculate the cumulative sum of the shifted q values. By comparing the current uj With the current record u max The difference between j The number of shifts of the current cumulative sum sum(j-1) is used to obtain the new cumulative sum sum(j) and the new greatest common factor u max .

[0178] By traversing the input data only once, the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of all input data can be calculated, which better unifies the operation process of normalization operation and Softmax operation.

[0179] Step 62, calculating the reciprocal of the cumulative sum of the exponential functions of the decimal parts of the exponents in the first exponential function of all input data to obtain a first normalization coefficient.

[0180] In one embodiment, by optimizing the Carmack algorithm, the constants and Newton iteration formula in the Carmack algorithm can be adjusted, so that the optimized Carmack algorithm can be used to calculate the reciprocal of the cumulative sum of the exponential functions of the decimal parts of the exponents in the first exponential function of all input data, thereby obtaining the first normalization coefficient.

[0181] Step 63: Using the first normalization coefficient, obtain the Softmax operation result corresponding to each input data.

[0182] In a specific implementation, after obtaining the first normalization coefficient, the Softmax operation result can be obtained using formula (19).

[0183] An embodiment of the present invention also provides a neural network, which includes the above-mentioned data processing device. The data processing device can be used as a Softmax operator to perform a Softmax operation, and can also be used as a normalization operator to perform a normalization operation. It can also be used as a Softmax operator and a normalization operator at the same time, and under external control, the operation mode is switched to perform a Softmax operation or a normalization operation.

[0184] When the data processing device acts as a Softmax operator and a normalization operator at the same time, the computational graph structure of the neural network can be simplified, and resources can be saved and resource utilization can be improved.

[0185] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the steps of any of the above methods.

[0186] In a specific implementation, the computer-readable storage medium may include: ROM, RAM, magnetic disk or optical disk, etc.

[0187] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and wherein the processor executes the steps of any of the above methods when running the computer program.

[0188] Regarding the various modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, or hardware modules / units, or they can be partially software modules / units and partially hardware modules / units. For example, for various devices and products applied to or integrated in a chip, the various modules / units included therein can all be implemented in the form of hardware such as circuits, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in a chip module, the various modules / units included therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units can be implemented in the form of hardware such as circuits. The element can be implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in the terminal, the various modules / units contained therein can be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits.

[0189] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A data processing device, characterized in that: The data processing device is suitable for performing Softmax operation; the data processing device comprises: An accumulation unit is adapted to receive input data, perform exponential function transformation on each input data to obtain a first exponential function corresponding to each input data; decompose the exponent of each first exponential function to obtain an integer part and a decimal part of the exponent in each first exponential function; extract the maximum value of the integer part of the exponent in each first exponential function; and calculate the exponential function accumulation sum of the decimal part of the exponent in each first exponential function; A power function operation unit, adapted to calculate the reciprocal of the first exponential function accumulation sum of the decimal part of the exponent in the first exponential function corresponding to each input data, to obtain a first normalization coefficient; The normalization unit is adapted to obtain the Softmax operation result corresponding to each input data by using the first normalization coefficient.

2. The data processing device according to claim 1, characterized in that The accumulation unit is suitable for calculating the first difference between the integer part of the exponent in the first exponential function of the j-th input data and the maximum value of the integer part of the exponent in the first exponential function of the previous j-1 input data, and using the first difference calculation result to perform the first exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the previous j-1 input data or the number of shifts of the first exponential function value of the decimal part of the first exponential function of the j-th input data, and performing shift and sum operations to obtain the first exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the previous j input data.

3. The data processing device according to claim 2, characterized in that When the first difference is greater than 0, a first shift operation is performed on the first exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j-1 input data using the first difference; and a sum operation is performed on the result of the first shift operation and the first exponential function value of the decimal part of the exponent in the first exponential function of the j-th input data to obtain the first exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j input data.

4. The data processing device according to claim 2, characterized in that When the first difference is less than 0, a second shift operation is performed on the first exponential function value of the decimal part of the exponent in the first exponential function of the j-th input data using the first difference; and a summation operation is performed on the result of the second shift operation and the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j-1 input data to obtain the exponential function accumulation sum of the decimal part of the exponent in the first exponential function values ​​of the first j input data.

5. The data processing device according to claim 1, characterized in that The accumulation unit is suitable for calculating the exponential function accumulation sum of the decimal part of the exponent in the first exponential function of the first j input data using the following formula: Where sum(j) represents the cumulative sum of the exponential functions of the decimal parts of the exponents in the first exponential function of the first j input data, sum(j-1) represents the cumulative sum of the exponential functions of the decimal parts of the exponents in the first exponential function of the first j-1 input data, and u j represents the integer part of the exponent in the first exponential function of the jth input data, p j The first exponential function value representing the fractional part of the exponent in the first exponential function of the jth input data, u max(j-1) Represents the maximum value of the integer part of the exponent in the first exponential function of the first j-1 input data.

6. The data processing device according to any one of claims 1 to 5, characterized in that: The data processing device is also suitable for performing a normalization operation, wherein the normalization operation and the Softmax operation reuse at least one of the accumulation unit, the power function operation unit and the normalization unit.

7. The data processing device according to claim 6, characterized in that When the data processing device performs a normalization operation, the accumulating unit is further adapted to calculate the square sum of each input data and the sum of the input data.

8. The data processing device according to claim 7, characterized in that The accumulation unit comprises: a first selector, a first multiplier, a second selector, a maximum value calculator, a first shifter, a second shifter, a first accumulator and a second accumulator, wherein: The output end of the first selector is connected to the first multiplier, the output end of the first multiplier is connected to the second selector, the output end of the second selector is connected to the maximum value calculator and the second shifter, and the output end of the maximum value calculator is connected to the first shifter and the second shifter; The output end of the first shifter and the output end of the second shifter are connected to the first accumulator; the output end of the first accumulator is connected to the first shifter; The output terminal of the second accumulator is connected to the input terminal of the second accumulator.

9. The data processing device according to claim 6, characterized in that When the data processing device performs a normalization operation, the power function operation unit is further adapted to calculate the mean and variance of the input data, and calculate the reciprocal square root function to obtain a second normalization coefficient.

10. The data processing device according to claim 9, characterized in that The power function operation unit comprises: a fixed-point subtractor, a first floating-point multiplier, a second floating-point multiplier, a floating-point subtractor and a third floating-point multiplier; wherein; An output end of the fixed-point subtractor is connected to the first floating-point multiplier; An output terminal of the first floating-point multiplier is connected to the second floating-point multiplier; An output terminal of the second floating-point multiplier is connected to the floating-point subtractor; An output terminal of the floating-point subtractor is connected to the third floating-point multiplier.

11. The data processing device according to claim 9, characterized in that The power function operation unit is suitable for obtaining the first normalization coefficient or the second normalization coefficient by using a Carmack algorithm.

12. The data processing device according to claim 6, characterized in that When the data processing device performs a normalization operation, the normalization unit is further adapted to obtain a normalization operation result using a second normalization coefficient.

13. The data processing device according to claim 12, characterized in that: The normalization unit comprises: a third shifter, a second multiplier and a first adder, wherein: An output end of the third shifter is connected to the second multiplier, and an output end of the second multiplier is connected to the first adder.

14. A data processing method, characterized in that: The data processing method is suitable for performing a Softmax operation; the method comprises: Receive input data, perform exponential function transformation on each input data, and obtain a first exponential function corresponding to each input data; decompose the exponent of each first exponential function to obtain the integer part and the decimal part of the exponent in each first exponential function; extract the maximum value of the integer part of the exponent in each first exponential function; calculate the exponential function cumulative sum of the decimal part of the exponent in each first exponential function; Calculate the reciprocal of the cumulative sum of the first exponential function of the decimal part of the exponent in the first exponential function corresponding to each input data to obtain a first normalization coefficient; The first normalization coefficient is used to obtain the Softmax operation result corresponding to each input data.

15. The data processing method according to claim 14, characterized in that: The step of calculating the cumulative sum of exponential functions of the fractional parts of the exponents in the first exponential functions comprises: Calculate the first difference between the integer part of the exponent in the first exponential function of the j-th input data and the maximum value of the integer part of the exponent in the first exponential function of the previous j-1 input data, and use the first difference calculation result to calculate the first exponential function cumulative sum of the decimal part of the exponent in the first exponential function of the previous j-1 input data or the number of shifts of the first exponential function value of the decimal part of the first exponential function of the j-th input data, and perform shift and sum operations to obtain the first exponential function cumulative sum of the decimal part of the exponent in the first exponential function of the previous j input data.

16. A neural network, characterized in that A data processing device comprising any one of claims 1 to 13.

Images