Method for generating softmax function based on circuit device and circuit device thereof

Abstract

A method for generating a Softmax function based on circuit elements includes: performing a subtraction calculation on an input variable and an offset, and performing an exponential calculation on the difference between the input variable and the offset to output an exponential function; applying a scaling factor to the exponential function and outputting a scaled exponential function; performing a summation calculation on the scaled exponential function and outputting the sum of the scaled exponential functions; performing a logarithmic calculation on the sum of the scaled exponential functions to output a logarithmic function, and performing a subtraction calculation on the logarithmic function and the input variable based on the logarithmic functions of the offset and the scaling factor to output the calculation result; and performing an exponential calculation on the calculation result to generate a Softmax function.

Description

Technical Field

[0001] This invention relates to a method and circuit elements for generating functions, and more particularly to a method and circuit elements for generating a Softmax function based on circuit elements. Prior Technology

[0002] The role of the Softmax function in AI models is to transform a set of numerical values ​​into a probability distribution, making the output intuitively represent the likelihood of each class. In multi-class classification problems, the last layer of a neural network typically outputs a set of raw numerical values, which can be positive, negative, or floating-point numbers of any range. However, to convert these values ​​into probabilities, a function is needed to adjust them so that their sum equals 1 and all values ​​fall between 0 and 1. Softmax is designed to do this.

[0003] The function is computed by taking the exponent of each value and dividing it by the sum of the exponents of all values, ensuring that each output is normalized to a probability. In this way, larger values ​​are amplified, smaller values ​​become relatively small, increasing the contrast between different categories and allowing the model to more clearly distinguish the most likely category when classifying. For example, in image classification applications, Softmax transforms the model's output into "what is the probability that this image is a cat, what is the probability that it is a dog, and what is the probability that it is a bird," allowing the model to make a final decision based on these probabilities.

[0004] However, for some applications, using the typical Softmax representation, the exponential distribution can easily exceed the data range of FP16. While subtracting large offsets can reduce the data range, it may lead to a loss of detail. Summary of the Invention

[0005] This invention provides a method for generating a Softmax function based on circuit elements, and the circuit elements thereof. The proposed numerically stable Softmax function uses logarithmic range reduction and avoids the exponential distribution from exceeding the FP16 range.

[0006] This invention provides a method for generating a Softmax function based on circuit elements. The method includes: performing a subtraction calculation on an input variable and an offset, and performing an exponential calculation on the difference between the input variable and the offset to output an exponential function; applying a scaling factor to the exponential function and outputting a scaled exponential function; performing a summation calculation on the scaled exponential function and outputting the sum of the scaled exponential functions; performing a logarithmic calculation on the sum of the scaled exponential functions to output a logarithmic function, and performing a subtraction calculation on the logarithmic function and the input variable based on the logarithmic functions of the offset and the scaling factor to output the calculation result; and performing an exponential calculation on the calculation result to generate a Softmax function.

[0007] This invention provides a circuit element for generating a Softmax function. The circuit element includes a subtraction and exponentiation calculation module, a scaling module, a summation module, a logarithmic and subtraction calculation module, and an exponentiation calculation module. The subtraction and exponentiation calculation module performs subtraction on the input variable and the offset, and performs exponential calculation on the difference between the input variable and the offset to output an exponential function. The scaling module applies a scaling factor to the exponential function and outputs a scaled exponential function. The summation module performs summation on the scaled exponential function and outputs the sum of the scaled exponential functions. The logarithmic and subtraction calculation module performs logarithmic calculation on the sum of the scaled exponential functions to output a logarithmic function, and performs subtraction on the logarithmic function and the input variable based on the logarithmic function of the offset and the scaling factor to output the calculation result. The exponentiation calculation module performs exponential calculation on the calculation result to generate the Softmax function.

[0008] Based on the above, in order to make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings. Simple Explanation of the Diagram

[0009] Figure 1 is a schematic diagram of the exponential distribution on the FP16 number line drawn according to relevant technologies, which is based on the application of the typical Softmax function. Figure 2 is a schematic diagram of the exponential distribution on the FP16 number line according to an embodiment of the present invention, which is based on the application of the numerically stable Softmax function with logarithmic range reduction. Figure 3 is a block diagram illustrating circuit elements for implementing numerically stable Softmax using logarithmic range reduction according to an embodiment of the present invention. Figure 4 is a flowchart illustrating the steps of a method for generating a Softmax function based on circuit elements according to an embodiment of the present invention. Implementation

[0010] The present invention is described in detail below with reference to embodiments, but the invention is not limited to the embodiments provided, and the embodiments provided may be appropriately combined. The terms "coupled / coupled" or "connected / connected" as used in this specification (including the claims) may refer to any direct or indirect connection means. For example, "a first device coupled to a second device" should be interpreted as "the first device is directly connected to the second device" or "the first device is indirectly connected to the second device through other devices or connection means." The term "signal" may refer to current, voltage, charge, temperature, data, electromagnetic waves, or any one or more signals. Furthermore, the term "and / or" may refer to "at least one of." For example, "a first signal and / or a second signal" should be interpreted as "at least one of a first signal and a second signal."

[0011] The softmax function is widely used in machine learning and pattern recognition. In related techniques, softmax is typically calculated using the following formula:

[0012] Where xi is the input variable, e is the natural constant, and n is the total number of inputs to the Softmax function. However, as shown in Figure 1, the exponential distribution Exp(xi) easily exceeds the data range of FP16. Although subtracting the large offset xoffset can reduce the data range, it may lead to a loss of detail.

[0013] To address this problem, this invention proposes using a numerically stable Softmax function with a logarithmically reduced range. The numerically stable Softmax function is derived by applying a scaling factor S to a typical Softmax function and taking its logarithm, and is expressed by the following formula: in

[0014] Where log is the natural logarithm, S is the scaling factor, and n is an integer greater than 1. In the proposed Softmax function, as shown in Figure 2, taking the logarithm can effectively compress the numerical range, and applying a sufficiently large scaling factor S to each exponential term before summation can prevent the exponent from being distributed out of range in the summation term.

[0015] Specifically, Figure 3 is a block diagram illustrating circuit elements for implementing a numerically stable Softmax using logarithmic range reduction according to an embodiment of the present invention. Referring to Figure 3, circuit element 100 may be a Softmax operation unit disposed on a chip. The Softmax operation unit 100 includes a subtraction and exponentiation calculation module 110, a scaling module 120, a summation module 130, a logarithmic and subtraction calculation module 140, and an exponentiation calculation module 150.

[0016] The subtraction and exponentiation calculation module 110 is used to receive the input variable xi, where the input variable xi is FP16 data. The subtraction and exponentiation calculation module 110 is further used to perform subtraction calculation on the input variable xi and the offset xoffset, and perform exponentiation calculation on the difference xi-xoffset to output the exponential function Exp(xi-xoffset).

[0017] Scaling module 120 receives the exponential function Exp(xi - xoffset) from subtraction and exponentiation calculation module 110. Scaling module 120 further applies a scaling factor S to the exponential function Exp(xi - xoffset) before summing and taking the logarithm, and outputs the scaled exponential function [Exp(xi - xoffset)] / S to summing module 130 and logarithm and subtraction calculation module 140.

[0018] The summation module 130 receives the scaling exponential function [Exp(xi-xoffset)] / S from the scaling module 120. The summation module 130 further performs a summation calculation on the scaling exponential function [Exp(xi-xoffset)] / S and outputs the sum of the scaling exponential function [Exp(xi-xoffset)] / S from i=1 to n, which can be expressed as follows:

[0019] The logarithm and subtraction calculation module 140 includes a logarithm calculation module 142 and a subtraction calculation module 144. The logarithm calculation module 142 receives the sum of the scaling exponential function [Exp(xi - xoffset)] / S from the summation module 130, and performs logarithmic calculation on the sum of the scaling exponential function [Exp(xi - xoffset)] / S to output a logarithmic function, which can be represented as follows:

[0020] The subtraction calculation module 144 is used to receive a logarithmic function and an input variable xi, and to perform a subtraction calculation on the logarithmic function and the input variable xi according to the logarithmic function of the offset xoffset and the scaling factor S to output the calculation result, which can be represented as follows: in

[0021] The exponentiation calculation module 150 receives the calculation result from the subtraction calculation module 144. The exponentiation calculation module 150 further performs exponentiation on the calculation result to generate a Softmax function, which can be represented as follows: in

[0022] Therefore, the numerically stable Softmax function of the present invention can be obtained by applying a scaling factor and obtaining the logarithm.

[0023] In this embodiment, the offset xoffset can be set to 11. Specifically, in FP16, the exponent distribution may exceed the range, leading to overflow. Therefore, when calculating the Softmax function, it is necessary to choose an appropriate xoffset to compress the numerical range so that the entire calculation remains within the range that FP16 can handle. The floating-point format of FP16 uses 5 bits to represent the exponent function, so its exponent range is -14 to 15. In other words, when calculating the exponent value, if the input value is too large, the result may exceed the maximum value that FP16 can represent, leading to overflow. According to the characteristics of FP16, its maximum safe exponent range is approximately exp(11), and the calculation result exceeding this range may become infinitely large. To avoid this situation, the Softmax function should not simply take the maximum value of the input value, but should subtract the safe range so that the final exponent value does not exceed the range of FP16. Therefore, the offset xoffset can be set to 11.

[0024] On the other hand, in this embodiment, the scaling factor S can be dynamically adjusted or set as a reference value. For example, if calculating the result of the Softmax function may still lead to floating-point overflow, the scaling factor S can be dynamically adjusted according to the output of the Softmax function to avoid floating-point overflow. In addition, in one embodiment, the scaling factor S can be set to a reference value based on design experience to ensure that the sum of the exponential functions is within the range of FP16 and to avoid small values ​​being truncated to 0.

[0025] Circuit element 100 can be implemented using various hardware circuits, such as Complex Programmable Logic Device (CPLD), Field Programmable Gate Array (FPGA), Application-Specific Integrated Circuit (ASIC), etc., but the present invention is not limited thereto. Sufficient teaching, suggestions, and implementation instructions can be obtained by referring to general knowledge in related art for the hardware structure of each computing module in circuit element 100.

[0026] Figure 4 is a flowchart illustrating the steps of a method for generating a Softmax function based on circuit elements according to an embodiment of the present invention. Referring to Figures 3 and 4, the method for generating the Softmax function in Figure 4 can be implemented at least based on the circuit element 100 in Figure 3, but the present invention is not limited thereto.

[0027] Taking circuit element 100 as an example, in step S100, a subtraction calculation is performed on the input variable xi and the offset xoffset, and an exponential calculation is performed on the difference xi-xoffset to output the exponential function Exp(xi-xoffset). In step S110, the scaling factor S is applied to the exponential function Exp(xi-xoffset), and the scaling exponential function [Exp Exp(xi-xoffset)] / S is output. In step S120, a summation calculation is performed on the scaling exponential function [Exp(xi-xoffset)] / S, and the sum of the scaling exponential functions [Exp(xi-xoffset)] / S from i=1 to n is output. In step S130, a logarithmic calculation is performed on the sum of the scaling exponential functions to output the logarithmic function, and a subtraction calculation is performed on the logarithmic function and the input variable xi according to the logarithmic function of the offset xoffset and the scaling factor S to output the calculation result. In step S140, an exponential calculation is performed on the calculation result to generate the Softmax function.

[0028] The method for generating the Softmax function based on circuit elements in this embodiment of the invention can be sufficiently taught, suggested and explained in the embodiments of Figures 1 to 3, and therefore will not be described in detail here.

[0029] In summary, in the embodiments of the present invention, the Softmax operation unit includes at least a scaling module and a logarithmic calculation module. A sufficiently large scaling factor S is applied to each exponential term before summation and logarithm taking, while logarithm taking effectively compresses the numerical range. Therefore, a numerically stable Softmax function with logarithmically reduced range can be generated for subsequent network layer calculations, and the exponential distribution will not exceed the FP16 range.

[0030] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0031] 100: Circuit elements 110: Subtraction and Exponentiation Calculation Module 120: Scaling Module 130: Summation Module 140: Logarithmic and Subtraction Calculation Module 142: Logarithmic Calculation Module 144: Subtraction Calculation Module 150: Index Calculation Module S100, S110, S120, S130, S140: Method Steps

Claims

1. A method for generating a Softmax function based on circuit elements, the method comprising: Perform a subtraction calculation on the input variable and the offset, and perform an exponential calculation on the difference between the input variable and the offset to output an exponential function; The scaling factor is applied to the exponential function, and the scaled exponential function is output; a summation calculation is performed on the scaled exponential function, and the sum of the scaled exponential functions is output; a logarithmic calculation is performed on the sum of the scaled exponential functions to output a logarithmic function, and a subtraction calculation is performed on the logarithmic function and the input variable according to the offset and the logarithmic function of the scaling factor to output the calculation result; and an exponential calculation is performed on the calculation result to generate the Softmax function.

2. The method for generating a Softmax function based on circuit elements as described in claim 1, wherein the input to the Softmax function is FP16 data.

3. The method for generating a Softmax function based on circuit elements as described in claim 1, wherein the scaling factor is adjusted according to the output of the Softmax function.

4. The method for generating a Softmax function based on circuit elements as described in claim 1, wherein the scaling factor is set to a reference value.

5. The method for generating a Softmax function based on circuit elements as described in claim 1, wherein the offset is set to 11.

6. The method for generating a Softmax function based on circuit elements as described in claim 1, wherein the Softmax function is expressed as: where xi is the input variable, xoffset is the offset, S is the scaling factor, n is the total number of inputs to the Softmax function, exp and e are natural constants, and log is the natural logarithm.

7. A circuit element for generating a Softmax function, said circuit element comprising: The system includes: a subtraction and exponentiation module for performing subtraction on the input variable and the offset, and performing exponentiation on the difference between the input variable and the offset to output an exponential function; a scaling module for applying a scaling factor to the exponential function and outputting a scaled exponential function; a summation module for performing summation on the scaled exponential function and outputting the sum of the scaled exponential function; and a logarithmic and subtraction module for performing logarithmic calculation on the sum of the scaled exponential function to output a logarithmic function, and performing subtraction on the logarithmic function and the input variable based on the logarithmic function of the offset and the scaling factor to output the calculation result. And an exponent calculation module, used to perform exponent calculation on the calculation result to generate the Softmax function.

8. The circuit element as described in claim 7, wherein the input to the Softmax function is FP16 data.

9. The circuit element as claimed in claim 7, wherein the scaling factor is adjusted according to the output of the Softmax function.

10. The circuit element as claimed in claim 7, wherein the scaling factor is set as a reference value.

11. The circuit element as claimed in claim 7, wherein the offset is set to 11.

12. The circuit element as claimed in claim 7, wherein the Softmax function is expressed as: where xi is the input variable, xoffset is the offset, S is the scaling factor, n is the total number of inputs to the Softmax function, exp and e are natural constants, and log is the natural logarithm.

13. The circuit element as claimed in claim 7, wherein the logarithmic and subtraction calculation module comprises: A logarithmic calculation module is used to perform the logarithmic calculation on the sum of the scaling exponential function to output the logarithmic function; And a subtraction calculation module, which performs a subtraction calculation on the logarithmic function and the input variable based on the logarithmic function of the offset and the scaling factor to output the calculation result.

Images