Circuit capable of configuring multiple activation functions and electronic equipment

Abstract

The invention discloses a circuit capable of configuring multiple activation functions and electronic equipment, and relates to the technical field of neural networks. The circuit comprises a configuration module, a plurality of activation function calculation modules and a selection module, and the activation function calculation modules are used for performing calculation according to input data and configuration parameters to generate calculation results; the configuration module is used for determining a target activation function calculation module based on a configuration file under the condition that it is determined that the activation function calculation module needs to be used according to the input data, and configuring the selection module and the target activation function calculation module based on the configuration file; and the selection module is used for outputting a calculation result of the target activation function calculation module. The circuit disclosed by the invention can be flexibly configured according to network requirements, and various mainstream activation function operators are supported at relatively low hardware cost, so that the requirements of low power consumption of an end side and real-time performance of network calculation are met.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a circuit and an electronic device that can be configured with multiple activation functions. Background Art

[0002] With the rapid development of artificial intelligence, various neural network models emerge in an endless stream, and more and more operators are applied to the calculation of neural networks. In the neural network algorithm, the activation function, as a nonlinear function, introduces nonlinear factors to convert the input signal into the output signal, so that the neural network can be trained to obtain complex patterns and relationships. The activation function introduces nonlinearity in each computing layer of the neural network, so that the neural network can approximate any complex function. Among them, the operator types of the activation function layer are also increasing, including Sigmoid (a mathematical function that is often used to map any real number to the (0,1) interval and has S-shaped curve characteristics), Tanh (Hyperbolic Tangent Function), ReLU (Rectified Linear Unit) and Leaky ReLU (Leaky Rectified Linear Unit). These activation functions are crucial to the performance and training of neural networks. Choose the appropriate activation function type according to the specific application and network structure.

[0003] The demand for neural network algorithms in edge computing applications on the end side is increasing, but due to the requirements of low power consumption and real-time performance, efficient hardware acceleration must be used to meet the performance requirements of neural network computing. However, the activation function operator of the neural network has a large amount of calculation, and some activation function operators are complex types. The use of general processor core calculations cannot meet the real-time requirements. At the same time, due to the requirement of low power consumption, it is impossible to deploy a GPU (Graphics Processing Unit) on the end side to run the neural network algorithm. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first purpose of the present application is to propose a circuit that can be configured with multiple activation functions, the circuit comprising: a configuration module, multiple activation function calculation modules and a selection module, wherein the activation function calculation module is used to calculate according to the input data and configuration parameters to generate calculation results; the configuration module is used to determine the target activation function calculation module based on the configuration file when it is determined that the activation function calculation module needs to be used according to the input data, and configure the selection module and the target activation function calculation module based on the configuration file; the selection module is used to output the calculation result of the target activation function calculation module. The circuit of the present application can be flexibly configured according to network needs, and can support a variety of mainstream activation function operators at a relatively low hardware cost, thereby meeting the requirements of low power consumption on the end side and real-time network computing.

[0005] The second objective of the present application is to provide an electronic device.

[0006] To achieve the above-mentioned purpose, the first aspect embodiment of the present application proposes a circuit that can be configured with multiple activation functions, the circuit comprising: a configuration module, multiple activation function calculation modules and a selection module, wherein the activation function calculation module is used to perform calculations according to input data and configuration parameters to generate calculation results; the configuration module is used to determine the target activation function calculation module based on the configuration file when it is determined according to the input data that the activation function calculation module needs to be used, and configure the selection module and the target activation function calculation module based on the configuration file; the selection module is used to output the calculation result of the target activation function calculation module.

[0007] According to one embodiment of the present application, the selection module is further configured to use the input data as output data when it is determined based on the input data that no activation function calculation is required.

[0008] According to one embodiment of the present application, multiple activation function calculation modules include: a first activation function calculation module, a second activation function calculation module and a third activation function calculation module, wherein the first activation function calculation module is used to calculate the ReLU type activation function, and the ReLU type activation function includes one of ReLU, ReLU6, LeakyReLU, PReLU and SoftShrinkage; the second activation function calculation module is used to calculate the Hard type activation function, and the Hard type activation function includes one of Hardswish, Hardsigmoid, HardTanh and HardShrink; the third activation function calculation module is used to calculate the exponential type activation function, and the exponential type activation function includes one of Sigmoid, Tanh, silu and ELU.

[0009] According to one embodiment of the present application, the first activation function calculation module includes: a first selection unit, a second selection unit and a third selection unit, the first input end of the first selection unit is used to receive input data, the first input end of the third selection unit is connected to the output end of the first selection unit, and the output end of the third selection unit is connected to the first input end of the second selection unit, wherein, in the case where the ReLU type activation function includes the ReLU activation function, the first selection unit is used to use the input data as the calculation result of the ReLU activation function when the input data is greater than or equal to a first preset value, or, when the input data is less than the first preset value, use the first preset value as the ReLU activation function. The third selection unit outputs the calculation result of the ReLU activation function to the second selection unit, and the second selection unit outputs the calculation result of the ReLU activation function; in the case that the ReLU type activation function includes the ReLU6 activation function, the third selection unit is used to use the calculation result of the ReLU activation function as the calculation result of the ReLU6 activation function when the calculation result of the ReLU activation function is less than the second preset value, or, when the calculation result of the ReLU activation function is greater than or equal to the second preset value, use the second preset value as the calculation result of the ReLU6 activation function, and the second selection unit is used to output the calculation result of the ReLU activation function.

[0010] According to one embodiment of the present application, the configuration parameters include LeakyReLU parameters or PreLU parameters, and the first activation function calculation module includes: a first shift register unit, a first multiplication unit, a second selection unit, a fourth selection unit and a fifth selection unit, the input ends of the first shift register unit and the first multiplication unit are both used to receive input data, the output end of the first shift register unit is connected to the first input end of the fifth selection unit, the output end of the first multiplication unit is connected to the second input end of the fifth selection unit, the output end of the fifth selection unit is connected to the first input end of the fourth selection unit, the second input end of the fourth selection unit is used to receive input data, and the output end of the fourth selection unit is connected to the second input end of the second selection unit, wherein, in the case where the ReLU type activation function includes the LeakyReLU activation function, the first shift register unit is used to shift the input data to obtain a first shift result, the first shift result is the product of the input data and the LeakyReLU parameter, and the fifth The selection unit outputs the first shift result to the fourth selection unit, and the fourth selection unit is used to use the input data as the calculation result of the LeakyReLU activation function when the input data is greater than or equal to the third preset value, or, when the input data is less than the third preset value, use the first shift result as the calculation result of the LeakyReLU activation function, and the second selection unit is used to output the calculation result of the LeakyReLU activation function; in the case where the ReLU type activation function includes the PReLU activation function, the first multiplication unit is used to obtain the multiplication result according to the product of the input data and the PreLU parameter, and the fifth selection unit outputs the multiplication result to the fourth selection unit, and the fourth selection unit is used to use the input data as the calculation result of the PReLU activation function when the input data is greater than or equal to the third preset value, or, when the input data is less than the third preset value, use the multiplication result as the calculation result of the PReLU activation function, and the second selection unit is used to output the calculation result of the PReLU activation function.

[0011] According to one embodiment of the present application, when the ReLU type activation function includes a SoftShrinkage activation function, the configuration parameters include a first SoftShrinkage parameter and a second SoftShrinkage parameter, and the first activation function calculation module includes: a second selection unit, a sixth selection unit, a seventh selection unit and a first addition unit, the first input end of the sixth selection unit is used to receive the first SoftShrinkage parameter, the second end of the sixth selection unit is used to receive the second SoftShrinkage parameter, the output end of the sixth selection unit is connected to the first input end of the first addition unit, the second input end of the first addition unit is used to receive input data, the output end of the first addition unit is connected to the first input of the seventh selection unit, and the output end of the seventh selection unit is connected to the third input end of the second selection unit, wherein the sixth selection unit is used to output the second SoftShrinkage parameter when the input data is greater than the first SoftShrinkage parameter, and the first addition unit is used to sum the input data and the second SoftShrinkage parameter as SoftShrink The seventh selection unit outputs the calculation result of the SoftShrinkage activation function to the second selection unit, and the second selection unit outputs the calculation result of the SoftShrinkage activation function; the sixth selection unit is also used to output the first SoftShrinkage parameter when the input data is less than the second SoftShrinkage parameter, the first addition unit is used to sum the input data and the first SoftShrinkage parameter as the calculation result of the SoftShrinkage activation function, the seventh selection unit outputs the calculation result of the SoftShrinkage activation function to the second selection unit, and the second selection unit outputs the calculation result of the SoftShrinkage activation function; the seventh selection unit is also used to use the fourth preset value as the calculation result of the SoftShrinkage activation function when the input data is greater than or equal to the second SoftShrinkage parameter and less than or equal to the first SoftShrinkage parameter, and the second selection unit outputs the calculation result oftShrinkage activation function.

[0012] According to one embodiment of the present application, the second activation function calculation module includes: a first multiplication and addition unit, a second multiplication unit, an eighth selection unit, a ninth selection unit, a tenth selection unit, an eleventh selection unit and a twelfth selection unit, the input end of the first multiplication and addition unit is used to receive input data, the output end of the first multiplication and addition unit is respectively connected to the input end of the second multiplication unit and the second input end of the eighth selection unit, the output end of the second multiplication unit is connected to the first input end of the eighth selection unit, the output end of the eighth selection unit is connected to the first input end of the eleventh selection unit, the first input end of the ninth selection unit is used to receive input data, and the second input end of the ninth selection unit is used to receive the fifth preset value, the output end of the ninth selection unit is connected to the first input end of the tenth selection unit, the second input end of the tenth selection unit is used to receive the sixth preset value, the output end of the tenth selection unit is connected to the second input end of the eleventh selection unit, and the output end of the eleventh selection unit is connected to the first input end of the twelfth selection unit, wherein, in the case where the Hard class activation function includes the Hardswish activation function, the first multiplication and addition unit is used to perform a multiplication and addition operation on the input data when the input data is greater than the eighth preset value and less than the ninth preset value to obtain a multiplication and addition operation result, and the second multiplication unit is used to obtain a second multiplication result according to the product of the multiplication and addition operation result and the input data, as the Hards The eighth selection unit outputs the calculation result of the Hardswish activation function to the eleventh selection unit; or, the ninth selection unit is used to use the input data as the calculation result of the Hardswish activation function when the input data is greater than or equal to the eighth preset value, and output the calculation result of the Hardswish activation function to the tenth selection unit, and the tenth selection unit outputs the calculation result of the Hardswish activation function to the eleventh selection unit; or, the tenth selection unit is used to use the sixth preset value as the calculation result of the Hardswish activation function when the input data is less than or equal to the ninth preset value, and output the calculation result of the Hardswish activation function to the tenth selection unit. The calculation result of the ish activation function is output to the eleventh selection unit; the eleventh selection unit outputs the calculation result of the Hardswish activation function to the twelfth selection unit, and the twelfth selection unit outputs the calculation result of the Hardswish activation function; when the Hard class activation function includes the Hardsigmoid activation function, the first multiplication and addition unit is used to perform a multiplication and addition operation on the input data when the input data is greater than the eighth preset value and less than the ninth preset value, to obtain a multiplication and addition operation result as the calculation result of the Hardsigmoid activation function, and the eighth selection unit outputs the calculation result of the Hardsigmoid activation function to the eleventh selection unit;Alternatively, the ninth selection unit is used to use the fifth preset value as the calculation result of the Hardsigmoid activation function when the input data is greater than or equal to the eighth preset value, and the tenth selection unit outputs the calculation result of the Hardsigmoid activation function to the eleventh selection unit; or, the tenth selection unit is used to use the sixth preset value as the calculation result of the Hardsigmoid activation function when the input data is less than or equal to the ninth preset value, and output the calculation result of the Hardsigmoid activation function to the eleventh selection unit; the eleventh selection unit outputs the calculation result of the Hardsigmoid activation function to the twelfth selection unit, and the twelfth selection unit outputs the calculation result of the Hardsigmoid activation function. ;

[0013] According to one embodiment of the present application, the configuration parameters include a first HardTanh parameter and a second HardTanh parameter, and the second activation function calculation module includes: a twelfth selection unit, a thirteenth selection unit, a fourteenth selection unit, a fifteenth selection unit, and a sixteenth selection unit, the first input end of the thirteenth selection unit is used to receive the first HardTanh parameter, the second input end of the thirteenth selection unit is used to receive the second HardTanh parameter, the output end of the thirteenth selection unit is connected to the second input end of the fourteenth selection unit, the first input end of the fourteenth selection unit and the first input end of the fifteenth selection unit are both used to receive input data, and the output end of the fourteenth selection unit is connected to the fifteenth selection unit. The second input end of the unit is connected, the output end of the fifteenth selection unit is connected to the first input end of the sixteenth selection unit, the second input end of the sixteenth selection unit is used to receive the seventh preset value, and the output end of the sixteenth selection unit is connected to the second input end of the twelfth selection unit, wherein, when the Hard class activation function includes a HardTanh activation function, the thirteenth selection unit is used to use the first HardTanh parameter as the calculation result of the HardTanh activation function when the input data is greater than the first HardTanh parameter, and the fourteenth selection unit outputs the calculation result of the HardTanh activation function to the fifteenth selection unit; or, the thirteenth selection unit 228 is used to When the second HardTanh parameter is greater than or equal to the second HardTanh parameter and less than or equal to the first HardTanh parameter, the second HardTanh parameter is used as the calculation result of the HardTanh activation function, and the fourteenth selection unit outputs the calculation result of the HardTanh activation function to the fifteenth selection unit; or, the fourteenth selection unit is used to use the input data as the calculation result of the HardTanh activation function when the input data is greater than or equal to the second HardTanh parameter and less than or equal to the first HardTanh parameter, and output the calculation result of the HardTanh activation function to the fifteenth selection unit; the fifteenth selection unit is used to output the calculation result of the HardTanh activation function to the sixteenth selection unit, and the sixteenth selection unit outputs the calculation result of the HardTanh activation function to the sixteenth selection unit. The calculation result of the dTanh activation function is output to the twelfth selection unit, and the calculation result of the HardTanh activation function is output by the twelfth selection unit; when the Hard type activation function includes the HardShrink activation function, the fifteenth selection unit is used to use the input data as the calculation result of the HardShrink activation function when the input data is greater than the tenth preset value or less than the eleventh preset value, and output the calculation result of the HardShrink activation function to the sixteenth selection unit; or, the sixteenth selection unit is used to use the seventh preset value as the calculation result of the HardShrink activation function when the input data is greater than or equal to the eleventh preset value and less than or equal to the tenth preset value;The sixteenth selection unit outputs the calculation result of the HardShrink activation function to the twelfth selection unit, and the twelfth selection unit outputs the calculation result of the HardShrink activation function. ;

[0014] According to one embodiment of the present application, the third activation function calculation module includes: a third multiplication unit, a second addition unit, a second shift register unit, an arithmetic logic unit, a seventeenth selection unit and an eighteenth selection unit, the input end of the third multiplication unit is used to receive input data, the output end of the third multiplication unit is respectively connected to the input end of the second addition unit and the first input end of the seventeenth selection unit, the output end of the second addition unit is connected to the input end of the second shift register unit, the output end of the second shift register unit is connected to the input end of the arithmetic logic unit, the output end of the arithmetic logic unit is connected to the first input end of the eighteenth selection unit, and the second input end of the seventeenth selection unit is used to receive the input data. According to the embodiment, the output end of the seventeenth selection unit is connected to the second input end of the eighteenth selection unit, wherein, when the exponential activation function includes the Tanh activation function, the third multiplication unit performs a product operation on the input data and the twelfth preset value to obtain a calculation result of the Tanh activation function, the seventeenth selection unit outputs the calculation result of the Tanh activation function to the eighteenth selection unit, and the eighteenth selection unit outputs the calculation result of the Tanh activation function; when the exponential activation function includes the Sigmoid activation function, the third multiplication unit performs a product operation on the input data and the thirteenth preset value to obtain a product result, and the second addition unit performs a sum operation on the product result and the fourteenth preset value to obtain To the summation result, the second shift register unit performs a shift operation on the summation result to obtain a second shift result, the arithmetic logic unit performs a first logic operation on the second shift result to obtain the calculation result of the Sigmoid activation function, and the eighteenth selection unit is used to output the calculation result of the Sigmoid activation function, wherein the first logic operation is to take the reciprocal of the second shift result and sum it with the fifteenth preset value and then take the reciprocal; when the exponential activation function includes the silu activation function, the arithmetic logic unit performs a second logic operation on the second shift result to obtain the calculation result of the silu activation function, and the eighteenth selection unit is used to output the calculation result of the silu activation function, wherein the second logic operation is to take the reciprocal of the second shift result and sum it with the fifteenth preset value and then take the reciprocal; when the exponential activation function includes the silu activation function, the arithmetic logic unit performs a second logic operation on the second shift result to obtain the calculation result of the silu activation function, and the eighteenth selection unit is used to output the calculation result of the silu activation function, wherein the second logic operation is The operation is the product of the result obtained by taking the inverse of the second shift result and summing it with the fifteenth preset value and then taking the inverse and the input data; when the exponential activation function includes the ELU activation function, the eighteenth selection unit is used to use the input data as the calculation result of the ELU activation function when the input data is greater than the sixteenth preset value; or, the eighteenth selection unit is used to perform a third logic operation on the second shift result by the arithmetic logic unit when the input data is less than or equal to the sixteenth preset value to obtain the calculation result of the ELU activation function, wherein the third logic operation is the difference between the second shift result and the seventeenth preset value multiplied by the eighteenth preset value; the eighteenth selection unit is used to output the calculation result of the ELU activation function.

[0015] In order to achieve the above-mentioned purpose, a second embodiment of the present application proposes an electronic device, including the aforementioned circuit that can be configured with multiple activation functions.

[0016] According to the circuit and electronic device that can be configured with multiple activation functions according to the embodiment of the present application, the circuit includes: a configuration module, multiple activation function calculation modules and a selection module, wherein the activation function calculation module is used to calculate according to input data and configuration parameters to generate calculation results; the configuration module is used to determine the target activation function calculation module based on the configuration file when it is determined that the activation function calculation module needs to be used according to the input data, and configure the selection module and the target activation function calculation module based on the configuration file; the selection module is used to output the calculation result of the target activation function calculation module. The circuit of the present application can be flexibly configured according to network needs, and can support multiple mainstream activation function operators at a relatively low hardware cost, thereby meeting the requirements of low power consumption on the end side and real-time network computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the structure of a circuit that can be configured with multiple activation functions according to some embodiments of the present application;

[0018] Figure 2 It is a schematic diagram of the structure of a circuit that can be configured with multiple activation functions according to other embodiments of the present application;

[0019] Figure 3 is a schematic diagram of the structure of a first activation function calculation module according to some embodiments of the present application;

[0020] Figure 4 is a schematic diagram of the structure of a second activation function calculation module according to some embodiments of the present application;

[0021] Figure 5 is a schematic diagram of the structure of a third activation function calculation module according to some embodiments of the present application;

[0022] Figure 6 It is a block diagram of an electronic device according to some embodiments of the present application. DETAILED DESCRIPTION

[0023] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0024] The circuits and electronic devices that can be configured with multiple activation functions according to the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0025] In some embodiments, reference Figure 1The circuit 100 of the present application that can be configured with multiple activation functions includes: a configuration module 1, multiple activation function calculation modules 2 and a selection module 3, wherein the activation function calculation module 2 is used to perform calculations according to input data and configuration parameters to generate calculation results; the configuration module 1 is used to determine the target activation function calculation module 2 based on the configuration file when it is determined according to the input data that the activation function calculation module 2 needs to be used, and configure the selection module 3 and the target activation function calculation module 2 based on the configuration file; the selection module 3 is used to output the calculation result of the target activation function calculation module 2.

[0026] Specifically, in order to meet the requirements of real-time and low power consumption, the neural network layer running in the edge computing AI scenario needs to input data into the activation function calculation module 2 for calculation. The circuit of the present application includes multiple activation function calculation modules 2, and multiple activation function calculation modules 2 correspond to multiple types of activation functions, such as ReLU type activation function, Hard type activation function, and exponential type activation function. Therefore, when the input data needs to use the activation function calculation module 2, the configuration module 1 determines the target activation function calculation module 2 based on the configuration file, and configures the selection module 3 and the target activation function calculation module 2 based on the configuration file, wherein the configuration file includes the configuration parameters of the activation function and the type of the target activation function calculation module 2. After the target activation function calculation module 2 is successfully configured, the target activation function calculation module 2 will calculate according to the input data and configuration parameters, generate the calculation result, and output the calculation result to the selection module 3, and the selection module 3 outputs the calculation result of the target activation function calculation module 2.

[0027] The circuit of the present application can be flexibly configured according to network needs, and can support a variety of mainstream activation function operators at a relatively low hardware cost, thereby meeting the requirements of low power consumption on the terminal side and real-time network computing.

[0028] In some embodiments, the selection module 3 is further configured to use the input data as output data when it is determined based on the input data that no activation function calculation is required.

[0029] Specifically, if the neural network layer running in the edge computing AI scenario needs to be accurately calculated, there is no need to input the data into the activation function calculation module 2 for calculation. Therefore, when the input data does not need to use the activation function calculation module 2, the configuration module 1 configures the selection module 3 based on the configuration file, and directly uses the input data as output data, which will be input into the processor core for accurate calculation.

[0030] In some embodiments, reference Figure 2, the multiple activation function calculation modules 2 include: a first activation function calculation module 21, a second activation function calculation module 22 and a third activation function calculation module 23, wherein the first activation function calculation module 21 is used to calculate the ReLU type activation function, and the ReLU type activation function includes one of the ReLU activation function, ReLU6 activation function, LeakyReLU activation function, PreLU (Parametric Rectified Linear Unit, parameterized streamline linear unit) activation function and SoftShrinkage activation function; the second activation function calculation module 22 is used to calculate the Hard type activation function, and the Hard type activation function includes one of the Hardswish activation function, Hardsigmoid activation function, HardTanh activation function and HardShrink activation function; the third activation function calculation module 23 is used to calculate the exponential type activation function, and the exponential type activation function includes one of the Sigmoid activation function, Tanh activation function, silu (Sigmoid Linear Unit, Sigmoid linear unit) activation function and ELU (Exponential Linear Units, exponential linear unit) activation function.

[0031] In some embodiments, reference Figure 3 The first activation function calculation module 21 includes: a first selection unit 211, a second selection unit 212 and a third selection unit 213, wherein the first input end of the first selection unit 211 is used to receive input data, the first input end of the third selection unit 213 is connected to the output end of the first selection unit 211, and the output end of the third selection unit 213 is connected to the first input end of the second selection unit 212, wherein, in the case where the ReLU type activation function includes the ReLU activation function, the first selection unit 211 is used to use the input data as the calculation result of the ReLU activation function when the input data is greater than or equal to a first preset value, or, in the case where the input data is less than the first preset value, use the first preset value as the ReLU activation function. U activation function, the third selection unit 213 outputs the calculation result of the ReLU activation function to the second selection unit 212, and the second selection unit 212 outputs the calculation result of the ReLU activation function; in the case that the ReLU type activation function includes the ReLU6 activation function, the third selection unit 213 is used to use the calculation result of the ReLU activation function as the calculation result of the ReLU6 activation function when the calculation result of the ReLU activation function is less than the second preset value, or, when the calculation result of the ReLU activation function is greater than or equal to the second preset value, use the second preset value as the calculation result of the ReLU6 activation function, and the second selection unit 212 is used to output the calculation result of the ReLU activation function.

[0032] Specifically, when the ReLU type activation function includes a ReLU activation function, the ReLU activation function calculation module includes a first selection unit 211, a third selection unit 213, and a second selection unit 212, and the ReLU activation function formula is as follows:

[0033] ReLU(x)=max(0,x)

[0034] Among them, ReLU(x) represents the calculation result of the ReLU activation function, and x represents the input data.

[0035] For example, refer to Figure 3 When the input data is greater than or equal to the first preset value (for example, 0), the first selection unit 211 directly outputs the input data as the calculation result of the ReLU activation function. When the input data is less than the first preset value (for example, 0), the first selection unit 211 outputs the first preset value (for example, 0) as the calculation result of the ReLU activation function. The third selection unit 213 outputs the calculation result of the ReLU activation function to the second selection unit 212, and the second selection unit 212 outputs the calculation result of the ReLU activation function.

[0036] Specifically, when the ReLU type activation function includes the ReLU6 activation function, the ReLU6 activation function calculation module includes a first selection unit 211, a second selection unit 212 and a third selection unit 213, and the ReLU6 activation function formula is as follows:

[0037] ReLU6(x)=min(max(0,x),6)

[0038] Among them, ReLU6(x) represents the calculation result of the ReLU6 activation function, and x represents the input data.

[0039] For example, refer to Figure 3 When the calculation result of the ReLU activation function is less than the second preset value (for example, 6), the third selection unit 213 outputs the calculation result of the ReLU activation function as the calculation result of the ReLU6 activation function. When the calculation result of the ReLU activation function is greater than or equal to the second preset value (for example, 6), the third selection unit 213 outputs the second preset value (for example, 6) as the calculation result of the ReLU6 activation function. The third selection unit 213 outputs the calculation result of the ReLU6 activation function to the second selection unit 212, and the second selection unit 212 outputs the calculation result of the ReLU6 activation function.

[0040] It should be noted that, when the ReLU type activation function includes the ReLU activation function, the third selection unit 213 is used to directly output the calculation result of the ReLU activation function. When the ReLU type activation function includes the ReLU6 activation function, the first selection unit 211 is reused, and the third selection unit 213 is used to output the calculation result of the ReLU6 activation function.

[0041] In some embodiments, reference Figure 3 , the configuration parameters include LeakyReLU parameters or PreLU parameters, the first activation function calculation module 21 includes: a first shift register unit 214, a first multiplication unit 217, a second selection unit 212, a fourth selection unit 216 and a fifth selection unit 215, the input ends of the first shift register unit 214 and the first multiplication unit 217 are both used to receive input data, the output end of the first shift register unit 214 is connected to the first input end of the fifth selection unit 215, the output end of the first multiplication unit 217 is connected to the second input end of the fifth selection unit 215, the output end of the fifth selection unit 215 is connected to the first input end of the fourth selection unit 216, the second input end of the fourth selection unit 216 is used to receive input data, and the output end of the fourth selection unit 216 is connected to the second input end of the second selection unit 212, wherein, in the case where the ReLU type activation function includes the LeakyReLU activation function, the first shift register unit 214 is used to shift the input data to obtain a first shift result, and the first shift result is the sum of the input data and the LeakyReLU parameter The product of the numbers, the fifth selection unit 215 outputs the first shift result to the fourth selection unit 216, the fourth selection unit 216 is used to use the input data as the calculation result of the LeakyReLU activation function when the input data is greater than or equal to the third preset value, or, when the input data is less than the third preset value, use the first shift result as the calculation result of the LeakyReLU activation function, and the second selection unit 212 is used to output the calculation result of the LeakyReLU activation function; in the case where the ReLU type activation function includes the PReLU activation function, the first multiplication unit 217 is used to obtain the multiplication result according to the product of the input data and the PreLU parameter, the fifth selection unit 215 outputs the multiplication result to the fourth selection unit 216, the fourth selection unit 216 is used to use the input data as the calculation result of the PReLU activation function when the input data is greater than or equal to the third preset value, or, when the input data is less than the third preset value, use the multiplication result as the calculation result of the PReLU activation function, and the second selection unit 212 is used to output the calculation result of the PReLU activation function.

[0042] Specifically, when the ReLU type activation function includes the LeakyReLU activation function, the LeakyReLU activation function calculation module includes a first shift register unit 214, a second selection unit 212, a fourth selection unit 216 and a fifth selection unit 215, and the LeakyReLU activation function formula is as follows:

[0043]

[0044] Among them, LeakyReLU(x) represents the calculation result of the LeakyReLU activation function, x represents the input data, and negative_slope represents the LeakyReLU parameter.

[0045] For example, refer to Figure 3 , the first shift register unit 214 performs shift processing on the input data, for example, right shifting 3 bits, to obtain a first shift result. In the case of right shifting 3 bits, the first shift result is the product of the input data and one eighth. The fifth selection unit 215 outputs the first shift result to the fourth selection unit 216. When the input data is greater than or equal to the third preset value (for example, 0), the fourth selection unit 216 outputs the input data as the calculation result of the LeakyReLU activation function. When the input data is less than the third preset value (for example, 0), the fourth selection unit 216 outputs the first shift result as the calculation result of the LeakyReLU activation function. The fourth selection unit 216 outputs the calculation result of the LeakyReLU activation function to the second selection unit 212, and the second selection unit 212 outputs the calculation result of the LeakyReLU activation function.

[0046] Specifically, when the ReLU type activation function includes a PReLU activation function, the PReLU activation function calculation module includes a first multiplication unit 217, a second selection unit 212, a fourth selection unit 216, and a fifth selection unit 215, and the PReLU activation function formula is as follows:

[0047]

[0048] Among them, PReLU(x) represents the calculation result of the PReLU activation function, x represents the input data, and α represents the PreLU parameter.

[0049] For example, refer to Figure 3The first multiplication unit 217 multiplies the input data by the PreLU parameter (for example, α) to obtain a multiplication result, and the fifth selection unit 215 outputs the multiplication result to the fourth selection unit 216. When the input data is greater than or equal to the third preset value (for example, 0), the fourth selection unit 216 directly outputs the input data as the calculation result of the PReLU activation function. When the input data is less than the third preset value (for example, 0), the fourth selection unit 216 outputs the multiplication result as the calculation result of the PReLU activation function. The fourth selection unit 216 outputs the calculation result of the PReLU activation function to the second selection unit 212, and the second selection unit 212 outputs the calculation result of the PReLU activation function.

[0050] It should be noted that the LeakyReLU activation function calculation module and the PReLU activation function calculation module reuse the fourth selection unit 216 and the fifth selection unit 215. When the ReLU activation function includes the LeakyReLU activation function, the fifth selection unit 215 is used to output the first shift result, and when the ReLU activation function includes the PReLU activation function, the fifth selection unit 215 is used to output the multiplication result; when the ReLU activation function includes the LeakyReLU activation function, the fourth selection unit 216 is used to output the input data when the input data is greater than or equal to the third preset value, and when the input data is less than the third preset value, output the first shift result, that is, the product of the input data and the LeakyReLU parameter, and when the ReLU activation function includes the PReLU activation function, the fourth selection unit 216 is used to output the input data when the input data is greater than or equal to the third preset value, and when the input data is less than the third preset value, output the product of the PreLU parameter and the input data.

[0051] In some embodiments, reference Figure 3When the ReLU type activation function includes a SoftShrinkage activation function, the configuration parameters include a first SoftShrinkage parameter and a second SoftShrinkage parameter. The first activation function calculation module 21 includes: a second selection unit 212, a sixth selection unit 218, a seventh selection unit 210, and a first addition unit 219. The first input end of the sixth selection unit 218 is used to receive the first SoftShrinkage parameter, the second end of the sixth selection unit 218 is used to receive the second SoftShrinkage parameter, and the output end of the sixth selection unit 218 is used to receive the second SoftShrinkage parameter. The sixth selection unit 218 is connected to the first input end of the first adding unit 219, the second input end of the first adding unit 219 is used to receive input data, the output end of the first adding unit 219 is connected to the first input of the seventh selecting unit 210, the output end of the seventh selecting unit 210 is connected to the third input end of the second selecting unit 212, wherein the sixth selecting unit 218 is used to output the second SoftShrinkage parameter when the input data is greater than the first SoftShrinkage parameter, and the first adding unit 219 is used to sum the input data and the second SoftShrinkage parameter as the SoftShrinkage parameter. The calculation result of the inkage activation function, the seventh selection unit 210 outputs the calculation result of the SoftShrinkage activation function to the second selection unit 212, and the second selection unit 212 outputs the calculation result of the SoftShrinkage activation function; the sixth selection unit 218 is also used to output the first SoftShrinkage parameter when the input data is less than the second SoftShrinkage parameter, the first addition unit 219 is used to sum the input data and the first SoftShrinkage parameter as the calculation result of the SoftShrinkage activation function, the seventh selection unit 210 outputs the calculation result of the SoftShrinkage activation function to the second selection unit 212, and the second selection unit 212 outputs the calculation result of the SoftShrinkage activation function; the seventh selection unit 210 is also used to use the fourth preset value as the calculation result of the SoftShrinkage activation function when the input data is greater than or equal to the second SoftShrinkage parameter and less than or equal to the first SoftShrinkage parameter, and the second selection unit 212 outputs the calculation result of the SoftShrinkage activation function.

[0052] Specifically, when the ReLU type activation function includes the SoftShrinkage activation function, the SoftShrinkage activation function calculation module includes a second selection unit 212, a sixth selection unit 218, a seventh selection unit 210 and a first addition unit 219, and the SoftShrinkage activation function formula is as follows:

[0053]

[0054] Among them, SoftShrinkage(x) represents the calculation result of the SoftShrinkage activation function, x represents the input data, λ represents the first SoftShrinkage parameter, and -λ represents the second SoftShrinkage parameter.

[0055] For example, refer to Figure 3 When the input data is greater than the first SoftShrinkage parameter (e.g., λ), the sixth selection unit 218 outputs the second SoftShrinkage parameter (e.g., -λ) to the first addition unit 219. The first addition unit 219 sums the input data with the second SoftShrinkage parameter (e.g., -λ) to obtain the calculation result of the SoftShrinkage activation function, and transmits the calculation result of the SoftShrinkage activation function to the seventh selection unit 210. The seventh selection unit 210 outputs the calculation result of the SoftShrinkage activation function to the second selection unit 212, and the second selection unit 212 outputs the calculation result of the SoftShrinkage activation function.

[0056] When the input data is less than the second SoftShrinkage parameter (e.g. -λ), the sixth selection unit 218 outputs the first SoftShrinkage parameter (e.g. λ) to the first addition unit 219, the first addition unit 219 sums the input data with the first SoftShrinkage parameter (e.g. λ) to obtain the calculation result of the SoftShrinkage activation function, and transmits the calculation result of the SoftShrinkage activation function to the seventh selection unit 210, the seventh selection unit 210 outputs the calculation result of the SoftShrinkage activation function to the second selection unit 212, and the second selection unit 212 outputs the calculation result of the SoftShrinkage activation function.

[0057] When the input data is greater than or equal to the second SoftShrinkage parameter (e.g., -λ) and less than or equal to the first SoftShrinkage parameter (e.g., λ), the sixth selection unit 218 will not output the first SoftShrinkage parameter or the second SoftShrinkage parameter to the first addition unit 219, and the first addition unit 219 will only output the input data to the seventh selection unit 210. When the input data is greater than or equal to the second SoftShrinkage parameter (e.g., -λ) and less than or equal to the first SoftShrinkage parameter (e.g., λ), the seventh selection unit 210 uses the fourth preset value (e.g., 0) as the calculation result of the SoftShrinkage activation function, and the second selection unit 212 outputs the calculation result of the SoftShrinkage activation function.

[0058] In some embodiments, reference Figure 4The second activation function calculation module 22 includes: a first multiplication and addition unit 221, a second multiplication unit 222, an eighth selection unit 223, a ninth selection unit 226, a tenth selection unit 227, an eleventh selection unit 224 and a twelfth selection unit 225, the input end of the first multiplication and addition unit 221 is used to receive input data, the output end of the first multiplication and addition unit 221 is respectively connected to the input end of the second multiplication unit 222 and the second input end of the eighth selection unit 223, the output end of the second multiplication unit 222 is connected to the first input end of the eighth selection unit 223, the output end of the eighth selection unit 223 is connected to the first input end of the eleventh selection unit 224 The first input end of the ninth selection unit 226 is connected to the first input end of the tenth selection unit 227, the second input end of the tenth selection unit 227 is used to receive the sixth preset value, the output end of the tenth selection unit 227 is connected to the second input end of the eleventh selection unit 224, and the output end of the eleventh selection unit 224 is connected to the first input end of the twelfth selection unit 225, wherein, in the case where the Hard class activation function includes the Hardswish activation function, the first multiplication and addition unit 221 When the input data is greater than the eighth preset value and less than the ninth preset value, the input data is multiplied and added to obtain a multiplication and addition operation result, the second multiplication unit 222 is used to obtain a second multiplication result according to the product of the multiplication and addition operation result and the input data as the calculation result of the Hardswish activation function, and the eighth selection unit 223 outputs the calculation result of the Hardswish activation function to the eleventh selection unit 224; or, the ninth selection unit 226 is used to use the input data as the calculation result of the Hardswish activation function when the input data is greater than or equal to the eighth preset value, and output the calculation result of the Hardswish activation function to the eleventh selection unit 224. Output to the tenth selection unit 227, the tenth selection unit 227 outputs the calculation result of the Hardswish activation function to the eleventh selection unit 224; or, the tenth selection unit 227 is used to use the sixth preset value as the calculation result of the Hardswish activation function when the input data is less than or equal to the ninth preset value, and output the calculation result of the Hardswish activation function to the eleventh selection unit 224; the eleventh selection unit 224 outputs the calculation result of the Hardswish activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the Hardswish activation function;When the Hard class activation function includes a Hardsigmoid activation function, the first multiplication-addition unit 221 is used to perform a multiplication-addition operation on the input data when the input data is greater than an eighth preset value and less than a ninth preset value, and obtain a multiplication-addition operation result as a calculation result of the Hardsigmoid activation function, and the eighth selection unit 223 outputs the calculation result of the Hardsigmoid activation function to the eleventh selection unit 224; or the ninth selection unit 226 is used to use the fifth preset value as the calculation result of the Hardsigmoid activation function when the input data is greater than or equal to the eighth preset value, and the tenth selection unit 224 outputs the calculation result of the Hardsigmoid activation function to the eleventh selection unit 224; Unit 227 outputs the calculation result of the Hardsigmoid activation function to the eleventh selection unit 224; or, the tenth selection unit 227 is used to use the sixth preset value as the calculation result of the Hardsigmoid activation function when the input data is less than or equal to the ninth preset value, and output the calculation result of the Hardsigmoid activation function to the eleventh selection unit 224; the eleventh selection unit 224 outputs the calculation result of the Hardsigmoid activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the Hardsigmoid activation function. ;

[0059] Specifically, when the Hard class activation function includes a Hardswish activation function, the Hardswish activation function calculation module includes a first multiplication and addition unit 221, a second multiplication unit 222, an eighth selection unit 223, a ninth selection unit 226, a tenth selection unit 227, an eleventh selection unit 224, and a twelfth selection unit 225, and the Hardswish activation function formula is as follows:

[0060]

[0061] Among them, Hardswish(x) represents the calculation result of the Hardswish activation function, and x represents the input data.

[0062] For example, refer to Figure 4When the input data is greater than the eighth preset value (for example, 3) and less than the ninth preset value (for example, -3), the first multiplication-addition unit 221 performs a multiplication-addition operation on the input data. For example, assuming that the input data is x, the input data is multiplied by one-sixth and then added by one-half to obtain a multiplication-addition operation result x / 6+1 / 2. The multiplication-addition operation result is input to the second multiplication unit 222. The second multiplication unit 222 performs a multiplication operation on the input data and the multiplication-addition operation result to obtain a second multiplication result x*(x+3) / 6 as a calculation result of the Hardswish activation function, and transmits the calculation result of the Hardswish activation function to the eighth selection unit 223. The eighth selection unit 223 outputs the calculation result of the Hardswish activation function to the eleventh selection unit 224. The eleventh selection unit 224 outputs the calculation result of the Hardswish activation function to the twelfth selection unit 225. The twelfth selection unit 225 outputs the calculation result of the Hardswish activation function.

[0063] When the input data is greater than or equal to the eighth preset value (for example, 3), the ninth selection unit 226 outputs the input data as the calculation result of the Hardswish activation function to the tenth selection unit 227, the tenth selection unit 227 outputs the calculation result of the Hardswish activation function to the eleventh selection unit 224, the eleventh selection unit 224 outputs the calculation result of the Hardswish activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the Hardswish activation function.

[0064] When the input data is less than or equal to the ninth preset value (for example, -3), the tenth selection unit 227 outputs the sixth preset value (for example, 0) as the calculation result of the Hardswish activation function to the tenth selection unit 227, the tenth selection unit 227 outputs the calculation result of the Hardswish activation function to the eleventh selection unit 224, the eleventh selection unit 224 outputs the calculation result of the Hardswish activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the Hardswish activation function.

[0065] Specifically, when the Hard class activation function includes a Hardsigmoid activation function, the Hardsigmoid activation function calculation module includes a first multiplication and addition unit 221, an eighth selection unit 223, a ninth selection unit 226, a tenth selection unit 227, an eleventh selection unit 224, and a twelfth selection unit 225, and the Hardsigmoid activation function formula is as follows:

[0066]

[0067] Wherein, Hardsigmoid(x) represents the calculation result of the Hardsigmoid activation function, and x represents the input data.

[0068] For example, refer to Figure 4 When the input data is greater than the eighth preset value (for example, 3) and less than the ninth preset value (for example, -3), the first multiplication-addition unit 221 performs a multiplication-addition operation on the input data. For example, assuming that the input data is x, the input data is multiplied by one-sixth and then added by one-half to obtain a multiplication-addition operation result x / 6+1 / 2 as the calculation result of the Hardsigmoid activation function, and the calculation result of the Hardsigmoid activation function is transmitted to the eighth selection unit 223, the eighth selection unit 223 outputs the calculation result of the Hardsigmoid activation function to the eleventh selection unit 224, the eleventh selection unit 224 outputs the calculation result of the Hardsigmoid activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the Hardsigmoid activation function.

[0069] When the input data is greater than or equal to the eighth preset value (for example, 3), the ninth selection unit 226 outputs the fifth preset value (for example, 1) as the calculation result of the Hardsigmoid activation function to the tenth selection unit 227, the tenth selection unit 227 outputs the calculation result of the Hardsigmoid activation function to the eleventh selection unit 224, the eleventh selection unit 224 outputs the calculation result of the Hardsigmoid activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the Hardsigmoid activation function.

[0070] When the input data is less than or equal to the ninth preset value (for example, -3), the tenth selection unit 227 outputs the sixth preset value (for example, 0) as the calculation result of the Hardsigmoid activation function to the tenth selection unit 227, the tenth selection unit 227 outputs the calculation result of the Hardsigmoid activation function to the eleventh selection unit 224, the eleventh selection unit 224 outputs the calculation result of the Hardsigmoid activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the Hardsigmoid activation function.

[0071] It should be noted that the Hardswish activation function calculation module and the Hardsigmoid activation function calculation module reuse the eighth selection unit 223, the ninth selection unit 226, the tenth selection unit 227, the eleventh selection unit 224 and the twelfth selection unit 225. In the case where the Hard class activation function includes the Hardswish activation function, the eighth selection unit 223 outputs the second multiplication result, for example, outputs x*(x+3) / 6, and in the case where the Hard class activation function includes the Hardsigmoid activation function, the eighth selection unit 223 outputs the multiplication and addition operation result, for example, outputs x / 6+1 / 2; in the case where the Hard class activation function includes the Hardswish activation function, the ninth selection unit 226 outputs the input data, and in the case where the Hard class activation function includes the Hardsigmoid activation function, the ninth selection unit 226 outputs the fifth preset value (e.g., 1); in the case where the Hard class activation function includes the Hardswish activation function, the tenth selection unit 227 outputs the input data when the input data is greater than or equal to the eighth preset value (e.g., 3), and outputs the sixth preset value (e.g., 0) when the input data is less than or equal to the ninth preset value (e.g., -3); in the case where the Hard class activation function includes the Hardsigmoid activation function, the tenth selection unit 227 outputs the fifth preset value (e.g., 1) when the input data is greater than or equal to the eighth preset value (e.g., 3), and outputs the sixth preset value (e.g., 0) when the input data is less than or equal to the ninth preset value (e.g., -3); in the Har In the case where the class d activation function includes the Hardswish activation function, the eleventh selection unit 224 outputs the second multiplication result x*(x+3) / 6 as the calculation result of the Hardswish activation function when the input data is greater than the eighth preset value (for example, 3) and less than the ninth preset value (for example, -3); when the input data is greater than or equal to the eighth preset value (for example, 3), the input data is output as the calculation result of the Hardswish activation function; when the input data is less than or equal to the ninth preset value (for example, -3), the sixth preset value (for example, 0) is output as the calculation result of the Hardswish activation function; in the case of the Hard class activation function, the eleventh selection unit 224 outputs the second multiplication result x*(x+3) / 6 as the calculation result of the Hardswish activation function when the input data is greater than or equal to the eighth preset value (for example, 3); When the function includes a Hardsigmoid activation function, the eleventh selection unit 224 outputs the multiplication-addition operation result x / 6+1 / 2 as the calculation result of the Hardsigmoid activation function when the input data is greater than an eighth preset value (for example, 3) and less than a ninth preset value (for example, -3); when the input data is greater than or equal to the eighth preset value (for example, 3), the eleventh selection unit 224 outputs the fifth preset value (for example, 1) as the calculation result of the Hardsigmoid activation function; when the input data is less than or equal to the ninth preset value (for example, -3), the eleventh selection unit 224 outputs the sixth preset value (for example, 0) as the calculation result of the Hardsigmoid activation function.

[0072] In some embodiments, reference Figure 4, the configuration parameters include a first HardTanh parameter and a second HardTanh parameter, the second activation function calculation module 22 includes: a twelfth selection unit 225, a thirteenth selection unit 228, a fourteenth selection unit 229, a fifteenth selection unit 220 and a sixteenth selection unit 2211, the first input end of the thirteenth selection unit 228 is used to receive the first HardTanh parameter, the second input end of the thirteenth selection unit 228 is used to receive the second HardTanh parameter, the output end of the thirteenth selection unit 228 is connected to the second input end of the fourteenth selection unit 229, the first input end of the fourteenth selection unit 229 and the first input end of the fifteenth selection unit 220 are both used to receive the input data According to the present invention, the output end of the fourteenth selection unit 229 is connected to the second input end of the fifteenth selection unit 220, the output end of the fifteenth selection unit 220 is connected to the first input end of the sixteenth selection unit 2211, the second input end of the sixteenth selection unit 2211 is used to receive the seventh preset value, and the output end of the sixteenth selection unit 2211 is connected to the second input end of the twelfth selection unit 225, wherein, when the Hard class activation function includes the HardTanh activation function, the thirteenth selection unit 228 is used to use the first HardTanh parameter as the calculation result of the HardTanh activation function when the input data is greater than the first HardTanh parameter, and the fourteenth selection unit 229 uses the HardTanh parameter as the calculation result of the HardTanh activation function. h activation function is output to the fifteenth selection unit 220; or, the thirteenth selection unit 228 is used to use the second HardTanh parameter as the calculation result of the HardTanh activation function when the input data is less than the second HardTanh parameter, and the fourteenth selection unit 229 outputs the calculation result of the HardTanh activation function to the fifteenth selection unit 220; or, the fourteenth selection unit 229 is used to use the input data as the calculation result of the HardTanh activation function when the input data is greater than or equal to the second HardTanh parameter and less than or equal to the first HardTanh parameter, and output the calculation result of the HardTanh activation function to the fifteenth selection unit 220 ; The fifteenth selection unit 220 is used to output the calculation result of the HardTanh activation function to the sixteenth selection unit 2211, the sixteenth selection unit 2211 outputs the calculation result of the HardTanh activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the HardTanh activation function; when the Hard type activation function includes the HardShrink activation function, the fifteenth selection unit 220 is used to use the input data as the calculation result of the HardShrink activation function when the input data is greater than the tenth preset value or less than the eleventh preset value, and output the calculation result of the HardShrink activation function to the sixteenth selection unit 2211;Alternatively, the sixteenth selection unit 2211 is used to use the seventh preset value as the calculation result of the HardShrink activation function when the input data is greater than or equal to the eleventh preset value and less than or equal to the tenth preset value; the sixteenth selection unit 2211 outputs the calculation result of the HardShrink activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the HardShrink activation function. ;

[0073] Specifically, when the Hard class activation function includes the HardTanh activation function, the HardTanh activation function calculation module includes a thirteenth selection unit 228, a fourteenth selection unit 229, a fifteenth selection unit 220, a sixteenth selection unit 2211 and a twelfth selection unit 225, and the HardTanh activation function formula is as follows:

[0074]

[0075] Among them, HardTanh(x) represents the calculation result of the HardTanh activation function, x represents the input data, max_val represents the first HardTanh parameter, and min_val represents the second HardTanh parameter.

[0076] For example, refer to Figure 4 When the input data is greater than the first HardTanh parameter (e.g., max_val), the thirteenth selection unit 228 outputs the first HardTanh parameter (e.g., max_val) as the calculation result of the HardTanh activation function to the fourteenth selection unit 229, the fourteenth selection unit 229 outputs the calculation result of the HardTanh activation function to the fifteenth selection unit 220, the fifteenth selection unit 220 is used to output the calculation result of the HardTanh activation function to the sixteenth selection unit 2211, the sixteenth selection unit 2211 outputs the calculation result of the HardTanh activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the HardTanh activation function.

[0077] When the input data is less than the second HardTanh parameter (for example, min_val), the thirteenth selection unit 228 outputs the second HardTanh parameter (for example, min_val) as the calculation result of the HardTanh activation function to the fourteenth selection unit 229, the fourteenth selection unit 229 outputs the calculation result of the HardTanh activation function to the fifteenth selection unit 220, the fifteenth selection unit 220 is used to output the calculation result of the HardTanh activation function to the sixteenth selection unit 2211, the sixteenth selection unit 2211 outputs the calculation result of the HardTanh activation function to the twelfth selection unit, and the twelfth selection unit 225 outputs the calculation result of the HardTanh activation function.

[0078] When the input data is greater than or equal to the second HardTanh parameter (for example, min_val) and less than or equal to the first HardTanh parameter (for example, max_val), the fourteenth selection unit 229 outputs the input data as the calculation result of the HardTanh activation function, the fourteenth selection unit 229 outputs the calculation result of the HardTanh activation function to the fifteenth selection unit 220, the fifteenth selection unit 220 is used to output the calculation result of the HardTanh activation function to the sixteenth selection unit 2211, the sixteenth selection unit 2211 outputs the calculation result of the HardTanh activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the HardTanh activation function.

[0079] Specifically, when the Hard class activation function includes the HardShrink activation function, the HardShrink activation function calculation module includes a fifteenth selection unit 220, a sixteenth selection unit 2211, and a twelfth selection unit 225, and the HardShrink activation function formula is as follows:

[0080]

[0081] Wherein, HardShrink(x) represents the calculation result of the HardShrink activation function, x represents input data, δ represents the tenth preset value, and -δ represents the eleventh preset value.

[0082] For example, refer to Figure 4When the input data is greater than the tenth preset value (for example, δ) or less than the eleventh preset value (for example, -δ), the fifteenth selection unit 220 outputs the input data as the calculation result of the HardShrink activation function, and outputs the calculation result of the HardShrink activation function to the sixteenth selection unit 2211, the sixteenth selection unit 2211 outputs the calculation result of the HardShrink activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the HardShrink activation function.

[0083] When the input data is greater than or equal to the eleventh preset value (for example, -δ) and less than or equal to the tenth preset value (for example, δ), the sixteenth selection unit 2211 outputs the seventh preset value (for example, 0) as the calculation result of the HardShrink activation function, and the sixteenth selection unit 2211 outputs the calculation result of the HardShrink activation function to the twelfth selection unit 225, and the twelfth selection unit 225 outputs the calculation result of the HardShrink activation function.

[0084] It should be noted that the HardTanh activation function calculation module and the HardShrink activation function calculation module reuse the fifteenth selection unit 220 and the sixteenth selection unit 2211. In the case where the Hard class activation function includes the HardTanh activation function, when the input data is greater than the first HardTanh parameter, the fifteenth selection unit 220 outputs the first HardTanh parameter (e.g., max_val) as the calculation result of the HardTanh activation function, when the input data is less than the second HardTanh parameter (e.g., min_val), the fifteenth selection unit 220 outputs the second HardTanh parameter as the calculation result of the HardTanh activation function, when the input data is greater than or equal to the second HardTanh parameter (e.g., min_val) and less than or equal to the first HardTanh parameter (e.g., max_val), the fifteenth selection unit 220 outputs the input data as the calculation result of the HardTanh activation function; in the case where the Hard class activation function includes the HardShrink activation function, the fifteenth selection unit 220 outputs the input data.

[0085] When the Hard class activation function includes the HardTanh activation function, the sixteenth selection unit 2211 outputs the calculation result of the HardTanh activation function; when the Hard class activation function includes the HardShrink activation function, when the input data is greater than the tenth preset value (for example, δ) or less than the eleventh preset value (for example, -δ), the sixteenth selection unit 2211 outputs the input data as the calculation result of the HardShrink activation function; when the input data is greater than or equal to the eleventh preset value (for example, -δ) and less than or equal to the tenth preset value (for example, δ), the sixteenth selection unit 2211 outputs the seventh preset value (for example, 0) as the calculation result of the HardShrink activation function.

[0086] In some embodiments, reference Figure 5The third activation function calculation module 23 includes: a third multiplication unit 231, a second addition unit 232, a second shift register unit 233, an arithmetic logic unit 234, a seventeenth selection unit 235 and an eighteenth selection unit 236, the input end of the third multiplication unit 231 is used to receive input data, the output end of the third multiplication unit 231 is respectively connected to the input end of the second addition unit 232 and the first input end of the seventeenth selection unit 235, the output end of the second addition unit 232 is connected to the input end of the second shift register unit 233, the output end of the second shift register unit 233 is connected to the input end of the arithmetic logic unit 234, and the output end of the arithmetic logic unit 234 is connected to the first input end of the eighteenth selection unit 236. The second input end of the seventeenth selection unit 235 is used to receive input data, and the output end of the seventeenth selection unit 235 is connected to the second input end of the eighteenth selection unit 236, wherein, when the exponential activation function includes a Tanh activation function, the third multiplication unit 231 performs a product operation on the input data and the twelfth preset value to obtain a calculation result of the Tanh activation function, and the seventeenth selection unit 235 outputs the calculation result of the Tanh activation function to the eighteenth selection unit 236, and the eighteenth selection unit 236 outputs the calculation result of the Tanh activation function; when the exponential activation function includes a Sigmoid activation function, the third multiplication unit 231 performs a product operation on the input data and the thirteenth preset value to obtain a multiplication The second adding unit 232 performs a sum operation on the multiplication result and the fourteenth preset value to obtain a sum result, the second shift register unit 233 performs a shift operation on the sum result to obtain a second shift result, the arithmetic logic unit 234 performs a first logic operation on the second shift result to obtain a calculation result of the Sigmoid activation function, and the eighteenth selecting unit 236 is used to output the calculation result of the Sigmoid activation function, wherein the first logic operation is to take the reciprocal of the second shift result and sum it with the fifteenth preset value and then take the reciprocal; when the exponential activation function includes the silu activation function, the arithmetic logic unit 234 performs a second logic operation on the second shift result to obtain the calculation result of the silu activation function, and the eighteenth selecting unit 236 is used to output the calculation result of the Sigmoid activation function. The selection unit 236 is used to output the calculation result of the silu activation function, wherein the second logic operation is the product of the result obtained by taking the inverse of the second shift result and summing it with the fifteenth preset value and then taking the inverse and the input data; when the exponential activation function includes the ELU activation function, the eighteenth selection unit 236 is used to use the input data as the calculation result of the ELU activation function when the input data is greater than the sixteenth preset value; or, the eighteenth selection unit 236 is used to perform a third logic operation on the second shift result by the arithmetic logic unit 234 when the input data is less than or equal to the sixteenth preset value to obtain the calculation result of the ELU activation function, wherein the third logic operation is the difference between the second shift result and the seventeenth preset value multiplied by the eighteenth preset value;The eighteenth selection unit 236 is used to output the calculation result of the ELU activation function.

[0087] Specifically, the third activation function calculation module 23 is used to calculate an exponential activation function, which includes one of a Sigmoid activation function, a Tanh activation function, a silu activation function, and an ELU activation function.

[0088] The Sigmoid activation function formula is as follows:

[0089]

[0090] Among them, Sigmoid(x) represents the calculation result of the Sigmoid activation function, and x represents the input data.

[0091] The Tanh activation function formula is as follows:

[0092]

[0093] Among them, Tanh(x) represents the calculation result of the Tanh activation function, and x represents the input data.

[0094] The silu activation function formula is as follows:

[0095]

[0096] Among them, silu(x) represents the calculation result of the silu activation function, and x represents the input data.

[0097] The ELU activation function formula is as follows:

[0098]

[0099] Among them, ELU(x) represents the calculation result of the ELU activation function, x represents the input data, and β represents the ELU parameter.

[0100] That is to say, the Sigmoid activation function, Tanh activation function, silu activation function and ELU activation function all contain e-exponent calculation. According to the fitting results, the e-exponent calculation can be simplified to multiplication and addition calculation within a certain range. For example, the fitting formula is as follows:

[0101] exp(x)=2 23 *(k*x+b)

[0102] Wherein, exp(x) represents the exponential calculation result, x represents the input data, k represents the thirteenth preset value, and b represents the fourteenth preset value.

[0103] Reference Figure 5The e index calculation module may include a third multiplication unit 231, a second addition unit 232, and a second shift register unit 233. The third multiplication unit 231 performs a product operation on the input data and a thirteenth preset value (for example, k) to obtain a product result kx, and outputs it to the second addition unit 232. The second addition unit 232 performs a sum operation on the product result kx and a fourteenth preset value (for example, b) to obtain a sum result kx+b, and outputs it to the second shift register unit 233. The second shift register unit 233 performs a shift operation on the sum result, for example, shifting 23 bits to the left to obtain a second shift result 2 23 *(k*x+b).

[0104] Furthermore, the Tanh activation function can be simplified according to the fitting formula, and the simplification process is as follows:

[0105]

[0106] Among them, Tanh(x) represents the calculation result of Tanh activation function, x represents input data, Indicates the twelfth preset value.

[0107] For example, refer to Figure 5 In the case where the exponential activation function includes a Tanh activation function, the Tanh activation function calculation module includes a third multiplication unit 231, a seventeenth selection unit 235, and an eighteenth selection unit 236. The third multiplication unit 231 performs multiplication on the input data and the twelfth preset value Perform the product operation to get As the calculation result of the Tanh activation function, the calculation result of the Tanh activation function is output to the seventeenth selection unit 235, the seventeenth selection unit 235 outputs the calculation result of the Tanh activation function to the eighteenth selection unit 236, and the eighteenth selection unit 236 outputs the calculation result of the Tanh activation function.

[0108] Continue to refer to Figure 5 , when the exponential activation function includes the Sigmoid activation function, the Sigmoid activation function calculation module includes a third multiplication unit 231, a second addition unit 232, a second shift register unit 233, an arithmetic logic unit 234 and an eighteenth selection unit 236. The third multiplication unit 231 performs a product operation on the input data and the thirteenth preset value (for example, k) to obtain a product result kx, and outputs it to the second addition unit 232. The second addition unit 232 performs a sum operation on the product result kx and the fourteenth preset value (for example, b) to obtain a sum result kx+b, and outputs it to the second shift register unit 233. The second shift register unit 233 performs a shift operation on the sum result, for example, shifting 23 bits to the left to obtain a second shift result exp(x)=2 23*(k*x+b), and output the second shift result to the arithmetic logic unit 234, the arithmetic logic unit 234 performs a first logic operation on the second shift result, that is, the second shift result is taken the inverse and summed with the fifteenth preset value (for example, 1) and then the inverse is taken to obtain the calculation result of the Sigmoid activation function And transmitted to the eighteenth selection unit 236, and the eighteenth selection unit 236 outputs the calculation result of the Sigmoid activation function.

[0109] Continue to refer to Figure 5 In the case where the exponential activation function includes the silu activation function, the silu activation function calculation module also includes a third multiplication unit 231, a second addition unit 232, a second shift register unit 233, an arithmetic logic unit 234 and an eighteenth selection unit 236. After the input data passes through the third multiplication unit 231, the second addition unit 232 and the second shift register unit 233, the second shift result exp(x)=2 is obtained. 23 *(k*x+b), and input the second shift result into the arithmetic logic unit 234 for a second logic operation, that is, the second shift result is reciprocally summed with the fifteenth preset value (e.g., 1), and then the reciprocal result is multiplied with the input data to obtain the calculation result of the silu activation function. And transmitted to the eighteenth selection unit 236, the eighteenth selection unit 236 outputs the calculation result of the silu activation function.

[0110] Continue to refer to Figure 5 , when the exponential activation function includes the ELU activation function, the ELU activation function calculation module includes a third multiplication unit 231, a second addition unit 232, a second shift register unit 233, an arithmetic logic unit 234, a seventeenth selection unit 235, and an eighteenth selection unit 236. After the input data passes through the third multiplication unit 231, the second addition unit 232, and the second shift register unit 233, the second shift result exp(x)=2 is obtained. 23 *(k*x+b), and input the second shift result into the arithmetic logic unit 234 for a third logic operation, that is, the difference between the second shift result and the seventeenth preset value (e.g., 1) is multiplied by the eighteenth preset value (e.g., β) to obtain β*(exp(x)-1), and the value is output to the eighteenth selection unit 236. When the input data is greater than the sixteenth preset value (e.g., 0), the eighteenth selection unit 236 outputs the input data as the calculation result of the ELU activation function, and when the input data is less than or equal to the sixteenth preset value (e.g., 0), the eighteenth selection unit 236 outputs β*(exp(x)-1) as the calculation result of the ELU activation function.

[0111] In addition, the third activation function calculation module 23 can also be implemented by a lookup table.

[0112] In this way, the circuit of the present application supports fast exponential calculation and dynamically adjusts the processing method of exponential calculation according to the actual scenario needs. Scenarios with low precision requirements can use fast calculation, and scenarios with high precision requirements can switch to processor calculation.

[0113] In summary, the circuit of the present application can improve the computing efficiency of the activation function operator in the mainstream neural network at a relatively low hardware cost, and meet the real-time requirements of the edge-side MCU (Microcontroller Unit) chip for the neural network algorithm. In this way, the circuit of the present application can reduce the computing time, thereby speeding up data processing and enabling edge devices to make decisions and respond faster; and compared with traditional CPU or GPU execution methods, the circuit of the present application usually has lower energy consumption when performing specific tasks, such as activation function calculation, which can extend the battery life of the device; at the same time, as part of edge computing, the circuit of the present application helps to complete more computing tasks locally, and the data is also processed and analyzed locally, reducing the risk of uploading data to the cloud and further enhancing data privacy protection; in addition, the circuit of the present application can support a variety of different mainstream activation functions to adapt to different AI application scenarios and neural network algorithms, such as supporting CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory, long short-term memory network) and Transformer (a deep learning model architecture widely used in fields such as natural language processing), etc., and can accelerate the computational efficiency of the Softmax probability function. In edge computing, it has significant advantages such as improving computing efficiency, reducing energy consumption, and enhancing real-time performance. It also has beneficial effects on improving user experience and promoting the popularization of AI. This enables edge computing to be more widely used in various fields, such as smart homes, fault detection, and smart security.

[0114] Corresponding to the above embodiment, the present application also proposes an electronic device.

[0115] See also Figure 6 As shown, the electronic device 600 of the present application includes the aforementioned circuit 100 that can be configured with multiple activation functions.

[0116] It should be pointed out that the above-mentioned explanation of the embodiments and beneficial effects of the circuit that can be configured with multiple activation functions is also applicable to the electronic device of the embodiment of the present application. In order to avoid redundancy, it will not be elaborated here.

[0117] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0118] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0119] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0120] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0121] In this application, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0122] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A circuit capable of configuring multiple activation functions, characterized in that: The circuit comprises: a configuration module, a plurality of activation function calculation modules and a selection module, wherein: The activation function calculation module is used to perform calculations based on input data and configuration parameters to generate calculation results; The configuration module is used to determine the target activation function calculation module based on the configuration file when it is determined according to the input data that the activation function calculation module needs to be used, and configure the selection module and the target activation function calculation module based on the configuration file; The selection module is used to output the calculation result of the target activation function calculation module.

2. The circuit capable of configuring multiple activation functions according to claim 1, characterized in that: The selection module is also used to use the input data as output data when it is determined based on the input data that no activation function calculation is required.

3. The circuit capable of configuring multiple activation functions according to claim 1, characterized in that: The plurality of activation function calculation modules include: a first activation function calculation module, a second activation function calculation module and a third activation function calculation module, wherein: The first activation function calculation module is used to calculate a ReLU type activation function, wherein the ReLU type activation function includes one of a ReLU activation function, a ReLU6 activation function, a LeakyReLU activation function, a PReLU activation function, and a SoftShrinkage activation function; The second activation function calculation module is used to calculate a Hard class activation function, wherein the Hard class activation function includes one of a Hardswish activation function, a Hardsigmoid activation function, a HardTanh activation function, and a HardShrink activation function; The third activation function calculation module is used to calculate an exponential activation function, and the exponential activation function includes one of a Sigmoid activation function, a Tanh activation function, a silu activation function, and an ELU activation function.

4. The circuit capable of configuring multiple activation functions according to claim 3, characterized in that: The first activation function calculation module includes: a first selection unit, a second selection unit and a third selection unit, wherein the first input end of the first selection unit is used to receive the input data, the first input end of the third selection unit is connected to the output end of the first selection unit, and the output end of the third selection unit is connected to the first input end of the second selection unit, wherein: In the case where the ReLU type activation function includes a ReLU activation function, the first selection unit is used to use the input data as a calculation result of the ReLU activation function when the input data is greater than or equal to a first preset value, or use the first preset value as a calculation result of the ReLU activation function when the input data is less than the first preset value, and the third selection unit outputs the calculation result of the ReLU activation function to the second selection unit, and the second selection unit outputs the calculation result of the ReLU activation function; In the case that the ReLU type activation function includes the ReLU6 activation function, the third selection unit is used to use the calculation result of the ReLU activation function as the calculation result of the ReLU6 activation function when the calculation result of the ReLU activation function is less than a second preset value, or to use the second preset value as the calculation result of the ReLU6 activation function when the calculation result of the ReLU activation function is greater than or equal to the second preset value, and the second selection unit is used to output the calculation result of the ReLU activation function.

5. The circuit capable of configuring multiple activation functions according to claim 3, characterized in that: The configuration parameters include LeakyReLU parameters or PreLU parameters, and the first activation function calculation module includes: a first shift register unit, a first multiplication unit, a second selection unit, a fourth selection unit and a fifth selection unit, the input ends of the first shift register unit and the first multiplication unit are both used to receive the input data, the output end of the first shift register unit is connected to the first input end of the fifth selection unit, the output end of the first multiplication unit is connected to the second input end of the fifth selection unit, the output end of the fifth selection unit is connected to the first input end of the fourth selection unit, the second input end of the fourth selection unit is used to receive the input data, and the output end of the fourth selection unit is connected to the second input end of the second selection unit, wherein, In the case where the ReLU type activation function includes a LeakyReLU activation function, the first shift register unit is used to perform a shift process on the input data to obtain a first shift result, where the first shift result is the product of the input data and the LeakyReLU parameter, and the fifth selection unit outputs the first shift result to the fourth selection unit, and the fourth selection unit is used to use the input data as a calculation result of the LeakyReLU activation function when the input data is greater than or equal to a third preset value, or, when the input data is less than the third preset value, use the first shift result as a calculation result of the LeakyReLU activation function, and the second selection unit is used to output the calculation result of the LeakyReLU activation function; In the case where the ReLU type activation function includes a PReLU activation function, the first multiplication unit is used to obtain a multiplication result based on the product of the input data and the PreLU parameter, the fifth selection unit outputs the multiplication result to the fourth selection unit, the fourth selection unit is used to use the input data as the calculation result of the PReLU activation function when the input data is greater than or equal to a third preset value, or, when the input data is less than the third preset value, use the multiplication result as the calculation result of the PReLU activation function, and the second selection unit is used to output the calculation result of the PReLU activation function.

6. The circuit capable of configuring multiple activation functions according to claim 3, characterized in that: When the ReLU type activation function includes a SoftShrinkage activation function, the configuration parameters include a first SoftShrinkage parameter and a second SoftShrinkage parameter, and the first activation function calculation module includes: a second selection unit, a sixth selection unit, a seventh selection unit and a first addition unit, the first input end of the sixth selection unit is used to receive the first SoftShrinkage parameter, the second end of the sixth selection unit is used to receive the second SoftShrinkage parameter, the output end of the sixth selection unit is connected to the first input end of the first addition unit, the second input end of the first addition unit is used to receive the input data, the output end of the first addition unit is connected to the first input end of the seventh selection unit, and the output end of the seventh selection unit is connected to the third input end of the second selection unit, wherein, The sixth selection unit is used to output the second SoftShrinkage parameter when the input data is greater than the first SoftShrinkage parameter, the first addition unit is used to sum the input data and the second SoftShrinkage parameter as the calculation result of the SoftShrinkage activation function, the seventh selection unit outputs the calculation result of the SoftShrinkage activation function to the second selection unit, and the second selection unit outputs the calculation result of the SoftShrinkage activation function; The sixth selection unit is further used for outputting the first SoftShrinkage parameter when the input data is less than the second SoftShrinkage parameter, the first addition unit is used for summing the input data and the first SoftShrinkage parameter as the calculation result of the SoftShrinkage activation function, the seventh selection unit outputs the calculation result of the SoftShrinkage activation function to the second selection unit, and the second selection unit outputs the calculation result of the SoftShrinkage activation function; The seventh selection unit is also used to use the fourth preset value as the calculation result of the SoftShrinkage activation function when the input data is greater than or equal to the second SoftShrinkage parameter and less than or equal to the first SoftShrinkage parameter, and the second selection unit outputs the calculation result of the oftShrinkage activation function.

7. The circuit capable of configuring multiple activation functions according to claim 3, characterized in that: The second activation function calculation module includes: a first multiplication and addition unit, a second multiplication unit, an eighth selection unit, a ninth selection unit, a tenth selection unit, an eleventh selection unit and a twelfth selection unit, the input end of the first multiplication and addition unit is used to receive the input data, the output end of the first multiplication and addition unit is respectively connected to the input end of the second multiplication unit and the second input end of the eighth selection unit, the output end of the second multiplication unit is connected to the first input end of the eighth selection unit, the output end of the eighth selection unit is connected to the first input end of the eleventh selection unit, the first input end of the ninth selection unit is used to receive the input data, the second input end of the ninth selection unit is used to receive the fifth preset value, the output end of the ninth selection unit is connected to the first input end of the tenth selection unit, the second input end of the tenth selection unit is used to receive the sixth preset value, the output end of the tenth selection unit is connected to the second input end of the eleventh selection unit, and the output end of the eleventh selection unit is connected to the first input end of the twelfth selection unit, wherein, In the case where the Hard class activation function includes a Hardswish activation function, the first multiplication-addition unit is used to perform a multiplication-addition operation on the input data to obtain a multiplication-addition operation result when the input data is greater than an eighth preset value and less than a ninth preset value, and the second multiplication unit is used to obtain a second multiplication result according to the product of the multiplication-addition operation result and the input data as a calculation result of the Hardswish activation function, and the eighth selection unit outputs the calculation result of the Hardswish activation function to the eleventh selection unit; or, the ninth selection unit is used to use the input data as a calculation result of the Hardswish activation function when the input data is greater than or equal to the eighth preset value. The calculation result of the Hardswish activation function is output to the tenth selection unit, and the tenth selection unit outputs the calculation result of the Hardswish activation function to the eleventh selection unit; or, the tenth selection unit is used to use the sixth preset value as the calculation result of the Hardswish activation function when the input data is less than or equal to the ninth preset value, and output the calculation result of the Hardswish activation function to the eleventh selection unit; the eleventh selection unit outputs the calculation result of the Hardswish activation function to the twelfth selection unit, and the twelfth selection unit outputs the calculation result of the Hardswish activation function; The Hard class activation function includes a Hardsigmoid activation function, the first multiplication-addition unit is used to perform a multiplication-addition operation on the input data when the input data is greater than an eighth preset value and less than a ninth preset value, to obtain a multiplication-addition operation result as a calculation result of the Hardsigmoid activation function, and the eighth selection unit outputs the calculation result of the Hardsigmoid activation function to the eleventh selection unit; or, the ninth selection unit is used to use the fifth preset value as the calculation result of the Hardsigmoid activation function when the input data is greater than or equal to the eighth preset value, The tenth selection unit outputs the calculation result of the Hardsigmoid activation function to the eleventh selection unit; or, the tenth selection unit is used to use the sixth preset value as the calculation result of the Hardsigmoid activation function when the input data is less than or equal to the ninth preset value, and output the calculation result of the Hardsigmoid activation function to the eleventh selection unit; the eleventh selection unit outputs the calculation result of the Hardsigmoid activation function to the twelfth selection unit, and the twelfth selection unit outputs the calculation result of the Hardsigmoid activation function.

8. The circuit capable of configuring multiple activation functions according to claim 3, characterized in that: The configuration parameters include a first HardTanh parameter and a second HardTanh parameter, and the second activation function calculation module includes: a twelfth selection unit, a thirteenth selection unit, a fourteenth selection unit, a fifteenth selection unit and a sixteenth selection unit, the first input end of the thirteenth selection unit is used to receive the first HardTanh parameter, the second input end of the thirteenth selection unit is used to receive the second HardTanh parameter, the output end of the thirteenth selection unit is connected to the second input end of the fourteenth selection unit, the first input end of the fourteenth selection unit and the first input end of the fifteenth selection unit are both used to receive the input data, the output end of the fourteenth selection unit is connected to the second input end of the fifteenth selection unit, the output end of the fifteenth selection unit is connected to the first input end of the sixteenth selection unit, the second input end of the sixteenth selection unit is used to receive the seventh preset value, and the output end of the sixteenth selection unit is connected to the second input end of the twelfth selection unit, wherein, When the Hard class activation function includes a HardTanh activation function, the thirteenth selection unit is used to use the first HardTanh parameter as the calculation result of the HardTanh activation function when the input data is greater than the first HardTanh parameter, and the fourteenth selection unit outputs the calculation result of the HardTanh activation function to the fifteenth selection unit; or, the thirteenth selection unit is used to use the second HardTanh parameter as the calculation result of the HardTanh activation function when the input data is less than the second HardTanh parameter, and the fourteenth selection unit outputs the calculation result of the HardTanh activation function to the fifteenth selection unit. The fifteenth selection unit; or, the fourteenth selection unit is used to use the input data as the calculation result of the HardTanh activation function when the input data is greater than or equal to the second HardTanh parameter and less than or equal to the first HardTanh parameter, and output the calculation result of the HardTanh activation function to the fifteenth selection unit; the fifteenth selection unit is used to output the calculation result of the HardTanh activation function to the sixteenth selection unit, and the sixteenth selection unit outputs the calculation result of the HardTanh activation function to the twelfth selection unit, and the twelfth selection unit outputs the calculation result of the HardTanh activation function; When the Hard class activation function includes a HardShrink activation function, the fifteenth selection unit is used to use the input data as the calculation result of the HardShrink activation function when the input data is greater than a tenth preset value or less than an eleventh preset value, and output the calculation result of the HardShrink activation function to the sixteenth selection unit; or, the sixteenth selection unit is used to use the seventh preset value as the calculation result of the HardShrink activation function when the input data is greater than or equal to the eleventh preset value and less than or equal to the tenth preset value; the sixteenth selection unit outputs the calculation result of the HardShrink activation function to the twelfth selection unit, and the twelfth selection unit outputs the calculation result of the HardShrink activation function.

9. The circuit capable of configuring multiple activation functions according to claim 3, characterized in that: The third activation function calculation module includes: a third multiplication unit, a second addition unit, a second shift register unit, an arithmetic logic unit, a seventeenth selection unit and an eighteenth selection unit, the input end of the third multiplication unit is used to receive the input data, the output end of the third multiplication unit is respectively connected to the input end of the second addition unit and the first input end of the seventeenth selection unit, the output end of the second addition unit is connected to the input end of the second shift register unit, the output end of the second shift register unit is connected to the input end of the arithmetic logic unit, the output end of the arithmetic logic unit is connected to the first input end of the eighteenth selection unit, the second input end of the seventeenth selection unit is used to receive the input data, and the output end of the seventeenth selection unit is connected to the second input end of the eighteenth selection unit, wherein, When the exponential activation function includes a Tanh activation function, the third multiplication unit performs a product operation on the input data and a twelfth preset value to obtain a calculation result of the Tanh activation function, the seventeenth selection unit outputs the calculation result of the Tanh activation function to the eighteenth selection unit, and the eighteenth selection unit outputs the calculation result of the Tanh activation function; When the exponential activation function includes a Sigmoid activation function, the third multiplication unit performs a multiplication operation on the input data and a thirteenth preset value to obtain a product result, the second addition unit performs a summation operation on the product result and a fourteenth preset value to obtain a summation result, the second shift register unit performs a shift operation on the summation result to obtain a second shift result, the arithmetic logic unit performs a first logic operation on the second shift result to obtain a calculation result of the Sigmoid activation function, and the eighteenth selection unit is used to output the calculation result of the Sigmoid activation function, wherein the first logic operation is to take the inverse of the second shift result, sum it with the fifteenth preset value, and then take the inverse; When the exponential activation function includes a silu activation function, the arithmetic logic unit performs a second logic operation on the second shift result to obtain a calculation result of the silu activation function, and the eighteenth selection unit is used to output the calculation result of the silu activation function, wherein the second logic operation is the product of the result obtained by taking the inverse of the second shift result and summing it with the fifteenth preset value and then taking the inverse, and the input data; When the exponential activation function includes the ELU activation function, the eighteenth selection unit is used to use the input data as the calculation result of the ELU activation function when the input data is greater than the sixteenth preset value; or, the eighteenth selection unit is used to, when the input data is less than or equal to the sixteenth preset value, the arithmetic logic unit performs a third logical operation on the second shift result to obtain the calculation result of the ELU activation function, wherein the third logical operation is the difference between the second shift result and the seventeenth preset value multiplied by the eighteenth preset value; the eighteenth selection unit is used to output the calculation result of the ELU activation function.

10. An electronic device, characterized in that: A circuit comprising a plurality of configurable activation functions as claimed in any one of claims 1 to 9.

Images