Lookup Table Generation Method, Device, Equipment and Storage Medium for Activation Function

Abstract

The present invention discloses a method, apparatus, device and storage medium for generating a lookup table of an activation function. The method includes: equally dividing the value range of the activation function according to a target deviation value to obtain a set of sampling points corresponding to the activation function; dividing the set of sampling points into multiple intervals; equally dividing the input data of the activation function in each interval according to a corresponding target spacing to obtain multiple sub-sets of sampling points; when the total storage space required for the multiple sub-sets of sampling points is less than or equal to a preset storage space, generating a lookup table of the activation function according to the multiple sub-sets of sampling points and storing the lookup table; when the total storage space is greater than the preset storage space, increasing the target deviation value and returning to execute the step of equally dividing the value range of the activation function according to the target deviation value to obtain a set of sampling points corresponding to the activation function. The present invention not only improves the accuracy of the data in the lookup table of the activation function, but also reduces the space storage of the lookup table.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a method, apparatus, device, and storage medium for generating a lookup table of an activation function. Background Art

[0002] An activation function is a function that operates on neurons in an artificial neural network and is responsible for mapping the input of a neuron to the output end. Usually, when generating a lookup table of an activation function, the value range of the input data of the activation function is evenly divided into 2 N (2 to the power of N) intervals, and then the calculation results of the activation function corresponding to each interval are stored in the RAM. When there is input data, according to which interval the input data falls into, the value corresponding to this interval is read from the RAM as the final calculation result.

[0003] Currently, in order to enable a chip (such as an artificial intelligence acceleration chip, an FPGA product, etc.) to store the lookup table data of the activation function, when generating the lookup table of the activation function, the value range of the input data of the activation function is usually evenly divided into fewer intervals to reduce the storage space required for storing the lookup table. Since activation functions are all non-linear functions, the value ranges of some intervals change rapidly, while those of some intervals change slowly. Although the storage space required for storing the lookup table is reduced, it will cause an increase in the error of the output result of the activation function and reduce the accuracy of the output result. Summary of the Invention

[0004] Embodiments of the present invention provide a method, apparatus, device, and storage medium for generating a lookup table of an activation function, aiming to solve the technical problem that when generating the lookup table of the activation function, evenly dividing the value range of the input data of the activation function into fewer intervals results in an increase in the error of the output result of the activation function.

[0005] Embodiments of the present invention provide a method for generating a lookup table of an activation function, where the method for generating the lookup table of the activation function includes:

[0006] Evenly divide the value range of the activation function according to a target deviation value to obtain a set of sampling points corresponding to the activation function;

[0007] Divide the set of sampling points into at least two intervals, where the line segment slopes of the activation functions corresponding to different intervals are different;

[0008] Evenly divide the input data of the activation function in each interval according to a corresponding target spacing to obtain multiple sub-sampling point sets, where the target spacing corresponding to the interval is inversely correlated with the line segment slope of the corresponding activation function;

[0009] When the first total storage space required for the multiple sets of subsampling points is less than or equal to the preset storage space, a lookup table of the input values and output values of the activation function is generated according to the multiple sets of subsampling points, and the lookup table is stored;

[0010] When the first total storage space required for the multiple sets of subsampling points is greater than the preset storage space, increase the target deviation value, and return to execute the step of equally dividing the value range of the activation function according to the target deviation value to obtain the set of sampling points corresponding to the activation function.

[0011] In one embodiment, the step of dividing the set of sampling points into at least two intervals includes:

[0012] Subtract the input data corresponding to two adjacent sampling points in the set of sampling points to obtain the distance between two adjacent sampling points in the set of sampling points;

[0013] Divide the sampling points with the distances within the same distance range into one interval to obtain at least two such intervals.

[0014] In one embodiment, the method for generating the lookup table of the activation function further includes:

[0015] Obtain the minimum distance between two adjacent sampling points in the set of sampling points;

[0016] Determine the target distance corresponding to each interval according to the minimum distance.

[0017] In one embodiment, the step of determining the target distance corresponding to each interval according to the minimum distance includes:

[0018] Determine the target distance corresponding to each interval according to the minimum distance and the line segment slope of the activation function corresponding to each interval;

[0019] Wherein, the ratio of the target distance corresponding to the interval with a larger line segment slope of the activation function to the target distance corresponding to the interval with a smaller line segment slope of the activation function is N, where N is greater than or equal to 2, and the larger the line segment slope, the smaller N is.

[0020] In one embodiment, the step of increasing the target deviation value includes:

[0021] Obtain the space difference between the first total storage space and the preset storage space;

[0022] When the space difference is greater than the preset threshold, increase the target deviation value by a first preset increment value;

[0023] When the spatial difference is less than or equal to the preset threshold, the target deviation value is increased by a second preset increment value, and the first preset increment value is greater than the second preset increment value.

[0024] In one embodiment, after the step of equally dividing the value range of the activation function according to the target deviation value to obtain the set of sampling points corresponding to the activation function, the method further includes:

[0025] Determine the second total storage space required for the sampling points in the set of sampling points;

[0026] When the second total storage space is less than or equal to the preset storage space, execute the step of dividing the set of sampling points into at least two intervals.

[0027] In one embodiment, after the step of determining the second total storage space of the sampling points in the set of sampling points, the method further includes:

[0028] When the second total storage space is greater than the preset storage space, increase the target deviation value, and return to execute the step of equally dividing the value range of the activation function according to the target deviation value to obtain the set of sampling points corresponding to the activation function.

[0029] In addition, to achieve the above object, the present invention further provides a lookup table generation device for an activation function, where the lookup table generation device for the activation function includes:

[0030] A value range division module, configured to equally divide the value range of the activation function according to the target deviation value to obtain the set of sampling points corresponding to the activation function;

[0031] An interval generation module, configured to divide the set of sampling points into at least two intervals, where the line segment slopes of the activation functions corresponding to different intervals are different;

[0032] A set generation module, configured to equally divide the input data of the activation function in each interval according to the corresponding target spacing to obtain a plurality of sub-sampling point sets, and the target spacing corresponding to the interval is inversely related to the line segment slope of the corresponding activation function;

[0033] A first storage module, configured to generate a lookup table of the input values and output values of the activation function according to the plurality of sub-sampling point sets and store the lookup table when the first total storage space required for the plurality of sub-sampling point sets is less than or equal to the preset storage space;

[0034] A second storage module, configured to increase the target deviation value when a first total storage space required for a plurality of the subsampling point sets is greater than a preset storage space, and return to execute the step of equally dividing the value range of the activation function according to the target deviation value to obtain the sampling point set corresponding to the activation function.

[0035] In addition, to achieve the above object, the present invention further provides a terminal device including: a memory, a processor, and a lookup table generation program of an activation function stored on the memory and executable on the processor. When the lookup table generation program of the activation function is executed by the processor, the steps of the above-mentioned activation function lookup table generation method are implemented.

[0036] In addition, to achieve the above object, the present invention further provides a storage medium, on which a lookup table generation program of an activation function is stored. When the lookup table generation program of the activation function is executed by a processor, the steps of the above-mentioned activation function lookup table generation method are implemented.

[0037] The technical solutions of a method, device, equipment, and storage medium for generating a lookup table of an activation function provided in the embodiments of the present invention at least have the following technical effects or advantages:

[0038] The present invention equally divides the value range of the activation function according to the target deviation value to obtain a sampling point set corresponding to the activation function, and then divides the sampling point set into multiple intervals. Based on the characteristics of the activation function, if the line segment slope of the activation function is large, the interval corresponding to the large line segment slope is finely divided, so that the interval corresponding to the large line segment slope includes more sampling points. If the line segment slope of the activation function is small, the interval corresponding to the small line segment slope is roughly divided, thereby obtaining a plurality of subsampling point sets. If the total storage space required for a plurality of subsampling point sets is less than or equal to the preset storage space, a lookup table of the input value and output value of the activation function is generated according to the plurality of subsampling point sets, and the lookup table is stored; if the first total storage space required for a plurality of subsampling point sets is greater than the preset storage space, the target deviation value is increased to reduce the total storage space required for the generated plurality of subsampling point sets, and then when the total storage space required for a plurality of subsampling point sets is less than or equal to the preset storage space, a lookup table of the input value and output value of the activation function is generated according to the plurality of subsampling point sets, and the lookup table is stored. In this way, after generating the lookup table of the activation function, the accuracy of the data in the lookup table can be improved, and the space storage of the lookup table can be reduced, which is beneficial to saving the storage resources of the chip and improving the speed of the chip to search for data through the lookup table, and solves the technical problem that when generating the lookup table of the activation function, the value range of the input data of the activation function is evenly divided into fewer intervals, resulting in an increase in the error of the output result of the activation function. Description of the Drawings

[0039] Figure 1 The structural schematic diagram of the hardware operating environment involved in the solution of the embodiment of the present invention;

[0040] Figure 2 The flowchart of the first embodiment of the method for generating the lookup table of the activation function of the present invention;

[0041] Figure 3 The function curve diagram of an activation function;

[0042] Figure 4 The specific flowchart of increasing the target deviation value in the method for generating the lookup table of the activation function of the present invention;

[0043] Figure 5 The flowchart of the second embodiment of the method for generating the lookup table of the activation function of the present invention;

[0044] Figure 6 The functional module diagram of the device for generating the lookup table of the activation function of the present invention. Detailed implementation manners

[0045] In order to better understand the above technical solution, the exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present invention can be more thoroughly understood and the scope of the present invention can be fully conveyed to those skilled in the art.

[0046] As Figure 1 shown, Figure 1 The structural schematic diagram of the hardware operating environment involved in the solution of the embodiment of the present invention.

[0047] It should be noted that Figure 1 it can be the structural schematic diagram of the hardware operating environment of the terminal device.

[0048] As Figure 1As shown in the figure, the terminal device may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0049] Those skilled in the art can understand that Figure 1 the structure of the terminal device shown in the figure does not constitute a limitation on the terminal device, and it may include more or fewer components than shown in the figure, or combine some components, or have a different component layout.

[0050] As Figure 1 shown in the figure, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and a look-up table generation program for activation functions. Among them, the operating system is a program for managing and controlling the hardware and software resources of the terminal device, and for running the look-up table generation program for activation functions and other software or programs.

[0051] In Figure 1 the terminal device shown in the figure, the user interface 1003 is mainly used to connect to the terminal and perform data communication with the terminal; the network interface 1004 is mainly used to connect to the background server and perform data communication with the background server; the processor 1001 may be used to call the look-up table generation program for activation functions stored in the memory 1005.

[0052] In this embodiment, the terminal device includes: a memory 1005, a processor 1001, and a look-up table generation program for activation functions stored on the memory 1005 and executable on the processor, where:

[0053] When the processor 1001 calls the look-up table generation program for activation functions stored in the memory 1005, the following operations are performed:

[0054] Evenly divide the value range of the activation function according to the target deviation value to obtain a set of sampling points corresponding to the activation function;

[0055] Divide the set of sampling points into at least two intervals, where the line segment slopes of the activation functions corresponding to different intervals are different;

[0056] For each interval, equally divide the input data of the activation function within the interval according to the corresponding target spacing, to obtain a plurality of sub-sampling point sets, where the target spacing corresponding to the interval is inversely correlated with the line segment slope of the corresponding activation function;

[0057] When the total first storage space required for the plurality of sub-sampling point sets is less than or equal to the preset storage space, generate a lookup table of the input values and output values of the activation function according to the plurality of sub-sampling point sets, and store the lookup table;

[0058] When the total first storage space required for the plurality of sub-sampling point sets is greater than the preset storage space, increase the target deviation value, and return to execute the step of equally dividing the value range of the activation function according to the target deviation value to obtain the sampling point set corresponding to the activation function.

[0059] When the processor 1001 calls the lookup table generation program of the activation function stored in the memory 1005, the following operations are also performed:

[0060] Subtract the input data corresponding to two adjacent sampling points in the sampling point set to obtain the spacing between two adjacent sampling points in the sampling point set;

[0061] Divide the sampling points with the spacing within the same spacing range into one interval to obtain at least two such intervals.

[0062] When the processor 1001 calls the lookup table generation program of the activation function stored in the memory 1005, the following operations are also performed:

[0063] Obtain the minimum spacing between two adjacent sampling points in the sampling point set;

[0064] Determine the target spacing corresponding to each interval according to the minimum spacing.

[0065] When the processor 1001 calls the lookup table generation program of the activation function stored in the memory 1005, the following operations are also performed:

[0066] Determine the target spacing corresponding to each interval according to the minimum spacing and the line segment slope of the activation function corresponding to each interval;

[0067] Wherein, the ratio of the target spacing corresponding to the interval with a large line segment slope of the activation function to the target spacing corresponding to the interval with a small line segment slope of the activation function is N, where N is greater than or equal to 2, and the larger the line segment slope, the smaller N is.

[0068] When the processor 1001 calls the lookup table generation program of the activation function stored in the memory 1005, the following operations are also performed:

[0069] Obtain the space difference between the first total storage space and the preset storage space;

[0070] When the space difference is greater than a preset threshold, increase the target deviation value by a first preset increment value;

[0071] When the space difference is less than or equal to the preset threshold, increase the target deviation value by a second preset increment value, where the first preset increment value is greater than the second preset increment value.

[0072] When the processor 1001 calls the lookup table generation program of the activation function stored in the memory 1005, the following operations are also performed:

[0073] Determine the second total storage space required for the sampling points in the sampling point set;

[0074] When the second total storage space is less than or equal to the preset storage space, perform the step of dividing the sampling point set into at least two intervals.

[0075] When the processor 1001 calls the lookup table generation program of the activation function stored in the memory 1005, the following operations are also performed:

[0076] When the second total storage space is greater than the preset storage space, increase the target deviation value, and return to execute the step of equally dividing the value range of the activation function according to the target deviation value to obtain the sampling point set corresponding to the activation function.

[0077] An embodiment of the lookup table generation method of the activation function is provided in the embodiments of the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than here.

[0078] As Figure 2 shown, in the first embodiment of the present invention, the lookup table generation method of the activation function of the present invention includes the following steps:

[0079] Step S210: Equally divide the value range of the activation function according to the target deviation value to obtain the sampling point set corresponding to the activation function.

[0080] In this embodiment, the activation function refers to a function that runs on the neurons of an artificial neural network, which is responsible for mapping the input of the neuron to the output end. Common activation functions include the tanh activation function, the sigmoid activation function, etc. This embodiment and the following various embodiments are described by taking the sigmoid activation function as an example. Among them, as Figure 3 shown, Figure 3It is the function curve of the Sigmoid activation function, and the mathematical expression of the Sigmoid activation function is:

[0081] The target deviation value can be understood as the precision of the corresponding value in the value range of the activation function, and the value range is the value interval or value range of the y of the activation function. For example, the target deviation value is 0.1, and the value precision of the values in the value range is one digit after the decimal point. Specifically, the target deviation value is a value preset by the user according to actual needs. Before generating the lookup table of the activation function, it is necessary to obtain the target deviation value set by the user, and then use the target deviation value to equally divide the value range of the activation function into N parts, each part corresponding to a sampling point, N parts corresponding to N sampling points, and N sampling points form the sampling point set corresponding to the activation function, that is, the sampling point set includes N sampling points. The sampling point refers to the data point (x, y) on the function curve of the activation function, x is the abscissa, which can also be understood as the input data of the activation function, and y is the ordinate, which can also be understood as the output data of the activation function. For example, Figure 3 If the target deviation value is 0.1 and the value range of the Sigmoid activation function is (0, 1), then the value range of the Sigmoid activation function is equally divided into 10 parts, which are [0, 0.1], (0.1, 0.2], (0.2, 0.3], (0.3, 0.4], (0.4, 0.5], (0.5, 0.6], (0.6, 0.7], (0.7, 0.8], (0.8, 0.9], (0.9, 1.0], that is, there are 10 sampling points corresponding to the data in these 10 parts.

[0082] Step S220: Divide the sampling point set into at least two intervals.

[0083] In this embodiment, since the degree of change of the line segment slope of some curves in the function curve of the activation function is large, that is, the slope is large, and the degree of change of the line segment slope of some curves is slow, that is, the slope is small. For the part with a large slope, the corresponding y value changes greatly, and for the part with a small slope, the corresponding y value changes slowly. For example, Figure 3 the y values corresponding to x less than -5 and x greater than 5 are almost infinitely close, and the change of the y values corresponding to x between -5 and 5 is very large. Therefore, based on the characteristics of the activation function, the sampling point set is divided into at least two intervals according to the line segment slope of the activation function curve. For example, the sampling points corresponding to the larger line segment slope are divided into one interval, the sampling points corresponding to the moderate line segment slope are divided into one interval, and the sampling points corresponding to the smaller line segment slope are divided into one interval, etc., so as to obtain multiple intervals. Among them, the obtained intervals are different, that is, the line segment slopes of the activation functions corresponding to different intervals are different.

[0084] Specifically, step S220 includes:

[0085] Take the difference between the input data corresponding to two adjacent sampling points in the set of sampling points to obtain the distance between two adjacent sampling points in the set of sampling points;

[0086] Divide the sampling points with the distances within the same distance range into one interval to obtain at least two such intervals.

[0087] First, calculate the distances between two adjacent sampling points in the set of sampling points. The distance between two adjacent sampling points is the difference between the input data (abscissa) corresponding to two adjacent sampling points, that is, take the difference between the input data corresponding to two adjacent sampling points in the set of sampling points to obtain the input data difference, and the obtained input data difference is the said distance. After obtaining the distances between each pair of adjacent sampling points in the set of sampling points, divide the sampling points in the set of sampling points with the distances within the same distance range into one interval, that is, divide the sampling points in the set of sampling points that satisfy the condition d < 2β into the first interval M1, divide the sampling points in the set of sampling points that satisfy 2β < d < 4β into the second interval M2,..., divide the sampling points in the set of sampling points that satisfy 2 n -1 β < d < 2 n β into the nth interval Mn, thus obtaining multiple intervals. Wherein, d is the distance between two adjacent sampling points, β is the minimum distance between two adjacent sampling points, n is the arrangement order of the intervals, and the line slopes of the activation functions corresponding to M1, M2,..., Mn gradually decrease, the line slope of the activation function corresponding to M1 is the largest, and the line slope of the activation function corresponding to Mn is the smallest.

[0088] For example, Figure 3 the activation function is divided into 10 parts and there are 10 sampling points. The abscissas corresponding to the ordinates in the positive semi-axis are approximately also Figure 3 0, 0.4, 0.9, 1.3, 2.1, etc. in [reference], and the abscissas corresponding to the ordinates in the negative semi-axis are not given. Wherein, the minimum distance between two adjacent abscissas is 0.4. Then, based on the characteristics of the activation function, divide the sampling points in the set of sampling points with distances less than 0.8 into the first interval M1. The first interval M1 is (-2.1, 2.1). Based on the division principle of the first interval M1, divide the sampling points in the set of sampling points with distances greater than 0.8 and less than 1.6 into the second interval M1,... divide the sampling points in the set of sampling points with distances greater than 2 n-1 ×0.4 and less than 2 n ×0.4 into the nth interval M1.

[0089] Step S230: Uniformly divide the input data of the activation function in each of the intervals according to the corresponding target spacing to obtain multiple sub-sampling point sets.

[0090] After obtaining multiple different intervals, determine the target spacing corresponding to each interval. The target spacing corresponding to each interval is determined according to the minimum spacing between two adjacent sampling points in the sampling point set. Determining the target spacing corresponding to each interval includes: obtaining the minimum spacing between two adjacent sampling points in the sampling point set, and then determining the target spacing corresponding to each interval according to the minimum spacing. Among them, after calculating the spacing between two adjacent sampling points in the sampling point set, multiple spacings will be obtained, and the minimum value among the multiple spacings is the minimum spacing, and then the target spacing corresponding to each interval is determined according to the minimum spacing, and the ratio of the target spacings between adjacent intervals is an integer.

[0091] Specifically, the target spacing corresponding to each interval is inversely correlated with the line segment slope of the corresponding activation function, that is, the larger the line segment slope of the activation function, the smaller the target spacing, the finer the interval division, the more sampling points in the sub-sampling point set obtained by the division, and the more accurate the output data corresponding to the part with a larger line segment slope in the generated look-up table. The smaller the line segment slope of the activation function, the larger the target spacing, the coarser the interval division, the fewer sampling points in the sub-sampling point set obtained by the division, and the output data corresponding to the part with a smaller line segment slope in the generated look-up table is almost very close, with a small error, that is, the storage space occupancy can be reduced. Then, based on the magnitude of the line segment slope of the activation function corresponding to different intervals, set the corresponding target spacing for each interval according to the minimum spacing. If the line segment slope of the activation function corresponding to the interval is large, set the target spacing corresponding to the interval to be small based on the minimum spacing. If the line segment slope of the activation function corresponding to the interval is small, set the target spacing corresponding to the interval to be large based on the minimum spacing. In this way, the intervals corresponding to larger line segment slopes can be finely divided, and the intervals corresponding to smaller line segment slopes can be coarsely divided, which is beneficial to saving storage space.

[0092] Further, in order to facilitate table lookup after generating the look-up table, the step of determining the target spacing corresponding to each interval according to the minimum spacing includes: determining the target spacing corresponding to each interval according to the minimum spacing and the line segment slope of the activation function corresponding to each interval. Among them, the ratio of the target spacing corresponding to the interval with a large line segment slope of the activation function to the target spacing corresponding to the interval with a small line segment slope of the activation function is N, where N is greater than or equal to 2, and the larger the line segment slope, the smaller N. For example, the minimum spacing is a, interval M1 is divided by 2 0 a, interval M2 is divided by 2 1 a, ..., interval Mn is divided by 2 n-1Partition a. The ratio of the target spacings corresponding to two adjacent intervals is 2. In this way, the intervals with a larger line segment slope of the activation function are divided more finely, and the intervals with a smaller line segment slope of the activation function are divided more roughly, which can improve the subsequent table lookup speed and data query accuracy.

[0093] Continuing with the above example, the first interval M1 is (-2.1, 2.1). Redivide M1 at an interval of 0.4. The input data (abscissa) in the first subsampling point set m1 obtained is (-2, -1.6, -1.2, -0.8, -0.4, 0, 0.4, 0.8, 1.2, 1.6, 2), that is, the input data of the sampling points in this subsampling point set is an integer multiple of 0.4, which is convenient for subsequent table lookup. After such division, the interval M1 is redivided. Then, similarly, redivide M2 at an interval of 0.8 to obtain the second subsampling point set m2,..., redivide Mn at an interval of 2 n-1 ×0.4 to obtain the nth subsampling point set mn.

[0094] Step S240: Determine whether the total first storage space required for the multiple subsampling point sets is less than or equal to the preset storage space; if so, execute step S250; if not, execute step S260.

[0095] Step S250: Generate a lookup table of the input values and output values of the activation function according to the multiple subsampling point sets, and store the lookup table.

[0096] Step S260: Increase the target deviation value, return to execute step S210, equally divide the value range of the activation function according to the target deviation value to obtain the sampling point set corresponding to the activation function.

[0097] In this embodiment, after obtaining multiple subsampling point sets, count the storage spaces of the sampling points in the multiple subsampling point sets to obtain the total first storage space required for the multiple subsampling point sets. Among them, the storage space of the sampling points in each subsampling point set is represented as Sn, and there are n subsampling point sets in total. Then the storage spaces of the sampling points in the n subsampling point sets are S1, S2,..., Sn respectively. The total first storage space is represented as S, and the preset storage space refers to the storage space of the chip, which is represented as W. S = S1 + S2 +,..., + Sn. The chip can be an artificial intelligence acceleration chip, an FPGA product, etc.

[0098] If S ≤ W, input the input data of the sampling points in each subsampled point set into the activation function to obtain the output data of the sampling points in each subsampled point set. Then, based on the input data and output data of each sampling point in each subsampled point set, generate a look-up table of the input values and output values of the activation function, and store the look-up table in the storage space of the chip. Among them, the accuracy of the output values of the generated look-up table can meet the user's requirements.

[0099] To further improve the accuracy of the output values of the look-up table, if S ≤ W, reduce the target deviation value, and then return to execute step S210 until multiple subsampled point sets are obtained again. Then, compare the new S and W. If the new S ≤ W and the reduced target deviation value is less than the set threshold, it means that when the storage space of the chip can store the multiple subsampled point sets obtained again, accurate output data of the sampling points can also be obtained. Then, based on the input data and output data of each sampling point in each subsampled point set, generate a look-up table of the input values and output values of the activation function, and store the look-up table in the storage space of the chip, which is beneficial to further improving the accuracy of the output values of the activation function in the look-up table.

[0100] If S > W, it means that the total first storage space required for multiple subsampled point sets is too large. If a look-up table is generated, the storage space of the chip cannot store the look-up table, that is, there is too much data in the look-up table. Then, the data in the look-up table needs to be reduced. Reducing the data in the look-up table can be achieved by increasing the target deviation value. After increasing the target deviation value, return to execute step S210. The sampling points in the obtained sampling point set will decrease, and the input data and output data of each sampling point in each subsequent subsampled point set will also decrease, and the total first storage space required for each subsampled point set will also decrease. That is, after executing step S210 one or more times in a loop, if the new S ≤ W, then based on the input data and output data of each sampling point in each subsampled point set, generate a look-up table of the input values and output values of the activation function, and store the look-up table in the storage space of the chip. In this way, when the chip can store a look-up table with more data, the accuracy of the output values of the activation function in the look-up table can also be improved.

[0101] In this embodiment, according to the above technical solution, based on the characteristics of the activation function, if the line segment slope of the activation function is large, the interval corresponding to the large line segment slope is finely divided so that the interval corresponding to the large line segment slope includes more sampling points. If the line segment slope of the activation function is small, the interval corresponding to the small line segment slope is roughly divided so that the interval corresponding to the small line segment slope includes fewer sampling points. After generating the look-up table of the activation function, it can not only improve the accuracy of the data in the look-up table, but also reduce the space storage of the look-up table, which is beneficial to saving the storage resources of the chip and further improving the speed of the chip to search for data through the look-up table.

[0102] Further, as shown in Figure 4 , the steps of increasing the target deviation value include:

[0103] Step S261: Obtain the space difference between the first total storage space and the preset storage space.

[0104] Step S262: When the space difference is greater than a preset threshold, increase the target deviation value by a first preset increment value.

[0105] Step S263: When the space difference is less than or equal to the preset threshold, increase the target deviation value by a second preset increment value, where the first preset increment value is greater than the second preset increment value.

[0106] Specifically, preset increment values for adjusting the target deviation value are set in advance. The preset increment values include a first preset increment value and a second preset increment value, and the first preset increment value is greater than the second preset increment value. If S > W, then calculate the difference between S and W, that is, the space difference, space difference = S - W. The space difference is used to measure the amount of increase in the target deviation value. If the target deviation value is greater than the preset threshold, it means that the amount of increase in the target deviation value can be larger, that is, increase the target deviation value by the first preset increment value each time. If the space difference is less than or equal to the preset threshold, it means that the amount of increase in the target deviation value cannot be too large, that is, increase the target deviation value by the second preset increment value each time. In this way, accurate adjustment of the target deviation value can be achieved, which is beneficial to improving the accuracy of subsequent sampling point set division and the accuracy of data in the generated look-up table.

[0107] As shown in Figure 5 , in the second embodiment of the present invention, the method for generating a look-up table of the activation function of the present invention includes the following steps:

[0108] Step S210: Evenly divide the value range of the activation function according to the target deviation value to obtain the sampling point set corresponding to the activation function;

[0109] Step S211: Determine the second total storage space required for the sampling points in the sampling point set;

[0110] Step S212: Judge whether the second total storage space is less than or equal to the preset storage space; if so, execute Step S220; if not, execute Step S213;

[0111] Step S213: Increase the target deviation value, and return to execute Step S210 to evenly divide the value range of the activation function according to the target deviation value to obtain the sampling point set corresponding to the activation function;

[0112] Step S220: Divide the set of sampling points into at least two intervals;

[0113] Step S230: Evenly divide the input data of the activation function in each of the intervals according to the corresponding target spacing to obtain multiple sub-sampling point sets;

[0114] Step S240: Determine whether the first total storage space required for the multiple sub-sampling point sets is less than or equal to a preset storage space; if so, execute Step S250; if not, execute Step S260;

[0115] Step S250: Generate a lookup table for the input values and output values of the activation function based on the multiple sub-sampling point sets and store the lookup table;

[0116] Step S260: Increase the target deviation value, return to execute Step S210, evenly divide the value range of the activation function according to the target deviation value to obtain the set of sampling points corresponding to the activation function.

[0117] In this embodiment, after obtaining the target deviation value set by the user, the value range of the activation function is evenly divided using the target deviation value to obtain the set of sampling points corresponding to the activation function, and then the total storage space required for the sampling points in the set of sampling points is calculated. The total storage space required for the sampling points in the set of sampling points is denoted as the second total storage space. Since the storage space of the chip is fixed and cannot be adjusted, after obtaining the set of sampling points corresponding to the activation function, it is necessary to consider whether the chip can store the sampling points in the set of sampling points. Specifically, the second total storage space is compared with the preset storage space. If the second total storage space is less than or equal to the preset storage space, it means that the chip can store the sampling points in the set of sampling points, that is, the chip can store the subsequent generated lookup table. If the second total storage space is greater than the preset storage space, it means that the division of the value range of the activation function is unreasonable, the data size of the sampling points in the set of sampling points is too large, and the chip cannot store the sampling points in the set of sampling points, that is, the chip cannot store the subsequent generated lookup table. Then it is necessary to ensure that the sampling points in the generated set of sampling points can be stored by the chip, that is, by increasing the target deviation value to reduce the number of sampling points in the generated set of sampling points. If after increasing the target deviation value once, the second total storage space is still greater than the preset storage space, then continue to increase the target deviation value until the second total storage space is less than or equal to the preset storage space. In this way, considering that the preset storage space of the chip is fixed, by adjusting the target deviation value, it is ensured that the subsequent generated lookup table can be stored by the chip, avoiding the occurrence of useless work.

[0118] For the specific implementation process of steps S220 - S260 in this embodiment, it is the same as that of steps S220 - S260 in the first embodiment, and will not be elaborated in this embodiment.

[0119] According to the above technical solution, in this embodiment, when generating the lookup table of the activation function, it is possible to avoid doing useless work, improve the accuracy of the data in the lookup table, reduce the space storage of the lookup table, which is beneficial to saving the storage resources of the chip, and further improve the speed of the chip to search for data through the lookup table.

[0120] Further, as Figure 6 shown, a lookup table generation device for an activation function provided by the present invention includes:

[0121] A value range division module 310, configured to equally divide the value range of the activation function according to a target deviation value to obtain a set of sampling points corresponding to the activation function;

[0122] An interval generation module 320, configured to divide the set of sampling points into at least two intervals, where the line segment slopes of the activation functions corresponding to different intervals are different;

[0123] A set generation module 330, configured to equally divide the input data of the activation function in each interval according to a corresponding target spacing to obtain a plurality of sub - sampling point sets, and the target spacing corresponding to the interval is inversely correlated with the line segment slope of the corresponding activation function;

[0124] A first storage module 340, configured to generate a lookup table of the input values and output values of the activation function according to the plurality of sub - sampling point sets and store the lookup table when the first total storage space required by the plurality of sub - sampling point sets is less than or equal to a preset storage space;

[0125] A second storage module 350, configured to increase the target deviation value and return to execute the step of equally dividing the value range of the activation function according to the target deviation value to obtain the set of sampling points corresponding to the activation function when the first total storage space required by the plurality of sub - sampling point sets is greater than the preset storage space.

[0126] The specific implementation manner of the lookup table generation device for the activation function of the present invention is basically the same as that of the embodiments of the above - mentioned lookup table generation method for the activation function, and will not be elaborated here.

[0127] Further, the present invention also provides a terminal device, including: a memory, a processor, and a lookup table generation program for the activation function stored on the memory and executable on the processor. When the lookup table generation program for the activation function is executed by the processor, the steps of the above - mentioned lookup table generation method for the activation function are implemented.

[0128] Furthermore, the present invention also provides a storage medium, on which a lookup table generation program for an activation function is stored. When the lookup table generation program for the activation function is executed by a processor, the steps of the above-mentioned lookup table generation method for the activation function are implemented.

[0129] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the specified functions in Figure 1 one or more of the flows Figure 1 or a plurality of flows and / or blocks

[0131] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified functions in Figure 1 one or more of the flows Figure 1 or a plurality of flows and / or blocks

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the specified functions in Figure 1 one or more of the flows Figure 1 or a plurality of flows and / or blocks

[0133] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim listing several devices, several of these devices may be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not denote any order. These words may be interpreted as names.

[0134] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made by those skilled in the art once they learn the basic inventive concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0135] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A method for generating a lookup table of activation functions, characterized in that, The method for generating the look-up table of the activation function includes: Equally divide the value range of the activation function according to the target deviation value to obtain the set of sampling points corresponding to the activation function; Divide the set of sampling points into at least two intervals, where the line slopes of the activation functions corresponding to different intervals are different; Obtain the minimum distance between two adjacent sampling points in the set of sampling points; Determine the target distance corresponding to each interval according to the minimum distance and the line slope of the activation function corresponding to each interval, where several distances are calculated, the minimum value among the several distances is the minimum distance, and the target distance corresponding to each interval is determined according to the minimum distance, and the ratio of the target distances between adjacent intervals is an integer. The ratio of the target distance corresponding to the interval with a larger line slope of the activation function to the target distance corresponding to the interval with a smaller line slope of the activation function is N, where N is a positive integer and N is greater than or equal to 2, and the larger the line slope, the smaller N; Equally divide the input data of the activation function in each interval according to the corresponding target distance to obtain multiple sub-sampling point sets, and the target distance corresponding to the interval is inversely related to the line slope of the corresponding activation function; When the total first storage space required by the multiple sub-sampling point sets is less than or equal to the preset storage space, generate the look-up table of the input values and output values of the activation function according to the multiple sub-sampling point sets, and store the look-up table; When the total first storage space required by the multiple sub-sampling point sets is greater than the preset storage space, increase the target deviation value, and return to execute the step of equally dividing the value range of the activation function according to the target deviation value to obtain the set of sampling points corresponding to the activation function; 2. The method according to claim 1, characterized in that, The step of dividing the set of sampling points into at least two intervals includes: Subtract the input data corresponding to two adjacent sampling points in the set of sampling points to obtain the distance between two adjacent sampling points in the set of sampling points; Divide the sampling points with the distances within the same distance range into one interval to obtain at least two such intervals.

3. The method according to claim 1, characterized in that, The step of increasing the target deviation value includes: Obtain the space difference between the total first storage space and the preset storage space; When the space difference is greater than the preset threshold, increase the target deviation value by a first preset increment value; When the space difference is less than or equal to the preset threshold, increase the target deviation value by a second preset increment value, and the first preset increment value is greater than the second preset increment value.

4. The method according to claim 1, wherein After the step of equally dividing the value range of the activation function according to the target deviation value to obtain the set of sampling points corresponding to the activation function, it further includes: Determine the total second storage space required by the sampling points in the set of sampling points; When the total second storage space is less than or equal to the preset storage space, execute the step of dividing the set of sampling points into at least two intervals.

5. The method according to claim 4, wherein After the step of determining the total second storage space of the sampling points in the set of sampling points, it further includes: When the second total storage space is greater than the preset storage space, increase the target deviation value, and return to execute the step of equally dividing the value range of the activation function according to the target deviation value to obtain the sampling point set corresponding to the activation function.

6. A lookup table generation device for an activation function, characterized in that The lookup table generation device for the activation function includes: A value range division module, configured to equally divide the value range of the activation function according to the target deviation value to obtain the sampling point set corresponding to the activation function; An interval generation module, configured to divide the sampling point set into at least two intervals, where the line segment slopes of the activation functions corresponding to different intervals are different, to obtain the minimum distance between two adjacent sampling points in the sampling point set, and determine the target distance corresponding to each interval according to the minimum distance and the line segment slope of the activation function corresponding to each interval. Among them, several distances are calculated, and the minimum value among the several distances is the minimum distance. The target distance corresponding to each interval is determined according to the minimum distance, and the ratio of the target distances between adjacent intervals is an integer. The ratio of the target distance corresponding to the interval with a larger line segment slope of the activation function to the target distance corresponding to the interval with a smaller line segment slope of the activation function is N, where N is a positive integer and N is greater than or equal to 2. The larger the line segment slope, the smaller N is; A set generation module, configured to equally divide the input data of the activation function in each interval according to the corresponding target distance to obtain a plurality of sub-sampling point sets, and the target distance corresponding to the interval is inversely correlated with the line segment slope of the corresponding activation function; A first storage module, configured to generate a lookup table of the input values and output values of the activation function according to the plurality of sub-sampling point sets and store the lookup table when the first total storage space required by the plurality of sub-sampling point sets is less than or equal to the preset storage space; A second storage module, configured to increase the target deviation value when the first total storage space required by the plurality of sub-sampling point sets is greater than the preset storage space, and return to execute the step of equally dividing the value range of the activation function according to the target deviation value to obtain the sampling point set corresponding to the activation function.

7. A terminal device, characterized in that, Includes: A memory, a processor, and a lookup table generation program for the activation function stored on the memory and executable on the processor. When the lookup table generation program for the activation function is executed by the processor, the steps of the lookup table generation method for the activation function according to any one of claims 1-5 are implemented.

8. A storage medium, characterized in that, A lookup table generation program for the activation function is stored thereon. When the lookup table generation program for the activation function is executed by the processor, the steps of the lookup table generation method for the activation function according to any one of claims 1-5 are implemented.

Images