Implementation Method, Structure, Computer Device and Medium of Neural Network Activation Function

Abstract

The embodiment of the present invention discloses a method, structure, computer device and medium for implementing a neural network activation function. The method includes: pre-storing a lookup table corresponding to a standby activation function in an AI chip, approximating the standby activation function with a piecewise function, and determining the input value and corresponding output value of the lookup table according to the piecewise function; obtaining the input data of the standby activation function, and determining the target input value of the lookup table according to the input data; looking up the table according to the target input value to obtain the corresponding target output value, and determining the output data of the standby activation function according to the target output value. By using a piecewise function to approximate the standby activation function and generating a corresponding lookup table, when the standby activation function is used for calculation, the calculation result is determined by looking up the table, the function of the neural network activation function is simply and quickly implemented, multiple calculations and storage are avoided, thereby improving the performance of the AI chip and saving power consumption.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of deep learning, and in particular, to a method for implementing a neural network activation function, a structure, a computer device, and a medium. Background Art

[0002] Artificial intelligence (AI) algorithms represented by deep learning are widely used in various industries and people's daily lives. These algorithms generally have characteristics such as large computational volume, complex tasks, and high precision. These characteristics of AI algorithms have promoted the rapid development of AI-accelerated customized chips. AI-accelerated chips usually perform targeted optimizations for various operations in AI algorithms to achieve higher performance. The principle of deep learning is to simulate a neural network, and its basic unit is a neuron. The activation function is the basic function of a neuron, which produces a non-linear output for the input of the neuron, thereby superimposing the effects of neurons in each layer to achieve the effect of capturing patterns. There are many common activation functions in deep learning, the most commonly used ones are ReLU (Rectified Linear Unit), Sigmoid, tanh, ReLU6, and leaky ReLU. In addition, there are also functions such as ELU (Exponential Linear Units), Mish, Exponential, Softsign, Softplus, and Swish.

[0003] As an essential component in AI algorithms, the activation function also needs to be implemented in the design of AI chips. To implement various forms of functions of the activation function, an ALU unit is usually used in AI chips to complete the calculation part of the function. The function of the ALU is designed to be relatively powerful, and various complex arithmetic operations can be achieved through combination and splitting. Considering the complexity and benefits of the design and implementation of AI chips, data formats such as int8, int16, or FP16 are usually adopted in actual AI chip designs. Since there are many function forms that the activation function can use, and some of the function forms are very complex, even using the simplest int8 data format, implementing the above various functions will consume a large amount of hardware resources and affect the performance of the AI chip, which is not conducive to improving the parallelism of the AI chip. Moreover, complex activation functions need to be split and executed step by step, resulting in a slow actual execution speed of the activation function, leading to low performance of the AI chip. At the same time, when using the ALU to implement the activation function step by step, it is necessary to read and write the buffer multiple times, which also increases the power consumption of the AI chip. Summary of the Invention

[0004] Embodiments of the present invention provide a method, structure, computer device, and medium for implementing a neural network activation function to simply and quickly implement the function of the neural network activation function, avoid multiple operations and storage, improve performance, and save power consumption.

[0005] In a first aspect, embodiments of the present invention provide a method for implementing a neural network activation function, the method including:

[0006] Pre-store a look-up table corresponding to the activation function to be used in the AI chip, approximate the activation function to be used with a piecewise function, and determine the input values and corresponding output values of the look-up table according to the piecewise function;

[0007] Obtain the input data of the activation function to be used, and determine the target input value of the look-up table according to the input data;

[0008] Look up the table according to the target input value to obtain the corresponding target output value, and determine the output data of the activation function to be used according to the target output value.

[0009] Optionally, multiple copies of the look-up table are stored separately;

[0010] Correspondingly, obtaining the input data of the activation function to be used and determining the target input value of the look-up table according to the input data includes:

[0011] Obtaining a plurality of the input data simultaneously according to a preset parallelism and respectively determining a plurality of the target input values;

[0012] Correspondingly, looking up the table according to the target input value to obtain the corresponding target output value includes:

[0013] Looking up the tables in different look-up tables simultaneously and concurrently according to a plurality of the target input values to obtain the respective corresponding target output values.

[0014] Optionally, pre-storing the look-up table corresponding to the activation function to be used in the AI chip includes:

[0015] Reusing the original RAM storage space of the AI chip, where the original RAM storage space is designed to store the input data and the output data, dividing the original RAM storage space into a data RAM storage space and a look-up table RAM storage space, and storing the look-up table in the look-up table RAM storage space;

[0016] Correspondingly, before obtaining the input data of the activation function to be used, it further includes:

[0017] Caching the input data into the data RAM storage space.

[0018] Optionally, the data formats of the input data and the look-up table are int16, the look-up table RAM storage space has a banked structure, and determining the target input value of the look-up table according to the input data includes:

[0019] Using the lower part of the input data as the target input value;

[0020] Looking up the table according to the target input value to obtain the corresponding target output value includes:

[0021] Copying the target input value according to the number of banks of the look-up table RAM storage space and respectively reading each bank of the look-up table RAM storage space using the target input value to simultaneously obtain a plurality of candidate output values;

[0022] Using the upper part of the input data as an output selection signal and determining the target output value from the plurality of candidate output values according to the output selection signal.

[0023] Optionally, before simultaneously obtaining a plurality of the input data according to a preset parallelism and respectively determining a plurality of the target input values, the method further includes:

[0024] Determining the preset parallelism according to the size of the original RAM storage space.

[0025] In a second aspect, an implementation structure of a neural network activation function provided by an embodiment of the present invention applies the method for implementing a neural network activation function provided by any embodiment of the present invention. The structure includes: a data RAM storage space, an address conversion module, a lookup table RAM storage space, a selector, and a data reconstruction module; wherein,

[0026] The data RAM storage space is used to store the input data of the activation function to be used.

[0027] The address conversion module is configured to take out the input data from the data RAM storage space, convert the input data into a target input value of a lookup table corresponding to the activation function to be used, and copy the target input value into a target number of copies and then give it to the lookup table RAM storage space.

[0028] The lookup table RAM storage space is a banked structure, and the number of banks is the target number of copies, and is used to store the lookup table, and respectively read each bank using the target input value to simultaneously obtain a plurality of candidate output values and give them to the selector.

[0029] The address conversion module is further configured to convert the input data into an output selection signal and give the output selection signal to the selector.

[0030] The selector is configured to determine a target output value from each of the candidate output values according to the output selection signal and give the target output value to the data reconstruction module.

[0031] The data reconstruction module is configured to collect and cache the target output value and reconstruct it into a data format required by the AI chip to obtain the output data of the activation function to be used.

[0032] Optionally, the lookup table RAM storage space is specifically used to store multiple copies of the replicated lookup table.

[0033] Correspondingly, the address conversion module is specifically configured to simultaneously receive a plurality of the input data according to a preset parallelism, respectively convert them into a plurality of the target input values and then give them to the lookup table RAM storage space, and respectively convert them into a plurality of the output selection signals and then give them to the selector.

[0034] Correspondingly, the lookup table RAM storage space is specifically used to perform table lookups in different lookup tables simultaneously and concurrently according to the multiple target input values, so as to obtain the output value to be selected corresponding to each target input value and provide it to the selector;

[0035] Correspondingly, the selector is specifically configured to determine the corresponding target output value from the corresponding output values to be selected according to each of the output selection signals.

[0036] Optionally, the address conversion module is specifically used to generate a read address of the data RAM storage space through a universal address generation unit, and fetch each input data from the data RAM storage space according to the execution order of the standby activation function.

[0037] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0038] one or more processors;

[0039] A memory for storing one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for implementing the neural network activation function provided by any embodiment of the present invention.

[0041] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for implementing a neural network activation function provided by any embodiment of the present invention.

[0042] The embodiment of the present invention provides a method for implementing a neural network activation function. First, a lookup table corresponding to a standby activation function is stored in advance in an AI chip, wherein the standby activation function is approximated by a piecewise function, and the input value and the corresponding output value of the lookup table can be determined based on the piecewise function. Then, the input data of the standby activation function is obtained, and the target input value of the lookup table is determined based on the input data. Then, a table is looked up based on the target input value to determine the corresponding target output value in the lookup table, thereby determining the output data of the standby activation function based on the target output value. The method for implementing a neural network activation function provided by the embodiment of the present invention approximates the standby activation function by using a piecewise function, and generates a corresponding lookup table, so that when the standby activation function is used for calculation, the calculation result is determined by looking up the table, thereby simply and quickly implementing the function of the neural network activation function, avoiding multiple calculations and storage, thereby improving the performance of the AI chip and saving power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1A flowchart of a method for implementing a neural network activation function provided in Embodiment 1 of the present invention;

[0044] Figure 2 A schematic diagram of the implementation structure of a neural network activation function provided in Embodiment 2 of the present invention;

[0045] Figure 3 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0047] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0048] Embodiment 1

[0049] Figure 1 This is a flow chart of a method for implementing a neural network activation function provided in the first embodiment of the present invention. This embodiment is applicable to the case where a neural network activation function is used for related calculations. This method can be implemented by the implementation structure of the neural network activation function provided in the embodiment of the present invention, which can generally be integrated into an AI chip. Figure 1 As shown, the specific steps include:

[0050] S11. Pre-store a lookup table corresponding to a standby activation function in the AI chip, wherein the standby activation function is approximated by a piecewise function, and determine an input value and a corresponding output value of the lookup table according to the piecewise function.

[0051] Specifically, the implementation method provided in this embodiment can be implemented on an AI chip based on a data flow architecture, so the required lookup table can be stored in the AI chip in advance, wherein the lookup table can be generated offline in advance by an external program according to the deep learning algorithm model to be executed as required, and stored in the external storage module DDR, then the AI chip can first read the lookup table data corresponding to the standby activation function from the DDR and store it in the lookup table storage unit. The lookup table is a commonly used hardware approximation implementation method, which can be obtained by approximating the complex function of the target through a piecewise function. The piecewise function can be used to first approximate the activation function to be used, and then the input value and output value of the corresponding lookup table can be determined according to the values of the independent variable and the dependent variable of the piecewise function. Assuming that the standby activation function is y=f(x), and the value range of the input data is [a,b], the range between a and b on the x-axis can be divided into n parts, each of which is Δx, and the y value corresponding to each segment can be y=f(a+Δx*i), where i=0,1,2,…, then the corresponding piecewise function can be obtained. Then each segment Δx can be represented by a number, and each number corresponds to an output value y n In this way, the input value and the corresponding output value of the lookup table can be obtained, and the storage space required for the lookup table can also be saved. The number can be the i value mentioned above, that is, it can be the storage address of the corresponding output value in the storage unit of the lookup table, so that the table can be simply looked up by the number.

[0052] S12: Obtain input data of the standby activation function, and determine a target input value of the lookup table according to the input data.

[0053] Specifically, when using the AI chip to perform calculations using the standby activation function, the currently required input data can be first read in the execution order of the standby activation function, and then the input data can be converted into the target input value of the lookup table according to the adopted piecewise function, that is, it can be the read address of the lookup table storage unit, so as to perform subsequent table lookup. Among them, the input data used by the standby activation function can also be pre-stored in the data storage unit in the AI chip, and specifically, the corresponding data can be read from the external storage module DDR for storage, and the output result can also be directly stored as the input data of the standby activation function when the AI chip executes the previous deep learning algorithm operation. When it is necessary to obtain the input data, the general address generation unit pbuf can be used to generate the read address of the data storage unit, and then the currently required input data can be taken out from the data storage unit according to the execution order of the standby activation function.

[0054] S13, looking up a table according to the target input value to obtain a corresponding target output value, and determining output data of the stand-by activation function according to the target output value.

[0055] Specifically, after determining the target input value, a table lookup can be performed according to the target input value, and the table lookup result obtained is the target output value. After obtaining the target output value, the target output value can be collected and cached, and data rearranged and adjusted, so as to finally reconstruct the output data that meets the data format requirements of the AI chip, and the output data can be output to the external storage module DDR. By continuously performing the above operations in the execution order of the standby activation function, the calculation process of the standby activation function can be completed.

[0056] On the basis of the above technical solution, optionally, the lookup table is copied multiple times and stored separately; accordingly, the obtaining of the input data of the standby activation function and determining the target input value of the lookup table based on the input data includes: simultaneously obtaining the multiple input data according to a preset parallelism, and separately determining the multiple target input values; accordingly, the table lookup based on the target input values to obtain the corresponding target output values includes: simultaneously and concurrently looking up in different lookup tables based on the multiple target input values to obtain the respective corresponding target output values.

[0057] Specifically, when the lookup table is stored in the lookup table storage unit, the lookup table can be copied and stored at the same time, so that multiple identical lookup tables are stored in the lookup table storage unit at the same time to achieve parallel computing, that is, to calculate the corresponding output data for multiple input data at the same time. Accordingly, multiple input data can be read from the data storage unit at the same time according to the preset parallelism, and the target input value corresponding to each input data can be determined respectively, and then the preset parallelism is used to perform table lookups in different lookup tables at the same time according to each target input value, so as to obtain the corresponding target output value, and then determine the corresponding output data. Among them, the number of lookup tables can be the same as the preset parallelism to maximize the utilization of the lookup tables.

[0058] Further optionally, the pre-storing of the lookup table corresponding to the standby activation function in the AI chip includes: reusing the original RAM storage space of the AI chip, the original RAM storage space being designed to store the input data and the output data, dividing the original RAM storage space into a data RAM storage space and a lookup table RAM storage space, and storing the lookup table in the lookup table RAM storage space; accordingly, before obtaining the input data of the standby activation function, it also includes: caching the input data in the data RAM storage space.

[0059] Specifically, the lookup table can be implemented using a storage unit (on-chip RAM), and the required RAM space size can be calculated based on the content to be stored in the lookup table and the corresponding data format. For int8 type data, there are 2^8=256 values in its input, so 256 output values need to be stored accordingly, and each output value is still int8 type data, so a total of 256*8bit, i.e., 256B of storage space is required, and if the lookup operation needs to be performed in parallel, a total storage space of 256B*N is required, where N is the preset parallelism. For int16 type data, there are 2^16 values in its input, so 2^16 output values need to be stored accordingly, and each output value is still int16 type data, so a total storage space of 2^16*16bit, i.e., 128KB of storage space is required, and if the lookup operation needs to be performed in parallel, a total storage space of 128KB*N is required, where N is the preset parallelism. In comparison, the RAM space required for int8 type data is smaller, and the AI chip can directly use on-chip RAM to implement the corresponding function for the table lookup operation, but for int16 type data, the required RAM space is very large, and if the on-chip RAM is used, the AI chip will be over-consumed. Therefore, in this embodiment, the original RAM storage space in the AI chip can be reused, and a part of the RAM storage space originally used to store input data and output data can be divided as the lookup table RAM storage space required for the int16 type data table lookup operation, which can be used as the above-mentioned lookup table storage unit, thereby eliminating the dedicated RAM storage space allocated for the lookup table, and then the corresponding lookup table is stored in the lookup table RAM storage space for standby. Of course, this storage method can also be used for int8 type data. Specifically, the original RAM storage space can be divided into two blocks, one as a data RAM storage space (data RAM), which is used to cache the input data of the standby activation function, which can be used as the above-mentioned data storage unit, and the other as a lookup table RAM storage space (table RAM), which is used to store the lookup table. Correspondingly, as described above, before obtaining the input data of the activation function to be used, the input data may also be first cached in the data RAM storage space.

[0060] Further optionally, the data formats of the input data and the lookup table are int16, and the RAM storage space of the lookup table is a banked structure. Determining the target input value of the lookup table according to the input data includes: using the lower segment part of the input data as the target input value; performing a lookup according to the target input value to obtain a corresponding target output value, including: copying the target input value according to the number of banks in the RAM storage space of the lookup table, and respectively using the target input value to read each bank in the RAM storage space of the lookup table to simultaneously obtain multiple candidate output values; using the upper segment part of the input data as an output selection signal, and determining the target output value from among the candidate output values according to the output selection signal.

[0061] Specifically, for data of the int16 type, an address conversion module can be used to convert the input data read from the data RAM storage space. Each input data is divided into two parts: the upper segment and the lower segment. The lower segment part is extracted as the target input value, which can be the read address of the RAM storage space of the lookup table. Since the RAM for caching input data is usually designed with a banked structure to achieve faster read speeds through parallel access, there are also multiple banks when it is reused as the RAM storage space of the lookup table. According to this structure, the address conversion module will simultaneously copy the obtained target input value according to the number of banks to obtain multiple corresponding read addresses, and then use these multiple read addresses to read multiple banks for each lookup table respectively to simultaneously obtain multiple candidate output values. At the same time, the address conversion module can extract the upper segment part of the input data as the output selection signal, and the multiple candidate output values can be selected by a selector according to this output selection signal to use one of the candidate output values as the target output value. For a solution that performs multiple lookup operations simultaneously with a preset parallelism, multiple groups of candidate output values can be selected simultaneously to obtain corresponding multiple target output values. Finally, a data reconstruction module can collect and rearrange the multiple target output values, thereby reconstructing the target output values into data in a specific output format required by the AI chip, and caching or outputting them to the off-chip storage module DDR.

[0062] Further optionally, before simultaneously acquiring the multiple input data according to the preset parallelism and respectively determining the multiple target input values, it also includes: determining the preset parallelism according to the size of the original RAM storage space. Specifically, different lookup table parallelisms can be implemented according to the different sizes of the original RAM storage space for caching input data set when the AI chip is designed. Exemplarily, assuming that the size of the original RAM storage space can allocate 0.5MB of RAM space as the lookup table RAM storage space, at this time, 0.5MB / 128KB=4 lookup tables (int16 type) can be stored in the lookup table RAM storage space, then four copies of the same lookup table data can be stored at the same time during the above-mentioned lookup table storage operation, and then four operations can be performed simultaneously each time when performing the table lookup operation, that is, a table lookup operation with a preset parallelism of 4 is implemented.

[0063] The technical solution provided by the embodiment of the present invention first stores the lookup table corresponding to the standby activation function in the AI chip in advance, wherein the standby activation function is approximated by a piecewise function, and the input value and the corresponding output value of the lookup table can be determined based on the piecewise function, and then the input data of the standby activation function is obtained, and the target input value of the lookup table is determined based on the input data, and then the table is looked up based on the target input value to determine the corresponding target output value in the lookup table, thereby determining the output data of the standby activation function based on the target output value. By using a piecewise function to approximate the standby activation function and generating a corresponding lookup table, when the standby activation function is used for calculation, the calculation result is determined by looking up the table, and the function of the neural network activation function is realized simply and quickly, avoiding multiple calculations and storage, thereby improving the performance of the AI chip and saving power consumption.

[0064] Embodiment 2

[0065] Figure 2 This is a schematic diagram of the implementation structure of the neural network activation function provided in the second embodiment of the present invention. This structure can generally be integrated into an AI chip, and the implementation method of the neural network activation function provided in any embodiment of the present invention can be applied, and it has the corresponding functional modules and beneficial effects of the execution method. Figure 2As shown in the figure, the structure includes: a data RAM storage space 21, an address conversion module 22, a lookup table RAM storage space 23, a selector 24, and a data reconstruction module 25. Among them, the data RAM storage space 21 is used to store the input data of the activation function to be used. The address conversion module 22 is used to take out the input data from the data RAM storage space 21, convert the input data into the target input value of the lookup table corresponding to the activation function to be used, and copy the target input value into the target number of copies and then give it to the lookup table RAM storage space 23. The lookup table RAM storage space 23 is a banked structure, and the number of banks is the target number of copies, which is used to store the lookup table, and read each bank using the target input value respectively to obtain multiple candidate output values at the same time and give them to the selector 24. The address conversion module 22 is also used to convert the input data into an output selection signal and give the output selection signal to the selector 24. The selector 24 is used to determine the target output value from each of the candidate output values according to the output selection signal and give the target output value to the data reconstruction module 25. The data reconstruction module 25 is used to collect and cache the target output value and reconstruct it into the data format required by the AI chip to obtain the output data of the activation function to be used.

[0066] Optionally, the lookup table RAM storage space 23 is specifically used to store multiple copies of the replicated lookup table. Correspondingly, the address conversion module 22 is specifically used to receive multiple input data according to the preset parallelism at the same time, convert them into multiple target input values respectively and then give them to the lookup table RAM storage space 23, and convert them into multiple output selection signals respectively and then give them to the selector 24. Correspondingly, the lookup table RAM storage space 23 is specifically used to perform lookups in different lookup tables simultaneously according to multiple target input values to obtain the candidate output values corresponding to each target input value and give them to the selector 24. Correspondingly, the selector 24 is specifically used to determine the corresponding target output value from the corresponding candidate output values according to each output selection signal.

[0067] Further optionally, the address conversion module 22 is specifically used to generate the read address of the data RAM storage space 21 through a general address generation unit and take out each input data from the data RAM storage space 21 according to the execution order of the activation function to be used.

[0068] The specific process can refer to the description in the above embodiments and will not be elaborated here.

[0069] The technical solution provided in the embodiment of the present invention uses a lookup table corresponding to the stand-by activation function to determine the calculation result by looking up the table when the stand-by activation function is used for calculation, thereby simply and quickly implementing the function of the neural network activation function, avoiding multiple calculations and storage, thereby improving the performance of the AI chip and saving power consumption.

[0070] Embodiment 3

[0071] Figure 3 The schematic diagram of the structure of the computer device provided for the third embodiment of the present invention shows a block diagram of an exemplary computer device suitable for implementing the implementation mode of the present invention. Figure 3 The computer device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. Figure 3 As shown, the computer device includes a processor 31, a memory 32, an input device 33 and an output device 34; the number of processors 31 in the computer device can be one or more. Figure 3 Taking a processor 31 as an example, the processor 31, memory 32, input device 33 and output device 34 in the computer device can be connected through a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0072] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the implementation method of the neural network activation function in the embodiment of the present invention. The processor 31 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 32, that is, implements the implementation method of the neural network activation function described above.

[0073] The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 32 may further include a memory remotely arranged relative to the processor 31, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0074] The input device 33 can be used to obtain the input data and look-up table data of the activation function to be used, and generate key signal inputs related to the user settings and function control of the computer device, etc. The output device 34 can be used to transmit the calculated output data, etc. to the outside.

[0075] Embodiment 4

[0076] Embodiment 4 of the present invention further provides a storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute a method for implementing a neural network activation function when executed by a computer processor. The method includes:

[0077] Pre-store a look-up table corresponding to the activation function to be used in the AI chip. The activation function to be used is approximated by a piecewise function, and the input values and corresponding output values of the look-up table are determined according to the piecewise function;

[0078] Obtain the input data of the activation function to be used, and determine the target input value of the look-up table according to the input data;

[0079] Perform a look-up according to the target input value to obtain the corresponding target output value, and determine the output data of the activation function to be used according to the target output value.

[0080] The storage medium can be any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media such as CD-ROM, floppy disk or magnetic tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. The storage medium can also include other types of memory or combinations thereof. Additionally, the storage medium can be located in the computer system in which the program is executed, or can be located in a different second computer system, and the second computer system is connected to the computer system through a network (such as the Internet). The second computer system can provide program instructions to the computer for execution. The term "storage medium" can include two or more storage media that can reside in different locations (such as in different computer systems connected through a network). The storage medium can store program instructions (such as specifically implemented as a computer program) that can be executed by one or more processors.

[0081] Of course, for a storage medium containing computer-executable instructions provided by the embodiments of the present invention, the computer-executable instructions are not limited to the method operations as described above, and can also execute the related operations in the method for implementing a neural network activation function provided by any embodiment of the present invention.

[0082] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0083] The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination of the above.

[0084] From the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software and the necessary general-purpose hardware. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as a floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk, or optical disc of a computer, including several instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0085] Note that the above is only the preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described here. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, it may include more other equivalent embodiments, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for implementing a neural network activation function, characterized in that Including: Storing in advance a lookup table corresponding to a to-be-activated function in an AI chip, approximating the to-be-activated function with a piecewise function, and determining input values and corresponding output values of the lookup table according to the piecewise function; Obtaining input data of the to-be-activated function, and determining a target input value of the lookup table according to the input data; Looking up the table according to the target input value to obtain a corresponding target output value, and determining output data of the to-be-activated function according to the target output value; A plurality of copies of the lookup table are stored separately; Correspondingly, obtaining the input data of the to-be-activated function, and determining the target input value of the lookup table according to the input data includes: Obtaining a plurality of the input data simultaneously according to a preset parallelism degree, and respectively determining a plurality of the target input values; Correspondingly, looking up the table according to the target input value to obtain a corresponding target output value includes: Looking up the table in different lookup tables simultaneously and concurrently according to a plurality of the target input values to obtain respective corresponding target output values.

2. The implementation method of the neural network activation function according to claim 1, wherein, Storing in advance the lookup table corresponding to the to-be-activated function in the AI chip includes: Reusing an original RAM storage space of the AI chip, where the original RAM storage space is designed to store the input data and the output data, splitting the original RAM storage space into a data RAM storage space and a lookup table RAM storage space, and storing the lookup table in the lookup table RAM storage space; Correspondingly, before obtaining the input data of the to-be-activated function, further including: Caching the input data into the data RAM storage space.

3. The implementation method of the neural network activation function according to claim 2, wherein The data format of the input data and the lookup table is int16, the lookup table RAM storage space has a banked structure, and determining the target input value of the lookup table according to the input data includes: Taking a lower segment part of the input data as the target input value; Looking up the table according to the target input value to obtain a corresponding target output value includes: Copying the target input value according to the number of banks of the lookup table RAM storage space, and respectively reading each bank of the lookup table RAM storage space using the target input value to simultaneously obtain a plurality of candidate output values; Taking a higher segment part of the input data as an output selection signal, and determining the target output value from among the plurality of candidate output values according to the output selection signal.

4. The implementation method of the neural network activation function according to claim 2, wherein Before obtaining a plurality of the input data simultaneously according to the preset parallelism degree and respectively determining a plurality of the target input values, further including: Determining the preset parallelism degree according to the size of the original RAM storage space.

5. An implementation structure of a neural network activation function, applying the implementation method of the neural network activation function as described in any one of claims 1-4, characterized in that Including: A data RAM storage space, an address conversion module, a lookup table RAM storage space, a selector, and a data reconstruction module; wherein, The data RAM storage space is used to store input data of the to-be-activated function; The address conversion module is used to take out the input data from the data RAM storage space, convert the input data into the target input value of the lookup table corresponding to the standby activation function, and copy the target input value into the target number of copies and give it to the lookup table RAM storage space; The RAM storage space of the lookup table is a bank structure, and the number of banks is the target number of copies, for storing the lookup table, and reading each bank using the target input value respectively, so as to simultaneously obtain multiple output values to be selected and give them to the selector; The address conversion module is also used to convert the input data into an output selection signal, and give the output selection signal to the selector; The selector is used to determine a target output value among the output values to be selected according to the output selection signal, and provide the target output value to the data reconstruction module; The data reconstruction module is used to collect and cache the target output value and reconstruct it into the data format required by the AI chip to obtain the output data of the stand-by activation function.

6. The implementation structure of the neural network activation function according to claim 5, wherein The lookup table RAM storage space is specifically used to store multiple copies of the lookup table; Accordingly, the address conversion module is specifically used to simultaneously receive a plurality of the input data according to a preset parallelism, and convert them into a plurality of the target input values and then send them to the lookup table RAM storage space, and convert them into a plurality of the output selection signals and then send them to the selector; Correspondingly, the lookup table RAM storage space is specifically used to perform table lookups in different lookup tables simultaneously and concurrently according to the multiple target input values, so as to obtain the output value to be selected corresponding to each target input value and provide it to the selector; Correspondingly, the selector is specifically configured to determine the corresponding target output value from the corresponding output values to be selected according to each of the output selection signals.

7. The implementation structure of the neural network activation function according to claim 6, wherein The address conversion module is specifically used to generate a read address of the data RAM storage space through a universal address generation unit, and to fetch each input data from the data RAM storage space according to the execution order of the standby activation function.

8. A computer device, characterized in that, include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for implementing the neural network activation function as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, the method for implementing a neural network activation function as described in any one of claims 1 to 4 is implemented.

Images