A hardware implementation method and device of a GAN network, a storage medium and a terminal

Abstract

The application discloses a kind of hardware implementation method, device, storage medium and terminal of GAN network, method includes: training GAN network obtains the floating-point number model after convergence, and exports the floating-point number model after convergence;According to the network parameter of GAN network, floating-point number model is converted into block floating-point model;The block floating-point convolution structure of block floating-point model is deployed on hardware, and convolution operation is carried out based on the block floating-point convolution structure after deployment.The hardware implementation method provided in the application has the advantages of simple implementation steps, less network parameter precision loss, low operation complexity and significantly reduced storage unit requirements, solves the problem that existing generative adversarial network has large resource overhead and slow inference speed when deployed on hardware.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of integrated circuit design, and in particular to a hardware implementation method and device of a GAN network, a storage medium and a terminal. BACKGROUND

[0002] In recent years, with the research process of artificial intelligence and convolutional neural networks, its functions are more and more developed, and the performance is more and more excellent. However, the problem of large neural network parameter quantity and high computational complexity seriously affects the practical application and development of neural networks in the market. The quantization method of integer arithmetic inference provides a quantization framework for the hardware deployment of convolutional neural networks. By quantizing the 32-bit floating-point representation of the weights and excitations of the neural network to 8-bit fixed points in a non-linear manner, the quantization of the network parameters is realized without retraining, and good results are achieved.

[0003] At present, the generative adversarial network (GAN) is a neural network model that has been widely studied in recent years. Through the mutual game between the generative model (Generative Model) and the discriminative model (Discriminative Model) during training, unsupervised learning or semi-supervised learning is realized. The generative adversarial network has developed from the most basic GAN, CGAN and DGAN network to more complex CycleGAN and DiscoGAN network, and has been widely studied in the direction of image generation, semantic segmentation, data enhancement, etc. and has achieved new results. The generative adversarial network contains a large number of convolution operations and U-shaped operation structures, so additional hardware acceleration module design is needed. SUMMARY

[0004] The embodiments of the present application provide a hardware implementation method, device, storage medium and terminal of a GAN network. In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not a general review, nor does it determine the key / important components or delineate the protection scope of these embodiments. Its only purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0005] In a first aspect, the embodiments of the present application provide a hardware implementation method of a GAN network, the method comprising:

[0006] training a GAN network to obtain a converged floating-point model, and exporting the converged floating-point model;

[0007] According to the network parameters of the GAN network, the floating-point number model is converted into a block floating-point model;

[0008] The block floating-point convolution structure of the block floating-point model is deployed on hardware, and a convolution operation is performed based on the deployed block floating-point convolution structure.

[0009] Optionally, the network parameters of the GAN network include weight parameters of the network and activation parameters of the network;

[0010] According to the network parameters of the GAN network, the floating-point number model is converted into a block floating-point model, comprising:

[0011] According to the weight parameters of the network, a first number of blocks are divided from the floating-point number model;

[0012] According to the activation parameters of the network, a second number of blocks are divided from the floating-point number model;

[0013] The shared exponent of each block in the first number of blocks and the second number of blocks is determined;

[0014] A shared exponent lookup table is generated based on the shared exponent of each block, and a block floating-point model is obtained.

[0015] Optionally, according to the weight parameters of the network, a first number of blocks are divided from the floating-point number model, comprising:

[0016] Each weight parameter existing in the floating-point number model is obtained;

[0017] Each weight parameter corresponds to a network layer, and the first number of blocks are generated by dividing the network layer into a block.

[0018] Optionally, according to the activation parameters of the network, a second number of blocks are divided from the floating-point number model, comprising:

[0019] Each activation parameter existing in the floating-point number model is obtained;

[0020] It is judged whether each activation parameter has a plurality of network layers associated therewith;

[0021] If not, each activation parameter corresponds to a network layer, and the second number of blocks is generated by dividing the network layer into a block;

[0022] Or,

[0023] If yes, each activation parameter corresponds to a network layer and a plurality of network layers associated therewith, and the second number of blocks is generated by dividing the network layer and the plurality of network layers into a block.

[0024] Optionally, the shared exponent of each block in the first number of blocks and the second number of blocks is determined, comprising:

[0025] The exponent set of each block in the first number of blocks and the second number of blocks is obtained;

[0026] arranging each index in the index set of each block in ascending order to obtain a sorted index of each block;

[0027] determining a maximum index at the end of the sorted index of each block as a shared index of each block.

[0028] Optionally, the convolution operation is performed based on the deployed block floating point convolution structure, including:

[0029] determining a first network layer of the convolution operation;

[0030] receiving an activation mantissa and a weight mantissa of the first network layer;

[0031] performing a fixed-point convolution and layer accumulation operation on the activation mantissa and the weight mantissa to obtain a convolution result;

[0032] obtaining an input activation index, an input weight index and an output activation index of a target network layer according to the shared index lookup table;

[0033] calculating a mantissa offset value according to the input activation index, the input weight index and the output activation index;

[0034] inputting the mantissa offset value into a shift register after combining the mantissa offset value with an activation function layer of the GAN network and shifting the convolution result in the shift register to obtain a shifted result;

[0035] outputting the shifted result and determining the shifted result as an input of a next network layer.

[0036] Optionally, the mantissa offset value is combined with the activation function layer of the GAN network and shifted in the shift register after the convolution result is inputted into the shift register to obtain a shifted result, including:

[0037] combining the mantissa offset value with the activation function layer of the GAN network to obtain a combined mantissa offset value;

[0038] when the sign bit of the convolution result is positive, shifting based on the combined mantissa offset value;

[0039] or,

[0040] when the sign bit of the convolution result is negative, determining 0 as the shifted result.

[0041] In a second aspect, an embodiment of the present application provides a hardware implementation device of a GAN network, the device including:

[0042] a model training and exporting module configured to train a GAN network to obtain a converged floating point number model and export the converged floating point number model;

[0043] The model conversion module is configured to convert the floating-point number model into a block floating-point model according to network parameters of the GAN network.

[0044] The model deployment operation module is configured to deploy a block floating-point convolution structure of the block floating-point model on hardware, and perform convolution operation based on the deployed block floating-point convolution structure.

[0045] In a third aspect, the embodiments of the present application provide a computer storage medium, which stores a plurality of instructions. The instructions are suitable for being loaded by a processor and performing the method steps described above.

[0046] In a fourth aspect, the embodiments of the present application provide a terminal, which can include a processor and a memory. The memory stores a computer program, which is suitable for being loaded by the processor and performing the method steps described above.

[0047] The technical solutions provided by the embodiments of the present application can have the following beneficial effects.

[0048] In the embodiments of the present application, the hardware implementation device of the GAN network first trains the GAN network to obtain a converged floating-point number model, and exports the converged floating-point number model. Then, the floating-point number model is converted into a block floating-point model according to network parameters of the GAN network. Finally, a block floating-point convolution structure of the block floating-point model is deployed on hardware, and convolution operation is performed based on the deployed block floating-point convolution structure. Since the floating-point number model is converted into the block floating-point model, a group of block floating-point format data only contains a shared exponent and a fixed-point mantissa of each number. Compared with the floating-point number format data, it requires less storage space and has lower operation complexity. Therefore, the hardware implementation method provided by the present application has the advantages of simple implementation steps, small network parameter precision loss, low operation complexity, and significantly reduced storage unit requirements, and solves the problems of large resource consumption and slow inference speed of the existing generative adversarial network in hardware deployment.

[0049] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0051] Figure 1 is a flowchart of a GAN network hardware implementation method provided by the embodiments of the present application;

[0052] Figure 2 is a schematic diagram of a U-shaped network structure in a generative adversarial network provided by the embodiments of the present application;

[0053] Figure 3 is a schematic diagram of block division according to an activation parameter provided by an embodiment of the present application;

[0054] Figure 4 is a schematic diagram of a convolution operation process provided by an embodiment of the present application;

[0055] Figure 5 is a schematic diagram of a convolution operation provided by an embodiment of the present application;

[0056] Figure 6 is a schematic diagram of a structure of a hardware implementation device of a GAN network provided by an embodiment of the present application;

[0057] Figure 7 is a schematic diagram of a structure of a terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following description and drawings are illustrative of the specific embodiments of the present application and are not intended to limit the generality of the present application.

[0059] It should be noted that the described embodiments are merely some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0060] The following description refers to the accompanying drawings. Unless otherwise indicated, same numbers in different drawings indicate same or similar elements. The implementations described in the following example embodiments are not meant to represent all implementations consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0061] In the description of the present application, it should be understood that the terms "first", "second", etc. are used only for the purpose of description, and cannot be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" means two or more. "And / or", which describes the relationship between the associated objects, means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The character " / " generally represents that the associated objects before and after are in an "or" relationship.

[0062] The application provides a hardware implementation method, device, storage medium and terminal of a GAN network to solve the problems in the related art. In the technical scheme provided by the application, the floating-point model is converted into a block floating-point model, so that a group of data in block floating-point format only contains a shared exponent and a fixed-point mantissa of each number, which requires less storage space and has lower operation complexity than data in floating-point format. Therefore, the hardware implementation method provided by the application has the advantages of simple implementation steps, small loss of network parameter precision, low operation complexity and significantly reduced requirement for storage units, solves the problems of large resource consumption and slow inference speed of the existing generative adversarial network in hardware deployment, and will be described in detail below by way of exemplary embodiments.

[0063] The application will be described in detail below with reference to the accompanying drawings. Figure 1 -Appendix Figure 5 The hardware implementation method of the GAN network provided by the embodiment of the application will be described in detail. The method can be implemented by relying on a computer program and can run on a hardware implementation device of the GAN network based on the von Neumann architecture. The computer program can be integrated in an application or can run as an independent tool application.

[0064] Please refer to Figure 1 A flowchart of the hardware implementation method of the GAN network provided by the embodiment of the application is provided. As shown in the figure, Figure 1 The method of the embodiment of the application can include the following steps:

[0065] S101, training the GAN network to obtain a converged floating-point model, and exporting the converged floating-point model;

[0066] In the embodiment of the application, the GAN network is first created, then the GAN network is trained by using training data, and a loss value is calculated according to a loss function. When the loss value reaches a preset threshold, a converged floating-point model is generated, or when the loss value does not reach the preset threshold, the loss value is back-propagated to update the model parameters, and the training is continued to converge the model parameters. After the training is completed, the converged floating-point model can be exported.

[0067] Specifically, the convergence process is implemented based on the pytorch platform, the Adam optimizer is used, the learning rate is 0.0002, the batch size is 1, and the number of training rounds is 200. The export process exports the GAN network generator model through the pytorch platform.

[0068] S102, converting the floating-point model into a block floating-point model according to the network parameters of the GAN network;

[0069] Among them, the block floating point (Block Floating) is a data format opposite to the floating point number and the fixed point number. If a group of floating point numbers is converted into a block floating point number, a shared exponent needs to be selected as the shared exponent of all data in the data block, and then each number in the block is shifted and aligned according to the difference between the shared exponent and the shared exponent of itself.

[0070] In the embodiments of the present application, the U-shaped network structure widely exists in the generative adversarial network, for example Figure 2 As shown in the figure, the structure is embodied as the output of the down-sampling convolution layer in the network not only serves as the input of the next layer, but also serves as the input of the up-sampling transposed convolution layer after several layers. The quantized generative adversarial network in the method quantizes the weights and activations of the network into the block floating point format.

[0071] Specifically, the network parameters of the GAN network include the weight parameters of the network and the activation parameters of the network.

[0072] Further, when the weights and activations of the network are quantized into the block floating point format, first, a first number of blocks are divided from the floating point number model according to the weight parameters of the network, and then a second number of blocks are divided from the floating point number model according to the activation parameters of the network, second, the shared exponent of each block in the first number of blocks and the second number of blocks is determined, and finally, a shared exponent lookup table is generated based on the shared exponent of each block to obtain a block floating point model.

[0073] Specifically, when the first number of blocks are divided from the floating point number model according to the weight parameters of the network, first, each weight parameter existing in the floating point number model is obtained, and then each weight parameter is divided into a block corresponding to the network layer to generate the first number of blocks.

[0074] Specifically, when the second number of blocks are divided from the floating point number model according to the activation parameters of the network, first, each activation parameter existing in the floating point number model is obtained, and then it is judged whether each activation parameter has a plurality of network layers associated therewith; if not, each activation parameter is divided into a block corresponding to the network layer to generate the second number of blocks; or, if yes, each activation parameter is divided into a block corresponding to the network layer and the plurality of network layers associated therewith to generate the second number of blocks.

[0075] Specifically, when the shared exponent of each block in the first number of blocks and the second number of blocks is determined, first, the exponent set of each block in the first number of blocks and the second number of blocks is obtained, then the exponents in each block are arranged in ascending order to obtain the sorted exponents of each block, and finally, the last largest exponent in the sorted exponents of each block is determined as the shared exponent of each block.

[0076] For example, for the weights of the network, because the convolution operation of each layer of the network requires complete weight parameters, the weights of each layer are taken as a block to avoid the need for alignment of the mantissa during the convolution process. For the activations of the network, considering the U-shaped structure commonly used in the generative adversarial network, the blocking manner of the activations can be divided into two categories: (1) if the result of a certain layer of the network is output only to the next layer, the activation of the next layer is taken as a block; (2) if the result is output to more than one layer, the convolution layers are taken together as a data block. As shown in FIG. 8, the output of the convolution layer A is diffused to the convolution layer B and the convolution layer C, so the activations of the convolution layer B and the convolution layer C are taken together as a block, sharing the same shared exponent. The design of the finally implemented block floating-point blocking structure fully takes into account the characteristics of the U-shaped structure of the generative adversarial network, avoiding the use of multiple shared exponents for the activations of the same layer of the network, thereby consuming additional hardware resources for the alignment of the mantissas between the activations during the convolution operation. Figure 3

[0077] S103, deploy the block floating-point convolution structure of the block floating-point model on the hardware, and perform convolution operation based on the deployed block floating-point convolution structure.

[0078] In the embodiments of the present application, after obtaining the block floating-point model, the block floating-point convolution structure of the block floating-point model can be deployed on the hardware, and convolution operation can be performed based on the deployed block floating-point convolution structure.

[0079] Specifically, when performing convolution operation based on the deployed block floating-point convolution structure, first, the first network layer of the convolution operation is determined, then the activation mantissa and the weight mantissa of the first network layer are received, and then the activation mantissa and the weight mantissa are subjected to fixed-point convolution and layer accumulation operation to obtain a convolution result, and then the input activation exponent, the input weight exponent and the output activation exponent of the target network layer are obtained according to the shared exponent lookup table, and then the mantissa offset value is calculated according to the input activation exponent, the input weight exponent and the output activation exponent, and then the mantissa offset value is combined with the activation function layer of the GAN network and input into the shift register for shifting to obtain a shifted result, and finally the shifted result is output and determined as the input of the next network layer.

[0080] It should be noted that the traditional network needs to design a separate ReLU module, and the present application can be completed together with the shift module by combining the mantissa offset value with the activation function layer of the GAN network, thereby improving the speed of the final hardware.

[0081] ​Further, after combining the mantissa offset value with the activation function layer of the GAN network and inputting the convolution result into the shift register for shifting, a shifted result is obtained. First, the mantissa offset value is combined with the activation function layer of the GAN network to obtain a combined mantissa offset value. When the sign bit of the convolution result is positive, the combined mantissa offset value is used for shifting. Or, when the sign bit of the convolution result is negative, 0 is determined as the shifted result.

[0082] It should be noted that the generated adversarial network convolution operation accelerator designed by the method can obtain the shared exponent of the input data block and the output data block by using the lookup table, so that the offset of the mantissa can be obtained only by fixed-point operation, without the need for frequent conversion between block floating-point numbers and floating-point numbers in hardware. At the same time, the commonly used activation function ReLU of the generated adversarial network can be integrated into the module for determining the mantissa offset by using a two-input selector, without consuming additional operation steps, thereby improving the operation speed of the convolution layer and the transposed convolution layer. Compared with the original fixed-point convolution accelerator, the block floating-point convolution accelerator implemented by the method does not need to increase the additional operation resource overhead in the serial path of the convolution operation, and fully guarantees the accuracy of the network, which is suitable for the hardware implementation of the generated adversarial network.

[0083] For example Figure 4 As shown in the convolution operation flowchart of one layer, the mantissa part of the activation and the weight is first subjected to general fixed-point convolution and layer accumulation operation (the right part of the figure). The left part of the figure is responsible for calculating the shift value required when aligning to the next block after each layer of convolution operation (shift value = input activation exponent + input weight exponent - output activation exponent) according to the shared exponent lookup table, and combining with the ReLU layer of the GAN network. If the sign bit of the convolution result is positive, the shift value calculated by the left module is used for shifting, and if it is negative, 0 is directly taken.

[0084] Further, the convolution operation flowchart is, for example Figure 5 As shown in the figure, it is a convolution operation, the left is a 5x5 input activation, the middle is a 3x3 input weight, and the right is a 3x3 output result. After multiplication of each point of the activation and the weight and then addition, the result of one convolution is obtained (the dark part refers to one convolution operation). The input weight needs to be slid on the input activation graph to cover the entire input activation graph. Layer accumulation is that there are many input activation graphs in the network, and each graph needs to be convolved with the weight. Each convolution result of the activation is accumulated, and the accumulated result is output to the next layer as the activation parameter.

[0085] The application utilizes software to pre-store block floating point shared exponents as a lookup table to accelerate the convolution operation speed of generative adversarial network inference in hardware deployment; meanwhile, the block floating point partitioning mode is optimized for the common U-shaped structure of the generative adversarial network to avoid additional mantissa alignment operations. The final performance test results show that the hardware implementation method has the advantages of simple implementation steps, small network parameter precision loss, low operation complexity and significantly reduced storage unit requirements, and solves the problems of large resource consumption and slow inference speed of the existing generative adversarial network in hardware deployment.

[0086] In the embodiment of the application, the hardware implementation device of the GAN network first trains the GAN network to obtain a converged floating point model, and exports the converged floating point model, then converts the floating point model into a block floating point model according to the network parameters of the GAN network, and finally deploys the block floating point convolution structure of the block floating point model on hardware and performs convolution operation based on the deployed block floating point convolution structure. Since the application converts the floating point model into a block floating point model, a group of block floating point format data only contains a shared exponent and a fixed point mantissa of each number, which requires less storage space and has lower operation complexity compared to floating point format data. Therefore, the hardware implementation method provided by the application has the advantages of simple implementation steps, small network parameter precision loss, low operation complexity and significantly reduced storage unit requirements, and solves the problems of large resource consumption and slow inference speed of the existing generative adversarial network in hardware deployment.

[0087] The following is an embodiment of the device of the application, which can be used to execute the method embodiment of the application. For details not disclosed in the device embodiment of the application, please refer to the method embodiment of the application.

[0088] Please refer to Figure 6 which shows the structure diagram of the hardware implementation device of the GAN network provided by an exemplary embodiment of the application. The hardware implementation device of the GAN network can be realized by software, hardware or a combination of the two to become all or part of the terminal. The device 1 includes a model training and exporting module 10, a model conversion module 20 and a model deployment and operation module 30.

[0089] The model training and exporting module 10 is used to train the GAN network to obtain a converged floating point model, and export the converged floating point model;

[0090] The model conversion module 20 is used to convert the floating point model into a block floating point model according to the network parameters of the GAN network;

[0091] The model deployment and operation module 30 is used to deploy the block floating point convolution structure of the block floating point model on hardware, and perform convolution operation based on the deployed block floating point convolution structure.

[0092] It should be noted that the hardware implementation device of the GAN network provided in the above embodiment is only used as an example to illustrate the division of the above functional modules when the hardware implementation method of the GAN network is executed. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the hardware implementation device of the GAN network provided in the above embodiment and the hardware implementation method of the GAN network belong to the same concept, which embodies the implementation process. Details refer to the method embodiment, which will not be repeated here.

[0093] The serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0094] In the embodiments of the present application, the hardware implementation device of the GAN network first trains the GAN network to obtain the converged floating point model, and exports the converged floating point model, then converts the floating point model into the block floating point model according to the network parameters of the GAN network, and finally deploys the block floating point convolution structure of the block floating point model on the hardware, and performs convolution operation based on the deployed block floating point convolution structure. Since the present application converts the floating point model into the block floating point model, a group of block floating point format data only contains a shared exponent and a fixed point mantissa of each number, which requires less storage space and has lower operation complexity compared with floating point format data. Therefore, the hardware implementation method provided by the present application has the advantages of simple implementation steps, small network parameter precision loss, low operation complexity and significantly reduced storage unit requirement, and solves the problems of large resource overhead and slow inference speed of the existing generative adversarial network in hardware deployment.

[0095] The present application also provides a computer readable medium having program instructions stored thereon, which, when executed by a processor, implement the hardware implementation method of the GAN network provided by each of the above method embodiments. The present application also provides a computer program product containing instructions, which, when executed on a computer, causes the computer to execute the hardware implementation method of the GAN network of each of the above method embodiments.

[0096] Please refer to Figure 7 The present application provides a terminal structure schematic diagram. As shown in Figure 7 The terminal 1000 can include at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.

[0097] The communication bus 1002 is used to realize the connection communication between the components.

[0098] The user interface 1003 can include a display screen, a camera, and optionally a standard wired interface and a wireless interface.

[0099] The network interface 1004 can optionally include a standard wired interface and a wireless interface (e.g., a WI-FI interface).

[0100] The processor 1001 can include one or more processing cores. The processor 1001 connects various parts of the electronic device 1000 through various interfaces and lines, executes various functions of the electronic device 1000 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 1001 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes operating systems, user interfaces, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 1001, but can be realized by a separate chip.

[0101] The memory 1005 can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1005 can include a program storage area and a data storage area, where the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above various method embodiments, etc.; the data storage area can store data involved in the above various method embodiments, etc. The memory 1005 can also be at least one storage device located away from the aforementioned processor 1001. As shown in Figure 7 The memory 1005 as a computer storage medium can include an operating system, a network communication module, a user interface module, and a hardware implementation application program of the GAN network.

[0102] In the terminal 1000 shown in Figure 7 In the terminal 1000 shown in

[0103] training the GAN network to obtain a converged floating point model, and exporting the converged floating point model;

[0104] converting the floating point model into a block floating point model according to network parameters of the GAN network;

[0105] deploying a block floating point convolution structure of the block floating point model on hardware, and performing convolution operation based on the deployed block floating point convolution structure.

[0106] In one embodiment, when the processor 1001 performs converting the floating point model into a block floating point model according to network parameters of the GAN network, it specifically performs the following operations:

[0107] dividing a first number of blocks from the floating point model according to weight parameters of the network;

[0108] dividing a second number of blocks from the floating point model according to activation parameters of the network;

[0109] determining a shared index of each block in the first number of blocks and the second number of blocks;

[0110] A shared index lookup table is generated based on the shared index of each block, and a block floating point model is obtained.

[0111] In one embodiment, the processor 1001, when performing the dividing of the first number of blocks from the floating point model according to the weight parameters of the network, specifically performs the following operations:

[0112] obtaining each weight parameter present in the floating point model;

[0113] dividing each weight parameter corresponding network layer into a block to generate the first number of blocks.

[0114] In one embodiment, the processor 1001, when performing the dividing of the second number of blocks from the floating point model according to the activation parameters of the network, specifically performs the following operations:

[0115] obtaining each activation parameter present in the floating point model;

[0116] determining whether each activation parameter has multiple network layers associated therewith;

[0117] if not, dividing each activation parameter corresponding network layer into a block to generate the second number of blocks;

[0118] or,

[0119] if yes, dividing each activation parameter corresponding network layer and the multiple network layers associated therewith into a block to generate the second number of blocks.

[0120] In one embodiment, the processor 1001, when performing the determining of the shared index of each block in the first number of blocks and the second number of blocks, specifically performs the following operations:

[0121] obtaining the index set of each block in the first number of blocks and the second number of blocks;

[0122] arranging each index in the index set of each block in ascending order to obtain the sorted index of each block;

[0123] determining the last largest index in the sorted index of each block as the shared index of each block.

[0124] In one embodiment, the processor 1001, when performing the convolution operation based on the deployed block floating point convolution structure, specifically performs the following operations:

[0125] determining a first network layer of the convolution operation;

[0126] receiving an activation mantissa and a weight mantissa of the first network layer;

[0127] performing fixed-point convolution and layer accumulation operations on the activation mantissa and the weight mantissa to obtain a convolution result;

[0128] According to the shared exponent lookup table, obtain an input activation exponent, an input weight exponent and an output activation exponent of the target network layer;

[0129] According to the input activation exponent, the input weight exponent and the output activation exponent, calculate a mantissa offset value;

[0130] Combine the mantissa offset value with an activation function layer of the GAN network, and input the combined result into a shift register to perform shifting, to obtain a shifted result;

[0131] Output the shifted result, and determine the shifted result as an input of a next network layer.

[0132] In the embodiments of the present application, the hardware implementation device of the GAN network first trains the GAN network to obtain a converged floating-point model, and exports the converged floating-point model, then converts the floating-point model into a block floating-point model according to the network parameters of the GAN network, and finally deploys the block floating-point convolution structure of the block floating-point model on the hardware, and performs convolution operation based on the deployed block floating-point convolution structure. Since the present application converts the floating-point model into the block floating-point model, a group of block floating-point format data only contains a shared exponent and a fixed-point mantissa of each number, which requires less storage space and has lower operation complexity compared to floating-point format data. Therefore, the hardware implementation method provided by the present application has the advantages of simple implementation steps, small network parameter precision loss, low operation complexity and significantly reduced storage unit requirements, and solves the problems of large resource overhead and slow inference speed of the existing generative adversarial network in hardware deployment.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program for hardware implementation of the GAN network can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory or a random access memory, etc.

[0134] The above only describes the preferred embodiments of the present application, and of course cannot limit the scope of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope of the present application.

Claims

1. A method for hardware implementation of a GAN network, the method comprising: The method comprises: training a GAN network to obtain a converged floating-point model, and exporting the converged floating-point model; converting the floating-point model into a block floating-point model according to network parameters of the GAN network; the network parameters of the GAN network comprise weight parameters of the network and activation parameters of the network; the conversion of the floating-point model into the block floating-point model according to the network parameters of the GAN network comprises: dividing a first number of blocks from the floating-point model according to the weight parameters of the network; dividing a second number of blocks from the floating-point model according to the activation parameters of the network; determining a shared exponent of each block in the first number of blocks and the second number of blocks; generating a shared exponent lookup table based on the shared exponent of each block to obtain a block floating-point model; deploying a block floating-point convolution structure of the block floating-point model on hardware, and performing convolution operation based on the deployed block floating-point convolution structure.

2. The method of claim 1, wherein, the division of the first number of blocks from the floating-point model according to the weight parameters of the network comprises: obtaining each weight parameter existing in the floating-point model; dividing a network layer corresponding to each weight parameter into a block to generate the first number of blocks.

3. The method of claim 1, wherein, the division of the second number of blocks from the floating-point model according to the activation parameters of the network comprises: obtaining each activation parameter existing in the floating-point model; judging whether a plurality of network layers associated with each activation parameter exist; if not, dividing a network layer corresponding to each activation parameter into a block to generate the second number of blocks; or, if yes, dividing a network layer corresponding to each activation parameter and the plurality of network layers associated therewith into a block to generate the second number of blocks.

4. The method of claim 1, wherein, the determination of the shared exponent of each block in the first number of blocks and the second number of blocks comprises: obtaining an exponent set of each block in the first number of blocks and the second number of blocks; arranging each exponent in the exponent set of each block in ascending order to obtain a sorted exponent of each block; determining a tailmost maximum exponent in the sorted exponent of each block as the shared exponent of each block.

5. The method of claim 1, wherein, the convolution operation based on the deployed block floating-point convolution structure comprises: determining a first network layer of convolution operation; receiving an activation mantissa and a weight mantissa of the first network layer; performing fixed-point convolution and layer accumulation operation on the activation mantissa and the weight mantissa to obtain a convolution result; obtaining an input activation exponent, an input weight exponent and an output activation exponent of the first network layer according to the shared exponent lookup table; calculating a mantissa offset value according to the input activation exponent, the input weight exponent and the output activation exponent; combining the mantissa offset value with an activation function layer of the GAN network, and inputting the combination into a shift register to perform shifting to obtain a shifted result; outputting the shifted result, and determining the shifted result as an input of a next network layer.

6. The method of claim 5, wherein, the combination of the mantissa offset value with the activation function layer of the GAN network, and the inputting of the combination into the shift register to perform shifting to obtain the shifted result comprises: Combine the mantissa offset value with an activation function layer of the GAN network to obtain a combined mantissa offset value; When the sign bit of the convolution result is positive, shift based on the combined mantissa offset value; Or, When the sign bit of the convolution result is negative, determine 0 as the shifted result. 7.A hardware-implemented apparatus of a GAN network implemented using the method of any one of claims 1-6. The device comprises: A model training derivation module configured to train a GAN network to obtain a converged floating point number model and derive the converged floating point number model; A model conversion module configured to convert the floating point number model into a block floating point model according to network parameters of the GAN network; A model deployment operation module configured to deploy a block floating point convolution structure of the block floating point model on hardware and perform convolution operation based on the deployed block floating point convolution structure.

8. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions, which are suitable for being loaded and executed by a processor to perform the method steps of any one of claims 1-6.

9. A terminal, characterized by comprising: Comprise: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded and executed by the processor to perform the method steps of any one of claims 1-6.

Images