Convolutional neural network acceleration device with security encryption and acceleration method

Abstract

The invention relates to a convolutional neural network acceleration device with security encryption and an acceleration method. The security processing unit stores a decryption key and encryption network parameters, the neural network acceleration unit comprises a convolution kernel unit, an activation function unit and a pooling module, the convolution kernel unit is used for carrying out convolution operation on decryption weight data and target input feature data, convolution operation result data is generated after the convolution operation, and the neural network acceleration unit is used for outputting the decryption weight data and the target input feature data. Performing an activation operation on the convolution operation result data by using an activation function unit so as to generate activation operation result data after the activation operation; and a pooling module is utilized to perform pooling operation on the activation operation result data so as to generate pooling result data after the pooling operation, and during the pooling operation, a decryption pooling core generated by decrypting the encryption pooling core by the security processing unit is utilized to perform pooling processing on the activation processing result data. According to the invention, the reasoning of the convolutional neural network can be accelerated, and the security of the convolutional neural network can be improved.

Description

Technical Field

[0001] The present invention relates to an acceleration device and an acceleration method, and in particular to a convolutional neural network acceleration device and an acceleration method with secure encryption. Background Art

[0002] As an important data classification tool, convolutional neural network (CNN) has a wide range of applications in information processing, pattern recognition, etc. A well-trained convolutional neural network can efficiently and accurately complete the classification and prediction of input data. At present, the research and training of neural network algorithms has become a hot topic in industry and academia. Therefore, a well-trained convolutional neural network has gradually become an important intellectual property of many data processing companies.

[0003] In neural network design, a high-performance convolutional neural network often requires more training resources, such as computing resources, human resources, and time resources used for network training. Therefore, training data sets, accelerators, network parameters, etc. are usually regarded as the intellectual property of convolutional neural networks and protected.

[0004] For specific recognition tasks, the weight parameters that are trained from massive amounts of data at a time-consuming cost are considered an important intellectual property. In particular, in areas involving national information security such as face recognition, the facial features trained by the terminal are also private and cannot be easily leaked.

[0005] In order to improve data security, encryption chips are generally used to protect data. Encryption chips are a general term for a type of security chip that integrates various symmetric and asymmetric algorithms internally and has an extremely high security level. It can ensure that the keys and information data stored internally will not be illegally read or tampered with.

[0006] For chip platform companies, how to protect the offline trained weight parameters to prevent them from being obtained by malware during the inference process is an important security consideration in the current industry chain cooperation. At the same time, the design of the pooling core is also very important for convolutional neural networks, so the pooling core also needs to be protected by encryption chips.

[0007] From the above description, it can be seen that for the trained convolutional neural network, how to effectively accelerate the reasoning of the convolutional neural network and achieve secure encryption during the accelerated reasoning process is a technical problem that urgently needs to be solved. Summary of the invention

[0008] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a convolutional neural network acceleration device and acceleration method with secure encryption, which can accelerate the reasoning of the convolutional neural network and improve the security of the convolutional neural network.

[0009] According to the technical solution provided by the present invention, a convolutional neural network acceleration device with secure encryption is provided, and the convolutional neural network acceleration device comprises:

[0010] A security processing unit, used to store a decryption key and an encrypted network parameter, wherein the encrypted network parameter is generated by encrypting the network parameter of the target convolutional neural network offline using the encryption key, the encryption key matches the decryption key, and the encrypted network parameter includes at least an encrypted weight parameter and an encrypted pooling core;

[0011] A neural network acceleration unit, used to perform convolution processing when the target convolutional neural network is working, at least includes a convolution kernel unit for performing convolution operations, an activation function unit for performing activation operations, and a pooling module for performing pooling operations, wherein:

[0012] The convolution kernel unit, activation function unit and pooling module are connected in sequence;

[0013] When performing convolution processing, the target input feature data and the decrypted weight data are loaded into the convolution core unit, so as to use the convolution core unit to perform convolution operation on the decrypted weight data and the target input feature data, and generate convolution operation result data after the convolution operation, wherein the target input feature data is generated by the security processing unit after decrypting the input feature source data, and the decrypted weight data is generated by the security processing unit after decrypting the encrypted weight parameter;

[0014] Performing an activation operation on the convolution operation result data using an activation function unit to generate activation operation result data after the activation operation;

[0015] The pooling module is used to perform a pooling operation on the activation operation result data to generate pooling result data after the pooling operation, wherein during the pooling operation, the activation processing result data is pooled using a decrypted pooling core generated by decrypting the encrypted pooling core by the security processing unit.

[0016] The convolution kernel unit includes a serialization module, a multiplication and accumulation module and an accumulator module connected in sequence, wherein:

[0017] During the convolution operation, the serialization module is used to rearrange the target input feature data and the decryption weight data respectively, where:

[0018] After rearranging the target input feature data, an input feature sub-target data group is generated, wherein the input feature sub-target data group includes a plurality of input feature sub-target data arranged in time sequence;

[0019] After rearranging the decryption weight data, a decryption weight sub-target data group is generated, wherein the decryption weight sub-target data group includes a plurality of decryption weight sub-target data arranged in time sequence;

[0020] Based on the corresponding time sequence characteristics of the input feature sub-target data and the decryption weight sub-target data, a multiplication and addition operation is performed using a multiplication and accumulator module to generate corresponding partial and result data after the multiplication and addition operation;

[0021] The accumulator module is used to perform accumulation operation on the partial sum result data output by the multiplication and accumulation module to generate convolution operation result data after the accumulation operation.

[0022] When the serialization module rearranges the target input feature data and decryption weight data, it includes:

[0023] Based on the stripe operation of the multiply-accumulate module, the target input feature data and the decrypted weight data are polled to split the target input feature data into a plurality of feature data packets and split the decrypted weight data into a plurality of weight data packets after the polling;

[0024] The split feature data packets are assembled into corresponding input feature sub-target data, and the split weight data packets are assembled into corresponding decryption weight sub-target data, wherein,

[0025] Arrange the feature data packets according to the polling timing sequence, and assemble the feature data packets arranged according to the polling timing sequence based on the multiplication and addition calculation bit width in the multiplication and accumulator module, so as to generate a corresponding input feature sub-target data after assembling;

[0026] Arrange the weight data packets according to the polling timing sequence, and assemble the corresponding weight data packets arranged according to the polling timing sequence based on the multiplication and addition calculation bit width in the multiplication and accumulator module to generate a corresponding decryption weight sub-target data after assembling;

[0027] The assembled input feature sub-target data and decryption weight sub-target data are loaded into the multiplier-accumulator module in a pipeline manner.

[0028] The serialization module includes a convolution data loader, a weight data loader and a convolution sequence generator, wherein:

[0029] The convolution data loader polls the target input feature data and sends the feature data packets generated by the polling split to the feature FIFO unit of the convolution sequence generator;

[0030] The weight data loader polls the decrypted weight data and sends the weight data packets generated by the polling split to the weight FIFO unit of the convolution sequence generator;

[0031] The convolution sequence generator assembles the feature data packets in the feature FIFO unit to generate input feature sub-target data after assembling, and sends the generated input feature sub-target data to the convolution data loader, so as to load the input feature sub-target data into the multiplication and accumulator module one by one by using the convolution data loader;

[0032] The convolution sequence generator assembles the weight data packets in the weight FIFO unit to generate decrypted weight sub-target data after assembly, and sends the generated decrypted weight sub-target data to the weight data loader, so that the decrypted weight sub-target data can be loaded one by one into the multiplier accumulator module using the weight data loader.

[0033] When the accumulator module performs accumulation operation on the partial sum result data, the convolution operation result source data is generated;

[0034] The generated convolution operation result source data is subjected to data restriction processing to generate convolution operation result data after the data restriction processing, wherein the data restriction processing performed on the convolution operation result source data includes a round operation and / or a saturate operation.

[0035] The activation function unit includes an activation function cache module and an activation module, wherein:

[0036] When performing an activation operation, the activation module reads the activation function from the activation function cache module, and uses the read activation function to perform an activation operation on the convolution operation result data, and generates activation operation result data after the activation operation;

[0037] The activation module loads the generated activation operation result data into the pooling module.

[0038] The convolutional neural network acceleration device also includes a data storage unit for data storage and a DMA module adapted to be connected to the data storage unit, wherein:

[0039] The DMA module is also connected to the security processing unit, and is connected to the convolution kernel unit through the input feature data buffer module;

[0040] During convolution processing, the input feature source data in the data storage unit is moved to the input feature data buffer module through the DMA module to form the target input feature data;

[0041] The convolution core unit reads the target input feature data from the input feature data cache module, and receives the decryption weight data loaded by the security processing unit, so as to perform a convolution operation on the target input feature data using the decryption weight data.

[0042] The security processing unit includes an encrypted data cache module, a one-time programmable module and a decryption module, wherein:

[0043] For the encrypted network parameters generated by offline encryption, the encrypted network parameters are transferred to the encrypted data cache module through the DMA module, so as to use the encrypted data cache module to buffer the encrypted network parameters;

[0044] Using a one-time programmable module to store the decryption key, and the one-time programmable module is configured to be unreadable by software;

[0045] The decryption module is adapted to be connected with the encrypted data cache module and the one-time programmable module.

[0046] During convolution processing, the decryption module reads the encrypted network parameters from the encrypted data cache module and reads the encryption key from the one-time programmable module. Thereafter, the encrypted network parameters are decrypted using the read encryption key, so that decrypted weight data and decrypted pooling cores can be generated after the decryption operation.

[0047] The decryption module sends the decryption weight data generated by decryption to the convolution kernel unit, and sends the decryption pooling kernel generated by decryption to the pooling module.

[0048] It also includes a control module, a control interface module adapted to be connected to the control module, and a data interface module, wherein:

[0049] The control module is connected with the data storage unit through the data interface module;

[0050] The control module is adaptively connected to the security processing unit and the neural network acceleration unit through the control interface module to configure the working status of the security processing unit and the neural network acceleration unit through the control interface module.

[0051] A convolutional neural network acceleration method with secure encryption, for any convolutional neural network, using the convolutional neural network acceleration device described above to perform convolution processing.

[0052] The advantages of the present invention are as follows: the encrypted network parameters are cached by a security processing unit, and the required decryption weight data and decryption pooling core can be decrypted and generated; the convolution processing is performed by a neural network acceleration unit; the activation function cache module in the convolution neural network acceleration unit can cache the activation function; during the activation operation, the calling and reading of the activation function can be reduced, thereby improving the speed of the convolution processing; the target input feature data and the decryption weight data are rearranged by a serialization module, so that the convolution kernel unit can adapt to the convolution processing of data with different bit widths, thereby accelerating the reasoning of the convolution neural network and improving the security of the convolution neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1This is a main block diagram of an embodiment of a neural network acceleration device of the present invention.

[0054] Figure 2 This is a structural block diagram of an embodiment of a neural network acceleration device of the present invention.

[0055] Figure 3 This is an embodiment of the present invention in which a serialization module and a multiplier-accumulator module are connected and coordinated.

[0056] Figure 4 The figure is a schematic diagram of the working process of an embodiment of the neural network acceleration device of the present invention. DETAILED DESCRIPTION

[0057] The present invention will be further described below in conjunction with specific drawings and embodiments.

[0058] In order to accelerate the reasoning of a convolutional neural network and improve the security of the convolutional neural network, the present invention provides a convolutional neural network acceleration device with secure encryption. Specifically, the convolutional neural network acceleration device includes:

[0059] A security processing unit, used to store a decryption key and an encrypted network parameter, wherein the encrypted network parameter is generated by encrypting the network parameter of the target convolutional neural network offline using the encryption key, the encryption key matches the decryption key, and the encrypted network parameter includes at least an encrypted weight parameter and an encrypted pooling core;

[0060] A neural network acceleration unit, used to perform convolution processing when the target convolutional neural network is working, at least includes a convolution kernel unit for performing convolution operations, an activation function unit for performing activation operations, and a pooling module for performing pooling operations, wherein:

[0061] The convolution kernel unit, activation function unit and pooling module are connected in sequence;

[0062] When performing convolution processing, the target input feature data and the decrypted weight data are loaded into the convolution core unit, so as to use the convolution core unit to perform convolution operation on the decrypted weight data and the target input feature data, and generate convolution operation result data after the convolution operation, wherein the target input feature data is generated by the security processing unit after decrypting the input feature source data, and the decrypted weight data is generated by the security processing unit after decrypting the encrypted weight parameter;

[0063] Performing an activation operation on the convolution operation result data using an activation function unit to generate activation operation result data after the activation operation;

[0064] The pooling module is used to perform a pooling operation on the activation operation result data to generate pooling result data after the pooling operation, wherein during the pooling operation, the activation processing result data is pooled using a decrypted pooling core generated by decrypting the encrypted pooling core by the security processing unit.

[0065] Figure 1 An embodiment of a convolutional neural network acceleration device of the present invention is shown in the figure. It can be seen from the figure that when the convolutional neural network acceleration device includes at least a security processing unit and a neural network acceleration unit, the security processing unit can be used to improve the security of the target convolutional neural network, and the neural network acceleration unit can be used to accelerate the reasoning process of the target convolutional neural network, wherein the target convolutional neural network is a convolutional neural network that needs to be protected and the reasoning process needs to be accelerated. It can be seen from the above description that the target convolutional neural network should be a trained neural network that can perform relevant data classification. The training method of the target convolutional neural network and the method of using the target convolutional neural network to classify data can be selected according to actual needs and will not be listed here.

[0066] For the target convolutional neural network generated through training, the network parameters of the target convolutional neural network can be determined and obtained. Specifically, when using a security processing unit to improve the security of a convolutional neural network, at least the network parameters of the target convolutional neural network should be encrypted, and thereafter, the encrypted network parameters corresponding to the target convolutional neural network and the decryption key used to decrypt the encrypted network parameters are stored using the security processing unit. Generally, the network parameters of a convolutional neural network may include weight parameters and / or pooling kernels, wherein the weight parameters refer to the weight information of all feature layers in the convolutional neural network, and the feature layers here may be the convolution layers in the target convolutional neural network; the pooling kernel is the pooling parameter used by the pooling module, and the weight parameters and the pooling kernel may be consistent with the prior art.

[0067] In specific implementation, after determining the network parameters of the target convolutional neural network, the network parameters can be encrypted using an encryption key to generate encrypted network parameters after encryption. For example, when the network parameters include weight parameters, the weight parameters are encrypted using an encryption key to obtain the corresponding encrypted weight parameters; when the network parameters include pooling cores, the pooling cores are encrypted using an encryption key to obtain the corresponding encrypted pooling cores. When the target convolutional neural network includes a pooling module, the weight parameters and the pooling cores can be encrypted at the same time to further improve the security of the target convolutional neural network. The following is a specific description of the encrypted network parameters including encrypted weight parameters and encrypted pooling cores.

[0068] When it is necessary to use the neural network acceleration unit to accelerate the reasoning of the convolutional neural network, in order to meet the reasoning of the convolutional neural network, the security processing unit should use the decryption key to decrypt the encrypted network parameters. Specifically, after decrypting the encrypted weight parameters, the corresponding decrypted weight data can be generated, and after decrypting the encrypted pooling core, the corresponding decrypted pooling core can be generated; thereafter, the encrypted weight data and the decrypted pooling core must be loaded into the neural network acceleration unit.

[0069] It can be understood that when a convolutional neural network is inferred, at least convolution processing is performed; therefore, when a neural network acceleration unit is used to accelerate the inference of a target convolutional neural network, the neural network acceleration unit can be configured to perform convolution processing when the target convolutional neural network is working, that is, when the target convolutional neural network is inferred, at least convolution processing is included.

[0070] Figure 1 , an embodiment of a neural network acceleration unit when performing convolution processing of a target convolutional neural network is shown in the figure. In the figure, the neural network acceleration unit includes a convolution kernel unit, an activation function unit and a pooling module, wherein the convolution kernel unit is used to perform convolution operation, the activation function unit is used to perform activation operation and the pooling module is used to perform pooling operation. It can be seen that the convolution processing performed by the neural network acceleration unit includes convolution operation, activation operation and pooling operation. Specifically, when performing the activation operation, a post-processing operation is performed on a single data element (matrix minimum unit) in the convolution operation result data, and the network can learn and represent more complex functional relationships by performing nonlinear transformation on the input; after performing the pooling operation, the size of the parameter matrix can be effectively reduced, thereby reducing the number of parameters in the last connected layer.

[0071] It should be noted that when the convolution kernel unit performs a convolution operation, it needs to obtain the target input feature data and the decryption weight data. After that, the convolution operation is performed on the decryption weight data and the target input feature data, and the convolution operation result data is generated after the convolution operation. The type of convolution kernel unit and the method and process of performing the convolution operation will be described in detail below.

[0072] When performing the activation operation, specifically, the activation function unit activates the convolution operation result data, and after the activation operation, the activation operation result data can be generated. Of course, when the activation function unit performs the activation operation, the activation function should be used, that is, the required activation function should be cached in the activation function unit. Figure 2 An embodiment of the activation function unit is shown in FIG. 1 , in which the activation function unit includes an activation function cache module and an activation module, wherein:

[0073] When performing an activation operation, the activation module reads the activation function from the activation function cache module, and uses the read activation function to perform an activation operation on the convolution operation result data, and generates activation operation result data after the activation operation;

[0074] The activation module loads the generated activation operation result data into the pooling module.

[0075] Specifically, the activation function cache module is used to cache the activation function required for the target convolutional neural network reasoning. Therefore, the type of activation function cached in the activation function cache module is related to the target convolutional neural network executing data classification, etc. During the activation operation, the activation module can read the corresponding activation function from the activation function cache module. Thereafter, the activation function can be used to perform an activation operation on the convolution operation result data, and the activation module can generate the activation operation result data, and the activation operation result data is loaded into the pooling module. It can be understood that when the activation function cache module is used to cache the activation function, the repeated activation function calls can be reduced, and the memory overhead of the activation function calls can be further reduced, and the speed of executing the convolution processing can be improved. The way and process of the activation module using the activation function to perform the activation operation on the convolution operation result data can be consistent with the prior art, which will not be repeated here.

[0076] In specific implementation, when the pooling module performs a pooling operation, it should obtain a decryption pooling core from the security processing unit, and then use the decryption pooling core to perform a pooling operation on the activation operation result data and generate pooling result data.

[0077] It should be noted that after the pooling module generates the pooling result data, the neural network acceleration unit completes the acceleration of the convolution operation of the target convolutional neural network during reasoning. From the above description, it can be seen that the encrypted network parameters of the target convolutional neural network can be securely protected by the security processing unit. The security processing unit can cache the encrypted weight parameters and the encrypted pooling core at the same time, which can reduce the area occupied and reduce the cost. Using the activation function cache module to cache the activation function can further accelerate the convolution processing of the target convolutional neural network during reasoning.

[0078] In one embodiment of the present invention, the convolutional neural network acceleration device further includes a data storage unit for data storage and a DMA module adapted to be connected to the data storage unit, wherein:

[0079] The DMA module is also connected to the security processing unit, and is connected to the convolution kernel unit through the input feature data buffer module;

[0080] During convolution processing, the input feature source data in the data storage unit is moved to the input feature data buffer module through the DMA module to form the target input feature data;

[0081] The convolution kernel unit reads the target input feature data from the input feature data cache module, and receives the decryption weight data loaded by the security processing unit, so as to perform a convolution operation on the target input feature data using the decryption weight data.

[0082] Depend on Figure 1 and Figure 2 It can be seen that Figure 1 This is a main block diagram of the convolutional neural network acceleration device of the present invention. Figure 2 The complete block diagram of the convolutional neural network acceleration device of the present invention is shown in FIG. Figure 2 It can be seen from the embodiment shown in the figure that the convolutional neural network acceleration device also includes a data storage unit and a DMA module. Figure 2 In the data storage unit, DRAM is Figure 2 An embodiment in which the data storage unit adopts DRAM (dynamic random access memory) is shown. Of course, the data storage unit can also adopt other storage forms. The DMA (Direct Memory Access) module is connected to the data storage unit and the security processing unit, and is connected through the input feature data cache module.

[0083] From the above description, it can be seen that the encrypted network parameters are generated by offline encryption of the network parameters of the target convolutional neural network. In order to store the encrypted network parameters in the security processing unit, the encrypted network parameters generated by offline encryption can be first loaded into the data storage unit. Thereafter, the encrypted network parameters are moved to the security processing unit through DMA, so that the security processing unit can be used to store the encrypted network parameters and perform corresponding decryption operations. For the situation of the security processing unit, please refer to Figure 2 and the corresponding instructions below.

[0084] The target input feature data specifically refers to the data object to be inferred by the target convolutional neural network. In order to form the target input feature data loaded into the convolution kernel, the input feature source data can be first stored in the data storage unit, and then the DMA module is used to load the input feature source data into the input feature data cache module. The input feature source data stored in the input feature data cache module forms the target input feature data. After the target input feature data is in the input feature data cache module, it can be read by the convolution kernel unit for convolution processing.

[0085] In a specific implementation, when performing convolution processing, the convolution kernel reads the target input feature data from the input feature data cache module and obtains the decryption weight data from the security processing unit. Thereafter, the decryption weight data can be used to perform convolution operations on the target input feature data. In a specific implementation, the method of using the DMA module to transfer the encrypted network parameters to the security processing unit and the method of transferring the input feature source data to the input feature data cache module can be consistent with the prior art.

[0086] In one embodiment of the present invention, it further comprises a control module, a control interface module adapted to be connected to the control module, and a data interface module, wherein:

[0087] The control module is connected with the data storage unit through the data interface module;

[0088] The control module is adaptively connected to the security processing unit and the neural network acceleration unit through the control interface module to configure the working status of the security processing unit and the neural network acceleration unit through the control interface module.

[0089] Figure 2 In the embodiment shown in, a control module is also included. The control module can adopt the existing commonly used control form. The control module can realize the working state configuration of the entire neural network acceleration device. For example, the control module can configure the working state of the security processing unit and the neural network acceleration unit. Specifically, when configuring the working state of the security processing unit, the security processing unit can store the encrypted network parameters transported by the DMA module, and use the decryption key to decrypt the encrypted network parameters; configuring the working state of the neural network acceleration unit at least includes performing the above-mentioned convolution processing. Of course, the control module can also store the encrypted network parameters and the input feature source data in the data storage unit via the data interface module, and thereafter control the DMA module to transport the encrypted network parameters and the input feature source data.

[0090] During specific implementation, the data interface module and the control interface module can adopt existing commonly used interface types. For example, the data interface module can store encrypted network parameters and input feature source data in the data storage unit, and the control interface module can control the working status of the DMA module, security processing unit and neural network acceleration unit.

[0091] In one embodiment of the present invention, the security processing unit includes an encrypted data cache module, a one-time programmable module and a decryption module, wherein:

[0092] For the encrypted network parameters generated by offline encryption, the encrypted network parameters are transferred to the encrypted data cache module through the DMA module, so as to use the encrypted data cache module to buffer the encrypted network parameters;

[0093] Using a one-time programmable module to store the decryption key, and the one-time programmable module is configured to be unreadable by software;

[0094] The decryption module is adapted to be connected with the encrypted data cache module and the one-time programmable module.

[0095] During convolution processing, the decryption module reads the encrypted network parameters from the encrypted data cache module and reads the encryption key from the one-time programmable module. Thereafter, the encrypted network parameters are decrypted using the read encryption key, so that decrypted weight data and decrypted pooling cores can be generated after the decryption operation.

[0096] The decryption module sends the decryption weight data generated by decryption to the convolution kernel unit, and sends the decryption pooling kernel generated by decryption to the pooling module.

[0097] Figure 2 An embodiment of a security processing unit is shown in FIG. As can be seen from the figure, the security processing unit may include an encrypted data cache module, a one-time programmable module, and a decryption module, wherein the encrypted data cache module may be used to store encrypted network parameters, and the one-time programmable module may be used to store decryption keys. When the one-time programmable module is configured to be software unreadable, it means that the decryption key stored in the one-time programmable module cannot be changed. It can be understood that the decryption key is a set of keys corresponding to the encryption key. Therefore, the form of the decryption key can be determined based on the encryption key for encrypting the network parameters. The encryption key and the decryption key can be consistent with the prior art, and will not be repeated here.

[0098] The decryption module is connected to the encrypted data cache module and the one-time programmable module. The decryption module can read the encrypted weight parameters and the encrypted pooling core from the encrypted data cache module, and can read the decryption key from the one-time programmable module. After that, the encrypted weight parameters can be decrypted using the decryption key to generate the decrypted weight parameters after decryption. When the encrypted pooling core is decrypted using the decryption key, the decrypted pooling core can be generated after decryption. From the above description, it can be seen that the decryption weight parameters generated by the decryption module should be loaded into the convolution core unit, and the decryption pooling core generated by the decryption should be loaded into the pooling module. The decryption module can adopt the existing commonly used form, and the way and process of the decryption module using the decryption key to perform corresponding decryption can be consistent with the existing ones.

[0099] It can be seen from the above description that when the control module is connected to the security processing unit through the control interface module, it specifically means that the control module is connected to the encrypted data cache module, the decryption module and the one-time programmable module through the control interface module to control the corresponding working states of the encrypted data cache module, the decryption module and the one-time programmable module. It can be understood that controlling the corresponding working states of the encrypted data cache module, the decryption module and the one-time programmable module specifically means completing the work of the above-mentioned security processing module.

[0100] The following is an example of the corresponding process of encrypting network parameters. Specifically:

[0101] The data block group length of the encryption algorithm corresponding to the encryption key is set to S. Thereafter, the weight parameters and pooling cores in the target convolutional neural network are respectively constructed as data block units with a unit length of S, and all data block units form a data group. Among them, the method of constructing the weight parameters and pooling cores as data block units with a unit length of S can be consistent with the prior art. For example, the encryption algorithm adopts the symmetric encryption algorithm AES128, and the corresponding data block group length S is 128 bits. The minimum encrypted data block is 128 bits. At this time, the decryption module should be the AES128 decryption module, that is, decryption is based on a 128-bit unit data block; generally, each database unit only contains the weight parameters or pooling core of one convolutional layer.

[0102] After forming a data group, an encryption algorithm can be used to encrypt each data block in the data group offline, wherein each data block is encrypted to form an encrypted data block, and the control module can store the encrypted data block in the data storage unit, and thereafter, the DMA module is configured to move the encrypted data block to the encrypted data cache module, and the decryption key is stored in the data storage unit, and then moved to the one-time programmable module through the DMA module. It can be understood that the decryption module decrypts each encrypted data block, and the data length of the decrypted data block obtained is also S. For example, when the packet length S of the above-mentioned data block is 128 bits, the data length of the decrypted data block is also 128 bits.

[0103] Generally, in the target convolutional neural network, each convolutional layer may have filters of multiple channels. In a convolutional layer, each batch of inputs will be calculated with the weight data of multiple channel filters. When the data bit width L=8 supported by the target convolutional neural network, the number of filters that need to be interleaved should be M=128 / 8=16, wherein data interleaving here specifically refers to the process of rearranging the positions of data on the premise of one-to-one correspondence between data sequences. The method and process of data interleaving may be consistent with the prior art, and shall be based on the operation that can satisfy the convolutional neural network reasoning process.

[0104] In one embodiment of the present invention, the convolution kernel unit includes a serialization module, a multiplication and accumulation module and an accumulator module connected in sequence, wherein:

[0105] During the convolution operation, the serialization module is used to rearrange the target input feature data and the decryption weight data respectively, where:

[0106] After rearranging the target input feature data, an input feature sub-target data group is generated, wherein the input feature sub-target data group includes a plurality of input feature sub-target data arranged in time sequence;

[0107] After rearranging the decryption weight data, a decryption weight sub-target data group is generated, wherein the decryption weight sub-target data group includes a plurality of decryption weight sub-target data arranged in time sequence;

[0108] Based on the corresponding time sequence characteristics of the input feature sub-target data and the decryption weight sub-target data, a multiplication and addition operation is performed using a multiplication and accumulator module to generate corresponding partial and result data after the multiplication and addition operation;

[0109] The accumulator module is used to perform accumulation operation on the partial sum result data output by the multiplication and accumulation module to generate convolution operation result data after the accumulation operation.

[0110] Figure 2 An embodiment of a convolution kernel unit is shown in the figure. In the figure, the convolution kernel unit includes a serialization module, a multiplication-accumulation module and an accumulator module, wherein the multiplication-accumulation module can be used to perform multiplication and addition operations, and the accumulator module can be used to perform addition operations.

[0111] In order to adapt to different data bit widths, the convolution kernel unit can use the serialization module to rearrange the data when performing convolution operations. Figure 2 In the embodiment, the serialization module is connected to the decryption module and the input feature data cache module, wherein the serialization module can read the target input feature data in the input feature data cache module, and can read the decryption weight data output by the decryption module, and rearrange the read target input feature data and decryption weight data, wherein after the target input feature data is rearranged, an input feature sub-target data group is generated, and after the decryption weight data is rearranged, a decryption weight sub-target data group is generated. The following is a specific example of the way in which the serialization module rearranges data.

[0112] In one embodiment of the present invention, when the serialization module rearranges the target input feature data and the decryption weight data, it includes:

[0113] Based on the stripe operation of the multiply-accumulate module, the target input feature data and the decrypted weight data are polled to split the target input feature data into a plurality of feature data packets and split the decrypted weight data into a plurality of weight data packets after the polling;

[0114] The split feature data packets are assembled into corresponding input feature sub-target data, and the split weight data packets are assembled into corresponding decryption weight sub-target data, wherein,

[0115] Arrange the feature data packets according to the polling timing sequence, and assemble the feature data packets arranged according to the polling timing sequence based on the multiplication and addition calculation bit width in the multiplication and accumulation module, so as to generate a corresponding input feature sub-target data after assembling;

[0116] Arrange the weight data packets according to the polling timing sequence, and assemble the corresponding weight data packets arranged according to the polling timing sequence based on the multiplication and addition calculation bit width in the multiplication and accumulator module to generate a corresponding decryption weight sub-target data after assembling;

[0117] The assembled input feature sub-target data and decryption weight sub-target data are loaded into the multiplier-accumulator module in a pipeline manner.

[0118] It should be noted that the input feature source data is stored in the data storage unit and should also be constructed into a data block with a data length of S using the above-mentioned network parameters. That is, the input feature source data should include multiple data blocks with a data length of S. From the above description, it can be seen that the target input feature data should also consist of multiple data blocks.

[0119] When the serialization module polls the target input feature data and decryption weight data, it should be based on the stripe operation of the multiplication and accumulation module, where the convolution kernel sliding on the feature map is called stripe. By adjusting the size of the stripe, the size of the feature map can be controlled, thereby affecting the performance of feature extraction. Therefore, the stripe operation of the multiplication and accumulation module specifically refers to the step length of the convolution kernel unit moving the input data when performing convolution operation. When the serialization module polls the target input feature data and decryption weight data, the corresponding bit data in each data block is obtained. For example, when the data length S of the data block is 128 bits, each polling can obtain 8 bits of data in each data block, that is, by polling each data block 16 times, all the data in the current data block can be obtained. At this time, the feature data packet contains 8 bits of data, and the weight data packet also contains 8 bits of data. Therefore, the situation of the feature data packet and the weight data packet can be determined according to the data length of the data block and the stripe operation of the multiplication and accumulation module.

[0120] As can be seen from the above description, during polling, multiple feature data packets can be generated in sequence, and then the multiple data packets are assembled into corresponding input feature sub-target data. Generally, the amount of data in the input feature sub-target data can be selected according to actual needs, such as a data width of 1024 bits. At this time, when a feature data packet contains 8 bits of data, the input feature sub-target data should include 128 feature data packets. When the 128 feature data packets are assembled, 64 data blocks with a data length of 16 bits can be formed. It can be seen that the input feature sub-target data can include 64 16-bit data blocks. For the assembly to form the decryption weight sub-target data, the specific assembly method can refer to the description of assembling the input feature sub-target data here, which will not be repeated here.

[0121] In order to satisfy the multiplication and addition operation of the multiplication and accumulator module, the input feature sub-target data and the decryption weight sub-target data can be synchronously loaded into the multiplication and accumulator module in a pipeline manner to improve the efficiency of the multiplication and addition calculation of the multiplication and accumulator module.

[0122] In specific implementation, the multiplier-accumulator module generally includes multiple identical MAC cores, wherein each MAC core contains multiple multipliers and multiple adders. For example, the multiplier-accumulator module may include 16 MAC cores, each MAC core includes 64 16-bit multipliers and 72 adders. The situations in the multiplier-accumulator module and the MAC core can be selected according to actual needs and are not listed one by one here.

[0123] When each MAC core includes 64 16-bit multipliers, for the int8 data format, each multiplier and adder can be divided into two computing units. The throughput of int8 is twice that of int16, and the output result is called a partial sum, where the int8 data format is an 8-bit binary number and the int16 data format is a 16-bit binary number. The multiplication and accumulation module performs multiplication and addition calculations on the input feature sub-target data and the decryption weight sub-target data to generate corresponding partial sum result data after the multiplication and addition operation. Specifically, during a stripe operation, the decryption weight sub-target data in the MAC core remains unchanged, and the input feature sub-target data slides along the input data cube (i.e., a point in the convolution kernel at the same position and a feature data point at a different position are continuously multiplied). The manner and process of the multiplication and accumulation module to perform multiplication and addition calculations can be consistent with the prior art and will not be repeated here.

[0124] In one embodiment of the present invention, the serialization module includes a convolution data loader, a weight data loader and a convolution sequence generator, wherein:

[0125] The convolution data loader polls the target input feature data and sends the feature data packets generated by the polling split to the feature FIFO unit of the convolution sequence generator;

[0126] The weight data loader polls the decrypted weight data and sends the weight data packets generated by the polling split to the weight FIFO unit of the convolution sequence generator;

[0127] The convolution sequence generator assembles the feature data packets in the feature FIFO unit to generate input feature sub-target data after assembling, and sends the generated input feature sub-target data to the convolution data loader, so as to load the input feature sub-target data into the multiplication and accumulator module one by one by using the convolution data loader;

[0128] The convolution sequence generator assembles the weight data packets in the weight FIFO unit to generate decrypted weight sub-target data after assembly, and sends the generated decrypted weight sub-target data to the weight data loader, so that the decrypted weight sub-target data can be loaded one by one into the multiplier accumulator module using the weight data loader.

[0129] Figure 3 An embodiment of the serialization module connected with the multiplier-accumulator module is shown in the figure. An embodiment of the serialization module is shown in the figure. In the figure, the serialization module may include a convolution data loader, a weight data loader and a convolution sequence generator, wherein the convolution data loader polls the target input feature data, and the decryption weight data is polled through the weight data loader. In order to be able to sort the feature data packets and weight data packets, a feature FIFO unit and a weight FIFO unit can be set in the convolution sequence generator, that is, by using the characteristics of FIFO, the feature data packets and weight data packets can be sorted in time sequence.

[0130] In specific implementation, a feature counter and a weight calculator are also set in the convolution sequence generator. The feature counter and the weight counter are preferably counters with decreasing counts. The feature counter can count the feature data packet, and the weight counter can count the weight data packet. When the corresponding counts of the feature counter and the weight counter are both 0, it indicates that the assembly of the feature data packet and the weight data packet is completed. Thereafter, the multiplier-accumulator module can be checked. When checking the multiplier-accumulator module, the credit value of the multiplier-accumulator module is mainly checked to determine whether to send the input feature sub-target number and the decrypted weight sub-target data to the multiplier-accumulator module.

[0131] It should be noted that the credit value of the multiplier-accumulator module and the method of checking the credit value of the multiplier-accumulator module can be consistent with the existing technology, specifically based on being able to determine the working status of the multiplier-accumulator module, such as determining the free space of the multiplier-accumulator module, etc., so as to coordinate the efficiency of data transmission between the sequence generator and the multiplier-accumulator module.

[0132] In one embodiment of the present invention, when the accumulator module performs an accumulation operation on the partial sum result data, the convolution operation result source data is generated;

[0133] The generated convolution operation result source data is subjected to data restriction processing to generate convolution operation result data after the data restriction processing, wherein the data restriction processing performed on the convolution operation result source data includes a round operation and / or a saturate operation.

[0134] Specifically, the accumulator module is used to accumulate the output of the multiplication and accumulation module, and after the accumulation, data limitation is performed, such as round operation and / or saturate operation, wherein, during the data limitation processing, the convolution operation result source data is first rounded (round operation), if the data overflows due to carry during the rounding process, the signal is generally saturated (saturation) processing, and the saturation processing is that the calculation result exceeds the maximum value of the data that can be stored in the required data format, and then the maximum value is used to represent the data.

[0135] During specific implementation, the accumulator module may adopt an existing commonly used form, specifically based on being able to implement accumulation operations and ultimately generate convolution operation result data.

[0136] In summary, a convolutional neural network acceleration method with secure encryption can be obtained. In one embodiment of the present invention, convolution processing is performed on any convolutional neural network using the convolutional neural network acceleration device described above.

[0137] Figure 4 A schematic diagram of an embodiment of the neural network acceleration device of the present invention is given in FIG. Figure 4 It can be seen that when using the neural network acceleration device, the parameters are initialized first, and then the encrypted weight parameters and the decryption key constituting the encrypted network parameters are written into the encrypted data cache module, and the activation function is written into the activation function cache module. After that, the DMA module and the registers of the one-time programmable module are configured to write the decryption key into the registers of the one-time programmable module.

[0138] During inference, the decryption module is used to decrypt the encrypted network parameters and provide the target input feature data. After that, the convolution kernel uses the decrypted weight data to perform convolution operation on the target input feature data, and the activation module is used to perform activation operation on the convolution operation result data. The pooling module is used to perform pooling operation on the activation operation result data. Finally, the pooling module stores the pooling result data in the data storage unit so that the control module can read the pooling result data through the data interface module. As can be seen from the above description, Figure 4 The DRAM in the figure is an embodiment of the above-mentioned data storage module, and the details can be referred to the above-mentioned corresponding description.

[0139] Specifically, the specific method and process of performing convolution processing using the convolutional neural network acceleration device can be referred to the corresponding description above, which will not be repeated here.

Claims

1. A convolutional neural network acceleration device with secure encryption, characterized in that: The convolutional neural network acceleration device comprises: A security processing unit, used to store a decryption key and an encrypted network parameter, wherein the encrypted network parameter is generated by encrypting the network parameter of the target convolutional neural network offline using the encryption key, the encryption key matches the decryption key, and the encrypted network parameter includes at least an encrypted weight parameter and an encrypted pooling core; A neural network acceleration unit, used to perform convolution processing when the target convolutional neural network is working, at least includes a convolution kernel unit for performing convolution operations, an activation function unit for performing activation operations, and a pooling module for performing pooling operations, wherein: The convolution kernel unit, activation function unit and pooling module are connected in sequence; When performing convolution processing, the target input feature data and the decrypted weight data are loaded into the convolution core unit, so as to use the convolution core unit to perform convolution operation on the decrypted weight data and the target input feature data, and generate convolution operation result data after the convolution operation, wherein the target input feature data is generated by the security processing unit after decrypting the input feature source data, and the decrypted weight data is generated by the security processing unit after decrypting the encrypted weight parameter; Performing an activation operation on the convolution operation result data using an activation function unit to generate activation operation result data after the activation operation; The pooling module is used to perform a pooling operation on the activation operation result data to generate pooling result data after the pooling operation, wherein during the pooling operation, the activation processing result data is pooled using a decrypted pooling core generated by decrypting the encrypted pooling core by the security processing unit.

2. The convolutional neural network acceleration device with secure encryption according to claim 1, characterized in that: The convolution kernel unit includes a serialization module, a multiplication and accumulation module and an accumulator module connected in sequence, wherein: During the convolution operation, the serialization module is used to rearrange the target input feature data and the decryption weight data respectively, where: After rearranging the target input feature data, an input feature sub-target data group is generated, wherein the input feature sub-target data group includes a plurality of input feature sub-target data arranged in time sequence; After rearranging the decryption weight data, a decryption weight sub-target data group is generated, wherein the decryption weight sub-target data group includes a plurality of decryption weight sub-target data arranged in time sequence; Based on the corresponding time sequence characteristics of the input feature sub-target data and the decryption weight sub-target data, a multiplication and addition operation is performed using a multiplication and accumulator module to generate corresponding partial and result data after the multiplication and addition operation; The accumulator module is used to perform accumulation operation on the partial sum result data output by the multiplication and accumulation module to generate convolution operation result data after the accumulation operation.

3. The convolutional neural network acceleration device with secure encryption according to claim 2, characterized in that: When the serialization module rearranges the target input feature data and decryption weight data, it includes: Based on the stripe operation of the multiply-accumulate module, the target input feature data and the decrypted weight data are polled to split the target input feature data into a plurality of feature data packets and split the decrypted weight data into a plurality of weight data packets after the polling; The split feature data packets are assembled into corresponding input feature sub-target data, and the split weight data packets are assembled into corresponding decryption weight sub-target data, wherein, Arrange the feature data packets according to the polling timing sequence, and assemble the feature data packets arranged according to the polling timing sequence based on the multiplication and addition calculation bit width in the multiplication and accumulation module, so as to generate a corresponding input feature sub-target data after assembling; Arrange the weight data packets according to the polling timing sequence, and assemble the corresponding weight data packets arranged according to the polling timing sequence based on the multiplication and addition calculation bit width in the multiplication and accumulator module to generate a corresponding decryption weight sub-target data after assembling; The assembled input feature sub-target data and decryption weight sub-target data are loaded into the multiplier-accumulator module in a pipeline manner.

4. The convolutional neural network acceleration device with secure encryption according to claim 3, characterized in that: The serialization module includes a convolution data loader, a weight data loader and a convolution sequence generator, wherein: The convolution data loader polls the target input feature data and sends the feature data packets generated by the polling split to the feature FIFO unit of the convolution sequence generator; The weight data loader polls the decrypted weight data and sends the weight data packets generated by the polling split to the weight FIFO unit of the convolution sequence generator; The convolution sequence generator assembles the feature data packets in the feature FIFO unit to generate input feature sub-target data after assembling, and sends the generated input feature sub-target data to the convolution data loader, so as to load the input feature sub-target data into the multiplication and accumulator module one by one by using the convolution data loader; The convolution sequence generator assembles the weight data packets in the weight FIFO unit to generate decrypted weight sub-target data after assembly, and sends the generated decrypted weight sub-target data to the weight data loader, so that the decrypted weight sub-target data can be loaded one by one into the multiplier accumulator module using the weight data loader.

5. The convolutional neural network acceleration device with secure encryption according to claim 3, characterized in that: When the accumulator module performs accumulation operation on the partial sum result data, the convolution operation result source data is generated; The generated convolution operation result source data is subjected to data restriction processing to generate convolution operation result data after the data restriction processing, wherein the data restriction processing performed on the convolution operation result source data includes a round operation and / or a saturate operation.

6. The convolutional neural network acceleration device with secure encryption according to any one of claims 1 to 5, characterized in that: The activation function unit includes an activation function cache module and an activation module, wherein: When performing an activation operation, the activation module reads the activation function from the activation function cache module, and uses the read activation function to perform an activation operation on the convolution operation result data, and generates activation operation result data after the activation operation; The activation module loads the generated activation operation result data into the pooling module.

7. The convolutional neural network acceleration device with secure encryption according to any one of claims 1 to 5, characterized in that: The convolutional neural network acceleration device also includes a data storage unit for data storage and a DMA module adapted to be connected to the data storage unit, wherein: The DMA module is also connected to the security processing unit, and is connected to the convolution kernel unit through the input feature data buffer module; During convolution processing, the input feature source data in the data storage unit is moved to the input feature data buffer module through the DMA module to form the target input feature data; The convolution kernel unit reads the target input feature data from the input feature data cache module, and receives the decryption weight data loaded by the security processing unit, so as to perform a convolution operation on the target input feature data using the decryption weight data.

8. The convolutional neural network acceleration device with secure encryption according to claim 7, characterized in that: The security processing unit includes an encrypted data cache module, a one-time programmable module and a decryption module, wherein: For the encrypted network parameters generated by offline encryption, the encrypted network parameters are transferred to the encrypted data cache module through the DMA module, so as to use the encrypted data cache module to buffer the encrypted network parameters; Using a one-time programmable module to store the decryption key, and the one-time programmable module is configured to be unreadable by software; The decryption module is adapted to be connected with the encrypted data cache module and the one-time programmable module. During convolution processing, the decryption module reads the encrypted network parameters from the encrypted data cache module and reads the encryption key from the one-time programmable module. Thereafter, the encrypted network parameters are decrypted using the read encryption key, so that decrypted weight data and decrypted pooling cores can be generated after the decryption operation. The decryption module sends the decryption weight data generated by decryption to the convolution kernel unit, and sends the decryption pooling kernel generated by decryption to the pooling module.

9. The convolutional neural network acceleration device with secure encryption according to claim 7, characterized in that: It also includes a control module, a control interface module adapted to be connected to the control module, and a data interface module, wherein: The control module is connected with the data storage unit through the data interface module; The control module is adaptively connected to the security processing unit and the neural network acceleration unit through the control interface module to configure the working status of the security processing unit and the neural network acceleration unit through the control interface module.

10. A convolutional neural network acceleration method with secure encryption, characterized in that: For any convolutional neural network, convolution processing is performed using the convolutional neural network acceleration device described in any one of claims 1 to 9.

Images