Convolutional neural network acceleration device and acceleration method with secure encryption

Abstract

The application relates to a convolutional neural network acceleration device and method with security encryption. The device comprises a security processing unit, a neural network acceleration unit, a convolution kernel unit, an activation function unit and a pooling module. The security processing unit stores a decryption key and encrypted network parameters. The neural network acceleration unit is used for performing convolution operation on decrypted weight data and target input feature data by using the convolution kernel unit, generating convolution operation result data after the convolution operation, performing activation operation on the convolution operation result data by using the activation function unit, and generating activation operation result data after the activation operation. The pooling module is used for performing pooling operation on the activation operation result data, and generating pooling result data after the pooling operation. During the pooling operation, the activation processing result data is processed by using a decrypted pooling kernel generated by the security processing unit. The application can accelerate the inference of the convolutional neural network and improve the security of the convolutional neural network.

Description

Technical Field

[0001] This invention relates to an acceleration device and method, and more particularly to a convolutional neural network acceleration device and method with secure encryption. Background Technology

[0002] As an important data classification tool, Convolutional Neural Networks (CNNs) have wide applications in information processing and pattern recognition. A well-trained CNN can efficiently and accurately classify and predict input data. Currently, the research and training of neural network algorithms has become a hot topic in both industry and academia. Therefore, a well-trained CNN has gradually become an important intellectual property for many data processing companies.

[0003] In neural network design, a high-performance convolutional neural network often requires significant training resources, such as computational, human, and time resources. Therefore, training datasets, accelerators, and network parameters are typically considered intellectual property of convolutional neural networks and are protected.

[0004] For specific recognition tasks, the weight parameters trained from massive amounts of data over a long period of time are considered an important form of intellectual property. In particular, in fields involving national information security, such as facial recognition, facial features trained on terminals are also private and cannot be easily disclosed.

[0005] To improve data security, encryption chips are generally used to protect data. Encryption chips are a general term for a type of security chip that integrates various symmetric and asymmetric algorithms and has an extremely high level of security. They can ensure that the keys and information data stored inside cannot be illegally read or tampered with.

[0006] For chip platform companies, protecting these offline-trained weight parameters to prevent them from being obtained by malware during inference is a crucial security consideration in current industry chain collaborations. Simultaneously, the design of the pooling kernel is also critical for convolutional neural networks; therefore, the pooling kernel also needs to be protected using encrypted chip technology.

[0007] As can be seen from the above description, how to effectively accelerate the inference of a trained convolutional neural network and achieve secure encryption during the accelerated inference process is a technical problem that urgently needs to be solved. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a secure and encrypted convolutional neural network acceleration device and acceleration method, which can accelerate the inference of convolutional neural networks and improve the security of convolutional neural networks.

[0009] According to the technical solution provided by the present invention, a convolutional neural network acceleration device with secure encryption is provided, the convolutional neural network acceleration device comprising:

[0010] A security processing unit is used to store decryption keys and encrypted network parameters, wherein the encrypted network parameters are generated by offline encryption of the network parameters of the target convolutional neural network using the encryption key, the encryption key matches the decryption key, and the encrypted network parameters include at least encrypted weight parameters and encrypted pooling kernels;

[0011] The neural network acceleration unit, used to perform convolution processing during the operation of the target convolutional neural network, includes at least a convolution kernel unit for performing convolution operations, an activation function unit for performing activation operations, and a pooling module for performing pooling operations.

[0012] The convolutional kernel unit, activation function unit, and pooling module are connected in sequence;

[0013] When performing convolution processing, the target input feature data and decryption weight data are loaded into the convolution kernel unit so that the convolution kernel unit can perform convolution operation on the decryption weight data and the target input feature data, and generate convolution operation result data after the convolution operation. The target input feature data is generated by the security processing unit after decrypting the input feature source data, and the decryption weight data is generated by the security processing unit after decrypting the encrypted weight parameters.

[0014] The activation function unit is used to activate the convolution result data to generate activation result data after the activation operation.

[0015] The activation operation result data is pooled using a pooling module to generate pooled result data after the pooling operation. During the pooling operation, the activation processing result data is pooled using a decrypted pooling core generated by the security processing unit decrypting the encrypted pooling core.

[0016] The convolutional kernel unit includes a serialization module, a multiply-accumulate module, and an accumulator module connected in sequence, wherein...

[0017] During convolution operations, the serialization module is used to rearrange the target input feature data and the decrypted weight data respectively.

[0018] After rearranging the target input feature data, an input feature sub-target data group is generated, wherein the input feature sub-target data group includes several input feature sub-target data arranged in time sequence;

[0019] After rearranging the decryption weight data, a decryption weight sub-target data group is generated, which includes several decryption weight sub-target data arranged in chronological order.

[0020] Based on the temporal arrangement characteristics of the input feature sub-target data and the decryption weight sub-target data, the multiply-accumulate module is used to perform multiply-accumulate operations to generate the corresponding partial and result data after the multiply-accumulate operations.

[0021] The accumulator module is used to perform an accumulation operation on the partial output and result data of the multiply-accumulator module to generate the convolution operation result data after the accumulation operation.

[0022] When the serialization module rearranges the target input feature data and decryption weight data, it includes:

[0023] Based on the stripe operation of the multiply-accumulate module, the target input feature data and decrypted weight data are polled to split the target input feature data into several feature data packets and the decrypted weight data into several weight data packets after polling.

[0024] The aforementioned split feature data packets are assembled into corresponding input feature sub-target data, and the split weight data packets are assembled into corresponding decryption weight sub-target data, wherein,

[0025] The feature data packets are arranged in polling order, and the corresponding feature data packets arranged in polling order are assembled based on the multiply-accumulate calculation bit width in the multiply-accumulate module, so as to generate a corresponding input feature sub-target data after assembly.

[0026] The weight data packets are arranged in polling order, and the corresponding weight data packets arranged in polling order are assembled based on the multiply-accumulate calculation bit width in the multiply-accumulate module, so as to generate a corresponding decryption weight subtarget data after assembly.

[0027] The assembled input feature sub-target data and decrypted weight sub-target data are loaded into the multiply-accumulate module in a pipeline manner.

[0028] The serialization module includes a convolutional data loader, a weight data loader, and a convolutional sequence generator, wherein...

[0029] The convolutional data loader polls the target input feature data and sends the feature data packets generated by the polling split to the feature FIFO unit of the convolutional sequence generator.

[0030] The weight data loader polls the decrypted weight data and sends the weight data packets generated by the polling split to the weight FIFO unit of the convolution sequence generator;

[0031] The convolution sequence generator assembles the feature data packets in the feature FIFO unit to generate input feature sub-target data after assembly, and sends the generated input feature sub-target data to the convolution data loader so that the convolution data loader can load the input feature sub-target data into the multiply-accumulate module one by one.

[0032] The convolution sequence generator assembles the weight data packets in the weight FIFO unit to generate decrypted weight subtarget data. The generated decrypted weight subtarget data is then sent to the weight data loader, which loads the decrypted weight subtarget data into the multiply-accumulate module one by one.

[0033] When the accumulator module performs accumulation operations on the partial and result data, it generates the source data of the convolution operation result;

[0034] The source data of the generated convolution operation result is subjected to data constraint processing to generate the convolution operation result data after data constraint processing. The data constraint processing of the source data of the convolution operation result includes round operation and / or saturate operation.

[0035] The activation function unit includes an activation function cache module and an activation module, wherein,

[0036] When performing the activation operation, the activation module reads the activation function from the activation function cache module, uses the read activation function to perform the activation operation on the convolution operation result data, and generates the activation operation result data after the activation operation.

[0037] The activation module loads the generated activation operation result data into the pooling module.

[0038] The convolutional neural network acceleration device also includes a data storage unit for data storage and a DMA module adapted and connected to the data storage unit, wherein...

[0039] The DMA module is also connected to the security processing unit and to the convolution kernel unit through the input feature data caching module;

[0040] During convolution processing, the input feature source data in the data storage unit is moved to the input feature data cache module through the DMA module to form the target input feature data;

[0041] The convolution kernel unit reads the target input feature data from the input feature data cache module and receives the decrypted weight data loaded by the security processing unit, so as to perform convolution operation on the target input feature data using the decrypted weight data.

[0042] The security processing unit includes an encrypted data caching module, a one-time programmable module, and a decryption module, wherein...

[0043] The encrypted network parameters generated offline are transferred to the encrypted data cache module via the DMA module so that the encrypted network parameters can be buffered by the encrypted data cache module.

[0044] The decryption key is stored using a one-time programmable module, and the one-time programmable module is configured to be unreadable by software.

[0045] The decryption module is adapted and connected to the encrypted data caching module and the one-time programmable module.

[0046] During convolution processing, the decryption module reads the encrypted network parameters from the encrypted data cache module and the encryption key from the one-time programmable module. Then, the read encryption key is used to decrypt the encrypted network parameters so that decrypted weight data and decrypted pooling kernel can be generated after the decryption operation.

[0047] The decryption module sends the decryption weight data generated by decryption to the convolution kernel unit, and sends the decryption pooling kernel generated by decryption to the pooling module.

[0048] It also includes a control module, a control interface module adapted and connected to the control module, and a data interface module, wherein,

[0049] The control module is adapted and connected to the data storage unit through the data interface module;

[0050] The control module is adapted to connect with the security processing unit and the neural network acceleration unit through the control interface module, so as to configure the working status of the security processing unit and the neural network acceleration unit through the control interface module.

[0051] A secure encryption method for accelerating convolutional neural networks involves performing convolution processing on any convolutional neural network using the aforementioned convolutional neural network acceleration device.

[0052] The advantages of this invention are as follows: A secure processing unit caches encrypted network parameters and can decrypt and generate the required decryption weight data and decryption pooling kernel. A neural network acceleration unit performs convolution processing. An activation function caching module within the convolutional neural network acceleration unit caches activation functions, reducing the number of activation function calls and improving the speed of convolution processing during activation operations. A serialization module rearranges the target input feature data and decryption weight data, enabling the convolution kernel unit to adapt to convolution processing of data with different bit widths. This accelerates the inference of the convolutional neural network and improves its security. Attached Figure Description

[0053] Figure 1This is a main block diagram of one embodiment of the neural network acceleration device of the present invention.

[0054] Figure 2 This is a structural block diagram of one embodiment of the neural network acceleration device of the present invention.

[0055] Figure 3 This is one embodiment of the connection and cooperation between the serialization module and the multiply-accumulate module of the present invention.

[0056] Figure 4 This is a schematic diagram illustrating the workflow of one embodiment of the neural network acceleration device of the present invention. Detailed Implementation

[0057] The present invention will be further described below with reference to specific accompanying drawings and embodiments.

[0058] To accelerate the inference process of convolutional neural networks (CNNs) and improve their security, this invention provides a CNN acceleration device with secure encryption. Specifically, the CNN acceleration device includes:

[0059] A security processing unit is used to store decryption keys and encrypted network parameters, wherein the encrypted network parameters are generated by offline encryption of the network parameters of the target convolutional neural network using the encryption key, the encryption key matches the decryption key, and the encrypted network parameters include at least encrypted weight parameters and encrypted pooling kernels;

[0060] The neural network acceleration unit, used to perform convolution processing during the operation of the target convolutional neural network, includes at least a convolution kernel unit for performing convolution operations, an activation function unit for performing activation operations, and a pooling module for performing pooling operations.

[0061] The convolutional kernel unit, activation function unit, and pooling module are connected in sequence;

[0062] When performing convolution processing, the target input feature data and decryption weight data are loaded into the convolution kernel unit so that the convolution kernel unit can perform convolution operation on the decryption weight data and the target input feature data, and generate convolution operation result data after the convolution operation. The target input feature data is generated by the security processing unit after decrypting the input feature source data, and the decryption weight data is generated by the security processing unit after decrypting the encrypted weight parameters.

[0063] The activation function unit is used to activate the convolution result data to generate activation result data after the activation operation.

[0064] The activation operation result data is pooled using a pooling module to generate pooled result data after the pooling operation. During the pooling operation, the activation processing result data is pooled using a decrypted pooling core generated by the security processing unit decrypting the encrypted pooling core.

[0065] Figure 1 The figure illustrates an embodiment of the convolutional neural network acceleration device of the present invention. As shown in the figure, when the convolutional neural network acceleration device includes at least a security processing unit and a neural network acceleration unit, the security processing unit can be used to improve the security of the target convolutional neural network, and the neural network acceleration unit can be used to accelerate the inference process of the target convolutional neural network. The target convolutional neural network is the convolutional neural network that needs to be protected and whose inference process needs to be accelerated. As can be seen from the above description, the target convolutional neural network should be a trained neural network that can perform relevant data classification. The training method of the target convolutional neural network and the data classification method using the target convolutional neural network can be selected according to actual needs, and will not be listed here.

[0066] For the trained target convolutional neural network, the network parameters of the target convolutional neural network can be determined and obtained. Specifically, when using a security processing unit to improve the security of the convolutional neural network, at least the network parameters of the target convolutional neural network should be encrypted. Subsequently, the security processing unit stores the encrypted network parameters corresponding to the target convolutional neural network and the decryption key used to decrypt the encrypted network parameters. Generally, the network parameters of the convolutional neural network may include weight parameters and / or pooling kernels. The weight parameters refer to the weight information of all feature layers within the convolutional neural network, where the feature layers can be the convolutional layers within the target convolutional neural network. The pooling kernel is the pooling parameter used by the pooling module; the weight parameters and pooling kernel can be consistent with existing technologies.

[0067] In practice, after determining the network parameters of the target convolutional neural network, these parameters can be encrypted using an encryption key to generate encrypted network parameters. For example, if the network parameters include weight parameters, encrypting the weight parameters with the encryption key yields the corresponding encrypted weight parameters; if the network parameters include pooling kernels, encrypting the pooling kernels with the encryption key yields the corresponding encrypted pooling kernels. When the target convolutional neural network includes a pooling module, both the weight parameters and the pooling kernels can be encrypted simultaneously to further enhance the security of the target convolutional neural network. The following explanation uses the example of encrypted network parameters including encrypted weight parameters and encrypted pooling kernels for further illustration.

[0068] When it is necessary to use the neural network acceleration unit to accelerate the inference of the convolutional neural network, in order to meet the inference requirements of the convolutional neural network, the security processing unit should use the decryption key to decrypt the encrypted network parameters. Specifically, after decrypting the encrypted weight parameters, the corresponding decrypted weight data can be generated, and after decrypting the encrypted pooling kernel, the corresponding decrypted pooling kernel can be generated. After that, the encrypted weight data and the decrypted pooling kernel need to be loaded into the neural network acceleration unit.

[0069] It is understandable that convolutional neural networks perform at least convolution processing during inference; therefore, when using a neural network acceleration unit to accelerate the inference of a target convolutional neural network, the neural network acceleration unit can be configured to perform the convolution processing that the target convolutional neural network performs, that is, the inference of the target convolutional neural network includes at least convolution processing.

[0070] Figure 1 The figure illustrates an embodiment of a neural network acceleration unit when performing convolution processing on a target convolutional neural network. The neural network acceleration unit includes a convolution kernel unit, an activation function unit, and a pooling module. The convolution kernel unit performs convolution operations, the activation function unit performs activation operations, and the pooling module performs pooling operations. Therefore, the convolution processing performed by the neural network acceleration unit includes convolution operations, activation operations, and pooling operations. Specifically, during the activation operation, post-processing operations are performed on individual data elements (the smallest unit of the matrix) within the convolution result data. By performing a nonlinear transformation on the input, the network can learn and represent more complex functional relationships. After performing the pooling operation, the size of the parameter matrix can be effectively reduced, thereby reducing the number of parameters in the final connection layer.

[0071] It should be noted that when a convolution kernel unit performs a convolution operation, it needs to obtain the target input feature data and the decrypted weight data. After that, it performs a convolution operation on the decrypted weight data and the target input feature data, and generates the convolution operation result data after the convolution operation. The type of convolution kernel unit and the method and process of performing convolution operations will be explained in detail below.

[0072] Activation operation specifically refers to the activation function unit activating the convolution result data and generating activation result data after the activation operation. Of course, the activation function unit should utilize activation functions when performing the activation operation; that is, the activation function unit should cache the required activation functions. Figure 2 The figure illustrates one embodiment of the activation function unit. The activation function unit includes an activation function cache module and an activation module.

[0073] When performing the activation operation, the activation module reads the activation function from the activation function cache module, uses the read activation function to perform the activation operation on the convolution operation result data, and generates the activation operation result data after the activation operation.

[0074] The activation module loads the generated activation operation result data into the pooling module.

[0075] Specifically, an activation function caching module is used to cache the activation functions required for inference by the target convolutional neural network. Therefore, the types of activation functions cached in the activation function caching module are related to data classification and other operations performed by the target convolutional neural network. During activation, the activation module can read the corresponding activation function from the activation function caching module. Subsequently, the activation function can be used to activate the convolution operation result data, and the activation module can generate activation operation result data, which is then loaded into the pooling module. It is understandable that caching activation functions using the activation function caching module reduces repeated calls to activation functions, further reducing memory overhead and improving the speed of convolution processing. The method and process by which the activation module uses activation functions to activate the convolution operation result data can be consistent with existing technologies and will not be elaborated here.

[0076] In practice, when the pooling module performs a pooling operation, it should obtain the decryption pooling core from the security processing unit. Then, it should use the decryption pooling core to perform a pooling operation on the activation operation result data and generate the pooling result data.

[0077] It should be noted that after the pooling module generates the pooling result data, the neural network acceleration unit completes the acceleration of convolution operations during inference of the target convolutional neural network. As explained above, the security processing unit can protect the encrypted network parameters of the target convolutional neural network. This unit can simultaneously cache encrypted weight parameters and encrypted pooling kernels, reducing area footprint and cost. Furthermore, caching activation functions using the activation function caching module can further accelerate convolution processing during inference of the target convolutional neural network.

[0078] In one embodiment of the present invention, the convolutional neural network acceleration device further includes a data storage unit for data storage and a DMA module adapted and connected to the data storage unit, wherein...

[0079] The DMA module is also connected to the security processing unit and to the convolution kernel unit through the input feature data caching module;

[0080] During convolution processing, the input feature source data in the data storage unit is moved to the input feature data cache module through the DMA module to form the target input feature data;

[0081] The convolution kernel unit reads the target input feature data from the input feature data cache module and receives the decrypted weight data loaded by the security processing unit, so as to perform convolution operation on the target input feature data using the decrypted weight data.

[0082] Depend on Figure 1 and Figure 2 It can be seen that, Figure 1 This is a main block diagram of the convolutional neural network acceleration device of the present invention. Figure 2 The diagram shows a complete block diagram of the convolutional neural network acceleration device of the present invention. Figure 2 As can be seen from the embodiments shown, the convolutional neural network acceleration device also includes a data storage unit and a DMA module. Figure 2 In this context, DRAM stands for Data Storage Unit, which is... Figure 2 The diagram illustrates one embodiment where the data storage unit uses DRAM (Dynamic Random Access Memory). Of course, the data storage unit can also employ other storage methods. The DMA (Direct Memory Access) module is connected to the data storage unit and the security processing unit, and is also connected via an input feature data cache module.

[0083] As explained above, the encrypted network parameters are generated offline by encrypting the network parameters of the target convolutional neural network. To store these encrypted network parameters in the secure processing unit, they can first be loaded into the data storage unit. Then, DMA is used to transfer the encrypted network parameters to the secure processing unit for storage and decryption operations. The details of the secure processing unit can be found in [reference needed]. Figure 2 And the corresponding explanations below.

[0084] Target input feature data specifically refers to the data object that the target convolutional neural network will use for inference. To form the target input feature data loaded into the convolutional kernel, the input feature source data can first be stored in a data storage unit. Then, a DMA module is used to load the input feature source data into an input feature data cache module. The input feature source data stored in the input feature data cache module forms the target input feature data. Once in the input feature data cache module, the target input feature data can be read by the convolutional kernel unit for convolution processing.

[0085] In practice, during convolution processing, the convolution kernel reads the target input feature data from the input feature data cache module and obtains the decrypted weight data from the security processing unit. Subsequently, the decrypted weight data can be used to perform convolution operations on the target input feature data. In practice, the method of using the DMA module to move the encrypted network parameters to the security processing unit and the method of moving the input feature source data to the input feature data cache module can be consistent with existing technologies.

[0086] In one embodiment of the present invention, it further includes a control module, a control interface module adapted and connected to the control module, and a data interface module, wherein,

[0087] The control module is adapted and connected to the data storage unit through the data interface module;

[0088] The control module is adapted to connect with the security processing unit and the neural network acceleration unit through the control interface module, so as to configure the working status of the security processing unit and the neural network acceleration unit through the control interface module.

[0089] Figure 2 The embodiment shown also includes a control module. The control module can employ commonly used control methods and can configure the overall operating state of the neural network acceleration device. For example, the control module can configure the operating states of the security processing unit and the neural network acceleration unit. Specifically, configuring the operating state of the security processing unit allows it to store the encrypted network parameters transferred by the DMA module and decrypt the encrypted network parameters using a decryption key. Configuring the operating state of the neural network acceleration unit includes at least performing the aforementioned convolution processing. Alternatively, the control module can also store the encrypted network parameters and input feature source data in the data storage unit via the data interface module, and then control the DMA module to transfer the encrypted network parameters and input feature source data.

[0090] In practical implementation, the data interface module and control interface module can adopt existing commonly used interface types. For example, the data interface module can store encrypted network parameters and input feature source data in the data storage unit, and the control interface module can control the working status of the DMA module, the security processing unit, and the neural network acceleration unit.

[0091] In one embodiment of the present invention, the security processing unit includes an encrypted data caching module, a one-time programmable module, and a decryption module, wherein,

[0092] The encrypted network parameters generated offline are transferred to the encrypted data cache module via the DMA module so that the encrypted network parameters can be buffered by the encrypted data cache module.

[0093] The decryption key is stored using a one-time programmable module, and the one-time programmable module is configured to be unreadable by software.

[0094] The decryption module is adapted and connected to the encrypted data caching module and the one-time programmable module.

[0095] During convolution processing, the decryption module reads the encrypted network parameters from the encrypted data cache module and the encryption key from the one-time programmable module. Then, the read encryption key is used to decrypt the encrypted network parameters so that decrypted weight data and decrypted pooling kernel can be generated after the decryption operation.

[0096] The decryption module sends the decryption weight data generated by decryption to the convolution kernel unit, and sends the decryption pooling kernel generated by decryption to the pooling module.

[0097] Figure 2 The diagram illustrates one embodiment of a security processing unit. As shown, the security processing unit may include an encrypted data cache module, a one-time programmable module, and a decryption module. The encrypted data cache module stores encrypted network parameters, and the one-time programmable module stores the decryption key. When the one-time programmable module is configured to be unreadable by software, the decryption key stored within it cannot be altered. It is understood that the decryption key is a set of keys corresponding to the encryption key. Therefore, based on the encryption key used to encrypt the network parameters, the form of the decryption key can be determined. The encryption and decryption key configurations are consistent with existing technologies and will not be elaborated further here.

[0098] The decryption module is connected to the encrypted data cache module and the one-time programmable module. The decryption module can read encrypted weight parameters and encrypted pooling kernels from the encrypted data cache module, and can also read the decryption key from the one-time programmable module. Subsequently, the decryption key can be used to decrypt the encrypted weight parameters to generate decrypted weight parameters. When decrypting the encrypted pooling kernel using the decryption key, a decrypted pooling kernel can be generated after decryption. As explained above, the decrypted weight parameters generated by the decryption module should be loaded into the convolutional kernel unit, and the decrypted pooling kernel generated by the decryption module should be loaded into the pooling module. The decryption module can adopt existing commonly used forms, and the method and process of the decryption module using the decryption key can be consistent with existing methods.

[0099] As explained above, when the control module connects to the security processing unit via the control interface module, it specifically means that the control module connects to the encrypted data cache module, the decryption module, and the one-time programmable module respectively via the control interface module, in order to control the corresponding working states of the encrypted data cache module, the decryption module, and the one-time programmable module. It can be understood that controlling the corresponding working states of the encrypted data cache module, the decryption module, and the one-time programmable module specifically means enabling the aforementioned security processing modules to perform their functions.

[0100] The following example illustrates the process of encrypting network parameters. Specifically:

[0101] The data block size of the encryption algorithm corresponding to the encryption key is set to S. Then, the weight parameters and pooling kernels in the target convolutional neural network are constructed into data block units with a unit length of S. All data block units form a data group. The way to construct the weight parameters and pooling kernels into data block units with a unit length of S can be consistent with existing technologies. For example, if the encryption algorithm uses the symmetric encryption algorithm AES128, the corresponding data block size S is 128 bits. The minimum encrypted data block is 128 bits. In this case, the decryption module should be the AES128 decryption module, that is, decryption is performed based on the 128-bit unit data block. Generally, each data unit contains only the weight parameters or pooling kernels of one convolutional layer.

[0102] After the data group is formed, each data block within the data group can be encrypted offline using an encryption algorithm. Each encrypted data block becomes an encrypted data block. The control module can store the encrypted data block in the data storage unit. Subsequently, the DMA module moves the encrypted data block to the encrypted data cache module, stores the decryption key in the data storage unit, and then moves it to the one-time programmable module via the DMA module. It can be understood that the decryption module decrypts each encrypted data block, and the resulting decrypted data block also has a data length of S. For example, if the block length S is 128 bits, then the decrypted data block will also have a data length of 128 bits.

[0103] Generally, within a target convolutional neural network, each convolutional layer can have multiple channels of filters. In a single convolutional layer, each batch of inputs is calculated with the weight data of multiple channel filters. When the target convolutional neural network supports a data bit width L=8, the number of filters that need to be interleaved should be M=128 / 8=16. Here, data interleaving specifically refers to the process of rearranging the positions of data under the premise that the data sequence corresponds one-to-one. The method and process of data interleaving can be consistent with existing technologies, and the specific requirements should be based on the computation that can meet the inference process of the convolutional neural network.

[0104] In one embodiment of the present invention, the convolution kernel unit includes a serialization module, a multiply-accumulate module, and an accumulator module connected in sequence, wherein,

[0105] During convolution operations, the serialization module is used to rearrange the target input feature data and the decrypted weight data respectively.

[0106] After rearranging the target input feature data, an input feature sub-target data group is generated, wherein the input feature sub-target data group includes several input feature sub-target data arranged in time sequence;

[0107] After rearranging the decryption weight data, a decryption weight sub-target data group is generated, which includes several decryption weight sub-target data arranged in chronological order.

[0108] Based on the temporal arrangement characteristics of the input feature sub-target data and the decryption weight sub-target data, the multiply-accumulate module is used to perform multiply-accumulate operations to generate the corresponding partial and result data after the multiply-accumulate operations.

[0109] The accumulator module is used to perform an accumulation operation on the partial output and result data of the multiply-accumulator module to generate the convolution operation result data after the accumulation operation.

[0110] Figure 2 The figure illustrates one embodiment of a convolution kernel unit. The convolution kernel unit includes a serialization module, a multiply-accumulate module, and an accumulator module. The multiply-accumulate module can perform multiplication-addition operations, and the accumulator module can perform addition operations.

[0111] To accommodate different data bit widths, the convolution kernel unit can utilize a serialization module to rearrange the data during convolution. Figure 2 In this module, the serialization module is connected to the decryption module and the input feature data buffer module. The serialization module can read the target input feature data in the input feature data buffer module and the decryption weight data output by the decryption module. It then performs data rearrangement on the read target input feature data and decryption weight data. After rearranging the target input feature data, an input feature sub-target data group is generated. After rearranging the decryption weight data, a decryption weight sub-target data group is generated. The following is a specific example illustrating how the serialization module performs data rearrangement.

[0112] In one embodiment of the present invention, when the serialization module rearranges the target input feature data and the decryption weight data, it includes:

[0113] Based on the stripe operation of the multiply-accumulate module, the target input feature data and decrypted weight data are polled to split the target input feature data into several feature data packets and the decrypted weight data into several weight data packets after polling.

[0114] The aforementioned split feature data packets are assembled into corresponding input feature sub-target data, and the split weight data packets are assembled into corresponding decryption weight sub-target data, wherein,

[0115] The feature data packets are arranged in polling order, and the corresponding feature data packets arranged in polling order are assembled based on the multiply-accumulate calculation bit width in the multiply-accumulate module, so as to generate a corresponding input feature sub-target data after assembly.

[0116] The weight data packets are arranged in polling order, and the corresponding weight data packets arranged in polling order are assembled based on the multiply-accumulate calculation bit width in the multiply-accumulate module, so as to generate a corresponding decryption weight subtarget data after assembly.

[0117] The assembled input feature sub-target data and decrypted weight sub-target data are loaded into the multiply-accumulate module in a pipeline manner.

[0118] It should be noted that the input feature source data stored in the data storage unit should also be constructed into a data block of length S using the above network parameters. That is, the input feature source data should include multiple data blocks of length S. As can be seen from the above description, the target input feature data should also consist of multiple data blocks.

[0119] When the serialization module polls the target input feature data and decryption weight data, it should be based on the stripe operation of the multiply-accumulate module. The sliding of the convolutional kernel on the feature map is called a stripe. By adjusting the size of the stripe, the size of the feature map can be controlled, thus affecting the performance of feature extraction. Therefore, the stripe operation of the multiply-accumulate module specifically refers to the stride of the input data movement when the convolutional kernel unit performs convolution operations. When the serialization module polls the target input feature data and decryption weight data, it obtains the corresponding bit data within each data block. For example, when the data block length S is 128 bits, each polling can obtain 8 bits of data within each data block. That is, after 16 polls, all data within the current data block can be obtained. At this time, the feature data packet contains 8 bits of data, and the weight data packet also contains 8 bits of data. Therefore, the state of the feature data packet and weight data packet can be determined based on the data block length and the stripe operation of the multiply-accumulate module.

[0120] As explained above, during polling, multiple feature data packets can be generated sequentially. These packets are then assembled into corresponding input feature sub-target data. Generally, the amount of data within the input feature sub-target data can be selected according to actual needs, such as a data width of 1024 bits. In this case, if a feature data packet contains 8 bits of data, the input feature sub-target data should include 128 feature data packets. Assembling these 128 feature data packets forms 64 data blocks with a length of 16 bits. Therefore, the input feature sub-target data can include 64 16-bit data blocks. For the specific assembly method to form the decryption weight sub-target data, please refer to the explanation of assembling the input feature sub-target data here; it will not be repeated here.

[0121] To meet the multiply-accumulate operation requirements of the multiply-accumulate module, the input feature sub-target data and the decryption weight sub-target data can be synchronously loaded into the multiply-accumulate module in a pipeline manner to improve the efficiency of the multiply-accumulate module in performing multiply-accumulate calculations.

[0122] In practice, the multiply-accumulate module typically includes multiple identical MAC cores. Each MAC core contains multiple multipliers and multiple adders. For example, the multiply-accumulate module may include 16 MAC cores, each containing 64 16-bit multipliers and 72 adders. The configuration of the multiply-accumulate module and MAC cores can be selected according to actual needs, and will not be listed here.

[0123] When each MAC core includes 64 16-bit multipliers, for int8 data format, each multiplier and adder can be divided into two computation units. The throughput of int8 is twice that of int16, and the output result is called a partial sum. Here, int8 data format is an 8-bit binary number, and int16 data format is a 16-bit binary number. The multiply-accumulate module performs multiply-add calculations on the input feature sub-target data and decryption weight sub-target data to generate the corresponding partial sum result data after the multiply-add operation. Specifically, during a stripe operation, the decryption weight sub-target data in the MAC core remains unchanged, while the input feature sub-target data slides along the input data cube (i.e., a point in the convolution kernel at the same position, and feature data points at different positions are continuously multiplied). The way and process of the multiply-accumulate module performs multiply-add calculations can be consistent with existing technologies and will not be elaborated here.

[0124] In one embodiment of the present invention, the serialization module includes a convolutional data loader, a weight data loader, and a convolutional sequence generator, wherein,

[0125] The convolutional data loader polls the target input feature data and sends the feature data packets generated by the polling split to the feature FIFO unit of the convolutional sequence generator.

[0126] The weight data loader polls the decrypted weight data and sends the weight data packets generated by the polling split to the weight FIFO unit of the convolution sequence generator;

[0127] The convolution sequence generator assembles the feature data packets in the feature FIFO unit to generate input feature sub-target data after assembly, and sends the generated input feature sub-target data to the convolution data loader so that the convolution data loader can load the input feature sub-target data into the multiply-accumulate module one by one.

[0128] The convolution sequence generator assembles the weight data packets in the weight FIFO unit to generate decrypted weight subtarget data. The generated decrypted weight subtarget data is then sent to the weight data loader, which loads the decrypted weight subtarget data into the multiply-accumulate module one by one.

[0129] Figure 3 The figure illustrates one embodiment of the serialization module connected and cooperating with a multiply-accumulate module, etc. The serialization module may include a convolutional data loader, a weight data loader, and a convolutional sequence generator. The convolutional data loader polls the target input feature data, and the weight data loader polls the decrypted weight data. To enable the sorting of feature data packets and weight data packets, feature FIFO units and weight FIFO units can be set within the convolutional sequence generator. That is, utilizing the characteristics of FIFO, feature data packets and weight data packets can be sorted in time sequence.

[0130] In specific implementation, a feature counter and a weight counter are also set in the convolution sequence generator. The feature counter and the weight counter should preferably be decrementing counters. The feature counter can count the feature data packets, and the weight counter can count the weight data packets. When the corresponding counts of the feature counter and the weight counter are both 0, it indicates that the assembly of the feature data packets and the weight data packets is complete. After that, the multiply-accumulate module can be checked. When checking the multiply-accumulate module, the credit value of the multiply-accumulate module is mainly checked to determine whether the input feature sub-target number and the decrypted weight sub-target data should be sent to the multiply-accumulate module.

[0131] It should be noted that the credit value of the multiply-accumulator module, as well as the method for checking the credit value of the multiply-accumulator module, can be consistent with existing technologies. Specifically, the working status of the multiply-accumulator module should be determined, such as the free space of the multiply-accumulator module, so as to coordinate the efficiency of data transmission between the sequence generator and the multiply-accumulator module.

[0132] In one embodiment of the present invention, when the accumulator module performs an accumulation operation on the partial and result data, it generates source data of the convolution operation result;

[0133] The source data of the generated convolution operation result is subjected to data constraint processing to generate the convolution operation result data after data constraint processing. The data constraint processing of the source data of the convolution operation result includes round operation and / or saturate operation.

[0134] Specifically, the accumulator module is used to accumulate the output of the multiply-accumulate module. After accumulation, data constraint is performed, such as rounding and / or saturating. During data constraint, the source data of the convolution operation result is first rounded. If data overflow occurs due to carry during rounding, the signal is usually saturated. Saturation means that if the calculation result exceeds the maximum value that the required data format can store, then the maximum value is used to represent this data.

[0135] In practice, the accumulator module can adopt a commonly used form, specifically one that can perform accumulation operations and ultimately generate convolution operation result data.

[0136] In summary, a secure and encrypted method for accelerating convolutional neural networks can be obtained. In one embodiment of the present invention, convolution processing is performed on any convolutional neural network using the aforementioned convolutional neural network acceleration device.

[0137] Figure 4 The diagram below illustrates one embodiment of the neural network acceleration device of the present invention. Figure 4 As can be seen, when using a neural network acceleration device, the parameters are first initialized. Then, the encryption weights that constitute the encryption network parameters and the decryption key are written into the encryption data cache module, and the activation function is written into the activation function cache module. Next, the registers of the DMA module and the one-time programmable module are configured to write the decryption key into the register of the one-time programmable module.

[0138] During inference, the decryption module decrypts the encrypted network parameters and provides target input feature data. Then, the convolution kernel performs a convolution operation on the target input feature data using the decrypted weight data. The activation module activates the convolution result data, and the pooling module pools the activation result data. Finally, the pooling module stores the pooling result data in the data storage unit so that the control module can read the pooling result data through the data interface module. As can be seen from the above description, Figure 4 The DRAM mentioned above is one embodiment of the aforementioned data storage module, and you can refer to the corresponding description above for details.

[0139] Specifically, the methods and processes for performing convolution processing using convolutional neural network acceleration devices can be found in the relevant descriptions above, and will not be repeated here.

Claims

1. A convolutional neural network acceleration device with secure encryption, characterized in that, The convolutional neural network acceleration device includes: A security processing unit is used to store decryption keys and encrypted network parameters, wherein the encrypted network parameters are generated by offline encryption of the network parameters of the target convolutional neural network using the encryption key, the encryption key matches the decryption key, and the encrypted network parameters include at least encrypted weight parameters and encrypted pooling kernels; The neural network acceleration unit, used to perform convolution processing during the operation of the target convolutional neural network, includes at least a convolution kernel unit for performing convolution operations, an activation function unit for performing activation operations, and a pooling module for performing pooling operations. The convolutional kernel unit, activation function unit, and pooling module are connected in sequence; When performing convolution processing, the target input feature data and decryption weight data are loaded into the convolution kernel unit so that the convolution kernel unit can perform convolution operation on the decryption weight data and the target input feature data, and generate convolution operation result data after the convolution operation. The target input feature data is generated by the security processing unit after decrypting the input feature source data, and the decryption weight data is generated by the security processing unit after decrypting the encrypted weight parameters. The activation function unit is used to activate the convolution result data to generate activation result data after the activation operation. The activation operation result data is pooled using a pooling module to generate pooled result data after the pooling operation. During the pooling operation, the activation processing result data is pooled using a decrypted pooling core generated by the security processing unit decrypting the encrypted pooling core.

2. The convolutional neural network acceleration device with secure encryption according to claim 1, characterized in that: The convolutional kernel unit includes a serialization module, a multiply-accumulate module, and an accumulator module connected in sequence, wherein... During convolution operations, the serialization module is used to rearrange the target input feature data and the decrypted weight data respectively. After rearranging the target input feature data, an input feature sub-target data group is generated, wherein the input feature sub-target data group includes several input feature sub-target data arranged in time sequence; After rearranging the decryption weight data, a decryption weight sub-target data group is generated, which includes several decryption weight sub-target data arranged in chronological order. Based on the temporal arrangement characteristics of the input feature sub-target data and the decryption weight sub-target data, the multiply-accumulate module is used to perform multiply-accumulate operations to generate the corresponding partial and result data after the multiply-accumulate operations. The accumulator module is used to perform an accumulation operation on the partial output and result data of the multiply-accumulator module to generate the convolution operation result data after the accumulation operation.

3. The convolutional neural network acceleration device with secure encryption according to claim 2, characterized in that, When the serialization module rearranges the target input feature data and decryption weight data, it includes: Based on the stripe operation of the multiply-accumulate module, the target input feature data and decrypted weight data are polled to split the target input feature data into several feature data packets and the decrypted weight data into several weight data packets after polling. The aforementioned split feature data packets are assembled into corresponding input feature sub-target data, and the split weight data packets are assembled into corresponding decryption weight sub-target data, wherein, The feature data packets are arranged in polling order, and the corresponding feature data packets arranged in polling order are assembled based on the multiply-accumulate calculation bit width in the multiply-accumulate module, so as to generate a corresponding input feature sub-target data after assembly. The weight data packets are arranged in polling order, and the corresponding weight data packets arranged in polling order are assembled based on the multiply-accumulate calculation bit width in the multiply-accumulate module, so as to generate a corresponding decryption weight subtarget data after assembly. The assembled input feature sub-target data and decrypted weight sub-target data are loaded into the multiply-accumulate module in a pipeline manner.

4. The convolutional neural network acceleration device with secure encryption according to claim 3, characterized in that, The serialization module includes a convolutional data loader, a weight data loader, and a convolutional sequence generator, wherein... The convolutional data loader polls the target input feature data and sends the feature data packets generated by the polling split to the feature FIFO unit of the convolutional sequence generator. The weight data loader polls the decrypted weight data and sends the weight data packets generated by the polling split to the weight FIFO unit of the convolution sequence generator; The convolution sequence generator assembles the feature data packets in the feature FIFO unit to generate input feature sub-target data after assembly, and sends the generated input feature sub-target data to the convolution data loader so that the convolution data loader can load the input feature sub-target data into the multiply-accumulate module one by one. The convolution sequence generator assembles the weight data packets in the weight FIFO unit to generate decrypted weight subtarget data. The generated decrypted weight subtarget data is then sent to the weight data loader, which loads the decrypted weight subtarget data into the multiply-accumulate module one by one.

5. The convolutional neural network acceleration device with secure encryption according to claim 3, characterized in that, When the accumulator module performs accumulation operations on the partial and result data, it generates the source data of the convolution operation result; The source data of the generated convolution operation result is subjected to data constraint processing to generate the convolution operation result data after data constraint processing. The data constraint processing of the source data of the convolution operation result includes round operation and / or saturate operation.

6. The convolutional neural network acceleration device with secure encryption according to any one of claims 1 to 5, characterized in that: The activation function unit includes an activation function cache module and an activation module, wherein, When performing the activation operation, the activation module reads the activation function from the activation function cache module, uses the read activation function to perform the activation operation on the convolution operation result data, and generates the activation operation result data after the activation operation. The activation module loads the generated activation operation result data into the pooling module.

7. The convolutional neural network acceleration device with secure encryption according to any one of claims 1 to 5, characterized in that: The convolutional neural network acceleration device also includes a data storage unit for data storage and a DMA module adapted and connected to the data storage unit, wherein... The DMA module is also connected to the security processing unit and to the convolution kernel unit through the input feature data caching module; During convolution processing, the input feature source data in the data storage unit is moved to the input feature data cache module through the DMA module to form the target input feature data; The convolution kernel unit reads the target input feature data from the input feature data cache module and receives the decrypted weight data loaded by the security processing unit, so as to perform convolution operation on the target input feature data using the decrypted weight data.

8. The convolutional neural network acceleration device with secure encryption according to claim 7, characterized in that: The security processing unit includes an encrypted data caching module, a one-time programmable module, and a decryption module, wherein... The encrypted network parameters generated offline are transferred to the encrypted data cache module via the DMA module so that the encrypted network parameters can be buffered by the encrypted data cache module. The decryption key is stored using a one-time programmable module, and the one-time programmable module is configured to be unreadable by software. The decryption module is adapted and connected to the encrypted data caching module and the one-time programmable module. During convolution processing, the decryption module reads the encrypted network parameters from the encrypted data cache module and the encryption key from the one-time programmable module. Then, the read encryption key is used to decrypt the encrypted network parameters so that decrypted weight data and decrypted pooling kernel can be generated after the decryption operation. The decryption module sends the decryption weight data generated by decryption to the convolution kernel unit, and sends the decryption pooling kernel generated by decryption to the pooling module.

9. The convolutional neural network acceleration device with secure encryption according to claim 7, characterized in that: It also includes a control module, a control interface module adapted and connected to the control module, and a data interface module, wherein, The control module is adapted and connected to the data storage unit through the data interface module; The control module is adapted to connect with the security processing unit and the neural network acceleration unit through the control interface module, so as to configure the working status of the security processing unit and the neural network acceleration unit through the control interface module.

10. A method for accelerating convolutional neural networks with secure encryption, characterized in that, For any convolutional neural network, convolution processing is performed using any one of the convolutional neural network acceleration devices according to claims 1 to 9.

Images