A neural network model inference method for improving security of intermediate data

Abstract

The application discloses a neural network model inference method for improving the security of intermediate data, and relates to the field of neural networks. In the method, the NPU encrypts the intermediate feature map data generated by each operator layer in real time during the neural network model inference process and stores the data in the encrypted form in the end-side SoC memory. The intermediate data leakage caused by the analysis of the memory data by attackers can be effectively prevented, the intermediate data output by the neural network model is unbreakable, the threat of privacy leakage such as member inference attack can be effectively resisted, the user privacy data is ensured not to be stolen, and the data security of the neural network model during the inference process is greatly improved, thereby providing a powerful guarantee for the information security of the user.

Description

Technical Field

[0001] This application relates to the field of neural networks, and in particular to a neural network model inference method for improving the security of intermediate data. Background Technology

[0002] With the widespread application of deep learning technology, deep learning accelerators (NPUs) play a crucial role in accelerating the operation of neural network algorithm models. The current mainstream architecture utilizes deep learning frameworks for model training, and the trained neural network algorithm model is then deployed to the edge SoC (System on Chip). Common deep learning frameworks include PyTorch, TensorFlow, and ONNX. Due to the compatibility issues between deep learning frameworks and edge SoCs in terms of model format, edge SoCs typically cannot directly run the model format trained by deep learning frameworks. Instead, they need to first use model compilation tools to compile the deep learning framework model format into the edge SoC model format. The compiled model file is stored in the memory of the edge SoC for inference by the NPU integrated into the edge SoC.

[0003] During model inference, the NPU generates a large amount of intermediate data, which is written into the memory of the edge SoC for subsequent inference. However, research has found that this intermediate data is vulnerable to attacks targeting memory data analysis, and thus vulnerable to member inference attacks. Attackers can use this intermediate data to steal sensitive information from the training data or the model's own attribute parameters, leading to data leakage and posing a security risk. Summary of the Invention

[0004] This application addresses the aforementioned problems and technical requirements by proposing a neural network model inference method to improve the security of intermediate data. The technical solution of this application is as follows:

[0005] A neural network model inference method to improve the security of intermediate data, the neural network model inference method comprising:

[0006] The NPU integrated in the edge SoC uses the built-in storage control unit to load the LayerPackage data block in the model file stored in the edge SoC memory, and uses the built-in parameter parsing unit to parse the layer information of each layer operator in the neural network model. The layer information of any layer operator includes the operator information of the current layer operator, the input feature map storage information of the current layer operator, and the output feature map storage information of the current layer operator.

[0007] For any integer parameter i, when the i-th layer operator is the first layer operator, the NPU uses the built-in storage control unit to load the model input data as the input data of the first layer operator. Otherwise, the NPU uses the built-in storage control unit to store information according to the input feature map of the i-th layer operator, loads the ciphertext of the input feature map of the i-th layer operator from the edge SoC memory into the NPU, and uses the built-in security operation unit to decrypt it inside the NPU and use it as the input data of the i-th layer operator.

[0008] The NPU uses its built-in operator execution unit to perform neural network operations on the input data of the i-th layer operator according to the operator information of the i-th layer operator to obtain the output data of the i-th layer operator;

[0009] When the i-th layer operator is the last layer operator, the NPU's built-in storage control unit outputs the output data of the i-th layer operator as the model output result. Otherwise, the NPU's built-in security operation unit encrypts the output data of the i-th layer operator inside the NPU to obtain the ciphertext of the output feature map of the i-th layer operator. Then, using the built-in storage control unit, the ciphertext of the output feature map of the i-th layer operator is written into the edge SoC memory as the ciphertext of the input feature map of the next layer operator based on the storage information of the output feature map of the i-th layer operator.

[0010] A further technical solution is that the NPU's built-in operator execution unit includes a neural network acceleration unit and an operator coprocessor (GCU). The NPU utilizes the built-in operator execution unit to perform neural network operations on the input data of the i-th layer operator, including:

[0011] The NPU uses its built-in storage control unit to load the soft operator executable file carried by the GCU package data block in the model file, and the operator coprocessor GCU executes the soft operator executable file.

[0012] When the i-th layer operator is a software operator, the operator coprocessor (GCU) performs neural network operations on the input data of the i-th layer operator according to the operator information of the i-th layer operator; when the i-th layer operator is a hardware operator, the neural network acceleration unit performs neural network operations on the input data of the i-th layer operator according to the operator information of the i-th layer operator.

[0013] Further technical solutions include: the neural network model inference method also includes:

[0014] The CPU integrated into the edge SoC parses the ModelPackage data block in the model file to obtain hash configuration information, and writes the hash configuration information into the NPU's hash control register for hash configuration;

[0015] After the NPU's built-in storage control unit loads the soft operator executable file carried by the GCUPackage data block in the model file into the NPU, the NPU's built-in security operation unit performs a hash operation verification on the soft operator executable file according to the hash configuration information in the hash control register. If the hash operation verification of the soft operator executable file by the NPU's built-in security operation unit passes, the GCU executes the soft operator executable file; otherwise, it returns an error flag through the hash control register.

[0016] A further technical solution involves the following: the hash configuration information includes the hash algorithm type and theoretical hash value; and the hash operation verification of the soft operator executable file based on the hash configuration information in the hash control register includes:

[0017] The soft operator executable file is hashed according to the hash algorithm type in the hash control register to obtain the actual hash value. When the actual hash value is consistent with the theoretical hash value in the hash control register, the hash operation of the soft operator executable file is determined to be successful; otherwise, the hash operation is determined to be unsuccessful.

[0018] Further technical solutions include: the neural network model inference method also includes:

[0019] The CPU integrated into the edge SoC parses the ModelPackage data block in the model file to obtain encryption / decryption configuration information, and writes the encryption / decryption configuration information into the NPU's encryption / decryption control register for encryption / decryption configuration.

[0020] The NPU's built-in secure computing unit decrypts the ciphertext of the input feature maps of each layer operator to obtain the input data, and encrypts the output data of each layer operator to obtain the ciphertext of the output feature maps, based on the pre-written key, initialization vector, and encryption / decryption configuration information in the encryption / decryption control register. Furthermore, the same initialization vector is used in the process of performing a model inference on the neural network model to obtain the model output result from the model input data, while different initialization vectors are used in different model inference processes.

[0021] A further technical solution involves the NPU's built-in secure computing unit decrypting the ciphertext of the input feature maps of each layer of operators and encrypting the output data of each layer of operators, including:

[0022] When the i-th layer operator is the first layer operator, the secure operation unit generates a random number as an initialization vector, uses the pre-written key and combines the initialization vector with the encryption and decryption configuration information in the encryption and decryption control register to encrypt the output data of the first layer operator to obtain the ciphertext of the output feature map of the first layer operator, and caches the initialization vector inside the secure operation unit;

[0023] When the i-th layer operator is an intermediate layer operator other than the first layer operator and the last layer operator, the secure operation unit reads the cached initialization vector, uses the pre-written key and combines it with the initialization vector and the encryption / decryption configuration information in the encryption / decryption control register to decrypt the ciphertext of the input feature map of the i-th layer operator to obtain the input data of the i-th layer operator, and encrypts the output data of the i-th layer operator to obtain the ciphertext of the output feature map of the i-th layer operator, and caches the initialization vector inside the secure operation unit;

[0024] When the i-th layer operator is the last layer operator, the secure operation unit reads the cached initialization vector, uses the pre-written key and combines it with the initialization vector and the encryption / decryption configuration information in the encryption / decryption control register to decrypt the ciphertext of the input feature map of the last layer operator to obtain the input data of the last layer operator, and destroys the currently used initialization vector.

[0025] The further technical solution is that the operator information of any layer of operators includes operator attribute parameters and operator weight storage information, and the operator attribute parameters indicate the operator type, operator level and operator architecture;

[0026] Neural network model inference methods include:

[0027] The NPU uses its built-in parameter parsing unit to parse the operator attribute parameters and operator weight storage information of the i-th layer operator. Based on the operator level indicated by the operator attribute parameters, it determines whether the i-th layer operator is a first-layer operator, a last-layer operator, or an intermediate-layer operator. Based on the operator architecture indicated by the operator attribute parameters of the i-th layer operator, it determines whether the i-th layer operator is a hardware operator or a software operator.

[0028] Further technical solutions include: the NPU also incorporates an input feature map FIFO, an output feature map FIFO, and a weight FIFO; the neural network model inference method also includes:

[0029] After the parameter parsing unit parses the hierarchical information of the i-th layer operator, the storage control unit loads the operator weights of the i-th layer operator from the edge SoC memory according to the operator weight storage information of the i-th layer operator and caches them in the weight FIFO. When the i-th layer operator is determined to be the first layer operator according to the operator hierarchy indicated by the operator attribute parameters, the storage control unit loads the model input data as the input data of the first layer operator and caches it in the input feature map FIFO. Otherwise, the storage control unit loads the ciphertext of the input feature map of the i-th layer operator from the edge SoC memory, and the secure operation unit decrypts the ciphertext of the input feature map to obtain the input data of the i-th layer operator and caches it in the input feature map FIFO.

[0030] Based on the operator architecture indicated by the operator attribute parameters, the neural network acceleration unit or operator coprocessor GCU is determined as the operator execution unit. The operator execution unit reads the input data of the i-th layer operator from the input feature map FIFO, reads the operator weights of the i-th layer operator from the weight FIFO, obtains the operator type of the i-th layer operator from the parameter parsing unit, and performs neural network operations on the input data of the i-th layer operator and the operator weights of the i-th layer operator in the input feature map FIFO according to the calculation method corresponding to the operator type of the i-th layer operator, and obtains the output data of the i-th layer operator which is cached in the output feature map FIFO.

[0031] When the operator at layer i is determined to be the last operator based on the operator hierarchy indicated by the operator attribute parameters, the storage control unit reads the output data of the i-th layer operator from the output feature map FIFO and outputs it as the model output result; otherwise, the secure operation unit reads the output data of the i-th layer operator from the output feature map FIFO and encrypts it to obtain the ciphertext of the output feature map. The storage control unit writes the ciphertext of the output feature map of the i-th layer operator into the edge SoC memory according to the storage information of the output feature map of the i-th layer operator.

[0032] Further technical solutions include: the neural network model inference method also includes:

[0033] The CPU integrated on the edge SoC parses the model file to obtain the storage information of the soft operator executable file, and writes the storage information of the soft operator executable file into the GCUPackage offset address register;

[0034] The NPU's built-in storage control unit reads the soft operator executable file storage information from the GCUPackage offset address register and loads the soft operator executable file from the edge SoC memory based on the soft operator executable file storage information.

[0035] The further technical solution is that the compiled model file includes ModelPackage data blocks, GCUPackage data blocks, GraphPackage data blocks, LayerPackage data blocks, and TensorPackage data blocks, wherein:

[0036] The ModelPackage data block carries encryption / decryption configuration information, hash configuration information, and basic model attribute information;

[0037] GCUPackage data blocks contain executable files for soft operators;

[0038] The GraphPackage data block records the total number of operator layers in the neural network model, the offset address of the LayerPackage data block, and the data size of the LayerPackage data block;

[0039] LayerPackage data blocks record the hierarchical information of each layer operator in the neural network model;

[0040] TensorPackage data blocks record the operator weights of each layer of the neural network model in the inference method of the neural network model.

[0041] The beneficial technical effects of this application are:

[0042] This application discloses a neural network model inference method to improve the security of intermediate data. In this method, during the neural network model inference process, the NPU encrypts the intermediate feature map data in real time and stores it in encrypted form in the edge SoC memory. This can effectively prevent intermediate data leakage caused by attackers analyzing the memory data, protect the intermediate data output by the neural network model from being cracked, effectively resist privacy leakage threats such as member inference attacks, ensure that user privacy data is not stolen, greatly improve the data security of the neural network model during the inference process, and provide strong protection for user information security.

[0043] This method employs a unique encryption, configuration, and collaborative processing mechanism during model compilation and runtime. The closely coordinated encryption and decryption workflow between the CPU and NPU involves all aspects of data loading, parsing, decryption, computation, encryption, and storage, ensuring the security of intermediate data throughout the entire inference process. It fills the gaps in existing technologies regarding the protection of intermediate data during NPU model inference and has high innovation and practicality. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the system architecture of an edge SoC in which the neural network model inference method of one embodiment of this application is applied.

[0045] Figure 2 This is a control block diagram of a neural network model inference method in one embodiment of this application.

[0046] Figure 3 This is a flowchart illustrating a neural network model inference method in one embodiment of this application.

[0047] Figure 4 This is a schematic diagram of the management process of the initialization vector in one embodiment of this application.

[0048] Figure 5 This is a data structure diagram of a model file in one embodiment of this application. Detailed Implementation

[0049] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0050] To improve the security of intermediate data generated by the NPU during model inference, this application discloses a neural network model inference method that enhances the security of intermediate data. This method is applied to an edge SoC, which integrates a CPU, NPU, model compiler, and memory. Please refer to... Figure 1 The system architecture diagram shown illustrates that the model compilation tool integrated into the edge SoC completes the compilation of the deep learning framework model format into the edge SoC model format, and stores the compiled model file in the edge SoC memory. The NPU integrated into the edge SoC performs neural network model inference according to the model file.

[0051] The compiled model file includes at least a LayerPackage data block. This block records the layer information of each operator in the neural network model. The layer information for any given operator includes the operator information, the input feature map storage information, and the output feature map storage information. The input feature map storage information for each operator includes the data type, address, and dimension of the input feature map. Similarly, the output feature map storage information includes the data type, address, and dimension of the output feature map. Since the output feature map of each operator in the neural network model is input into the next operator, the output feature map storage information of each operator is consistent with the input feature map storage information of the next operator.

[0052] The edge-side SoC-integrated NPU includes a built-in storage control unit 210, a parameter parsing unit 220, a secure computing unit 230, and an operator execution unit 240. The neural network model inference method executed by the NPU according to the model file includes the following steps; please refer to [reference needed]. Figure 2 The control block diagram shown and Figure 3 The flowchart shown below:

[0053] In step 310, the NPU uses the built-in storage control unit 210 to load the LayerPackage data block in the model file stored in the terminal SoC memory, and uses the built-in parameter parsing unit 220 to parse the layer information of each layer operator in the neural network model, thereby obtaining the operator information, input feature map storage information and output feature map storage information of each layer operator.

[0054] Step 320: When the i-th layer operator is a first-layer operator, the NPU uses its built-in storage control unit 210 to load the model input data directly as the input data for the first-layer operator without decryption, where i is an integer parameter. When the i-th layer operator is not a first-layer operator, the NPU uses its built-in storage control unit 210 to load the ciphertext of the input feature map of the i-th layer operator from the edge SoC memory into the NPU, based on the input feature map storage information of the i-th layer operator. The NPU's built-in secure computing unit 230 decrypts the ciphertext of the input feature map of the i-th layer operator within the NPU to obtain the input data of the i-th layer operator.

[0055] Step 330: For any i-th layer operator, the NPU uses the built-in operator execution unit 240 to perform neural network operations on the input data of the i-th layer operator according to the operator information of the i-th layer operator parsed by the parameter parsing unit 220 to obtain the output data of the i-th layer operator.

[0056] Step 340: When the i-th layer operator is the last layer operator, the NPU's built-in storage control unit 210 does not encrypt the output data of the i-th layer operator, but directly uses its output as the model output result, thereby completing a complete neural network model inference process. When the i-th layer operator is not the last layer operator, the output data of the i-th layer operator needs to be temporarily stored for subsequent inference. In this case, the NPU's built-in secure operation unit 230 encrypts the output data of the i-th layer operator inside the NPU to obtain the ciphertext of the output feature map of the i-th layer operator. Then, the NPU's built-in storage control unit 210 writes the ciphertext of the output feature map of the i-th layer operator into the edge-side SoC memory according to the storage information of the output feature map of the i-th layer operator, as the ciphertext of the input feature map of the next layer operator.

[0057] Therefore, the intermediate feature maps generated by the NPU during the neural network model inference process are stored in encrypted form in the edge SoC memory, which effectively prevents data leakage caused by attackers analyzing the memory data, protects the intermediate results of the model output from being cracked, and greatly improves the data security of the neural network model during the inference process.

[0058] The operator information of any layer operator in the neural network model includes operator attribute parameters and operator weight storage information. The operator weight storage information indicates the address and dimension of the operator weights. The operator attribute parameters at least indicate the operator type. In addition, the operator attribute parameters also include a flag bit to indicate the operator level. The operator weights of each layer operator in the neural network model are contained in the model file and stored in the edge SoC. Specifically, the compiled model file also includes a TensorPackage data block, which records the operator weights of each layer operator in the neural network model. In the above embodiment, after the parameter parsing unit 220 parses the level information of any i-th layer operator in the neural network model in step 310, the operator attribute parameters, operator weight storage information, input feature map storage information, and output feature map storage information of the i-th layer operator can be determined. The NPU also has built-in input feature map FIFO 250, output feature map FIFO 260, and weight FIFO 270. The neural network model inference process implemented by the NPU specifically includes the following process, please refer to Figure 2 :

[0059] The storage control unit 210 loads the operator weights of the i-th layer operator from the edge SoC memory according to the operator weight storage information of the i-th layer operator and caches them in the weight FIFO 270. After caching, the storage information of the operator weights of the i-th layer operator in the weight FIFO 270 is provided to the operator execution unit 240 so that the operator execution unit 240 can read the operator weights of the i-th layer operator according to the storage information.

[0060] Furthermore, the operator level can be determined based on the operator attribute parameters of the i-th layer operator obtained from the analysis. When the i-th layer operator is determined to be a first-layer operator based on the operator level indicated by the operator attribute parameters, the storage control unit 210 loads the model input data as the input data of the first-layer operator, without decryption, and directly caches it in the input feature map FIFO 250. When the i-th layer operator is not a first-layer operator, the storage control unit 210 loads the encrypted input feature map of the i-th layer operator from the edge SoC memory according to the input feature map storage information of the i-th layer operator, and the secure operation unit 230 decrypts the encrypted input feature map to obtain the input data of the i-th layer operator and caches it in the input feature map FIFO 250. In either case, after caching, the storage control unit 210 or the secure operation unit 230 will provide the storage information of the input data of the i-th layer operator in the input feature map FIFO 250 to the operator execution unit 240 so that the operator execution unit 240 can read the input data of the i-th layer operator according to the stored information.

[0061] Then, the operator execution unit 240 reads the input data of the i-th layer operator from the input feature map FIFO 250, reads the operator weights of the i-th layer operator from the weight FIFO 270, obtains the operator type of the i-th layer operator from the parameter parsing unit 220, and performs neural network operations on the input data and operator weights of the i-th layer operator in the input feature map FIFO according to the calculation method corresponding to the operator type of the i-th layer operator, and obtains the output data of the i-th layer operator, which is cached in the output feature map FIFO 260. Similarly, the operator execution unit 240 provides the storage information of the output data of the i-th layer operator in the output feature map FIFO 260 to the storage control unit 210 or the safe operation unit 230, so that the storage control unit 210 or the safe operation unit 230 can read the output data of the i-th layer operator according to the storage information.

[0062] When the operator at layer i is determined to be the last operator based on the operator hierarchy indicated by the operator attribute parameters, there is no need to encrypt the output data. Instead, the storage control unit 210 directly reads the output data of the i-th layer operator from the output feature map FIFO 260 and outputs it as the model output result. When the i-th layer operator is not the last operator, the secure operation unit 230 reads the output data of the i-th layer operator from the output feature map FIFO 260 and encrypts it to obtain the ciphertext of the output feature map. Then, the storage control unit 210 writes the ciphertext of the output feature map of the i-th layer operator into the edge SoC memory according to the storage information of the output feature map of the i-th layer operator.

[0063] In one embodiment, the compiled model file also includes a ModelPackage data block, which carries encryption / decryption configuration information. The CPU integrated into the edge SoC parses the model file to obtain the encryption / decryption configuration information and writes it into the NPU's encryption / decryption control register for encryption / decryption configuration.

[0064] In step 320, the NPU's built-in secure computing unit 230 uses a pre-written key combined with the encryption / decryption configuration information in the encryption / decryption control register to decrypt the ciphertext of the input feature map of the i-th layer operator to obtain the input data of the i-th layer operator. And in step 340, it uses the pre-written key combined with the encryption / decryption configuration information in the encryption / decryption control register to encrypt the output data of the i-th layer operator to obtain the ciphertext of the output feature map of the i-th layer operator.

[0065] The encryption / decryption configuration information in this embodiment includes the encryption / decryption algorithm type and mode used. The encryption / decryption algorithm type is SM4 or AES symmetric algorithm, and the encryption / decryption mode is ECB, CBC, CFB, or OFB mode. The secure computing unit 230, in conjunction with a pre-written key, performs encryption / decryption operations on the intermediate feature map generated during inference according to the encryption / decryption algorithm type and mode in the encryption / decryption configuration information. The key used by the secure computing unit 230 is not updated in real time, but is written to the chip's internal secure key storage area using a manufacturer-specific key writing tool at the chip's factory. This secure key storage area is inaccessible to the CPU; only the secure computing unit 230 can read the key from this secure key storage area.

[0066] To enhance encryption and decryption security, the confidentiality of the intermediate feature map is protected during transmission and storage by increasing the randomness of the ciphertext and resisting specific cryptographic attacks. An initialization vector is introduced to work with the key to perform encryption and decryption operations on the intermediate feature map. Specifically, the secure operation unit 230 decrypts the input feature map ciphertext of each layer operator to obtain the input data, and encrypts the output feature map ciphertext of each layer operator to obtain the output feature map ciphertext, based on the pre-written key, initialization vector, and encryption / decryption configuration information in the encryption / decryption control register. This initialization vector is a random number; in practical applications, a 128-bit random number is typically used.

[0067] The same initialization vector is used during a single inference process on the neural network model, while different initialization vectors are used for different inference processes. A single inference process refers to the process of completing the neural network operations of each layer operator in the neural network model to obtain the model's output from the input data. This ensures that even when performing two different inference processes on the same input data, the resulting input / output feature map ciphertexts will be different, further enhancing the security of the intermediate feature maps. For details, please refer to [link / reference]. Figure 4 The flowchart shown below:

[0068] When the i-th layer operator is the first layer operator, the secure operation unit 230 generates a random number as the initialization vector IV, uses the pre-written key and combines the initialization vector IV with the encryption and decryption configuration information in the encryption and decryption control register to encrypt the output data of the first layer operator to obtain the ciphertext of the output feature map of the first layer operator, and caches the initialization vector inside the secure operation unit 230.

[0069] When the i-th layer operator is an intermediate layer operator other than the first layer operator and the last layer operator, the secure operation unit 230 reads the cached initialization vector IV, uses the pre-written key and combines the initialization vector IV with the encryption / decryption configuration information in the encryption / decryption control register to decrypt the ciphertext of the input feature map of the i-th layer operator to obtain the input data of the i-th layer operator, and encrypts the output data of the i-th layer operator to obtain the ciphertext of the output feature map of the i-th layer operator, and continues to cache the initialization vector IV inside the secure operation unit 230.

[0070] When the i-th layer operator is the last layer operator, the secure operation unit 230 reads the cached initialization vector IV, uses the pre-written key, and combines the initialization vector IV with the encryption / decryption configuration information in the encryption / decryption control register to decrypt the ciphertext of the input feature map of the last layer operator to obtain the input data of the last layer operator, and destroys the currently used initialization vector IV. Then, it reads the initialization vector IV generated after the first layer operator is encountered in the next inference process.

[0071] In one embodiment, the operator execution unit 240 built into the NPU includes a neural network acceleration unit and an operator coprocessor (GCU). The GCU needs to execute a soft operator executable file to implement corresponding functions. This soft operator executable file is contained in the model file and stored in the edge SoC memory. Therefore, in this embodiment, the compiled model file also includes a GCUPackage data block, which contains the soft operator executable file. The NPU uses its built-in storage control unit 210 to load the soft operator executable file carried in the GCUPackage data block of the model file, thereby enabling the GCU to execute the soft operator executable file to implement the corresponding processor functions.

[0072] When the operator execution unit 240 performs neural network operations on the input data of the i-th layer operator, it is executed collaboratively by the neural network acceleration unit and the operator coprocessor GCU. Specifically:

[0073] When the i-th layer operator is a software operator, the operator coprocessor GCU performs neural network operations on the input data of the i-th layer operator according to the operator information of the i-th layer operator. That is, the operator coprocessor GCU operation part cannot be hardware-hardened for operators.

[0074] When the i-th layer operator is a hardware operator, the neural network acceleration unit performs neural network operations on the input data of the i-th layer operator according to the operator information of the i-th layer operator.

[0075] To distinguish whether the i-th layer operator is a hardware operator or a software operator, in one embodiment, the operator attribute parameters of any layer operator also include a flag bit to indicate the operator architecture of that layer operator. The i-th layer operator is determined to be a hardware operator or a software operator based on the operator architecture indicated by the operator attribute parameters, and the neural network acceleration unit or operator coprocessor GCU is determined to be used as the operator execution unit to perform neural network operations based on the operator architecture indicated by the operator attribute parameters.

[0076] In another embodiment, to further improve the security and reliability of the model inference process, the ModelPackage data block in the model file also carries hash configuration information. The CPU integrated into the edge SoC parses the ModelPackage data block in the model file to obtain the hash configuration information and writes the hash configuration information into the NPU's hash control register for hash configuration.

[0077] After the NPU's built-in storage control unit 210 loads the soft operator executable file carried by the GCUPackage data block from the model file into the NPU, the NPU's built-in security operation unit 230 first performs a hash operation verification on the loaded soft operator executable file according to the hash configuration information in the hash control register. Only when the security operation unit 230 verifies the hash operation of the soft operator executable file does the GCU execute the soft operator executable file; otherwise, it returns an error flag through the hash control register.

[0078] In this embodiment, the hash configuration information includes the hash algorithm type and the theoretical hash value, and the hash algorithm type is either SM3 or SHA256 hash algorithm. The secure operation unit 230 performs hash operation verification on the soft operator executable file, including:

[0079] The soft operator executable file is hashed according to the hash algorithm type in the hash control register to obtain the actual hash value, and then compared with the theoretical hash value. If the actual hash value matches the theoretical hash value in the hash control register, the hash operation verification of the soft operator executable file is considered successful; otherwise, the hash operation verification is considered unsuccessful.

[0080] In a practical application embodiment, the compiled model file includes ModelPackage data blocks, GCUPackage data blocks, GraphPackage data blocks, LayerPackage data blocks, and TensorPackage data blocks stored sequentially. Please refer to... Figure 5 The data structure shown includes:

[0081] (1) The ModelPackage data block carries encryption / decryption configuration information and hash configuration information. In one embodiment, the ModelPackage data block also carries basic model attribute information, including format identifier, model version information, compiler version information, and model file size.

[0082] like Figure 5 As shown, the ModelPackage data block includes, in sequence, a Magic Number field, a Model version field, a Compiler version field, a Security data structure, and a Binary Size field. The Magic Number field carries a format identifier, which can be used to determine if the model file is correct. The Model version field carries the model version information of the model compiler integrated into the edge SoC. The Compiler version field carries the compiler version information. The Security data structure carries encryption / decryption configuration information and hash configuration information. The Binary Size field carries the model file size.

[0083] The Security data structure includes the following fields: crypto enable, crypto algs, crypto mode, hash enable, hash algs, and hash value. The crypto enable field carries encryption / decryption enabling information, used to configure the NPU to perform intermediate feature map encryption / decryption operations. The crypto algs field configures the encryption / decryption algorithm type, and the crypto mode field configures the encryption / decryption algorithm mode. The hash enable field carries hash enabling information, used to configure the NPU to perform hash verification operations. The hash algs field configures the hash algorithm type, and the hash value field configures the theoretical hash value.

[0084] (2) The GCU Package data block contains the executable files of soft operators.

[0085] (3) The GraphPackage data block records the total number of operator layers in the neural network model, the offset address of the LayerPackage data block, and the data size of the LayerPackage data block. For example... Figure 5As shown, the GraphPackage data block includes a layer number field, a layer offset field, and a layer size field. The layer number field carries the total number of operator layers, the layer offset field carries the offset address of the LayerPackage data block, and the layer size field carries the data size of the LayerPackage data block.

[0086] (4) LayerPackage data blocks record the layer information of each operator in the neural network model. LayerPackage data blocks sequentially include the Node data structures of each operator layer in the neural network model, such as... Figure 5 Taking a neural network model containing a total of N operator layers as an example, the LayerPackage data block includes Node 1 to Node N corresponding to the 1st operator layer to the Nth operator layer.

[0087] Each field in the Node data structure of any operator layer carries, in sequence, the operator attribute parameter, the input feature map data type, the output feature map data type, the input feature map data address, the input feature map data dimension, the output feature map data address, the output feature map data dimension, the operator weight address, and the operator weight dimension.

[0088] (5) TensorPackage data blocks record the operator weights of each layer of the neural network model. TensorPackage data blocks sequentially include the Node Tensor data structures corresponding to each operator layer in the neural network model, such as... Figure 5 Taking a neural network model with a total of N operator layers as an example, the TensorPackage data block sequentially includes the Node1 Tensor data structure to the NodeN Tensor data structure corresponding to the 1st operator layer to the Nth operator layer. The Node Tensor data structure corresponding to each operator layer carries the operator weights of that operator layer.

[0089] The model compiler in the edge SoC compiles the data into the form of... Figure 5The model file is shown and stored at a specified address in the memory of the edge SoC. The CPU integrated into the edge SoC parses the information carried in the predefined MagicNumber, Model version, Compiler version, Security data structure, and BinarySize fields in the ModelPackage data block of the model file. The basic model attribute information carried in each field is used for information retrieval. The information carried in each field of the Security data structure is extracted to obtain encryption / decryption configuration information and hash configuration information. The NPU is then configured for encryption / decryption according to the encryption / decryption configuration information, and the NPU is also configured for hashing according to the hash configuration information.

[0090] After the CPU parses the ModelPackage data block, it can determine the address of the GCUPackage data block in memory based on the pre-determined data size of the ModelPackage data block, thus obtaining the soft operator executable file storage information. This soft operator executable file storage information is then written to the GCUPackage offset address register. The NPU's built-in storage control unit reads the soft operator executable file storage information from the GCUPackage offset address register and loads the soft operator executable file from the edge SoC memory based on this information.

[0091] The CPU also parses the GraphPackage data blocks in the model file, obtaining the total number of operator layers, the offset address of the LayerPackage data block, and the data size of the LayerPackage data block carried by each field. Based on the offset address of the LayerPackage data block, the address of the LayerPackage data block in the edge SoC memory can be located, and the LayerPackage offset address register can be configured. Based on the data size of the LayerPackage data block, the address of the TensorPackage data block in the edge SoC memory can be located.

[0092] During NPU initialization, a 64-bit memory base address register is configured. The Node Tensor data structures corresponding to each operator layer within the TensorPackage data block are loaded by the storage control unit 210 using the address (i.e., offset address) of the corresponding node in the Node data block of the corresponding operator layer in the LayerPackage data block plus the base address of the memory base address register for DMA loading.

[0093] In a LayerPackage data block, the Node data block is located by the storage control unit 210 based on the result of the memory base address register + LayerPackage offset address register + Node index. The LayerPackage offset address register is configured according to the offset address of the LayerPackage data block in the GraphPackage data block. The Node index is recorded by the NPU's internal accumulator and is flipped to 0 after the last layer operator is executed, restarting the count.

[0094] The above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations that can be directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. A neural network model inference method to improve the security of intermediate data, characterized in that, The neural network model inference method includes: The NPU integrated in the edge SoC uses the built-in storage control unit to load the LayerPackage data block in the model file stored in the edge SoC memory, and uses the built-in parameter parsing unit to parse the layer information of each layer operator in the neural network model. The layer information of any layer operator includes the operator information of the current layer operator, the input feature map storage information of the current layer operator, and the output feature map storage information of the current layer operator. For any integer parameter i, when the i-th layer operator is the first layer operator, the NPU uses the built-in storage control unit to load the model input data as the input data of the first layer operator. Otherwise, the NPU uses the built-in storage control unit to store information according to the input feature map of the i-th layer operator, loads the ciphertext of the input feature map of the i-th layer operator from the edge SoC memory into the NPU, and uses the built-in security operation unit to decrypt it inside the NPU and use it as the input data of the i-th layer operator. The NPU uses its built-in operator execution unit to perform neural network operations on the input data of the i-th layer operator according to the operator information of the i-th layer operator to obtain the output data of the i-th layer operator; When the i-th layer operator is the last layer operator, the NPU's built-in storage control unit outputs the output data of the i-th layer operator as the model output result. Otherwise, the NPU's built-in security operation unit encrypts the output data of the i-th layer operator inside the NPU to obtain the ciphertext of the output feature map of the i-th layer operator. Then, using the built-in storage control unit, the ciphertext of the output feature map of the i-th layer operator is written into the edge SoC memory as the ciphertext of the input feature map of the next layer operator based on the storage information of the output feature map of the i-th layer operator.

2. The neural network model inference method according to claim 1, characterized in that, The NPU's built-in operator execution unit includes a neural network acceleration unit and an operator coprocessor (GCU). The NPU uses the built-in operator execution unit to perform neural network operations on the input data of the i-th layer operator, including: The NPU uses its built-in storage control unit to load the soft operator executable file carried in the GCUPackage data block of the model file, and the operator coprocessor GCU executes the soft operator executable file. When the i-th layer operator is a software operator, the operator coprocessor (GCU) performs neural network operations on the input data of the i-th layer operator according to the operator information of the i-th layer operator; when the i-th layer operator is a hardware operator, the neural network acceleration unit performs neural network operations on the input data of the i-th layer operator according to the operator information of the i-th layer operator.

3. The neural network model inference method according to claim 2, characterized in that, The neural network model inference method further includes: The CPU integrated into the edge SoC parses the ModelPackage data block in the model file to obtain hash configuration information, and writes the hash configuration information into the NPU's hash control register for hash configuration; After the NPU's built-in storage control unit loads the soft operator executable file carried by the GCUPackage data block in the model file into the NPU, the NPU's built-in security operation unit performs a hash operation verification on the soft operator executable file according to the hash configuration information in the hash control register. If the NPU's built-in security operation unit passes the hash operation verification of the soft operator executable file, the GCU executes the soft operator executable file; otherwise, it returns an error flag through the hash control register.

4. The neural network model inference method according to claim 3, characterized in that, Hash configuration information includes the hash algorithm type and theoretical hash value. The hash operation verification of the soft operator executable file based on the hash configuration information in the hash control register includes: The soft operator executable file is hashed according to the hash algorithm type in the hash control register to obtain the actual hash value. When the actual hash value is consistent with the theoretical hash value in the hash control register, the hash operation of the soft operator executable file is determined to be successful; otherwise, the hash operation is determined to be unsuccessful.

5. The neural network model inference method according to claim 1, characterized in that, The neural network model inference method further includes: The CPU integrated into the edge SoC parses the ModelPackage data block in the model file to obtain encryption / decryption configuration information, and writes the encryption / decryption configuration information into the NPU's encryption / decryption control register for encryption / decryption configuration. The NPU's built-in secure computing unit decrypts the ciphertext of the input feature maps of each layer operator to obtain the input data, and encrypts the output data of each layer operator to obtain the ciphertext of the output feature maps, based on the pre-written key, initialization vector, and encryption / decryption configuration information in the encryption / decryption control register. Furthermore, the same initialization vector is used in the process of performing a model inference on the neural network model to obtain the model output result from the model input data, while different initialization vectors are used in different model inference processes.

6. The neural network model inference method according to claim 5, characterized in that, The NPU's built-in secure computing unit decrypts the ciphertext of the input feature maps for each layer of operators and encrypts the output data for each layer of operators, including: When the i-th layer operator is the first layer operator, the secure operation unit generates a random number as an initialization vector, uses the pre-written key and combines the initialization vector with the encryption and decryption configuration information in the encryption and decryption control register to encrypt the output data of the first layer operator to obtain the ciphertext of the output feature map of the first layer operator, and caches the initialization vector inside the secure operation unit; When the i-th layer operator is an intermediate layer operator other than the first layer operator and the last layer operator, the secure operation unit reads the cached initialization vector, uses the pre-written key and combines it with the initialization vector and the encryption / decryption configuration information in the encryption / decryption control register to decrypt the ciphertext of the input feature map of the i-th layer operator to obtain the input data of the i-th layer operator, and encrypts the output data of the i-th layer operator to obtain the ciphertext of the output feature map of the i-th layer operator, and caches the initialization vector inside the secure operation unit; When the i-th layer operator is the last layer operator, the secure operation unit reads the cached initialization vector, uses the pre-written key and combines it with the initialization vector and the encryption / decryption configuration information in the encryption / decryption control register to decrypt the ciphertext of the input feature map of the last layer operator to obtain the input data of the last layer operator, and destroys the currently used initialization vector.

7. The neural network model inference method according to claim 2, characterized in that, The operator information of any layer of operators includes operator attribute parameters and operator weight storage information. The operator attribute parameters indicate the operator type, operator level and operator architecture. The neural network model inference method includes: The NPU uses its built-in parameter parsing unit to parse the operator attribute parameters and operator weight storage information of the i-th layer operator. Based on the operator level indicated by the operator attribute parameters, it determines whether the i-th layer operator is a first-layer operator, a last-layer operator, or an intermediate-layer operator. Based on the operator architecture indicated by the operator attribute parameters of the i-th layer operator, it determines whether the i-th layer operator is a hardware operator or a software operator.

8. The neural network model inference method according to claim 7, characterized in that, The NPU also incorporates an input feature map FIFO, an output feature map FIFO, and a weight FIFO; the neural network model inference method further includes: After the parameter parsing unit parses the hierarchical information of the i-th layer operator, the storage control unit loads the operator weights of the i-th layer operator from the edge SoC memory according to the operator weight storage information of the i-th layer operator and caches them in the weight FIFO. When the i-th layer operator is determined to be the first layer operator according to the operator hierarchy indicated by the operator attribute parameters, the storage control unit loads the model input data as the input data of the first layer operator and caches it in the input feature map FIFO. Otherwise, the storage control unit loads the ciphertext of the input feature map of the i-th layer operator from the edge SoC memory, and the secure operation unit decrypts the ciphertext of the input feature map to obtain the input data of the i-th layer operator and caches it in the input feature map FIFO. Based on the operator architecture indicated by the operator attribute parameters, the neural network acceleration unit or operator coprocessor GCU is determined as the operator execution unit. The operator execution unit reads the input data of the i-th layer operator from the input feature map FIFO, reads the operator weights of the i-th layer operator from the weight FIFO, obtains the operator type of the i-th layer operator from the parameter parsing unit, and performs neural network operations on the input data of the i-th layer operator and the operator weights of the i-th layer operator in the input feature map FIFO according to the calculation method corresponding to the operator type of the i-th layer operator, and obtains the output data of the i-th layer operator which is cached in the output feature map FIFO. When the operator at layer i is determined to be the last operator based on the operator hierarchy indicated by the operator attribute parameters, the storage control unit reads the output data of the i-th layer operator from the output feature map FIFO and outputs it as the model output result; otherwise, the secure operation unit reads the output data of the i-th layer operator from the output feature map FIFO and encrypts it to obtain the ciphertext of the output feature map. The storage control unit writes the ciphertext of the output feature map of the i-th layer operator into the edge SoC memory according to the storage information of the output feature map of the i-th layer operator.

9. The neural network model inference method according to claim 2, characterized in that, The neural network model inference method further includes: The CPU integrated on the edge SoC parses the model file to obtain the storage information of the soft operator executable file, and writes the storage information of the soft operator executable file into the GCUPackage offset address register; The NPU's built-in storage control unit reads the soft operator executable file storage information from the GCUPackage offset address register and loads the soft operator executable file from the edge SoC memory based on the soft operator executable file storage information.

10. The neural network model inference method according to claim 1, characterized in that, The compiled model file includes ModelPackage data blocks, GCUPackage data blocks, GraphPackage data blocks, LayerPackage data blocks, and TensorPackage data blocks, among which: The ModelPackage data block carries encryption / decryption configuration information, hash configuration information, and basic model attribute information; GCUPackage data blocks contain executable files for soft operators; The GraphPackage data block records the total number of operator layers in the neural network model, the offset address of the LayerPackage data block, and the data size of the LayerPackage data block; LayerPackage data blocks record the hierarchical information of each layer operator in the neural network model; TensorPackage data blocks record the operator weights of each layer in a neural network model.

Images