Fpga-oriented layered homomorphic encryption neural network inference deployment optimization method

Abstract

The application discloses a layered homomorphic encryption neural network inference deployment optimization method for FPGA, and belongs to the field of neural network optimization and secure computation, and comprises the following steps: step 1, adaptively block an input image and extract statistical features; step 2, based on a collaborative fusion optimization paradigm, implement CKKS homomorphic encryption-aware topological pruning on a neural network model; step 3, implement hierarchical parameter genetic homomorphic encryption on the pruned model: taking network layer order as a link, independently generating encrypted parameters for each layer, adaptively generating the first layer parameters from image block statistical features, and evolving and genetically tuning subsequent layer parameters through the previous layer parameters to form an encrypted parameter genetic flow; and step 4, deploying the optimized model to an FPGA platform to realize efficient privacy protection inference. The application can collaboratively optimize algorithm accuracy, homomorphic encryption characteristics and FPGA hardware performance, and achieves a good performance balance in a privacy protection scene.

Description

Technical Field

[0001] This invention belongs to the field of neural network optimization and secure computing, specifically relating to a hierarchical homomorphic encrypted neural network inference deployment optimization method for FPGA. Background Technology

[0002] With the widespread application of deep learning in image-intensive tasks (such as medical imaging and autonomous driving), the issue of image data privacy protection has become increasingly prominent. To address this challenge, encrypted neural network inference technology has emerged, which can complete computations without decrypting sensitive image data, thus achieving privacy protection in the neural network inference process.

[0003] Among them, the CKKS (Cheon-Kim-Kim-Song) homomorphic encryption scheme has become the preferred technology for neural network inference in privacy-preserving scenarios due to its unique ability to support near-floating-point arithmetic. This scheme not only supports floating-point operations common in image neural networks but also allows addition and multiplication operations within the cryptographic domain without frequent noise management intervention, thus achieving efficient privacy-preserving inference while maintaining image processing accuracy. However, CKKS faces challenges in practical deployments, including high computational complexity and large storage overhead, especially on resource-constrained edge devices where its efficiency is relatively low.

[0004] To address these challenges, existing technologies have proposed various locality optimization methods. For example, parameter quantization and multiplication depth optimization reduce computational overhead; batch processing and single-instruction multiple-data techniques improve throughput; and hardware acceleration (such as GPUs) enhances the efficiency of encryption operations. However, these methods still have limitations: parameter quantization may lead to a decrease in model accuracy; multiplication depth optimization is limited by network structure; and hardware acceleration solutions mostly rely on general-purpose platforms such as GPUs, lacking customized designs for resource-constrained and efficient FPGA (Field-Programmable Gate Array) platforms. FPGAs have significant advantages in edge computing due to their high parallelism, low power consumption, and flexibility, but existing CKKS homomorphic encryption neural network inference optimization schemes for FPGAs are relatively scarce. Especially during encryption inference, the lack of sufficient integration of privacy protection mechanisms makes it difficult to balance data security and real-time performance.

[0005] In terms of neural network pruning, traditional pruning methods do not fully consider the computational characteristics of CKKS homomorphic encryption (such as multiplication depth and noise growth), resulting in limited improvement in encryption inference efficiency. In terms of deployment optimization, existing solutions also do not fully consider the network layer order characteristics, lack a mechanism for dynamic adjustment of encryption parameters at different levels, and do not fully utilize the parallel computing capabilities and hardware programmability of FPGAs, resulting in suboptimal inference performance and resource utilization. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a hierarchical homomorphic encryption neural network inference deployment optimization method for FPGAs, suitable for privacy-preserving inference in resource-constrained environments. First, the input image is adaptively segmented and statistical features are extracted to drive the initial configuration of subsequent hierarchical encryption parameters. Based on a collaborative fusion optimization paradigm, considering both model accuracy and homomorphic computational overhead, topological pruning with CKKS homomorphic encryption awareness is implemented on the neural network. Next, hierarchical parameter genetic homomorphic encryption is applied to the pruned network: using the network layer order as a link, encryption parameters are generated independently at each layer. The parameters of the first layer are adaptively generated from the statistical features of image segments, and the parameters of subsequent layers are tuned through evolutionary genetic processes of the previous layer's parameters, forming a genetic flow of encryption parameters. Finally, the optimized model is deployed to an FPGA platform to achieve efficient privacy-preserving inference. This invention can collaboratively optimize algorithm accuracy, homomorphic encryption characteristics, and FPGA hardware performance, achieving a good performance balance in privacy-preserving scenarios.

[0007] The technical solution of the present invention is as follows:

[0008] A hierarchical homomorphic encrypted neural network inference deployment optimization method for FPGA includes the following steps:

[0009] Step 1: Perform adaptive block segmentation on the input image and calculate and extract statistical features;

[0010] Step 2: Based on the collaborative fusion optimization paradigm, perform topology pruning of the neural network model using CKKS homomorphic encryption sensing;

[0011] Step 3: Implement hierarchical parameter genetic homomorphic encryption on the pruned model: Using the network hierarchy as the link, each layer generates encryption parameters independently. The parameters of the first layer are adaptively generated by the statistical features of image blocks, and the parameters of subsequent layers are tuned by the evolutionary genetic tuning of the parameters of the previous layer to form a genetic flow of encryption parameters.

[0012] Step 4: Deploy the optimized model to the FPGA platform to achieve efficient privacy-preserving inference.

[0013] Furthermore, the specific process of step 1 is as follows:

[0014] Step 1.1: Before segmenting, normalize the input image;

[0015] Step 1.2: Determine the size of the sub-blocks after dividing the input image. ,in, The height of the input image sub-block; The width of the input image sub-block; The number of color channels in the input image; the size of the sub-blocks must satisfy FPGA resource constraints, CKKS encryption constraints, and neural network computation constraints.

[0016] Step 1.3: Normalize the image Divided into The sub-blocks are flattened into vectors;

[0017] Each sub-block after division is defined as:

[0018] ;

[0019] in, for In the height direction Width direction is The sub-block at the location; This is the index of the sub-block in the height direction; This is the index of the sub-block in the width direction; , These represent the total number of sub-blocks in the image along the height and width directions, respectively.

[0020] each sub-block Flatten into a one-dimensional vector through vectorization operations. ;

[0021] Step 1.4: For each flattened one-dimensional vector of sub-blocks, calculate five numerical distribution features: mean, variance, maximum value, minimum value, and dynamic range. Summarize these numerical distribution features to obtain three global features, including the global mean. Global variance Global dynamic range :

[0022] ;

[0023] in, The total number of sub-blocks in the input image; For sub-blocks The mean;

[0024] ;

[0025] in, For sub-blocks The variance;

[0026] ;

[0027] in, To find the maximum value of the one-dimensional vectors of all sub-blocks, take the maximum value. To find the minimum value of the one-dimensional vector that iterates through all sub-blocks, take the minimum value. , Sub-blocks The maximum and minimum values ​​of a one-dimensional vector.

[0028] Furthermore, the specific process of step 2 is as follows:

[0029] Step 2.1: Construct the neural network model to be pruned and establish a computational overhead model under the CKKS homomorphic encryption environment; the neural network model contains several convolutional layers and several fully connected layers; calculate the total computational overhead of each layer. :

[0030] ;

[0031] in, , , , These are the cost weights for multiplication, addition, rotation, and bootstrapping, respectively. , , , These represent the number of multiplication, addition, rotation, and bootstrapping operations, respectively.

[0032] Step 2.2: Use channel-level or neuron-level weight norms as a magnitude evaluation metric;

[0033] For a convolutional layer, the weight tensor is... For each output channel The weight norm is calculated using the L2 norm.

[0034] For a fully connected layer, the weight matrix is: For each output neuron The weight norm is calculated using the L2 norm.

[0035] Step 2.3: Calculate the total computational cost for each convolutional layer and fully connected layer of the model;

[0036] Step 2.4: Based on the weight norm of the channel or neuron and the computational overhead of CKKS homomorphic encryption, comprehensively evaluate the retention value of each channel or neuron and output the priority index of each channel or neuron.

[0037] Step 2.5: Perform topology pruning on the neural network model;

[0038] Step 2.6: Verify the model complexity after pruning;

[0039] Step 2.7: Fine-tune the pruned model.

[0040] Furthermore, the specific process of step 2.3 is as follows:

[0041] For convolutional layers, the number of multiplication, addition, and rotation operations and the multiplication depth are calculated based on the input and output dimensions and the convolution kernel parameters, respectively.

[0042] For fully connected layers, the number of multiplication, addition, and rotation operations and the multiplication depth are calculated based on the number of input and output neurons and the batch size, respectively.

[0043] Based on the above indicators and the total computational cost formula in step 2.1, calculate the total computational cost of each layer.

[0044] The specific process of step 2.4 is as follows:

[0045] The total computational cost of the current layer is evenly distributed to each output channel or neuron to obtain the average homomorphic computational cost. Combined with the weight norm of each output channel or neuron in step 2.2, the priority index of its output channel or neuron is calculated. Those with higher values ​​are preferentially retained during the pruning process.

[0046] The specific process of step 2.5 is as follows:

[0047] According to the pruning ratio Determine the number of channels or neurons retained in each layer. ;

[0048] Based on the priority index of the channels or neurons output in step 2.4, sort all output channels or neurons in the current layer in descending order; select the top priority ones. The channel or neuron indexes constitute a reserved index set;

[0049] Based on the retained index set, the corresponding output channels or neurons are extracted from the original weights to generate new pruned weights, and the number of output channels or neurons in the current layer is updated accordingly. ;

[0050] If a bias term exists in the current layer, the bias vector is updated synchronously based on the same set of reserved indices, and the number of input channels or neurons in the next layer is adjusted.

[0051] Furthermore, the specific process of step 3 is as follows:

[0052] Step 3.1: Preprocess the pruned neural network weight matrix and bias vector by numerical normalization and format adjustment to meet the encoding requirements of the CKKS homomorphic encryption scheme for plaintext polynomials; the specific process is as follows:

[0053] Step 3.1.1, calculate the first... Layer weight dynamic range and bias dynamic range And normalize it to get the first Layer normalized weight matrix Bias vector Then based on , Calculate the first Dynamic range of weights after layer normalization Bias dynamic range :

[0054] ;

[0055] ;

[0056] in, For traversal All rows With all columns Take the maximum value of the elements in the list; For traversal All rows With all columns The minimum value is taken from the elements in the set. For traversing vectors All elements Take the maximum value; For traversing vectors All elements Take the minimum value;

[0057] Step 3.1.2, place the first Layer normalized weight matrix and bias vector Adjust to a format suitable for encoding;

[0058] Step 3.2: The first-layer encryption parameters are adaptively generated based on the image block statistical features extracted in Step 1;

[0059] Step 3.3: Define the inter-layer encryption parameter genetic tuning function and execute the encryption parameter genetic flow;

[0060] Step 3.4: Perform layer-by-layer CKKS homomorphic encryption on the weight matrix and bias vector of each layer based on the genetic flow parameters;

[0061] Step 3.5: Verify the correctness and accuracy of the encryption parameters for each layer, and optimize the parameters to suit FPGA resources and inference requirements.

[0062] Furthermore, the specific process of step 3.2 is as follows:

[0063] Step 3.2.1, Scaling factor of the first layer Set to:

[0064] ;

[0065] in, It is a constant; and These are the standard deviation and dynamic range of the weight matrix after normalization of the first convolutional layer, respectively.

[0066] Set the initial modulus of the first layer. for:

[0067] ;

[0068] in, This is the magnification factor calculated for the first layer;

[0069] Step 3.2.2: Verify the parameter accuracy of the CKKS homomorphic encryption scheme by evaluating the encryption calculation of the first layer of the neural network model;

[0070] Determine whether the relative error before and after encryption is less than the maximum permissible relative error. If satisfied, the verification ends; otherwise, adjustments are performed.

[0071] like If the value exceeds the preset threshold, a logarithmic transformation or outlier truncation is applied to the sub-block data, and then the statistical characteristics and encryption parameters are recalculated according to the process in step 1.4.

[0072] like If the value is less than or equal to a preset threshold, then increase the value. or Then return to step 3.2.1 to regenerate and verify the parameters until the error meets the requirements;

[0073] Finally, the first layer of encryption parameters is generated. .

[0074] Furthermore, the specific process of step 3.3 is as follows:

[0075] Define the genetic tuning function and execute the encrypted parameter genetic stream; genetic tuning function The core mechanism is as follows:

[0076] ;

[0077] in, For the first The set of encryption parameters for the layer; For the first The set of encryption parameters for the layer, For the first Layer scaling factor For the first The initial modulus of the layer; These are the internal cryptographic parameters of the genetic tuning function; For the first Context information of the layer;

[0078] It consists of two sub-functions that handle the scaling factor and the initial modulus, respectively. and constitute:

[0079] Genetic tuning of scaling factors is achieved through the scaling factor genetic tuner function. calculate:

[0080] ;

[0081] in, For the first Layer scaling factor;

[0082] Genetic tuning of the initial modulus is achieved through the initial modulus genetic tuning subfunction. calculate:

[0083] ;

[0084] And it satisfies:

[0085] ;

[0086] in, For the first The initial modulus of the layer; For the first Amplification factor for layer calculation; For the first The dynamic range of weights after normalization in a convolutional layer;

[0087] Using the first layer's encryption parameter set as the initial seed, the algorithm iterates along the neural network layer indices and applies a genetic tuning function. Dynamically generate encryption parameters for each layer;

[0088] The specific process of step 3.4 is as follows:

[0089] Homomorphic encryption is performed on the parameters of each layer of the neural network using a genetic flow of encrypted parameters.

[0090] Use the Layer scaling factor The pre-processed first Layer Weight subvectors With the bias subvectors Encoded as a CKKS plaintext polynomial;

[0091] Use system public key The generated plaintext polynomial is homomorphically encrypted to obtain the ciphertext.

[0092] The encrypted text is transmitted and stored in external memory.

[0093] Furthermore, the specific process of step 4 is as follows:

[0094] Step 4.1: For the ALINX Black Gold AX7020 development board, evaluate its hardware resource constraints; define the resource constraint set. , , , , These correspond to constraints on the number of logic units, flip-flops, digital signal processing modules, and block memory, respectively. Based on the determined CKKS encryption parameters and the pruned model size, the resource requirements are estimated: digital signal processing module requirements, block memory requirements, logic unit quantity requirements, and flip-flop quantity requirements. Among these, the CKKS encryption parameters include the polynomial ring dimension and the modular chain length.

[0095] Step 4.2: Design a dedicated hardware acceleration engine for CKKS homomorphic computation on the FPGA. Through a unified dynamic parameter configuration interface, load dynamically encrypted parameters between neural network layers to achieve privacy-preserving inference through hardware and software collaboration.

[0096] Step 4.3: Implement a hierarchical encrypted inference pipeline architecture, design a dual-pipeline synchronization mechanism based on a finite state machine control pipeline and a hardware acceleration engine data pipeline, and map the neural network model to the hardware pipeline.

[0097] Step 4.4: Design a storage scheduling subsystem based on double buffering and prefetching strategies;

[0098] Step 4.5: Implement and deploy the hardware system physically, using the Vivado HLS toolchain to generate bitstream files;

[0099] Step 4.6: Execute the complete inference process on the FPGA and test the functional correctness of the encrypted neural network model on the FPGA and evaluate its performance.

[0100] Furthermore, the specific process of step 4.2 is as follows:

[0101] Step 4.2.1: Design a parallel homomorphic multiply-accumulate unit;

[0102] Ciphertext weight polynomial and input activation polynomial Loaded into a two-port block memory, and subjected to parallel number-theoretical transformations via a butterfly arithmetic network, the result is transformed to the number-theoretical transformation domain. and ;in, and They are polynomials and The representation in the number-theoretical transformation domain after transformation by number-theoretical transformation;

[0103] In the transformation domain of number theory, for and Perform point-by-point modular multiplication on the corresponding coefficients;

[0104] The dot product result is transformed back to coefficient representation through parallel inverse number theory transformation to obtain the homomorphic multiplication result, which is then fed into the subsequent unit;

[0105] Step 4.2.2: Design a configurable analog-to-digital switching unit;

[0106] Obtain the first from the genetic flow parameters The layered modular chain, and based on the Barrett reduction algorithm, the ciphertext polynomial is removed from the current modular chain. Switch to target module ;

[0107] Step 4.2.3: Design an efficient encrypted text rotation unit;

[0108] Based on rotation step Dynamically load the corresponding Galois key and polynomial ring dimension ;

[0109] The address generator determines the step size. Parallel computing New index for each slot ;

[0110] use Multiple parallel memory banks, read at once Data for each slot, and with the Galois key. Perform Galois automorphism mapping homomorphic multiplication in parallel homomorphic multiply-add units;

[0111] Using a configurable crossbar switch network of an FPGA, the mapped data is distributed according to the first... New index for each slot The final data rearrangement is performed, and the entire rotation operation is completed with a single parallel memory write.

[0112] Furthermore, the specific process of step 4.3 is as follows:

[0113] Step 4.3.1: Design a dual-pipeline synchronization mechanism;

[0114] The control pipeline schedules the computation of each layer according to the network hierarchy using a finite state machine.

[0115] The data pipeline consists of a hardware acceleration engine that performs ciphertext computations in the order of network layers;

[0116] The control pipeline and the data pipeline are driven by a pair of handshake signals to achieve synchronization of the two pipelines;

[0117] Step 4.3.2: The neural network model... Each layer, mapped to a data pipeline according to its computational order, is... The pipeline consists of consecutive stages, with encrypted data being transferred between stages via a first-in-first-out queue buffer; each pipeline stage dynamically configures a shared hardware acceleration engine based on the layer type of the neural network model it implements.

[0118] The specific process of step 4.4 is as follows:

[0119] Step 4.4.1: To enable parameter switching in the control pipeline, a double-buffering mechanism is used in the FPGA's on-chip block memory to store encrypted parameters.

[0120] When the control pipeline uses a buffer to configure the current layer parameters, the prefetch strategy simultaneously loads the next layer's set of encrypted parameters into another buffer.

[0121] The cached parameters are directly mapped to the hardware acceleration engine through a dedicated configuration register interface;

[0122] Step 4.4.2: To meet the data bandwidth requirements of multi-level concurrent execution in a hierarchical pipeline, a data prefetching mechanism is designed. This mechanism drives data prefetching by controlling the pipeline. When the state machine enters the... When the layer is in operation, the prefetch controller will... The required cryptographic model weights for the layer are preloaded from off-chip double-rate synchronous dynamic random access memory to on-chip block memory.

[0123] The beneficial technical effects brought about by this invention are as follows.

[0124] (1) Significantly reduces the computational complexity of CKKS encrypted inference. This invention designs a neural network topology pruning strategy based on a collaborative fusion optimization paradigm to remove channels or neurons that have little impact on model accuracy but high computational overhead, effectively reducing the computational load of the neural network model. Combined with the adaptive CKKS parameter configuration mechanism, it significantly improves the efficiency of encrypted inference.

[0125] (2) Significantly improves the adaptability and security of encrypted neural network inference. This invention avoids encryption redundancy caused by parameter mismatch by adaptively generating the first layer parameters and dynamically evolving the subsequent layer parameters, thereby minimizing noise growth, reducing decryption errors and computation latency, while maintaining model accuracy and facilitating applications of highly sensitive data.

[0126] (3) Significantly improves privacy-preserving inference performance on resource-constrained edge devices. This invention fully utilizes the high parallelism and reconfigurability of FPGAs, and builds an optimized homomorphic encryption operation kernel and a dedicated inference acceleration pipeline based on the Vivado HLS tool. At the same time, the CKKS fully homomorphic encryption mechanism effectively prevents data leakage and ensures that the entire inference process is executed efficiently within the encrypted domain. Attached Figure Description

[0127] Figure 1 This is a flowchart of the hierarchical homomorphic encrypted image neural network inference optimization method for FPGA of the present invention.

[0128] Figure 2 This is a comparison chart of the accuracy of the pruned model and the original model according to the present invention. Detailed Implementation

[0129] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0130] like Figure 1 As shown, a hierarchical homomorphic encrypted neural network inference deployment optimization method for a field-programmable gate array (FPGA) hardware platform includes the following steps:

[0131] Step 1: Adaptively segment the input image and extract statistical features to drive the initial configuration of subsequent layered encryption parameters. The specific process is as follows:

[0132] Step 1.1: Before segmentation, the input image is normalized to map pixel values ​​to a fixed interval, thus unifying the benchmark for statistical feature extraction. The normalization formula is:

[0133] ;

[0134] in, The image is after normalization; To specify the horizontal position of image pixels in the grid; To specify the position of image pixels in the vertical direction within the grid; This is the index of the color channel; Represents the spatial coordinates of the input image and channels The original pixel value at that location; and These represent the maximum and minimum pixel values ​​of the image, respectively.

[0135] Step 1.2: Determine the input image block parameters. The neural network used in this method is a multi-layer structure, containing... Each layer, after training, can perform inference tasks. The neural network used in this embodiment is a convolutional neural network (CNN), which contains two convolutional layers and two fully connected layers. Determining the block parameters aims to select an appropriate sub-block size. ,in, The height of the input image sub-block; The width of the input image sub-block; The number of color channels in the input image;

[0136] Sub-block size The choice must satisfy the following three constraints:

[0137] FPGA resource constraints: Total number of elements in the sub-block It needs to match the capacity of the FPGA's embedded memory (BRAM) and the parallel computing capabilities of the digital signal processor (DSP) unit;

[0138] CKKS encryption constraints: CKKS encryption uses a polynomial ring. Encryption is performed, among which, Let be the set of integers, representing the coefficients of the polynomial; Let be a variable in the polynomial ring; The sub-block dimension is a polynomial ring dimension; the total number of elements in the sub-block must be compatible with the number of slots in the polynomial ring dimension; the flattening vector of the sub-block (total number of elements) ) should meet To achieve efficient Single Instruction Multiple Data (SIMD) packaging;

[0139] Neural network computational constraints: The size of the sub-block should be compatible with the input dimension of the first layer of the neural network or the size of the convolution kernel.

[0140] Step 1.3: Adaptive image block processing. The normalized image is divided into... The sub-blocks are flattened into vectors to accommodate CKKS encryption.

[0141] Normalized image Each sub-block after division Defined as:

[0142] ;

[0143] in, for In the height direction Width direction is The sub-block at the location; Indicates the index of the sub-block in the height direction; Indicates the index of the sub-block in the width direction; This represents the total number of sub-blocks in the image along the height direction; This represents the total number of sub-blocks in the width direction of the image; Tensor slice operators are used to slice tensors from... Extract sub-blocks; where the starting row index is... The end row index is The starting column index is The index of the end column is The last one This indicates that the entire channel dimension is selected;

[0144] If the boundary is insufficient It can be filled with zero padding, with a padding value of 0 or a copy of the boundary value;

[0145] each sub-block Flatten it into a one-dimensional vector using vectorization operations:

[0146] ;

[0147] in, Indicates a sub-block A flattened one-dimensional vector; For vectorization operations;

[0148] Flattening operations are performed in either row-priority or channel-priority order:

[0149] ;

[0150] ;

[0151] in, For sub-blocks The flattened one-dimensional vector One element; This is the index of an element in the flattened one-dimensional vector; Indicates a sub-block Pixel index in the height direction, Indicates a sub-block Pixel index in the width direction, Indicates a sub-block Index in the channel direction.

[0152] Step 1.4: Perform statistical feature calculations, extract five numerical distribution features for each sub-block, and summarize the global features to provide a basis for the CKKS encryption parameters of the first layer of the neural network. For each sub-block, after flattening, one-dimensional... Calculate the following numerical distribution characteristics:

[0153] sub-block mean :

[0154] ;

[0155] sub-block variance :

[0156] ;

[0157] sub-block Maximum and minimum values ​​of a one-dimensional vector:

[0158] ;

[0159] ;

[0160] in, For sub-blocks The maximum value of a one-dimensional vector is obtained by... Operation from sub-block Select the element with the largest value from all elements in the array; For sub-blocks The minimum value of a one-dimensional vector is found by... Operation from sub-block Select the element with the smallest value from all elements in the array;

[0161] sub-block dynamic range :

[0162] ;

[0163] By summarizing the statistical characteristics of all sub-blocks, three global features are obtained:

[0164] global mean :

[0165] ;

[0166] in, This represents the total number of sub-blocks in the input image. ;

[0167] Global variance :

[0168] ;

[0169] Global dynamic range :

[0170] ;

[0171] in, This represents the maximum value of the one-dimensional vector of all sub-blocks, and the maximum value is taken. This means traversing all the one-dimensional vectors of the sub-blocks and taking the minimum value.

[0172] Step 2: Based on the collaborative fusion optimization paradigm, simultaneously considering model accuracy and homomorphic computational overhead, implement CKKS homomorphic encrypted sensing topology pruning on the neural network model to remove channels or neurons that have little impact on accuracy but high computational overhead.

[0173] Step 2.1: First, construct the neural network model to be pruned, and simultaneously establish a computational overhead model under the CKKS homomorphic encryption environment. The neural network model to be pruned uses a typical convolutional neural network (CNN) as an example, with a model structure containing two convolutional layers and two fully connected layers. The computational overhead model is based on quantifying the operational complexity of each convolutional layer and each fully connected layer in CKKS encryption inference, considering key indicators including multiplication operations, addition operations, rotation operations, multiplication depth, and bootstrapping operations. Define the cost weights for each operation, including the multiplication cost weight. Additive cost weighting Rotation cost weight Bootstrapping cost weight And set a maximum multiplication depth threshold. Using these metrics, the total computational cost of constructing each convolutional layer and each fully connected layer is determined. :

[0174] ;

[0175] in, , , , These represent the number of operations: multiplication, addition, rotation, and bootstrapping, respectively.

[0176] Step 2.2: Quantify the magnitude of the influence of weights in the neural network model by using channel-level or neuron-level weight norms as the magnitude evaluation index.

[0177] For a convolutional layer, the weight tensor is... ,in , , , These are, respectively, the number of output feature maps generated by the convolutional layer, the number of input feature maps fed into the convolutional layer, the size of the convolutional kernel in terms of spatial height, and the size of the convolutional kernel in terms of spatial width; for each output channel The weight norm is calculated using the L2 norm:

[0178] ;

[0179] in, For the convolutional layer The weight norm of each output channel; Indicates a fixed output channel. And select the subtensor formed by the weight elements at all input channels, all kernel heights, and all kernel widths; Indicates the convolutional layer's... The weights of each output channel, , , These correspond to the input channel number, kernel height, and width of the convolutional layer, respectively. Represents the L2 norm;

[0180] For a fully connected layer, the weight matrix is: , , These correspond to the number of neurons output by the fully connected layer and the number of neurons input to the fully connected layer, respectively. For each output neuron... The weight norm is calculated using the L2 norm:

[0181] ;

[0182] in, For the fully connected layer The weight norm of each output neuron; Indicates that the fixed output neuron is And select a subvector consisting of all weight elements in the dimension of the input neuron; Indicates the fully connected layer number 1 The weight elements of each output neuron. The input neuron number corresponding to the fully connected layer.

[0183] Step 2.3: Based on the general layer computational overhead formula defined in Step 2.1, further refine and calculate the total computational overhead for each convolutional layer and fully connected layer. Analyze the computational complexity of convolutional and fully connected layers under the CKKS homomorphic encryption environment, quantifying the number of multiplication operations, addition operations, rotation operations, and multiplication depth.

[0184] For a convolutional layer, assuming the input tensor size is... , , , , These correspond to the batch size, number of input channels, height of the input image, and width of the input image, respectively, with the output tensor size being... , , These correspond to batch size, number of output channels, height of output feature map, and width of output feature map, respectively. The kernel size is... , , The formulas for calculating the height and width of the convolution kernel are as follows:

[0185] Calculate the number of multiplication operations in a convolutional layer :

[0186] ;

[0187] Calculate the number of addition operations in the convolutional layer :

[0188] ;

[0189] in, The total number of elements in the output feature map represents the total number of output values ​​that need to be calculated. The number of multiplication operations required for each output element; The number of addition operations required for each output element;

[0190] Calculate rotation operands Rotation operations specifically include basic rotation, channel rotation, output rotation, and spatial rotation.

[0191] Calculate the basic rotation operands :

[0192] ;

[0193] Calculate channel rotation operands :

[0194] ;

[0195] in, This is the floor function; This is the channel packing factor, with a default value of 4, representing the size of the channel groups processed in parallel.

[0196] Calculate the output rotation operands :

[0197] ;

[0198] in, The size of the regrouping group for the output channel, the default value is 8;

[0199] Computation space rotation operands :

[0200] ;

[0201] Total number of rotation operations in convolutional layers for:

[0202] ;

[0203] Calculate the multiplication depth of the convolutional layer The base multiplication depth of a convolutional layer is 1. If the number of input channels is large (e.g., ...), the multiplication depth will be increased. Processing needs to be done in batches, increasing the depth:

[0204] ;

[0205] in, A binary indicator function to identify the bias term; if a bias term exists, If no bias term exists, ;

[0206] For a fully connected layer, the input dimension is... , representing the number of input neurons; the output dimension is , indicating the number of output neurons; batch size is The calculation formula is as follows:

[0207] Calculate the number of multiplication operations in a fully connected layer :

[0208] ;

[0209] Calculate the number of addition operands in a fully connected layer :

[0210] ;

[0211] Calculate the rotation operands of a fully connected layer :

[0212] ;

[0213] in, This is the scaling factor for the fully connected layer; It is a rounding function;

[0214] Calculate the multiplication depth of the fully connected layer :

[0215] ;

[0216] in, is the grouping factor, representing the parameters used to group input neurons in the fully connected layer computation.

[0217] Based on the above metrics, calculate the total computational cost of each convolutional layer and fully connected layer. :

[0218] If this layer is a convolutional layer, then:

[0219] ;

[0220] in, This represents the total computational cost of the convolutional layer. The maximum multiplication depth threshold; This is the bootstrap factor for the convolutional layer, indicating whether bootstrapping is required; if... This indicates a need for self-initiation. This indicates that bootstrapping is not required;

[0221] If this layer is a fully connected layer, then:

[0222] ;

[0223] in, This represents the total computational cost of the fully connected layer. The maximum multiplication depth threshold; This is the bootstrap factor for fully connected layers, indicating whether bootstrapping is required. This indicates a need for self-initiation. This indicates that bootstrapping is not required;

[0224] Step 2.4: Based on the weight norm of channels or neurons and the computational overhead of CKKS homomorphic encryption, construct a quantification evaluation mechanism for pruning strategy priority to comprehensively evaluate the retention value of each channel or neuron.

[0225] For convolutional layers, their total computational cost is... The homomorphic computation cost for each output channel is then amortized across it. ,in, This represents the number of output channels of the convolutional layer; combined with the weight norm of each output channel of the convolutional layer calculated in step 2.2. Calculate its output channel priority index :

[0226] ;

[0227] in, It is a small constant introduced to avoid division by zero;

[0228] For fully connected layers, their total computational overhead is... The homomorphic computation cost of each output neuron is obtained by amortizing the cost across all output neurons. ,in, This represents the number of output neurons in the fully connected layer; combined with the weight norm of each output neuron in the fully connected layer calculated in step 2.2. Calculate the priority index of its output neuron. :

[0229] ;

[0230] The higher the priority value, the greater the contribution of the homomorphic computation cost per unit of channel or neuron to the model accuracy, and it should be given priority to be retained during the pruning process.

[0231] Step 2.5: Perform topology pruning on the neural network model. Based on the preset pruning ratio... Determine the number of channels or neurons retained in each layer. ,in, This refers to the original number of channels or neurons.

[0232] Perform pruning operations on different types of layers respectively:

[0233] For convolutional layers, the output channel priority index is calculated based on step 2.4. The output channels of this layer are sorted in descending order. The convolutional layer retains its index set. Defined as the top priority The set consisting of channel indices:

[0234] ;

[0235] in, This indicates the priority ranking among all channels. The original index of the bit;

[0236] Based on the reserved index set From the original weight tensor The corresponding output channels are extracted from the middle slices to generate the pruned new weight tensor. :

[0237] ;

[0238] At the same time, update the number of output channels of this layer to ;

[0239] If the current convolutional layer has a bias term Then based on the same set of reserved indexes Synchronously update the bias vector:

[0240] ;

[0241] in, After pruning and The corresponding new bias vector;

[0242] At the same time, adjust the number of input channels in the next layer (e.g., for a convolutional layer) or the number of input neurons (e.g., for a fully connected layer).

[0243] For fully connected layers, the priority index of the output neurons calculated in step 2.4 is used. Sort all output neurons in the current layer in descending order. The fully connected layer retains its index set. Defined as the top priority The set of neuron indices:

[0244] ;

[0245] in, This indicates the priority ranking among all neurons. The original index of the bit;

[0246] Based on the same set of reserved indexes From the original weight matrix Extracting the corresponding output neurons (rows) from the middle slices, thereby generating a new pruned weight matrix. :

[0247] ;

[0248] At the same time, update the number of output neurons. ;

[0249] If the fully connected layer has a bias term Then based on the same set of reserved indexes Synchronously update the bias vector:

[0250] ;

[0251] in, After pruning and The corresponding new bias vector;

[0252] At the same time, adjust the number of input channels in the next layer (if it is a convolutional layer) or the number of input neurons (if it is a fully connected layer).

[0253] Step 2.6: After pruning the neural network model, recalculate the total CKKS complexity of the pruned model based on the calculation method in Step 2.3. By comparing the total complexity before and after pruning, and statistically analyzing the reduction in the number of multiplication operations, rotation operations, and multiplication depth, the effectiveness of pruning in reducing the homomorphic computational complexity of CKKS can be quantitatively verified.

[0254] Step 2.7: To mitigate the accuracy loss caused by pruning, the pruned model is fine-tuned using an adaptive moment estimation optimization algorithm, with the information gain loss function as the benchmark. For the goal:

[0255] ;

[0256] in, For the first The first sample Real labels for each category; For the first The first sample The model predicts probabilities for each category; For batch size, This represents the number of categories.

[0257] The model was iteratively optimized for several rounds on a subset of the training dataset of the public image dataset, and its performance was evaluated using a validation set. After fine-tuning, the model accuracy was restored to a level close to that before pruning, ensuring that the pruning optimization did not significantly affect the performance.

[0258] Step 3: Implement hierarchical parameter genetic homomorphic encryption on the pruned model: Using the network hierarchy as a link, each layer independently generates encryption parameters. The parameters of the first layer are adaptively generated from the statistical features of image blocks, and the parameters of subsequent layers are tuned through evolutionary genetic modification of the parameters of the previous layer, forming a genetic flow of encryption parameters. The specific process is as follows:

[0259] Step 3.1: Preprocess the pruned neural network weight matrix and bias vector by numerical normalization and format adjustment to meet the encoding requirements of the CKKS homomorphic encryption scheme for plaintext polynomials. The specific process is as follows:

[0260] Step 3.1.1: Use a normalization method to constrain the dynamic range of the weights and biases to reduce the quantization error of CKKS encryption. Calculate the... Layer weight dynamic range :

[0261] ;

[0262] in, For the first Layer weight matrix; To traverse the matrix All rows With all columns Take the maximum value of the elements in the list; To traverse the matrix All rows With all columns The minimum value is taken from the elements in the set.

[0263] Calculate the first Layer bias dynamic range :

[0264] ;

[0265] in, For the first Layer bias vector; For traversing vectors All elements Take the maximum value; For traversing vectors All elements Take the minimum value;

[0266] The normalization formula is:

[0267] ;

[0268] ;

[0269] in, , The first Layer-normalized weight matrix and bias vector; To avoid small values ​​that are divided by zero, the normalized weights and biases are usually in the range of [-1, 1] or [0, 1].

[0270] No. Dynamic range of weights after layer normalization for:

[0271] ;

[0272] in, For traversal All rows With all columns Take the maximum value of the elements in the list; For traversal All rows With all columns The minimum value is taken from the elements in the set.

[0273] No. Layer normalized bias dynamic range for:

[0274] ;

[0275] in, For traversing vectors All elements Take the maximum value; For traversing vectors All elements Take the minimum value.

[0276] Step 3.1.2, place the first Layer normalized weight matrix and bias vector Adjust to a format suitable for encoding.

[0277] The matrix is ​​flattened to transform it into a vector sequence suitable for encoding. This operation maps the matrix elements to one-dimensional vectors in row-major order.

[0278] ;

[0279] in, For the first The number of rows in the weight matrix after layer normalization; For the first The number of columns in the weight matrix after layer normalization; This indicates a flattening operation based on row priority. Represents the weight matrix of the first element. Line number The specific element values ​​of the column, From 1 to row index, From 1 to Column index; Represents the construction and transpose of a vector;

[0280] If the length of the flattened vector Exceeding the maximum capacity of a single encoding in the CKKS scheme Then it needs to be partitioned into Subvectors:

[0281] ;

[0282] The sub-vector sequence after partitioning is ,in, For the first The first layer Each subvector; The actual length does not exceed This ensures that each subvector can be encoded individually into a CKKS ciphertext; where The index of the subvector, with values ​​ranging from 1 to... ;

[0283] Normalized bias vector It is already a one-dimensional structure and requires no additional flattening operation. If its length... Exceeding the capacity of a single encoding Then the partition is Subvectors:

[0284] ;

[0285] The sub-vector sequence after partitioning is ,in, For the first The first layer Each subvector; The actual length does not exceed ,in The index of the subvector, with values ​​ranging from 1 to... .

[0286] Step 3.2: Adaptively generate the first-layer encryption parameters of the neural network based on the image block statistical features extracted in Step 1. The specific process is as follows:

[0287] Step 3.2.1: Set the scaling factor for the first layer. This allows it to cover the dynamic range, numerical distribution, and dynamic range of the first-layer weights of the input image.

[0288] ;

[0289] in, It is a constant; and These are the standard deviation and dynamic range of the weight matrix after normalization of the first convolutional layer, respectively.

[0290] CKKS employs layered modules to manage noise growth in the ciphertext. The module chain allocation strategy dynamically allocates modules based on the actual homomorphic computation depth of each neural network layer, resulting in a total ciphertext module. It can be viewed as the product of a series of smaller prime numbers:

[0291] ;

[0292] in, The total length of the modular chain is determined by the total computational depth of the neural network (i.e., the total number of layers);

[0293] For the Layered neural networks (where The number of moduli allocated is equal to the multiplication depth required for that layer, and the continuous subsets of moduli are allocated layer by layer in a cumulative order starting from the beginning of the moduli chain until all layers are covered, thereby ensuring that the noise growth of each layer is effectively controlled within the range of the allocated moduli.

[0294] The initial modulus of the first layer of encryption in a neural network satisfy:

[0295] ;

[0296] in, This is the magnification factor calculated for the first layer;

[0297] Step 3.2.2: Verify the parameter accuracy of the CKKS homomorphic encryption scheme by evaluating the encryption computation of the first layer of the neural network model. Assume that the encryption computation of the first layer of the neural network specifically employs encryption matrix multiplication. ; Calculate the relative error between the decryption result and the plaintext result. :

[0298] ;

[0299] in, and These are CKKS encryption and decryption operations, respectively. The input data matrix; This is the weight matrix of the first layer of the neural network; Represents the L2 norm;

[0300] Determine whether the relative error satisfies the following conditions. , The maximum permissible relative error is defined; if this condition is met, the verification ends; otherwise, adjustments are performed.

[0301] like If the value exceeds the preset threshold, a logarithmic transformation or outlier truncation can be applied to the sub-block data, and then the statistical characteristics and encryption parameters can be recalculated according to the process in step 1.4.

[0302] like If the value is less than or equal to a preset threshold, then increase the value. or Then return to step 3.2.1 to regenerate and verify the parameters until the error meets the requirements;

[0303] At this point, the first-layer encryption parameter set has been generated. This serves as the initial seed for the genetic flow of encryption parameters in subsequent layers.

[0304] Step 3.3: To realize the dependency and dynamic evolution of encryption parameters between neural network layers, define a secure genetic tuning function. And based on this, execute the encrypted parameter genetic stream.

[0305] Genetic tuning function It is a deterministic function whose input is the previous layer (the first layer). The set of encryption parameters for the layer The output is the current layer (the first layer). The set of encryption parameters for the layer The core mechanism of this function is as follows:

[0306] ;

[0307] in, For the first The set of encryption parameters for the layer, For the first Layer scaling factor For the first The modulus of the layer; The internal cryptographic parameters of the genetic tuning function are defined as a one-way SHA-256 hash function to ensure forward security during the evolutionary process; For the first Contextual information of the layer, including the first Dynamic range of weights in the normalized weight matrix of a convolutional layer Standard deviation and magnification factor This is used to achieve adaptive adjustment of parameters and network characteristics;

[0308] It consists of two sub-functions that handle the scaling factor and the initial modulus, respectively. and The structure is as follows. The specific tuning rules for each parameter are as follows:

[0309] Genetic tuning of scaling factors is achieved through the scaling factor genetic tuner function. calculate:

[0310] ;

[0311] in, It is a constant; For the first The dynamic range of the weight moments after normalization of the convolutional layer;

[0312] Genetic tuning of the initial modulus is achieved through the initial modulus genetic tuning subfunction. The calculation shows that the function performs modulus decrementing while ensuring that the noise constraint of homomorphic multiplication is satisfied:

[0313] ;

[0314] And it must meet the following conditions:

[0315] ;

[0316] in, For the first The prime factors in the modulus chain consumed by layer homomorphic computation. This operation achieves an evolutionary decrease in the modulus;

[0317] The first-layer encryption parameter set generated in step 3.2 As the initial seed, along the neural network layer number Iterate and apply the genetic tuning function. Dynamically generate unique encryption parameters for each layer:

[0318] ;

[0319] This iterative process constructs a sequence of encrypted parameters. This refers to the genetic flow of encryption parameters, which ensures deep topological coupling between the encryption system and the neural network structure.

[0320] Step 3.4: Perform layer-by-layer CKKS homomorphic encryption on the weight matrix and bias vector of each layer of the neural network based on the encrypted parameter genetic flow.

[0321] Using the genetic flow generated in step 3.3, homomorphic encryption is performed on the parameters of each neural network layer. This process ensures that the encryption of each layer uses its unique encryption parameters that have a genetic dependency on the predecessor layer.

[0322] For the neural network layer( ), obtain from the encrypted parameter genetic stream the result of step 3.2 (when ) or step 3.3 (when Generated and exclusive to the first Layer encryption parameter set The core parameter is the scaling factor. With the initial modulus ;

[0323] Use the Layer scaling factor The preprocessed weight subvectors and bias subvectors are encoded into CKKS plaintext polynomials:

[0324] ;

[0325] ;

[0326] in, For the first Layer Weight subvectors Encoded plaintext polynomial; For the first Layer bias subvectors Encoded plaintext polynomial; For CKKS encoding functions; These are formal variables in the ring;

[0327] Use system public key Homomorphic encryption is applied to the generated plaintext polynomial:

[0328] ;

[0329] ;

[0330] in, For the first Layer The CKKS ciphertext is encoded and encrypted using each weighted subvector. For the first Layer The CKKS ciphertext is encoded and encrypted using bias subvectors; This is a CKKS encryption function;

[0331] All generated ciphertext The data is transferred and stored in the FPGA's external memory, preparing it for subsequent efficient privacy-preserving inference computations to be performed on the FPGA.

[0332] Step 3.5: Verify the correctness and accuracy of the encryption parameters for each layer, and optimize the parameters to suit FPGA resources and inference requirements. The specific process is as follows:

[0333] Use test vectors that conform to the first-level input specifications. Execute the first one respectively The plaintext forward computation and homomorphic encryption computation of each layer are performed, and the ciphertext result is decrypted. The relative error between the decryption result and the plaintext result is calculated. :

[0334] ;

[0335] in, Represents the L2 norm; To prevent division by zero of constants; For the first The plaintext vector obtained after decrypting the ciphertext result by layer; For the first Plaintext calculation results of the layer;

[0336] like ,in If the preset maximum permissible error threshold is used, then the first... The layer encryption precision verification passes; otherwise, the verification fails; if it fails, the scaling factor is increased. or initial modulus Then, repeat the verification process from steps 3.3 to 3.5; if the conditions are met, the verification process ends.

[0337] Step 4: Deploy the optimized model to the FPGA platform to achieve efficient privacy-preserving inference. This implementation uses the ALINX Black Gold AX7020 FPGA development board as the target hardware platform and employs the Vivado HLS (High-Level Synthesis) tool for design and optimization. The specific process is as follows:

[0338] Step 4.1: Evaluate the hardware resource constraints of the ALINX Black Gold AX7020 development board. Define the set of resource constraints. , , , , These correspond to constraints on the number of logic units, flip-flops, digital signal processing modules, and block memory, respectively. Based on the CKKS encryption parameters determined in step 3 (including the polynomial ring dimension)... Modular chain length Based on the size of the pruned neural network model obtained in step 2, estimate the resource requirements:

[0339] Digital Signal Processing Module (DSP) Requirements :

[0340] ;

[0341] in, This represents the total number of homomorphic operations in the pruned model. This is an empirical coefficient, usually set to 100, representing the approximate number of DSPs required for each core computing unit or each large-scale multiplier.

[0342] Block RAM (BRAM) requirements :

[0343] ;

[0344] in, This represents the cumulative storage size of the encrypted data and intermediate results. This represents the depth of each BRAM module in the FPGA.

[0345] Logic Unit Count (LUT) Requirements :

[0346] ;

[0347] in, To estimate the number of LUTs required for the computational complexity of each unit;

[0348] Trigger Count (FF) Requirements :

[0349] ;

[0350] in, To estimate the number of FFs required for the computational complexity of each unit;

[0351] Compare the estimated resource requirements with the resource constraints on the FPGA development board and perform constraint checks: , , , The resource assessment is considered successful only if all inequalities are true simultaneously; otherwise, the encryption parameters are adjusted or the global pruning ratio is increased. Then, a new resource assessment will be conducted until all constraints are met.

[0352] Step 4.2: Design a dedicated hardware acceleration engine for CKKS homomorphic encryption computation on the FPGA. This engine consists of three core modules: a parallel homomorphic multiply-accumulate unit (PMAC), a configurable analog-to-digital switching unit (CMSU), and an efficient ciphertext rotation unit (ECRU). Through a unified dynamic parameter configuration interface, it loads the dynamic encryption parameters between neural network layers generated in Step 3, achieving hardware-software co-operational privacy-preserving inference. The design of each module is as follows:

[0353] Step 4.2.1: Design a parallel homomorphic multiply-add unit. This unit is the core module for performing homomorphic multiply-add operations between the ciphertext weights and the input data. It adopts a parallel polynomial multiplication architecture based on number theory transformation (NTT). The specific execution steps are as follows:

[0354] Ciphertext weight polynomial and input activation polynomial Loaded into the dual-port BRAM and through a The butterfly operation network at the level performs NTT in parallel, transforms it to the NTT domain, and obtains... and ;in, For polynomials The representation in the number-theoretical transformation domain after transformation by number-theoretical transformation; For polynomials The representation in the number-theoretical transformation domain after transformation by number-theoretical transformation;

[0355] In the NTT domain, for and Perform point-by-point modular multiplication on the corresponding coefficients:

[0356] ;

[0357] in, For coefficient index; For modulo operation; and They are and The first under the number-theoretic transformation domain One coefficient; Modulus;

[0358] The result of the dot product By using the parallel inverse NTT (INTT), the coefficient representation is transformed back to obtain the homomorphic multiplication result. And sent to the subsequent unit, in which It is an inverse number theory transformation.

[0359] Step 4.2.2: Design a configurable analog-to-digital switching unit. This unit is responsible for performing analog-to-digital switching and rescaling after homomorphic multiplication to control noise. Its execution flow is as follows:

[0360] Obtain the first from the genetic flow parameters Layer Modular Chain And based on the Barrett reduction algorithm, the ciphertext polynomial is reduced from the current modulus. Switch to target module For ciphertext polynomials in modulus Each coefficient below The core operations are as follows:

[0361] Scale adjustment:

[0362] ;

[0363] in, These are the coefficients after scaling. Scaling factor In modulus Modular inverse;

[0364] Approximate quotient calculation:

[0365] ;

[0366] in, This is the approximate quotient obtained through calculation; For pre-calculated constants, ; For accuracy parameters, ;

[0367] Output result:

[0368] ;

[0369] in, These are the output coefficients; The modulus of the ciphertext before the operation was performed, i.e., the current modulus; This is the next modulus that the ciphertext needs to switch to after the operation is performed, i.e., the target modulus;

[0370] Verify output coefficients The entire encrypted polynomial, after processing, is output, and its modulus has been successfully switched to... The noise level decreased as expected.

[0371] Step 4.2.3: Design an efficient ciphertext rotation unit (ECRU). This unit provides dedicated hardware support for the cyclic shifting of ciphertext slots to achieve data transformation along the channel dimension in the encryption model. The execution steps are as follows:

[0372] Based on rotation step Dynamically load the corresponding Galois key. and polynomial ring dimension ;

[0373] The address generator is based on the step size. Parallel computing New index for each slot ;

[0374] use Multiple parallel memory banks, read at once Data for each slot, and the Galois key. Perform Galois automorphism mapping homomorphic multiplication in the PMAC unit;

[0375] Using a configurable crossbar switch network of an FPGA, the mapped data is distributed according to the first... New index for each slot The final data rearrangement is performed, and the entire rotation operation is completed with a single parallel memory write.

[0376] Step 4.3: Implement a hierarchical encrypted inference pipeline architecture. Design a dual-pipeline synchronization mechanism based on a finite state machine-based control pipeline and a hardware acceleration engine-based data pipeline to map the neural network model to the hardware pipeline, achieving dynamic adaptation between the encrypted parameter genetic flow and the fixed hardware architecture. The specific process is as follows:

[0377] Step 4.3.1: Design a dual-pipeline synchronization mechanism to coordinate the control pipeline and data pipeline through handshake signals, ensuring the timing synchronization of the encryption context and the computation task;

[0378] The control pipeline uses a finite state machine (FSM) to implement the control logic. Scheduled according to network layers, the FSM sequentially performs parameter configuration, waits for data readiness, issues a computation trigger signal, and transitions to the next layer upon receiving computation completion feedback. The state transitions of the FSM are indexed by the currently processed neural network layer. Driven by handshake signals from the data pipeline;

[0379] The data pipeline consists of the hardware acceleration engine described in step 4.2, which performs ciphertext computation according to the network layer order, with the neural network's first... The conditions for starting layer computing are that the trigger signal of the control pipeline and the data ready signal of the previous layer are received simultaneously.

[0380] The control pipeline and the data pipeline are driven by a pair of handshake signals, ensuring that computation only begins when both the control and data layers are ready, thus achieving synchronization of the two pipelines.

[0381] Step 4.3.2: Map the neural network model onto the data pipeline and integrate the core computational operations. The specific mapping and integration process is as follows:

[0382] Neural network Each layer, mapped to a data pipeline according to its computational order, is... Each pipeline stage is a continuously connected pipeline stage, meaning that each pipeline stage is a dynamically configurable processing unit with the Parallel Homomorphic Multiply-Accumulate Unit (PMAC) of step 4.2 as its core and optionally integrating a Configurable Analog-to-Digital Switching Unit (CMSU).

[0383] The data pipeline stages communicate with each other via a high-bandwidth FIFO (First-In-First-Out) buffer on the FPGA, enabling a "later-stage triggered, earlier-stage preloaded" data flow pattern: when the... While the first level is being calculated, its output is directly written to the second level. A FIFO buffer between levels. Once the amount of data in this FIFO buffer reaches a preset threshold, the next level is triggered. The computational preparation at the level ensures that the required input data is preloaded into its local buffer in advance;

[0384] To ensure smooth operation of the production line, its first... Level and First Interstage FIFO buffer depth for:

[0385] ;

[0386] in, It is the first Level of computational latency; It is the first The time it takes to load one unit of data at each level. During level calculation, the first The required input data for each layer has been prefetched, minimizing the latency of data transfer between layers.

[0387] Each pipeline stage dynamically configures a shared hardware acceleration engine based on the layer type of the neural network it implements. The integration process is as follows:

[0388] Convolutional or fully connected operation stages perform homomorphic multiplication by scheduling PMAC units; convolutional layers simulate sliding windows by performing parallel ciphertext rotation operations through ECRU units; fully connected layers are directly mapped to dot product operations.

[0389] The activation function unit acts as a coprocessor after a specific level, performing CKKS-compatible nonlinear approximation calculations on the output ciphertext of PMAC, and the calculation results are directly written into the input FIFO buffer of the next level.

[0390] After homomorphic multiplication, the inter-stage scheduling nodes for analog-to-digital switching are controlled by the CMSU unit, which schedules the pipeline to perform analog-to-digital switching and rescaling between specified stages to control noise.

[0391] Step 4.4: Design a storage scheduling subsystem based on double buffering and prefetching strategies to ensure the loading of encrypted parameter genetic streams and the data delivery of hierarchical pipelines; the specific process is as follows:

[0392] Step 4.4.1: To implement the parameter switching of the control pipeline in step 4.3.1, a double-buffering mechanism is used in the on-chip BRAM of the FPGA to encrypt the parameters. The specific process is as follows:

[0393] Two parameter buffers are allocated in the on-chip BRAM of the FPGA; when the control pipeline uses one of the buffers (which stores...), it is called a parameter buffer. When configuring the hardware acceleration engine in one cache area, the prefetching strategy simultaneously loads the next layer of encryption parameter sets into another cache area. ;

[0394] The cached parameters are directly mapped to the PMAC, CMSU, and ECRU units of the hardware acceleration engine through a dedicated configuration register interface implemented on the FPGA.

[0395] Step 4.4.2: To meet the data bandwidth requirements of multi-level concurrent execution in a hierarchical pipeline, a data prefetching mechanism driven by the control pipeline is designed. The specific process is as follows:

[0396] When the state machine enters the... When the layer is in operation, the prefetch controller is triggered, and the DMA controller retrieves the first... The required cryptographic model weights for each layer are preloaded from off-chip DDR (Double Data Rate Synchronous Dynamic Random Memory) to on-chip BRAM. This mechanism hides the data loading time within the computation time of the previous layer, ensuring the smooth execution of the hierarchical pipeline.

[0397] Step 4.5: Perform the physical implementation and deployment of the hardware system, using the Vivado HLS toolchain to achieve high-level synthesis, system integration, place and route, and generate bitstream files that meet the constraints. The specific process is as follows:

[0398] First, in Vivado HLS, the core loop is synthesized into a fully pipelined hardware architecture by inserting compilation instructions, and reusable register-transfer-level (RTL) intellectual property (IP) cores are generated for PMAC, CMSU, and ECRU. After synthesis, the resource consumption of the generated RTL module is evaluated. Ensure that it meets the set of resource constraints defined in step 4.1. ;

[0399] Next, multiple RTL IP cores generated by HLS are integrated into the Vivado IP Integrator to build the complete hierarchical pipeline architecture described in step 4.3, and place-and-route is performed.

[0400] Then, static timing analysis is performed to verify whether the digital circuit design after placement and routing meets the target clock frequency constraint. If the target clock frequency constraint is not met, the process returns to the high-level synthesis stage in step 4.5.1. In HLS, compilation instructions are inserted or adjusted cyclically for the core calculations, splitting the original lengthy combinational logic path into multiple clock cycles. The synthesis and implementation process is then iterated again until the timing is completely converged.

[0401] Finally, the Vivado implementation tool was used to convert the final placement and routing results into a bitstream file. This bitstream file was then configured into the FPGA via the JTAG interface connecting the host and the FPGA development board, completing the deployment of the hardware system and laying the hardware foundation for subsequent execution of efficient privacy-preserving inference and achieving a performance leap.

[0402] Step 4.6: Execute the complete neural network inference process on the FPGA, and test the functional correctness of the encrypted neural network model on the FPGA and evaluate its performance. The specific process is as follows:

[0403] Step 4.6.1: Execute the neural network inference process, the specific process is as follows:

[0404] When the system starts up, the ARM processor sends a global startup signal to the FPGA; the control pipeline state machine responds, loading the first-layer encryption parameter set generated in step 1 from the off-chip DDR memory. The data is transmitted to various hardware acceleration engines; meanwhile, privacy-protected input image data and encrypted model weight parameters are also transmitted or preloaded into the FPGA and DDR via high-speed interfaces.

[0405] Data propagates layer by layer along the hierarchical pipeline constructed in step 4.3. For the first... layer( From 1 to The control assembly line state machine ensures that the first... Layer encryption parameter set Once the data from the previous layer has been loaded and is ready, a trigger signal is issued; the data pipeline starts the next... Layer computation performs homomorphic multiply-accumulate operations in the PMAC unit, and performs homomorphic nonlinear approximation calculations and analog-to-digital switching operations as needed; the output of this layer is written to the FIFO buffer of the next layer, driving the pipeline forward;

[0406] After the output layer completes its calculations, the encrypted result is transmitted back to the ARM processor via the DMA controller and sent to the client for decryption, completing one inference cycle. The entire model and intermediate data are encrypted, effectively ensuring security.

[0407] Step 4.6.2: Execute the test procedure to confirm its functional correctness and performance evaluation. On the standard test set, run FPGA encryption inference and CPU plaintext inference in parallel. For the same input, decrypt the FPGA output. and CPU output Compare and calculate their relative error. :

[0408] ;

[0409] in, This represents the total number of samples used for validation on the standard test dataset. To avoid small values ​​when divided by zero; For the first The FPGA output for decrypting a sample; For the first CPU output for each sample;

[0410] like (in If the maximum permissible error threshold preset in step 3.5.1 is met, then the accuracy verification passes. Otherwise, backtrack to step 3.5 to adjust the encryption parameters or to step 2 to optimize the model pruning strategy;

[0411] The performance evaluation metrics are as follows:

[0412] Throughput :

[0413] ;

[0414] in, Batch size; For inference delay;

[0415] Calculation speedup :

[0416] ;

[0417] in, This refers to the time consumed by CPU inference; This refers to the time consumed by FPGA inference;

[0418] resource utilization rate :

[0419] ;

[0420] in, This refers to the actual amount of resources used. The total amount of resources provided for the FPGA platform.

[0421] This invention provides a hierarchical homomorphic encryption neural network inference deployment optimization method for Field Programmable Gate Array (FPGA) hardware platforms. First, the input image is adaptively segmented and statistical features are extracted. Based on a collaborative fusion optimization paradigm, topological pruning with CKKS homomorphic encryption awareness is implemented on the neural network. Next, hierarchical parameter genetic homomorphic encryption is implemented on the pruned network: using the network layer order as a link, each layer independently generates encryption parameters. The parameters of the first layer are adaptively generated from the statistical features of the image segments, and the parameters of subsequent layers are tuned through evolutionary genetic processes of the previous layer's parameters, forming a genetic flow of encryption parameters. Finally, the optimized model is deployed to the FPGA platform to achieve efficient privacy-preserving inference. Experiments and performance evaluations are conducted using a convolutional neural network (CNN) trained on the MNIST dataset as an example. This dataset contains 70,000 handwritten digit images, specifically 60,000 training images and 10,000 test images. The images contain handwritten Arabic numerals from 0 to 9. The dataset is formed after standardization of the original collected samples.

[0422] In the adaptive block partitioning phase, the experimentally selected sub-block size was: Total number of elements in the sub-block: Statistical features were extracted; during the topology pruning stage, the comparison results of model complexity before and after pruning are shown in Table 1 below. The model complexity was significantly compressed. The pruned model was then trained, as follows: Figure 2 As shown, the test results indicate that the accuracy of the pruned model reached 99.37%, which is better than the accuracy of the unpruned model (also known as the original model) of 99.35%, thus verifying the effectiveness of the pruning method.

[0423] Table 1 Comparison of model complexity before and after pruning

[0424] .

[0425] Through a hierarchical parameter genetic encryption mechanism, encryption time is reduced by 18%. Subsequent layer parameters are dynamically evolved through genetic tuning, improving encryption efficiency while ensuring noise control. Finally, the optimized model is deployed on the AX7020 FPGA platform, achieving efficient privacy-preserving inference with the help of a hierarchical pipeline and a dedicated hardware acceleration engine. The comparison of FPGA resource usage before and after model pruning is shown in Table 2 below. The results show that the pruned model can significantly reduce resource requirements without significantly affecting accuracy.

[0426] Table 2 Comparison of FPGA resource usage before and after model pruning.

[0427] .

[0428] Experimental results demonstrate that this invention can synergistically optimize the algorithm accuracy of the model, the homomorphic encryption characteristics of CKKS, and the inference performance improved by FPGA hardware, providing an efficient and practical solution for privacy-preserving neural network inference.

[0429] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A hierarchical homomorphic encrypted neural network inference deployment optimization method for FPGA, characterized in that, Includes the following steps: Step 1: Perform adaptive block segmentation on the input image and calculate and extract statistical features; Step 2: Based on the collaborative fusion optimization paradigm, perform topology pruning of the neural network model using CKKS homomorphic encryption sensing; the specific process is as follows: Step 2.1: Construct the neural network model to be pruned and establish a computational overhead model under the CKKS homomorphic encryption environment; the neural network model contains several convolutional layers and several fully connected layers; calculate the total computational overhead of each layer. : ； in, , , , These are the cost weights for multiplication, addition, rotation, and bootstrapping, respectively. , , , These represent the number of multiplication, addition, rotation, and bootstrapping operations, respectively. Step 2.2: Use channel-level or neuron-level weight norms as a magnitude evaluation metric; For a convolutional layer, the weight tensor is... For each output channel The weight norm is calculated using the L2 norm. For a fully connected layer, the weight matrix is: For each output neuron The weight norm is calculated using the L2 norm. Step 2.3: Calculate the total computational cost for each convolutional and fully connected layer of the model; the specific process is as follows: For convolutional layers, the number of multiplication, addition, and rotation operations and the multiplication depth are calculated based on the input and output dimensions and the convolution kernel parameters, respectively. For fully connected layers, the number of multiplication, addition, and rotation operations and the multiplication depth are calculated based on the number of input and output neurons and the batch size, respectively. Based on the above indicators and the total computational cost formula in step 2.1, calculate the total computational cost of each layer. Step 2.4: Based on the weight norm of the channel or neuron and the computational cost of CKKS homomorphic encryption, comprehensively evaluate the retention value of each channel or neuron and output the priority index of each channel or neuron; the specific process is as follows: The total computational cost of the current layer is evenly distributed to each output channel or neuron to obtain the average homomorphic computational cost. Combined with the weight norm of each output channel or neuron in step 2.2, the priority index of its output channel or neuron is calculated. Those with higher values ​​are preferentially retained during the pruning process. Step 2.5: Perform topology pruning on the neural network model; the specific process is as follows: According to the pruning ratio Determine the number of channels or neurons retained in each layer. ; Based on the priority index of the channels or neurons output in step 2.4, sort all output channels or neurons in the current layer in descending order; select the top priority ones. The channel or neuron indexes constitute a reserved index set; Based on the retained index set, the corresponding output channels or neurons are extracted from the original weights to generate new pruned weights, and the number of output channels or neurons in the current layer is updated accordingly. ; If a bias term exists in the current layer, the bias vector is updated synchronously based on the same set of retained indices, and the number of input channels or neurons in the next layer is adjusted. Step 2.6: Verify the model complexity after pruning; Step 2.7: Fine-tune the pruned model; Step 3: Implement hierarchical parameter genetic homomorphic encryption on the pruned model: Using the network hierarchy as the link, each layer generates encryption parameters independently. The parameters of the first layer are adaptively generated by the statistical features of image blocks, and the parameters of subsequent layers are tuned by the evolutionary genetic tuning of the parameters of the previous layer to form a genetic flow of encryption parameters. Step 4: Deploy the optimized model to the FPGA platform to achieve efficient privacy-preserving inference.

2. The hierarchical homomorphic encrypted neural network inference deployment optimization method for FPGA as described in claim 1, characterized in that, The specific process of step 1 is as follows: Step 1.1: Before segmenting, normalize the input image; Step 1.2: Determine the size of the sub-blocks after dividing the input image. ,in, The height of the input image sub-block; The width of the input image sub-block; The number of color channels in the input image; the size of the sub-blocks must satisfy FPGA resource constraints, CKKS encryption constraints, and neural network computation constraints. Step 1.3: Normalize the image Divided into The sub-blocks are flattened into vectors; Each sub-block after division is defined as: ； in, for In the height direction Width direction is The sub-block at the location; This is the index of the sub-block in the height direction; This is the index of the sub-block in the width direction; , These represent the total number of sub-blocks in the image along the height and width directions, respectively. each sub-block Flatten into a one-dimensional vector through vectorization operations. ; Step 1.4: For each flattened one-dimensional vector of sub-blocks, calculate five numerical distribution features: mean, variance, maximum value, minimum value, and dynamic range. Summarize these numerical distribution features to obtain three global features, including the global mean. Global variance Global dynamic range : ； in, The total number of sub-blocks in the input image; For sub-blocks The mean; ； in, For sub-blocks The variance; ； in, To find the maximum value of the one-dimensional vectors of all sub-blocks, take the maximum value. To find the minimum value of the one-dimensional vector that iterates through all sub-blocks, take the minimum value. , Sub-blocks The maximum and minimum values ​​of a one-dimensional vector.

3. The hierarchical homomorphic encrypted neural network inference deployment optimization method for FPGA as described in claim 2, characterized in that, The specific process of step 3 is as follows: Step 3.1: Preprocess the pruned neural network weight matrix and bias vector by numerical normalization and format adjustment to meet the encoding requirements of the CKKS homomorphic encryption scheme for plaintext polynomials; the specific process is as follows: Step 3.1.1, calculate the first... Layer weight dynamic range and bias dynamic range And normalize it to obtain the first Layer normalized weight matrix Bias vector Then based on , Calculate the first Dynamic range of weights after layer normalization Bias dynamic range : ； ； in, For traversal All rows With all columns Take the maximum value of the elements in the list; For traversal All rows With all columns The minimum value is taken from the elements in the set. For traversing vectors All elements Take the maximum value; For traversing vectors All elements Take the minimum value; Step 3.1.2, place the first Layer normalized weight matrix and bias vector Adjust to a format suitable for encoding; Step 3.2: The first-layer encryption parameters are adaptively generated based on the image block statistical features extracted in Step 1; Step 3.3: Define the inter-layer encryption parameter genetic tuning function and execute the encryption parameter genetic flow; Step 3.4: Perform layer-by-layer CKKS homomorphic encryption on the weight matrix and bias vector of each layer based on the genetic flow parameters; Step 3.5: Verify the correctness and accuracy of the encryption parameters for each layer, and optimize the parameters to suit FPGA resources and inference requirements.

4. The hierarchical homomorphic encrypted neural network inference deployment optimization method for FPGA as described in claim 3, characterized in that, The specific process of step 3.2 is as follows: Step 3.2.1, Scaling factor of the first layer Set to: ； in, It is a constant; and These are the standard deviation and dynamic range of the weight matrix after normalization of the first convolutional layer, respectively. Set the initial modulus of the first layer. for: ； in, This is the magnification factor calculated for the first layer; Step 3.2.2: Verify the parameter accuracy of the CKKS homomorphic encryption scheme by evaluating the encryption calculation of the first layer of the neural network model; Determine whether the relative error before and after encryption is less than the maximum permissible relative error. If satisfied, the verification ends; otherwise, adjustments are performed. like If the value exceeds the preset threshold, a logarithmic transformation or outlier truncation is applied to the sub-block data, and then the statistical characteristics and encryption parameters are recalculated according to the process in step 1.

4. like If the value is less than or equal to a preset threshold, then increase the value. or Then return to step 3.2.1 to regenerate and verify the parameters until the error meets the requirements; Finally, the first layer of encryption parameters is generated. .

5. The hierarchical homomorphic encrypted neural network inference deployment optimization method for FPGA as described in claim 4, characterized in that, The specific process of step 3.3 is as follows: Define the genetic tuning function and execute the encrypted parameter genetic stream; genetic tuning function The core mechanism is as follows: ； in, For the first The set of encryption parameters for the layer; For the first The set of encryption parameters for the layer, For the first Layer scaling factor For the first The initial modulus of the layer; These are the internal cryptographic parameters of the genetic tuning function; For the first Context information of the layer; It consists of two sub-functions that handle the scaling factor and the initial modulus, respectively. and constitute: Genetic tuning of scaling factors is achieved through the scaling factor genetic tuner function. calculate: ； in, For the first Layer scaling factor; Genetic tuning of the initial modulus is achieved through the initial modulus genetic tuning subfunction. calculate: ； And it satisfies: ； in, For the first The initial modulus of the layer; For the first Amplification factor for layer calculation; For the first The dynamic range of weights after normalization in a convolutional layer; Using the first layer's encryption parameter set as the initial seed, the algorithm iterates along the neural network layer indices and applies a genetic tuning function. Dynamically generate encryption parameters for each layer; The specific process of step 3.4 is as follows: Homomorphic encryption is performed on the parameters of each layer of the neural network using a genetic flow of encrypted parameters. Use the Layer scaling factor The pre-processed first Layer Weight subvectors With the bias subvectors Encoded as a CKKS plaintext polynomial; Use system public key The generated plaintext polynomial is homomorphically encrypted to obtain the ciphertext. The encrypted text is transmitted and stored in external memory.

6. The hierarchical homomorphic encrypted neural network inference deployment optimization method for FPGA as described in claim 5, characterized in that, The specific process of step 4 is as follows: Step 4.1: For the ALINX Black Gold AX7020 development board, evaluate its hardware resource constraints; define the resource constraint set. , , , , These correspond to constraints on the number of logic units, flip-flops, digital signal processing modules, and block memory, respectively. Based on the determined CKKS encryption parameters and the pruned model size, the resource requirements are estimated: digital signal processing module requirements, block memory requirements, logic unit quantity requirements, and flip-flop quantity requirements. Among these, the CKKS encryption parameters include the polynomial ring dimension and the modular chain length. Step 4.2: Design a dedicated hardware acceleration engine for CKKS homomorphic computation on the FPGA. Through a unified dynamic parameter configuration interface, load dynamically encrypted parameters between neural network layers to achieve privacy-preserving inference through hardware and software collaboration. Step 4.3: Implement a hierarchical encrypted inference pipeline architecture, design a dual-pipeline synchronization mechanism based on a finite state machine control pipeline and a hardware acceleration engine data pipeline, and map the neural network model to the hardware pipeline. Step 4.4: Design a storage scheduling subsystem based on double buffering and prefetching strategies; Step 4.5: Implement and deploy the hardware system physically, using the Vivado HLS toolchain to generate bitstream files; Step 4.6: Execute the complete inference process on the FPGA and test the functional correctness of the encrypted neural network model on the FPGA and evaluate its performance.

7. The hierarchical homomorphic encrypted neural network inference deployment optimization method for FPGA as described in claim 6, characterized in that, The specific process of step 4.2 is as follows: Step 4.2.1: Design a parallel homomorphic multiply-accumulate unit; Ciphertext weight polynomial and input activation polynomial Loaded into a two-port block memory, and subjected to parallel number-theoretical transformations via a butterfly arithmetic network, the result is transformed to the number-theoretical transformation domain. and ;in, and They are polynomials and The representation in the number-theoretical transformation domain after a number-theoretical transformation; In the transformation domain of number theory, for and Perform point-by-point modular multiplication on the corresponding coefficients; The dot product result is transformed back to coefficient representation through parallel inverse number theory transformation to obtain the homomorphic multiplication result, which is then fed into the subsequent unit; Step 4.2.2: Design a configurable analog-to-digital switching unit; Obtain the first from the genetic flow parameters The layered modular chain, and based on the Barrett reduction algorithm, the ciphertext polynomial is removed from the current modular chain. Switch to target module ; Step 4.2.3: Design an efficient encrypted text rotation unit; Based on rotation step Dynamically load the corresponding Galois key and polynomial ring dimension ; The address generator determines the step size. Parallel computing New index for each slot ; use Multiple parallel memory banks, read at once Data for each slot, and with the Galois key. Perform Galois automorphism mapping homomorphic multiplication in parallel homomorphic multiply-add units; Using a configurable crossbar switch network of an FPGA, the mapped data is distributed according to the first... New index for each slot The final data rearrangement is performed, and the entire rotation operation is completed with a single parallel memory write.

8. The hierarchical homomorphic encrypted neural network inference deployment optimization method for FPGA as described in claim 7, characterized in that, The specific process of step 4.3 is as follows: Step 4.3.1: Design a dual-pipeline synchronization mechanism; The control pipeline schedules the computation of each layer according to the network hierarchy using a finite state machine. The data pipeline consists of a hardware acceleration engine that performs ciphertext computations in the order of network layers; The control pipeline and the data pipeline are driven by a pair of handshake signals to achieve synchronization of the two pipelines; Step 4.3.2: The neural network model... Each layer, mapped to a data pipeline according to its computational order, is... The pipeline consists of consecutive stages, with encrypted data being transferred between stages via a first-in-first-out queue buffer; each pipeline stage dynamically configures a shared hardware acceleration engine based on the layer type of the neural network model it implements. The specific process of step 4.4 is as follows: Step 4.4.1: To enable parameter switching in the control pipeline, a double-buffering mechanism is used in the FPGA's on-chip block memory to store encrypted parameters. When the control pipeline uses a buffer to configure the current layer parameters, the prefetch strategy simultaneously loads the next layer's set of encrypted parameters into another buffer. The cached parameters are directly mapped to the hardware acceleration engine through a dedicated configuration register interface; Step 4.4.2: To meet the data bandwidth requirements of multi-level concurrent execution in a hierarchical pipeline, a data prefetching mechanism is designed. This mechanism drives data prefetching by controlling the pipeline. When the state machine enters the... When the layer is in operation, the prefetch controller will... The required cryptographic model weights for the layer are preloaded from off-chip double-rate synchronous dynamic random access memory to on-chip block memory.

Images