Software and hardware collaborative graph neural network privacy reasoning acceleration method and device
By employing a hybrid CKKS and BFV approach and co-designing hardware and software, we optimized data encoding and partitioning strategies, designed a dedicated hardware accelerator, and addressed the issues of computational and storage imbalance, high activation layer complexity, and limited applicability in privacy inference within graph neural networks. This improved efficiency and hardware utilization, and expanded the scope of applications.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-13
- Publication Date
- 2026-03-10
AI Technical Summary
Existing graph neural network privacy inference schemes based on fully homomorphic encryption suffer from problems such as unbalanced computation and storage resources, high computational complexity of activation layers, insufficient hardware-software coordination, and limited applicability of graph models, resulting in low efficiency and low utilization of hardware resources.
A hybrid scheme of CKKS and BFV is adopted, combined with batch nonlinear activation of lookup tables. Through hardware and software co-design, the data encoding and block strategy are optimized, and a dedicated shifter and a hardware accelerator with high resource reuse rate are designed to support inference of various GNN structures.
It achieves a performance improvement of 3.3×–4.2× on a 7nm process, a 119x increase in energy-delay-area, a 40% reduction in power consumption, supports multiple GNN structures, and has inference accuracy that is only 0.2%–0.7% lower than plaintext, thus expanding its application range.
Smart Images
Figure CN121638449A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of graph neural network privacy protection, and particularly relates to a hardware-aided graph neural network privacy inference acceleration method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND
[0002] As an important model for processing graph structured data, graph neural network (GNN) has been widely used in social network recommendation, knowledge graph, biological information analysis and other fields. With the increasing demand for privacy protection, privacy protection inference technology based on fully homomorphic encryption (FHE) has become a research hotspot. Fully homomorphic encryption has the characteristics of "use but not visible", and can perform calculations while keeping data encrypted, thus effectively protecting the privacy of user graph data. The commonly used fully homomorphic encryption schemes at present include the CKKS scheme and the BFV scheme based on ring learning with errors (RLWE), and the TFHE scheme based on learning with errors (LWE).
[0003] The existing graph neural network privacy inference based on fully homomorphic encryption mainly includes three types of operations: linear combination, adjacency aggregation and nonlinear activation. Among them, combination is a ciphertext-plaintext matrix multiplication of node features and model weights; aggregation is a ciphertext-ciphertext matrix multiplication of adjacency matrix and node features; and activation is a nonlinear mapping achieved by polynomial approximation or lookup table. Existing technologies such as CryptoGCN and Penguin mainly optimize linear layers, but use square activation or quadratic polynomial fitting to simplify the activation function, resulting in a significant decrease in model accuracy. If you want to improve accuracy, you need to use high-order polynomials (such as sign function requiring 343-order polynomial) to fit the activation function, which significantly increases the homomorphic multiplication depth required for inference, and must use bootstrapping technology to refresh the ciphertext, thus requiring larger encryption parameters (such as N=65536, polynomial modulus length > 1500 bits) to ensure security, resulting in huge computation and storage overhead.
[0004] To address the above challenges, a series of software-hardware collaborative fully homomorphic encryption acceleration architectures have been proposed. For example, Figure 1As shown, the accelerator of general-purpose homomorphic encryption such as CraterLake, SHARP, etc. mainly faces CKKS calculation, and generally needs to configure a large on-chip buffer of 180MB-512MB to meet the large parameter memory bandwidth requirement, resulting in high chip area and power consumption, and limited acceleration effect on GNN and other specific workloads. The first soft and hard collaborative graph neural network privacy inference acceleration scheme, PPGNN, uses CKKS to perform linear layer operation, and introduces the programmable bootstrap (PBS) of TFHE to implement the nonlinear layer. Although this scheme reduces the storage pressure by reducing the parameter size (the polynomial length is reduced to 4096), the single instruction single data (SISD) computing characteristics of TFHE make the nonlinear layer need to process data point by point, which greatly increases the computing amount.
[0005] Although the existing work has made some progress, there are still the following deficiencies:
[0006] Hardware and software design is fragmented: algorithm side optimization focuses on reducing computing complexity, but does not fully consider hardware structure characteristics; general-purpose accelerator lacks targeted support for graph neural network privacy inference load, resulting in limited overall performance.
[0007] Computing complexity imbalance: PPGNN relies on the programmable bootstrap of TFHE to implement the nonlinear layer, although it reduces the parameter size, but because its computing mode cannot be batch processed, the operation amount increases exponentially, for example, on the Pubmed dataset, the single layer calculation of the graph neural network needs to perform tens of thousands of programmable bootstrap operations.
[0008] Low utilization of hardware resources: existing acceleration schemes need to prepare large-capacity high-bandwidth storage for a small number of bootstrap operations, but bootstrap only accounts for less than 10% of the total calculation amount of inference, causing hardware resources to be idle for a long time.
[0009] Poor adaptability of graph model: the aggregation method of PPGNN only supports additive aggregation and cannot handle more general GNN structures such as weighted graphs.
[0010] The above problems seriously restrict the efficiency and practicality of GNN inference under FHE. SUMMARY
[0011] The present application aims at the following main problems existing in the existing graph neural network (GNN) privacy inference based on fully homomorphic encryption (FHE):
[0012] 1. Imbalance between computational and storage resources: Traditional solutions separate bootstrapping from nonlinear mapping, which requires extremely large encryption parameters. Linear layers are forced to operate under large parameters, resulting in huge computational and storage overhead.
[0013] 2. High computational complexity of activation layer: Existing PPGNN uses programmable bootstrapping based on TFHE to achieve nonlinear activation. Since the computation mode cannot be batch processed, the amount of computation increases explosively.
[0014] 3. Insufficient hardware and software synergy: Existing accelerators are generally based on general-purpose fully homomorphic encryption acceleration architectures, which cannot effectively utilize the characteristics of graph neural network inference load, resulting in low hardware resource utilization.
[0015] 4. Limited applicability to graph models: PPGNN only supports additive aggregation and cannot handle more general graph models such as weighted graphs.
[0016] The technical objective of this invention is to provide a GNN privacy inference acceleration system and device based on a hybrid homomorphic encryption algorithm. Through hardware and software co-design, the system reduces parameter scale and computational complexity, improves inference efficiency and energy efficiency, and maintains model accuracy and broader applicability of GNN.
[0017] To address the shortcomings of existing technologies, such as Figure 6 As shown, this invention proposes a hardware-software co-operational graph neural network privacy inference acceleration method, which includes:
[0018] In the input data encryption step, the client uses fully homomorphic encryption to encrypt the adjacency matrix and node features of the graph data, obtaining the ciphertext of the node features and the ciphertext of the adjacency matrix, and uploads them to the cloud server, which has a neural network model for performing inference tasks.
[0019] The homomorphic combination operation step involves the cloud server performing matrix multiplication on the ciphertext of the node's features and the plaintext of the weights of a specified layer in the neural network model to obtain the combination result.
[0020] In the format conversion step, the cloud server converts the combined result from the slot space to the coefficient space to obtain the conversion result;
[0021] In the homomorphic aggregation step, the cloud server performs polynomial operations in the coefficient space to perform matrix multiplication on the transformation result and the adjacency matrix ciphertext to obtain the result polynomial; during the matrix multiplication process, when the data length of the transformation result exceeds the dimension of a single polynomial, it is calculated in batches according to a preset block division rule and the block results are merged in the first few coefficients of the result polynomial.
[0022] The homomorphic activation step first extracts the valid data from the resulting polynomial using a shifter and converts the valid data into LWE format. Then, it is packaged into the plaintext space and evaluated according to the pre-generated lookup table polynomial to achieve nonlinear activation and bootstrapping, thus obtaining the activation result. The lookup table records the mapping relationship of the plaintext space after nonlinear activation.
[0023] The iterative output step is repeated until the homomorphic combination operation step is completed and the homomorphic activation step is completed, and the inference of all network layers in the neural network model is completed. The final activation result is returned to the client, and the client decrypts it to obtain the inference output result of the neural network model.
[0024] The aforementioned hardware-software co-operated graph neural network privacy inference acceleration method, wherein the homomorphic aggregation step includes:
[0025] Using the transformation result as feature data, when the length of the feature data is less than or equal to the dimension that a single polynomial can accommodate, matrix multiplication is performed by multiplying the polynomial corresponding to the ciphertext of the adjacency matrix with the polynomial of the feature data in the coefficient space. Specifically, this polynomial multiplication includes:
[0026] The feature data is assigned to the corresponding coefficients using the following formula: ; ;
[0027] In the formula The polynomial represents the number of features that a single polynomial can accommodate, and F represents the number of features that the graph data has.
[0028] Using row-major encoding, the feature V[i][j] in the i-th row and j-th column of the feature data is placed on the (i × F + j)-th coefficient of the polynomial v;
[0029] Reverse the t-th row A[t][i] of the adjacency matrix ciphertext A and encode it onto the coefficients of the polynomial a_t at intervals of F:
[0030] a_t[N - i × F mod N] = A[t][i]; The adjacency matrix ciphertext A is encoded row by row on a polynomial, so the entire adjacency matrix ciphertext A includes multiple polynomials. In a_t, t is the index of the polynomial, representing the encoding of the t-th row of the adjacency matrix ciphertext A;
[0031] Performing homomorphic polynomial multiplication a_t · v yields the resulting polynomial.
[0032] The aforementioned hardware-software co-operated graph neural network privacy inference acceleration method uses the transformation result as feature data. When the data length of the feature data is greater than the dimension N that a single polynomial can accommodate, the following steps are performed:
[0033] Generate a pair of block parameters ( , ): ; ;
[0034] In the formula, N is the dimension that a single polynomial can accommodate, and H is the number of nodes in the graph data;
[0035] The feature data V of size H×F and the ciphertext A of size H×H adjacency matrix are split into multiple sub-matrices; the feature data V is divided into The eigenvalue submatrix; the ciphertext A of this adjacency matrix is also divided into... Adjacency submatrices;
[0036] For each H_c×F_c eigenmatrix and its corresponding adjacent submatrix, perform homomorphic polynomial multiplication to obtain multiple sub-results. Combine all sub-results using homomorphic addition to obtain the result polynomial.
[0037] like Figure 7 As shown, this invention also proposes a hardware-software co-operated graph neural network privacy inference acceleration device, which includes:
[0038] In the input data encryption step, the client uses fully homomorphic encryption to encrypt the adjacency matrix and node features of the graph data, obtaining the ciphertext of the node features and the ciphertext of the adjacency matrix, and uploads them to the cloud server, which has a neural network model for performing inference tasks.
[0039] The homomorphic combination operation step involves the cloud server performing matrix multiplication on the ciphertext of the node's features and the plaintext of the weights of a specified layer in the neural network model to obtain the combination result.
[0040] In the format conversion step, the cloud server converts the combined result from the slot space to the coefficient space to obtain the conversion result;
[0041] In the homomorphic aggregation step, the cloud server performs polynomial operations in the coefficient space to perform matrix multiplication on the transformation result and the adjacency matrix ciphertext to obtain the result polynomial; during the matrix multiplication process, when the data length of the transformation result exceeds the dimension of a single polynomial, it is calculated in batches according to a preset block division rule and the block results are merged in the first few coefficients of the result polynomial.
[0042] The homomorphic activation step first extracts the valid data from the resulting polynomial using a shifter and converts the valid data into LWE format. Then, it is packaged into the plaintext space and evaluated according to the pre-generated lookup table polynomial to achieve nonlinear activation and bootstrapping, thus obtaining the activation result. The lookup table records the mapping relationship of the plaintext space after nonlinear activation.
[0043] The iterative output step is repeated until the homomorphic combination operation step is completed and the homomorphic activation step is completed, and the inference of all network layers in the neural network model is completed. The final activation result is returned to the client, and the client decrypts it to obtain the inference output result of the neural network model.
[0044] The aforementioned hardware-software co-operated graph neural network privacy inference acceleration device, wherein the homomorphic aggregation step includes:
[0045] Using the transformation result as feature data, when the length of the feature data is less than or equal to the dimension that a single polynomial can accommodate, matrix multiplication is performed by multiplying the polynomial corresponding to the ciphertext of the adjacency matrix with the polynomial of the feature data in the coefficient space. Specifically, this polynomial multiplication includes:
[0046] The feature data is assigned to the corresponding coefficients using the following formula: ; ;
[0047] In the formula The polynomial represents the number of features that a single polynomial can accommodate, and F represents the number of features that the graph data has.
[0048] Using row-major encoding, the feature V[i][j] in the i-th row and j-th column of the feature data is placed on the (i × F + j)-th coefficient of the polynomial v;
[0049] Reverse the t-th row A[t][i] of the adjacency matrix ciphertext A and encode it onto the coefficients of the polynomial a_t at intervals of F:
[0050] a_t[N - i × F mod N] = A[t][i]; The adjacency matrix ciphertext A is encoded row by row on a polynomial, so the entire adjacency matrix ciphertext A includes multiple polynomials. In a_t, t is the index of the polynomial, representing the encoding of the t-th row of the adjacency matrix ciphertext A;
[0051] Performing homomorphic polynomial multiplication a_t · v yields the resulting polynomial.
[0052] The aforementioned hardware-software co-operated graph neural network privacy inference acceleration device uses the transformation result as feature data. When the data length of the feature data is greater than the dimension N that a single polynomial can accommodate, the following steps are performed:
[0053] Generate a pair of block parameters ( , ): ; ;
[0054] In the formula, N is the dimension that a single polynomial can accommodate, and H is the number of nodes in the graph data;
[0055] The feature data V of size H×F and the ciphertext A of size H×H adjacency matrix are split into multiple sub-matrices; the feature data V is divided into The eigenvalue submatrix; the ciphertext A of this adjacency matrix is also divided into... Adjacency submatrices;
[0056] For each H_c×F_c eigenmatrix and its corresponding adjacent submatrix, perform homomorphic polynomial multiplication to obtain multiple sub-results. Combine all sub-results using homomorphic addition to obtain the result polynomial.
[0057] The present invention also proposes a client for the aforementioned hardware-software co-operated graph neural network privacy inference acceleration device.
[0058] The present invention also proposes an electronic device, including the aforementioned hardware and software co-operated graph neural network privacy inference acceleration device, which may be connected to an information display device for displaying the inference output result with user-set display parameters, attributes or through an artificial intelligence model.
[0059] The present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the hardware-software co-operated graph neural network privacy inference acceleration method.
[0060] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the hardware-software co-operated graph neural network privacy inference acceleration method.
[0061] As can be seen from the above solutions, the advantages of the present invention are:
[0062] 1. Improved inference efficiency: At 7nm process, the Uranus acceleration system achieves a performance improvement of 3.3×–4.2× compared to the state-of-the-art GNN accelerator PPGNN, and more than 9× compared to the general-purpose CKKS accelerator.
[0063] 2. Improved energy efficiency and resource utilization: Energy-Delay-Area Area (EDAP) is more than 119 times higher than PPGNN, the chip area is about half that of PPGNN, and power consumption is reduced by more than 40%, avoiding the problem of long-term idle large-capacity on-chip storage.
[0064] 3. Guaranteeing inference accuracy: By using the LUT activation method of the BFV scheme, the accuracy of ciphertext inference on the Cora, Citeseer and Pubmed datasets is only 0.2%–0.7% lower than that of plaintext inference, which is far better than the traditional CKKS polynomial fitting scheme.
[0065] 4. Wider range of applications: Supports multiple GNN structures (including weighted graphs and GNNs requiring regularization), making it easy to deploy in cloud inference scenarios.
[0066] This invention outperforms existing solutions in reducing hardware resource consumption, improving computational efficiency, and ensuring data privacy and model accuracy. Attached Figure Description
[0067] Figure 1 An accelerated architecture diagram for existing fully homomorphic encryption technologies;
[0068] Figure 2 This is a schematic diagram illustrating the performance bottleneck of privacy-preserving inference in graph neural networks.
[0069] Figure 3 This is a schematic diagram of the algorithm framework of the present invention;
[0070] Figure 4 This is a schematic diagram of the hardware acceleration device structure;
[0071] Figure 5 A schematic diagram of a specific embodiment of the assignment to multiple polynomials;
[0072] Figure 6 This is a flowchart of the method of the present invention;
[0073] Figure 7 This is a block diagram of the device of the present invention;
[0074] Figure 8 This is a schematic diagram of the structure of the first electronic device of the present invention;
[0075] Figure 9 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;
[0076] Figure 10 This is a schematic diagram of the structure of the second electronic device of the present invention.
[0077] Figure label:
[0078] A - First electronic device;
[0079] B-A hardware-software co-operational graph neural network privacy inference acceleration device;
[0080] C-Data acquisition equipment;
[0081] D-Information display device;
[0082] 1000 - Second electronic device;
[0083] Ⅰ-Computational Unit;
[0084] II-ROM;
[0085] III-RAM;
[0086] N-bus;
[0087] V-Interface;
[0088] VI - Input Unit;
[0089] VII - Output Unit;
[0090] VIII - Storage medium;
[0091] IX - Communication Unit. Detailed Implementation
[0092] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.
[0093] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0094] The processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).
[0095] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.
[0096] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.
[0097] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.
[0098] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0099] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.
[0100] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.
[0101] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.
[0102] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0103] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0104] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0105] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0106] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0107] The inventor has long been engaged in research on hardware and software collaborative acceleration of fully homomorphic encryption. For example... Figure 2 As shown, the analysis of existing solutions reveals that the performance bottleneck of privacy inference in graph neural networks mainly stems from the computation of linear layers with large parameters. Specifically, existing methods separate the bootstrapping process from the nonlinear mapping, resulting in excessively large homomorphic multiplication depths required for inference. This necessitates the use of extremely large parameters, forcing even linear layers with relatively small depths to operate under large parameters, leading to excessively high overall computational and storage overhead. On the other hand, while PPGNN reduces the parameter size by introducing programmable bootstrapping via TFHE, it sacrifices batch processing capabilities, causing a surge in computational complexity, thus making its bootstrapping the bottleneck for inference.
[0108] Through in-depth analysis and multiple rounds of experimentation, the inventors proposed a software framework based on a hybrid CKKS and BFV scheme: The CKKS scheme handles linear combinations and adjacency aggregations, achieving efficient matrix operations; the BFV scheme supports batch processing of nonlinear activations based on lookup tables (LUTs), merging bootstrapping and nonlinear computation into a single polynomial evaluation operation, thus completing inference with small parameters while maintaining high accuracy; and by optimizing data encoding and block partitioning strategies, the extraction and rotation overhead in the aggregation stage is reduced. Furthermore, this invention designs a matching reconfigurable hardware architecture, incorporating a homomorphic multifunctional computing unit (HMU) and a dedicated shifter, improving module reusability and hardware utilization.
[0109] These improvements enable a balance between parameter size, computational complexity, and hardware resources while ensuring model accuracy, significantly improving the inference efficiency of FHE-based GNNs.
[0110] In summary, to achieve the above-mentioned technical effects, the present invention proposes the following key technical points:
[0111] Key Point 1: A GNN inference framework based on a hybrid CKKS+BFV scheme. The CKKS scheme is used in the linear combination and adjacency aggregation stages, while the BFV scheme is employed in the nonlinear activation stage. Bootstrapping and nonlinear activation are fused into polynomial evaluation based on lookup tables (LUTs), supporting Single Instruction Multiple Data (SIMD) batch processing. Technical benefits: While maintaining accuracy, the multiplication depth required for inference is significantly reduced, allowing the overall inference process to be completed with smaller parameters, reducing computational load and storage overhead, and improving the computational efficiency of nonlinear layers.
[0112] Key Point 2: Optimized Encoding and Blocking Strategies. A coefficient-space-based aggregation and data rearrangement strategy is proposed, concentrating effective computation results in the first few terms of the ciphertext. When the number of nodes and features exceeds the polynomial dimension, block processing is employed, followed by efficient merging of results after aggregation. Technical Effects: This reduces the number of ciphertext rotation and extraction operations, lowers bandwidth and latency overhead during homomorphic matrix multiplication and aggregation, and improves computational efficiency and hardware utilization.
[0113] Key Point 3: Reconfigurable Acceleration Architecture with Hardware and Software Collaboration. An accelerator is designed to match the aforementioned algorithm framework, including a homomorphic multi-functional computing unit with high resource reuse, and a dedicated shifter for efficient conversion between RLWE and LWE ciphertexts. The computing cluster and storage module are connected via an on-chip network to achieve flexible data flow scheduling. Technical Benefits: This avoids the waste of large-capacity storage required for a small number of bootstrapping operations in general-purpose fully homomorphic encryption accelerators, achieving up to 2048-way computational parallelism. Under the same process technology and frequency, it achieves 3.3–4.2 times faster inference than PPGNN, while significantly reducing power consumption and area.
[0114] Key Point 4: Support for scalable graph models. Aggregation operations are implemented using standard matrix multiplication, supporting weighted graphs and GNN models with regularization terms, rather than being limited to additive aggregation. Technical effects: Enhances the system's adaptability to various GNN structures and expands the applicability of this technology in multiple application scenarios such as social networks, recommendation systems, and bioinformatics.
[0115] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.
[0116] This invention comprises two parts: an algorithm framework and a hardware acceleration device.
[0117] Algorithm framework ( Figure 3It includes three main operational phases: homomorphic combination, homomorphic aggregation, and homomorphic activation.
[0118] Hardware acceleration device ( Figure 4 It contains four computing clusters, each containing a data mapping region (for rotation, extraction, and analog-to-digital switching) and a computing region (for homomorphic addition, multiplication, number theory transformations, and radix conversion, etc.). The clusters are interconnected through an on-chip network and connected to high-bandwidth storage and on-chip cache.
[0119] The overall GNN inference process of this invention is as follows:
[0120] Step 1: Input Data Encryption. The client uses fully homomorphic encryption to encrypt the adjacency matrix and initial node features of the graph, and then uploads them to the cloud server. Graph data in real-world applications is very diverse in type and content. For example, in an online shopping scenario, each node in the graph data can represent a user, whose features can be modeled as the frequency of purchases of existing products. Edges can then be modeled as the user's social relationships, that is, using the idea that "what my friends like, I might also like" to optimize the recommendation process.
[0121] Step 2: Homomorphic combination operation. The server performs matrix multiplication on the ciphertext of node features and the plaintext of model weights to obtain the combination result; this stage adopts slot coding and block strategy to reduce rotation operations.
[0122] The node features and adjacency matrix (in the case of edges) of the graph data provided by the user need to be encrypted to avoid privacy leaks. The iteration of GNN is mainly divided into two steps: updating features (parameters trained by the model, also known as weights), aggregating the updated features with adjacent nodes through edges, and finally introducing nonlinearity through activation functions to improve the model's expressive power.
[0123] Slot coding is a common encoding method for homomorphic encryption algorithms (including BGV, BFV, CKKS, etc.) based on the RLWE problem. It encodes a large amount of data on many points of a polynomial, so that multiplying two polynomials is equivalent to multiplying on these values. It should be noted that if the encoding is on the coefficients of the polynomial, the multiplication of the polynomial will result in convolution on the data. Therefore, slot coding is the commonly used encoding method.
[0124] The goal of step 2 is to efficiently compute the encrypted text matrix multiplication between the node feature ciphertext (V) and the model weight plaintext (W). This invention employs slot coding in the CKKS encryption scheme for this step. Matrix multiplication under slot coding can be performed using the relatively mature BSGS (Baby-Step Giant-Step) method, which reduces the O(N) rotation operation to O(\sqrt{N}) complexity. It's important to note that changing the position of data within a slot is a highly complex operation in homomorphic encryption algorithms. On the other hand, the block partitioning strategy addresses situations where the data volume is too large for matrix multiplication to be completed within a single ciphertext. It divides the data into multiple smaller matrices, calculates the matrix multiplication using the BSGS method for each, and adds the corresponding positions, thereby reducing the number of homomorphic rotations.
[0125] Step 3: Format Conversion. The combined result is converted from slot space to coefficient space to prepare for subsequent aggregation operations. This conversion process can use the S2C algorithm, the basic principle of which is to perform a homomorphic NTT transformation.
[0126] Step 4: Homomorphic aggregation operation. Perform ciphertext-ciphertext matrix multiplication between the adjacency matrix and node features in the coefficient space to propagate graph structure information; utilize the convolution property of polynomial operations in the coefficient space to reduce rotation and extraction overhead; when the number of nodes and features exceeds the polynomial dimension, calculate in batches according to preset block rules and merge the block results in the first few coefficients of the resulting polynomial.
[0127] Since the overhead of ciphertext multiplication is much greater than that of ciphertext multiplication, this invention reduces the overhead of homomorphic aggregation operation, a bottleneck operation, by leveraging the properties of polynomial operations. Since the adjacency matrix (i.e., the edge case in the user's graph data) can be encrypted and sent by the user in advance according to a specified method, the adjacency matrix in the coefficient space can be used directly in step 4.
[0128] In essence, aggregation involves multiplying the user-provided adjacency matrix with the node feature matrix generated in the previous step to incorporate edge information into the reasoning process. Under homomorphic conditions, both are ciphertext. Since the plaintext of the CKKS algorithm is a polynomial, when we place the data onto the coefficients of the polynomial through the previous transformation, we can utilize the properties of polynomial multiplication. By carefully arranging the positions of the data in the two polynomials, we can directly calculate the result of multiplying and accumulating multiple positions with a simple polynomial multiplication. For cases with excessively long data, this invention can also distribute the data across the corresponding coefficients of multiple polynomials, and finally, through simple addition, accumulate the scattered results, thus addressing the key challenge of processing large-scale graph data with limited polynomial dimensions. Specifically:
[0129] This process involves meticulous data encoding, block partitioning strategies, and a deep utilization of the properties of homomorphic operations, particularly polynomial multiplication.
[0130] The aforementioned "excessive data length" is a precise technical concept relative to the FHE encryption parameters, rather than a simple hardware performance bottleneck. In this step, the key data are the node feature matrix V (from the previous step's "homomorphic combination" and converted to the coefficient space) and the adjacency matrix A (ciphertext). If the graph data has H nodes and F features, then the "data length" of the feature matrix V can be considered as H × F. In the CKKS scheme, data is encoded into a polynomial. This polynomial has a fixed maximum dimension, denoted as N (e.g., in the implementation of this invention, N = 32768). When the total data length H × F exceeds the dimension N that a single polynomial can accommodate, a "excessive data length" situation occurs, meaning homomorphic aggregation cannot be completed within a single polynomial.
[0131] Key data and coding formula (basic case of H × F ≤ N):
[0132] The goal of this invention is to achieve matrix multiplication (A · V) in the coefficient space through polynomial multiplication (a (the polynomial corresponding to the adjacency matrix A) · v (the polynomial corresponding to the node feature V)). Polynomial multiplication is essentially a "convolution" operation, while matrix multiplication is a "dot product" operation. This invention proposes a data rearrangement strategy (an optimized encoding strategy) that makes the result of convolution equivalent to matrix multiplication.
[0133] The process of assigning data to corresponding coefficients is as follows: ; . It is the t-th polynomial encoded by the adjacency matrix (adjacency matrix ciphertext A), where N is the degree of the polynomial, F is the number of features of each node, and A is the adjacency matrix. The first row, by traversing i and t, gives the correspondence between the coefficients of the multiple polynomials encoded by the adjacency matrix. V is the polynomial encoded by the node features, and V is the node feature matrix (feature data). The second line, by traversing i and j, gives the correspondence between the coefficients of the node feature matrix encoded into a polynomial.
[0134] 1. Assignment of node feature v (V[i][j]):
[0135] Using standard "row-major" encoding, the feature V[i][j] in the i-th row and j-th column is placed on the (i × F + j)-th coefficient of the polynomial v, i.e., v[i × F + j] = V[i][j].
[0136] 2. Allocation of adjacency matrix a (A[t][i]):
[0137] This is the key to the invention. To calculate the t-th row of the resulting matrix (i.e., the dot product of the t-th row of A and V), the t-th row A[t][i] of A needs to be reversed and encoded onto the coefficients of the polynomial a_t at an interval of F (matching the row spacing of V). This interval is F when arranging the data of the adjacent matrix on the coefficients of the polynomial. Arranging the data at this interval ensures that the calculated result is exactly the result of the corresponding matrix multiplication.
[0138] a_t[N - i × F mod N] = A[t][i]
[0139] It is important to note that in this invention, v and V represent different meanings. v represents a polynomial, and V represents a node feature matrix (feature data). Although the polynomial v is encoded by the matrix V, their data types are different. When performing the homomorphic polynomial multiplication a_t · v to obtain the resulting polynomial, due to the aforementioned ingenious arrangement, the calculated dot product result A[t][0]V[0][j] + A[t][1]V[1][j] + ... will automatically and centrally appear in the first F coefficients of the resulting polynomial. This greatly reduces the overhead of subsequent extraction of effective data.
[0140] Blocking strategy: Handling "data length too long" (H × F > N):
[0141] This invention does not employ more complex or expensive homomorphic operations, but instead uses efficient "block processing." This invention introduces a set of preset block rules, defining a pair of block parameters ( , ): ; .
[0142] This ensures that the data size H_c × F_c of any block is always less than or equal to the polynomial dimension N (H_c × F_c ≤ N).
[0143] 2. How to "assign to multiple polynomials":
[0144] The original feature matrix V of size H × F and the adjacency matrix A of size H × H are divided into multiple submatrices (blocks).
[0145] Feature data V is divided into Each block.
[0146] The adjacency matrix A is also divided into blocks accordingly.
[0147] Key point: Each H_c × F_c eigenmatrix and its corresponding adjacent submatrix are independently assigned to a new set of polynomials (ciphertext) according to the encoding formula in "point 2" above.
[0148] For example Figure 5 As shown:
[0149] The first block (A_{00} and V_0) is encoded as a_{00} and v_0.
[0150] The second block (A_{01} and V_1) is encoded as a_{01} and v_1.
[0151] 4. The implementation and technical difficulties of "simple addition"
[0152] "Calculate in batches and combine the results in the first few coefficients of the resulting polynomial."
[0153] 1. Calculate in batches:
[0154] The cloud server performs homomorphic polynomial multiplication on each set of block polynomials.
[0155] v'_{00} = a_{00} · v_{0} (Calculates a partial result for the first block)
[0156] v'_{01} = a_{01} · v_{1}$ (Calculates a partial result for the second block)
[0157] ...and so on
[0158] 2. Result of "simple addition":
[0159] Due to the collaborative design of the block division rules (H_c, F_c) and the encoding formula, the partial results generated by different block computations are naturally aligned in the position of the result polynomial.
[0160] Therefore, merging all block results does not require complex rotations or rearrangements, but only a simple homomorphic addition (Hadd).
[0161] v'_0 = v'_{00} + v'_{01} + ...
[0162] Technical difficulty and core value:
[0163] The brilliance of this technology lies in the fact that it guarantees:
[0164] 1. Additivity: It ensures that partial results from block computation can be correctly merged through a simple HAdd operation.
[0165] 2. Result concentration: It ensures that even after a "simple addition", the effective data of the final combined result is still concentrated in the first F_c coefficients of the result polynomial.
[0166] This "optimized encoding and block strategy" is one of the core elements of this invention. Through complex algorithm design (data arrangement and block rules), it decomposes a large-scale matrix multiplication problem into a series of parallel homomorphic operation tasks that only require polynomial multiplication and simple addition. This significantly reduces the number of ciphertext rotation and extraction operations, greatly reducing bandwidth and latency overhead, and is key to achieving high-performance hardware and software co-acceleration.
[0167] Step 5: Homomorphic Activation Operation. First, the valid data is extracted using a shifter and converted into vector-type ciphertext, such as LWE format. LWE stands for Learning with Errors. LWE format ciphertext consists of a vector and a scalar, unlike the ciphertext form based on the RLWE problem mentioned earlier, which uses two polynomials. Then, the vector-type ciphertext is packaged into the BFV plaintext space and evaluated according to the pre-generated lookup table polynomials to achieve nonlinear activation and bootstrapping, preparing for the next layer of inference.
[0168] BFV is a fully homomorphic encryption scheme based on the RLWE problem. Here, it refers to the plaintext space packed into the ciphertext of the BFV algorithm. Essentially, it realizes the conversion from the CKKS algorithm to the BFV algorithm through homomorphic decryption.
[0169] The polynomial output from step 4 does not contain every coefficient that is desired. However, since the features and adjacency matrices are arranged in the manner specified in this invention, it is possible to determine which positions of the coefficients are the desired results, and these coefficients are the valid parts.
[0170] Furthermore, this conversion is a mature RLWE->LWE conversion, which essentially involves extracting the coefficients of the polynomial in a certain order and with certain symbols, and then performing a homomorphic decryption operation using the CKKS key encrypted with BFV.
[0171] The evaluation here is to evaluate a polynomial. The lookup table maintains the one-to-one mapping relationship of the entire plaintext space after nonlinear activation. By taking values at different points, an interpolation polynomial can be determined. Therefore, after evaluating this polynomial, the activation result of the pre-output can be obtained.
[0172] Step 6: Iteration and Output. Repeat steps 2–5 until all network layers have been inferred. Return the final ciphertext result to the client, which will then decrypt it to obtain the inference output.
[0173] III. Hardware Device Examples
[0174] like Figure 4 As shown, the hardware acceleration device of the present invention includes:
[0175] HMU unit: It consists of multiple reconfigurable processing elements (PEs). Each PE contains modular addition and modular multiplication units. It can be configured to switch to support NTT, BConv, MA, MM and other operations to achieve resource reuse.
[0176] The Shifter unit is used to efficiently complete the extraction and conversion between RLWE and LWE. It adopts a cycle-by-cycle shift and symbol discrimination strategy, so that the average extraction delay is close to 1 cycle.
[0177] Automorphism unit: Used to support index mapping operations such as polynomial rotation.
[0178] On-chip caching and NoC: Enables high-speed data interaction and storage between the computing cluster and HBM.
[0179] The aforementioned hardware architecture is closely matched with the algorithm framework, ensuring parallelism while avoiding the waste of reserving large storage capacity for a small number of operations, as is common in traditional accelerators.
[0180] IV. Optional Implementation Methods
[0181] In specific application scenarios, only the CKKS scheme can be enabled to complete linear layer inference, which is suitable for tasks that do not require high accuracy but pursue extremely low latency.
[0182] For ultra-large-scale graph data with a number of nodes or features that is much greater than the multinomial dimension, parallel processing can be achieved by adding block-level layers.
[0183] For different activation functions (such as Sigmoid and Tanh), the corresponding BFV lookup table polynomials can be pre-calculated offline and replaced without changing the hardware structure.
[0184] When deployed on higher bandwidth storage systems (such as HBM3), the number of clusters can be increased to achieve higher parallelism.
[0185] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.
[0186] like Figure 7 As shown, this invention also proposes a hardware-software co-operated graph neural network privacy inference acceleration device, which includes:
[0187] In the input data encryption step, the client uses fully homomorphic encryption to encrypt the adjacency matrix and node features of the graph data, obtaining the ciphertext of the node features and the ciphertext of the adjacency matrix, and uploads them to the cloud server, which has a neural network model for performing inference tasks.
[0188] The homomorphic combination operation step involves the cloud server performing matrix multiplication on the ciphertext of the node's features and the plaintext of the weights of a specified layer in the neural network model to obtain the combination result.
[0189] In the format conversion step, the cloud server converts the combined result from the slot space to the coefficient space to obtain the conversion result;
[0190] In the homomorphic aggregation step, the cloud server performs polynomial operations in the coefficient space to perform matrix multiplication on the transformation result and the adjacency matrix ciphertext to obtain the result polynomial; during the matrix multiplication process, when the data length of the transformation result exceeds the dimension of a single polynomial, it is calculated in batches according to a preset block division rule and the block results are merged in the first few coefficients of the result polynomial.
[0191] The homomorphic activation step first extracts the valid data from the resulting polynomial using a shifter and converts the valid data into LWE format. Then, it is packaged into the plaintext space and evaluated according to the pre-generated lookup table polynomial to achieve nonlinear activation and bootstrapping, thus obtaining the activation result. The lookup table records the mapping relationship of the plaintext space after nonlinear activation.
[0192] The iterative output step is repeated until the homomorphic combination operation step is completed and the homomorphic activation step is completed, and the inference of all network layers in the neural network model is completed. The final activation result is returned to the client, and the client decrypts it to obtain the inference output result of the neural network model.
[0193] The aforementioned hardware-software co-operated graph neural network privacy inference acceleration device, wherein the homomorphic aggregation step includes:
[0194] Using the transformation result as feature data, when the length of the feature data is less than or equal to the dimension that a single polynomial can accommodate, matrix multiplication is performed by multiplying the polynomial corresponding to the ciphertext of the adjacency matrix with the polynomial of the feature data in the coefficient space. Specifically, this polynomial multiplication includes:
[0195] The feature data is assigned to the corresponding coefficients using the following formula: ; .
[0196] In the formula The polynomial represents the number of features that a single polynomial can accommodate, and F represents the number of features that the graph data has.
[0197] Using row-major encoding, the feature V[i][j] in the i-th row and j-th column of the feature data is placed on the (i × F + j)-th coefficient of the polynomial v;
[0198] Reverse the t-th row A[t][i] of the adjacency matrix ciphertext A and encode it onto the coefficients of the polynomial a_t at intervals of F:
[0199] a_t[N - i × F mod N] = A[t][i]; The adjacency matrix ciphertext A is encoded row by row on a polynomial, so the entire adjacency matrix ciphertext A includes multiple polynomials. In a_t, t is the index of the polynomial, representing the encoding of the t-th row of the adjacency matrix ciphertext A;
[0200] Performing homomorphic polynomial multiplication a_t · v yields the resulting polynomial.
[0201] The aforementioned hardware-software co-operated graph neural network privacy inference acceleration device uses the transformation result as feature data. When the data length of the feature data is greater than the dimension N that a single polynomial can accommodate, the following steps are performed:
[0202] Generate a pair of block parameters ( , ): ; .
[0203] In the formula, N is the dimension that a single polynomial can accommodate, and H is the number of nodes in the graph data;
[0204] The feature data V of size H×F and the ciphertext A of size H×H adjacency matrix are split into multiple sub-matrices; the feature data V is divided into The eigenvalue submatrix; the ciphertext A of this adjacency matrix is also divided into... Adjacency submatrices;
[0205] For each H_c×F_c eigenmatrix and its corresponding adjacent submatrix, perform homomorphic polynomial multiplication to obtain multiple sub-results. Combine all sub-results using homomorphic addition to obtain the result polynomial.
[0206] The present invention also proposes a client for the aforementioned hardware-software co-operated graph neural network privacy inference acceleration device.
[0207] like Figure 8 As shown, in another embodiment of the present invention, a first electronic device A is also proposed, including the aforementioned hardware and software co-operated graph neural network privacy inference acceleration device B.
[0208] like Figure 9As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D via wired or wireless information transmission schemes. The data acquisition device C is used to collect and acquire graph data, which can be user application privacy graph data, such as shopping privacy graph data containing historical consumption records, browsing records, and customer service communication records from a shopping platform. The neural network model is a recommendation model. The information display device D is used to display the recommendation results obtained by the present invention, such as recommended product categories, recommended product models, and recommended product links. This ensures that user privacy shopping data will not be leaked or misused.
[0209] The information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data. This information display mechanism can be manually preset, for example, visualizing the data output by the first electronic device A. It can present the user with specified key information, such as inference results or activation results, based on user-defined display parameters and / or attributes. Display parameters could be, for example, the data range to be displayed, and display attributes could be, for example, the font, color, or whether scrolling is enabled. This allows the user to understand this information more promptly without having to access secondary pages or scroll through pages, saving user effort. Alternatively, this information display mechanism can be an artificial intelligence (AI) display model, which can learn the user's key information interests based on previous usage habits, such as viewing time, number of clicks, and number of edits, and then automatically present the user with rich and necessary key information.
[0210] The present invention also provides a computer program product, which includes a computer program that can be stored on a readable storage medium. When the computer program is executed by a processor, the computer is able to execute the hardware-software co-operated graph neural network privacy inference acceleration method provided by the above methods.
[0211] In another embodiment, the present invention also proposes a storage medium VIII for storing a computer program that executes the hardware-software co-operated graph neural network privacy inference acceleration method. It should be understood that the storage medium in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).
[0212] Figure 10 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present invention is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.
[0213] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from storage medium VIII into random access memory (RAM) III. The RAM III may also store various programs and data required for the operation of the device 1000. The computing unit I, ROM II, and RAM III are interconnected via bus IV. An input / output (I / O) interface V is also connected to bus IV.
[0214] Multiple components in the second electronic device 1000 are connected to I / O interface V, including: input unit VI, such as a keyboard, mouse, etc.; output unit VII, such as various types of displays, speakers, etc.; storage medium VIII, such as a disk, optical disk, etc.; and communication unit IX, such as a network card, modem, wireless transceiver, etc. Communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0215] The computing unit I can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S6. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to perform methods by any other suitable means (e.g., by means of firmware).
[0216] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.
Claims
1. A hardware-software collaborative graph neural network privacy inference acceleration method, characterized in that, The method comprises the following steps: An input data encryption step, in which a client encrypts an adjacency matrix and node features of graph data using full homomorphic encryption to obtain node feature ciphertext and adjacency matrix ciphertext and uploads them to a cloud server having a neural network model for performing an inference task; A homomorphic combination operation step, in which the cloud server performs matrix multiplication on the node feature ciphertext and weight plaintext of a specified layer of the neural network model to obtain a combination result; A format conversion step, in which the cloud server converts the combination result from slot space to coefficient space to obtain a conversion result; A homomorphic aggregation step, in which the cloud server performs polynomial operation in the coefficient space to perform matrix multiplication on the conversion result and the adjacency matrix ciphertext to obtain a result polynomial; In the matrix multiplication, when the data length of the conversion result exceeds the dimension of a single polynomial, the matrix multiplication is calculated in batches according to a preset blocking rule and the batch results are combined in the first plurality of coefficients of the result polynomial; A homomorphic activation step, in which effective data in the result polynomial is extracted by a shifter and converted into LWE format, and then packed into plaintext space, evaluated according to a pre-generated lookup table polynomial to realize nonlinear activation and bootstrap, and an activation result is obtained; The lookup table records the mapping relationship after nonlinear activation of the plaintext space; An iteration output step, in which the homomorphic combination operation step to the homomorphic activation step are repeated until the inference of all network layers in the neural network model is completed, and the final activation result is returned to the client, and the inference output result of the neural network model is obtained by decryption of the client.
2. The hardware-software co-designed graph neural network privacy inference acceleration method of claim 1, wherein, The homomorphic aggregation step comprises: When the data length of the feature data is less than or equal to the dimension that can be accommodated by a single polynomial, the matrix multiplication is realized by multiplying the polynomial corresponding to the adjacency matrix ciphertext and the polynomial of the feature data in the coefficient space, and the polynomial multiplication specifically comprises: The data of the feature data is distributed to the corresponding coefficients by the following formula: wherein represents a polynomial, N is the dimension that a single polynomial can accommodate, and F is the number of features that the graph data has; The feature V[i][j] in the i-th row and j-th column of the feature data is placed in the (i×F+j)th coefficient of the polynomial v by using row-major order encoding; The t-th row A[t][i] of the adjacency matrix ciphertext A is coded in reverse order and at intervals of F to the coefficients of the polynomial a_t: a_t[N-i×F mod N]=A[t][i]; the adjacency matrix ciphertext A is coded on the polynomial by row, so the entire adjacency matrix ciphertext A comprises a plurality of polynomials, t in a_t is the subscript of the polynomial, representing the coding of the data in the t-th row of the adjacency matrix ciphertext A; The homomorphic polynomial multiplication a_t·v is performed to obtain the result polynomial.
3. The hardware-software co-designed graph neural network privacy inference acceleration method of claim 2, wherein, When the data length of the feature data is greater than the dimension N that can be accommodated by a single polynomial, the following steps are performed: Generate a pair of block parameters ( , ): In the formula, N is the dimension that can be accommodated by a single polynomial, and H is the number of nodes of the graph data; The feature data V of size HxF and the adjacency matrix ciphertext A of size HxH are cut into sub-matrices; the feature data V is divided into feature sub-matrices; The adjacency matrix ciphertext A is also divided into adjacency sub-matrices; Each H_c x F_c feature sub-matrix and the corresponding adjacency sub-matrix perform homomorphic polynomial multiplication to obtain a plurality of sub-results, and combine all the sub-results by homomorphic addition to obtain a result polynomial.
4. A hardware-software co-designed graph neural network privacy inference acceleration apparatus, characterized in that, It comprises: An input data encryption step, in which a client encrypts the adjacency matrix and node features of graph data using fully homomorphic encryption to obtain node feature ciphertext and adjacency matrix ciphertext and uploads them to a cloud server, which has a neural network model for performing an inference task; A homomorphic combination operation step, in which the cloud server performs matrix multiplication on the node feature ciphertext and the weight plaintext of a specified layer in the neural network model to obtain a combination result; A format conversion step, in which the cloud server converts the combination result from slot space to coefficient space to obtain a conversion result; A homomorphic aggregation step, in which the cloud server performs polynomial operations in the coefficient space to perform matrix multiplication on the conversion result and the adjacency matrix ciphertext to obtain a result polynomial; Wherein, when the data length of the conversion result exceeds the dimension of a single polynomial during the matrix multiplication, the data is calculated in batches according to a preset blocking rule and the batch results are combined in the first plurality of coefficients of the result polynomial; A homomorphic activation step, in which the effective data in the result polynomial is first extracted by a shifter and converted into LWE format, then packed into plaintext space, evaluated according to a pre-generated lookup table polynomial, and nonlinear activation and bootstrap are realized to obtain an activation result; Wherein, the lookup table records the mapping relationship after nonlinear activation of the plaintext space; An iteration output step, in which the homomorphic combination operation step to the homomorphic activation step are repeated until the inference of all network layers in the neural network model is completed, and the final activation result is returned to the client, and the inference output result of the neural network model is obtained by the client after decryption.
5. The hardware-software co-designed graph neural network privacy inference acceleration device of claim 4, wherein, The homomorphic aggregation step comprises: When the data length of the feature data is less than or equal to the dimension that can be accommodated by a single polynomial, the matrix multiplication is realized by multiplying the polynomial corresponding to the adjacency matrix ciphertext and the polynomial of the feature data in the coefficient space, and the polynomial multiplication specifically comprises: The data of the feature data is distributed to the corresponding coefficients by the following formula: wherein represents a polynomial, N is the dimension that a single polynomial can accommodate, and F is the number of features that the graph data has; The feature V[i][j] in the i-th row and j-th column of the feature data is placed in the (i x F + j)-th coefficient of the polynomial v by using row-major order encoding; The t-th row A[t][i] of the adjacency matrix ciphertext A is coded in reverse order and at intervals of F to the coefficients of the polynomial a_t: a_t[N - i x F mod N] = A[t][i]; The adjacency matrix ciphertext A is coded on the polynomial by row, so the entire adjacency matrix ciphertext A includes a plurality of polynomials, and t in a_t is the subscript of the polynomial, representing the coding of the data in the t-th row of the adjacency matrix ciphertext A; The homomorphic polynomial multiplication a_t · v is performed to obtain the result polynomial.
6. The hardware-software co-designed graph neural network privacy inference acceleration device of claim 5, wherein, When the data length of the feature data is greater than the dimension N that can be accommodated by a single polynomial, the following steps are performed: Generate a pair of block parameters ( , ): Wherein N is the dimension of a single polynomial can accommodate, H is the number of nodes of the graph data; The feature data V of size HxF and the adjacency matrix ciphertext A of size HxH are cut into sub-matrices; the feature data V is divided into feature sub-matrices; The adjacency matrix ciphertext A is also divided into adjacency sub-matrices; Each H_c x F_c feature sub-matrix and the corresponding adjacency sub-matrix perform homomorphic polynomial multiplication to obtain a plurality of sub-results, and all sub-results are combined by homomorphic addition to obtain a result polynomial.
7. A client for any of the hardware and software collaborative graph neural network privacy inference acceleration devices described in claims 4-6.
8. An electronic device, comprising: An electronic device comprising any of the hardware and software collaborative graph neural network privacy inference acceleration devices described in claims 4-6, or connected with an information display device for displaying the inference output result with user-set display parameters, attributes, or through an artificial intelligence model.
9. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the hardware and software collaborative graph neural network privacy inference acceleration method of any of claims 1-3.
10. A computer program product comprising a computer program, characterized in that, The computer program which, when executed by a processor, implements the steps of the hardware and software collaborative graph neural network privacy inference acceleration method of any of claims 1-3.