A spatio-temporal graph convolution network privacy inference method based on homomorphic encryption

Abstract

This invention discloses a privacy-preserving inference method based on homomorphic encryption using a spatiotemporal graph convolutional network (SPCRN). The method includes: a server receiving ciphertext data uploaded by a client; encrypting the original time-series data using the client's public key; performing ciphertext inference on the ciphertext data using an optimized SPCRN; during ciphertext inference, selective ciphertext rotation is performed on the ciphertext data using spatiotemporal convolutional layers in the optimized SPCRN, and activation processing is applied to the rotated features using activation layers in the optimized SPCRN to obtain an encrypted inference result; and the encrypted inference result is returned to the client, allowing the client to decrypt the ciphertext data using its private key to obtain a plaintext prediction result. By performing selective ciphertext rotation and targeted activation during ciphertext inference, redundant ciphertext computation is reduced while ensuring the effectiveness of spatiotemporal feature extraction, thus improving the efficiency of ciphertext inference.

Description

Technical Field

[0001] This invention relates to the field of machine learning, and more particularly to a privacy-preserving inference method for spatiotemporal graph convolutional networks based on homomorphic encryption. Background Technology

[0002] In existing research on privacy-preserving inference using graph convolutional networks, some solutions improve overall computational efficiency by refining data packaging mechanisms and ciphertext encoding strategies.

[0003] A key technical problem with existing technologies is that when dealing with complex and dynamically changing spatiotemporal graph networks, the ciphertext rotation operations involved in homomorphic encryption and the resulting key switching overhead remain enormous, becoming a core bottleneck severely restricting the real-time inference performance of the model. In other words, existing methods struggle to achieve a good balance between data privacy and security and ciphertext computation efficiency.

[0004] Therefore, how to reduce redundant ciphertext calculations and improve ciphertext reasoning efficiency is a technical problem that still needs to be solved. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a privacy-preserving inference method for spatiotemporal graph convolutional networks based on homomorphic encryption. The technical problem to be solved is how to reduce redundant ciphertext computation and improve ciphertext inference efficiency.

[0006] A first aspect of this invention provides a privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption, applied to a server, comprising:

[0007] Receive encrypted data uploaded by the client; the encrypted data is obtained by encrypting the original time-series data based on the client's public key;

[0008] Ciphertext inference is performed on the ciphertext data based on an optimized spatiotemporal graph convolutional network. During ciphertext inference, selective ciphertext rotation is performed on the ciphertext data based on the spatiotemporal convolutional layers in the optimized spatiotemporal graph convolutional network, and the rotated features are activated based on the activation layers in the optimized spatiotemporal graph convolutional network to obtain the encrypted inference result.

[0009] The encrypted inference result is returned to the client, so that the client can decrypt the ciphertext data based on the private key to obtain the plaintext prediction result.

[0010] Furthermore, when the spatiotemporal convolutional layers of the spatiotemporal graph convolutional network perform selective ciphertext rotation, the following steps are included:

[0011] Receive a set of rotation keys generated by the client; the set of rotation keys corresponds to different time offsets;

[0012] The optimized spatiotemporal graph convolutional network is trained to obtain the learnable rotation mask and the corresponding effective time offset.

[0013] During the ciphertext inference phase, a corresponding rotation key is selected from the rotation key set based on the effective time offset, and selective ciphertext rotation is performed on the ciphertext data.

[0014] Further, the rotating mask is trained, including the following steps:

[0015] Features at different time steps are input into the spatiotemporal convolutional layer, and the temporal convolution output is calculated based on the convolution kernel and ciphertext rotation operation of the spatiotemporal convolutional layer.

[0016] During model training, the parameters of the rotation mask and convolution kernel are optimized through backpropagation;

[0017] Based on the optimized rotation mask, determine and retain the effective time offsets that contribute to the temporal convolution output and meet the preset conditions.

[0018] Furthermore, the activation layer of the optimized spatiotemporal graph convolutional network uses a second-order orthogonal polynomial as the activation function to perform activation processing on the features after ciphertext rotation.

[0019] Furthermore, before performing activation processing on the features after ciphertext rotation, the following steps are included:

[0020] An exponential moving average (EMA) input calibration mechanism is introduced into the activation layer of the spatiotemporal graph convolutional network to normalize the input data of the activation layer to the effective fitting range of the polynomial.

[0021] Furthermore, the graph convolutional layers in the optimized spatiotemporal graph convolutional network are configured with learnable gating variables at the node level.

[0022] Activation processing is performed on the features after ciphertext rotation, including the following steps:

[0023] Based on the gating variables, determine and control whether nodes in the graph convolutional layer need to be activated using polynomials.

[0024] Furthermore, the optimized spatiotemporal graph convolutional network is obtained by optimizing the original spatiotemporal convolutional network.

[0025] Furthermore, the method also includes training the optimized spatiotemporal graph convolutional network using knowledge distillation.

[0026] The method for training the optimized spatiotemporal graph convolutional network using knowledge distillation includes the following steps:

[0027] The original spatiotemporal graph convolutional network is used as the teacher model, and the optimized spatiotemporal graph convolutional network is used as the student model. Knowledge distillation training is performed on multiple student models based on the teacher model.

[0028] During the restorative optimization phase of distillation training, the learnable rotation mask and learnable gating variables are frozen.

[0029] A second aspect of this invention provides a privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption, applied to a client, comprising:

[0030] Receive raw time-series data input by the user, encrypt the raw time-series data using a public key, and generate ciphertext data;

[0031] Generate the set of rotation keys required for homomorphic encryption;

[0032] The encrypted data and the set of rotation keys are sent to the server so that the server can perform encrypted inference and selective encrypted rotation operations on the encrypted data based on an optimized spatiotemporal graph convolutional network.

[0033] Receive the encrypted reasoning result returned by the server, which is obtained by the server through ciphertext reasoning;

[0034] Using the private key corresponding to the public key, the encrypted reasoning result is decrypted to obtain the plaintext prediction result, which is then output.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The server of this invention first receives ciphertext data encrypted by the client using a public key, and then performs ciphertext inference on the ciphertext data through an optimized spatiotemporal graph convolutional network. During the inference process, the spatiotemporal convolutional layer of the network performs selective ciphertext rotation on the ciphertext data, and the activation layer performs activation processing on the rotated features to obtain the encrypted inference result. Finally, the encrypted inference result is returned to the client, so that the client can use the private key to decrypt and obtain the plaintext prediction result, thus realizing privacy inference in the ciphertext state.

[0037] By introducing a learnable rotation mechanism into temporal convolution and activating the rotation features, unnecessary nonlinear computations are reduced. Compared with the implementation that directly retains all time offsets and all activation positions, the computational overhead and latency in the homomorphic encryption process can be significantly reduced. While ensuring the spatiotemporal feature extraction effect, redundant ciphertext computation is reduced and the efficiency of ciphertext inference is improved.

[0038] 2. This invention uses a second-order Chebyshev polynomial with an input calibration mechanism as the activation function to replace the traditional nonlinear activation function ReLU. It also constrains the coefficients and expression of the polynomial through coefficient quantization constraints to adapt to the ciphertext computation environment. This can better suppress numerical drift and noise accumulation, and is therefore more conducive to maintaining computational stability in the homomorphic encryption inference process.

[0039] 3. This invention introduces a restorative knowledge distillation method for low-overhead structures, which performs targeted optimization of the student network under the premise of fixed rotation mask structure and gating layout. Compared with conventional knowledge distillation or simply supervised training, this method can more effectively restore the model's discriminative ability without destroying the model's low-overhead structure, thereby achieving higher plaintext inference accuracy while ensuring lower homomorphic encryption execution cost.

[0040] This invention solves the technical problem of how to reduce redundant ciphertext calculations and improve ciphertext reasoning efficiency. Attached Figure Description

[0041] The accompanying drawings, which form part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0042] The invention will be more clearly understood with reference to the accompanying drawings and the following detailed description, wherein:

[0043] Figure 1 A flowchart illustrating a privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption, provided in this embodiment of the invention. Figure 1 ;

[0044] Figure 2 A flowchart illustrating a privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption, provided in this embodiment of the invention. Figure 2 ;

[0045] Figure 3 An interactive schematic diagram illustrating a privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption, provided in an embodiment of the present invention;

[0046] Figure 4 This is a diagram illustrating an optimized spatiotemporal graph convolutional network architecture provided in an embodiment of the present invention. Detailed Implementation

[0047] The technical methods in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0048] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of the present invention are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0049] It should also be understood that in the embodiments of the present invention, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0050] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more unless explicitly defined or given contrary instructions in the context.

[0051] The Rectified Linear Unit (ReLU) is a commonly used nonlinear activation function in artificial neural networks.

[0052] Galois is a special auxiliary key in homomorphic encryption used to perform data rotation and permutation operations in the ciphertext state.

[0053] A fully homomorphic encryption scheme (Cheon-Kim-Kim-Song, CKKS) is a cryptographic technique that allows arbitrary computations to be performed directly on encrypted data. An encryption scheme is called fully homomorphic if it supports an unlimited number of addition and multiplication operations on the ciphertext, and the ciphertext after decryption is equivalent to the result of performing the same operations on the original plaintext.

[0054] Spatio-Temporal Graph Convolutional Network (T-GCN) is a deep learning model specifically designed to process data that simultaneously possesses graph structure and temporal dynamics.

[0055] Furthermore, the term "and / or" in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this invention generally indicates that the preceding and following related objects have an "or" relationship.

[0056] It should also be understood that the description of the various embodiments in this invention emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0057] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0058] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0059] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0060] In the existing technology, CKKS includes A complex plaintext message vector First by scaling factor Magnify and encode the packaged polynomial ring The encrypted text is in the form of... It can directly perform addition (CAdd), scalar multiplication (PMult), ciphertext multiplication (CMult), and ciphertext rotation (Rotation) in the ciphertext field.

[0061] However, ciphertext computation suffers from two major performance bottlenecks. The first is ciphertext multiplication (CMult). Multiplying two ciphertexts causes the scale factor to expand dramatically. It must be divided by through a rescaling operation. Scaling back to the original scale, the rescaling operation in homomorphic encryption is shown in equation (1):

[0062]

[0063] Each rescaling operation consumes one homomorphic multiplication level. Increasing the multiplication depth forces the system to use a larger polynomial order during initialization. Sum of coefficients modulus This leads to an exponential increase in overall computational complexity.

[0064] Secondly, there is ciphertext rotation. This is used to achieve cyclic shifting of elements within a vector. CKKS requires calling automorphism transformation. This operation not only requires performing a large ciphertext matrix multiplication, but also relies on a pre-generated set of Galois keys. This greatly increases the memory usage and communication overhead of both the client and server.

[0065] In existing technologies, in skeleton motion recognition tasks, data is represented as a spatiotemporal tensor composed of three-dimensional spatial coordinates and time frames. The ST-GCN backbone network extracts features by stacking spatiotemporal units. Its spatial graph convolutional layers are used to aggregate features of adjacent keypoints within the same frame, which can be abstracted as formula (2):

[0066]

[0067] in To incorporate the adjacency matrix with self-loops. In CKKS, this only involves efficient plaintext-ciphertext multiplication (PMult) and addition. Its temporal convolutional layers are used to extract action features across frames, within a time window. The inner sliding is as shown in formula (3):

[0068]

[0069] In the homomorphic encryption domain, this data step offset in the time dimension The aforementioned expensive ciphertext rotation operation is necessary. This is achieved by increasing the number of rotations and Galois key pairs required as the convolution kernel size and time step offset increase.

[0070] In addition, traditional ST-GCN is widely used. As an activation function: However, such discontinuous, non-polynomial functions exist in polynomial rings. It cannot be calculated directly.

[0071] In order for graph neural networks to run legally in the CKKS environment, HE-friendly low-order polynomials must be used to approximate and replace ReLU, as shown in Equation (4):

[0072]

[0073] Evaluation of quadratic terms At this time, a ciphertext-ciphertext multiplication needs to be performed, which inevitably consumes 1 level of homomorphic multiplication depth.

[0074] If a full polynomial replacement is performed on all nodes in a deep network, and the network retains... If there are multiple nonlinear layers, the depth of homomorphic multiplication will reach [a certain value]. This not only leads to excessively high inference latency, rendering it useless, but the superposition of deep polynomials also easily causes feature distribution shifts and gradient explosion, resulting in a severe degradation of the model's classification accuracy. Therefore, theoretically modeling and dynamically identifying the nonlinear redundancy of the network and pruning polynomial layers to reduce multiplication depth are the core prerequisites for achieving efficient CKKS secure inference.

[0075] It is evident that, in addition to the increased memory usage and communication overhead caused by the increased ciphertext rotation operations between the client and server, existing technologies also suffer from problems such as linear growth in the number of Galois keys, feature distribution shift, and gradient explosion.

[0076] To address the aforementioned technical problems, the present invention is described in detail through several embodiments.

[0077] Example 1

[0078] Example 1 illustrates a privacy-preserving inference method for spatiotemporal graph convolutional networks based on homomorphic encryption, provided by this invention, applied to the server side. Figure 1 A flowchart illustrating a privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption, provided in this embodiment of the invention. Figure 1 ,like Figure 1 As shown, the method includes steps S1 to S3:

[0079] S1. Receive ciphertext data uploaded by the client; the ciphertext data is obtained by encrypting the original time-series data based on the client's public key.

[0080] In this embodiment of the invention, the original time-series data is the raw spatiotemporal sequence information collected by the client without encryption.

[0081] S2. Ciphertext inference is performed on the ciphertext data based on the optimized spatiotemporal graph convolutional network. During the ciphertext inference, the spatiotemporal convolutional layer in the optimized spatiotemporal graph convolutional network performs selective ciphertext rotation on the ciphertext data, and the activation layer in the optimized spatiotemporal graph convolutional network activates the rotated features to obtain the encrypted inference result.

[0082] S3. Return the encrypted reasoning result to the client so that the client can decrypt the ciphertext data based on the private key and obtain the plaintext prediction result.

[0083] In this embodiment of the invention, when the spatiotemporal convolutional layer of the spatiotemporal graph convolutional network performs selective ciphertext rotation in step S2, it includes steps S21 to S23:

[0084] S21. Receive the set of rotation keys generated by the client; the set of rotation keys corresponds to different time offsets.

[0085] It is understandable that the ciphertext rotation operation is only used to align features corresponding to different time offsets in time sequence to facilitate subsequent convolutional feature extraction. This process only adjusts the position of the ciphertext and does not change the encryption structure of the ciphertext itself. Therefore, it can be directly decrypted by the client after inference is completed, and the rotation operation will not have an adverse effect on the decryption process and decryption result.

[0086] S22. Train the optimized spatiotemporal graph convolutional network to obtain a learnable rotation mask and the corresponding effective time offset.

[0087] In this embodiment of the invention, a tapmask rotation mode is introduced, using a learnable binary mask `tap_aux` to filter and control the time offset to be retained. (Mask) Each element in the array corresponds to a rotation offset. By training the mask, it is possible to adaptively retain the rotation offsets that contribute significantly to the inference results, thus achieving selective rotation of the ciphertext.

[0088] S23. During the ciphertext reasoning stage, select the corresponding rotation key from the rotation key set according to the effective time offset, and perform selective ciphertext rotation on the ciphertext data.

[0089] Understandably, during the model training phase, the time offset effective for inference is learned and determined through the rotation mask. This offset corresponds one-to-one with the keys in the pre-generated rotation key set on the client. Therefore, during ciphertext inference, the matching rotation key can be directly selected based on the effective offset, and rotation is only performed on necessary sequential features, thereby achieving accurate and efficient selective ciphertext rotation.

[0090] In this embodiment of the invention, training the rotating mask in step S22 includes steps S221 to S223:

[0091] S221. Input features with different time steps into the spatiotemporal convolutional layer, and calculate the temporal convolution output based on the convolution kernel and ciphertext rotation operation of the spatiotemporal convolutional layer.

[0092] In this embodiment of the invention, the calculation formula for the rotation operation is formula (5):

[0093]

[0094] in, It is a rotated convolution kernel. The input data at time step Features at the location. By introducing the STE Binary activation function, the mask... It is converted into a binary form (0 or 1) to determine whether to perform a rotation calculation in each rotation operation.

[0095] S222. During model training, the parameters of the rotation mask and convolution kernel are optimized through backpropagation.

[0096] In this embodiment of the invention, the initial mask value of the model can be set to all 1s or all 0s, which is specifically controlled by the parameter rot_init_keep_all that controls the mask initialization strategy; during the training process, the mask parameters are optimized through the backpropagation algorithm, thereby adaptively determining the rotation offset that contributes to the extraction of convolutional features.

[0097] S223. Based on the optimized rotation mask, determine and retain the effective time offset that contributes to the temporal convolution output and meets the preset conditions.

[0098] In this embodiment of the invention, after the model training is completed, a rotation execution plan is generated based on the optimized rotation mask to determine the actual rotation offset used by each convolutional layer in encrypted inference.

[0099] The rotation plan is obtained through the following formula (6):

[0100]

[0101] The aforementioned rotation scheme is used to guide the generation and selection of Galois keys during homomorphic encryption inference, thereby further reducing the size of the key set and lowering inference latency.

[0102] In this embodiment of the invention, the activation layer of the optimized spatiotemporal graph convolutional network in step S2 uses a second-order orthogonal polynomial as the activation function to perform activation processing on the features after ciphertext rotation.

[0103] For example, the present invention uses a second-order Chebyshev polynomial as the activation function, thereby replacing the traditional ReLU activation function.

[0104] Understandably, in homomorphic encryption environments, the traditional ReLU activation function suffers from poor numerical stability, easily leading to decreased encryption precision or even numerical overflow in deep neural networks. In contrast, the Chebyshev polynomial possesses excellent numerical stability, effectively avoiding the precision loss and numerical overflow issues that occur with ReLU during homomorphic encryption inference, making it more suitable for ciphertext domain computation.

[0105] In this embodiment of the invention, the second-order Chebyshev polynomial is as shown in formula (7):

[0106]

[0107] The activation calculation for this polynomial is given by formula (8):

[0108]

[0109] in, and For trainable parameters, This is the result after normalizing the input data. This method allows the activation function to efficiently express complex nonlinear characteristic relationships and avoids the numerical instability and excessive computational cost problems of the traditional ReLU activation function in homomorphic encryption inference scenarios.

[0110] In this embodiment of the invention, before performing activation processing on the features after ciphertext rotation, step S224 is included:

[0111] S224. Introduce the exponential moving average (EMA) input calibration mechanism in the activation layer of the spatiotemporal graph convolutional network to normalize the input data of the activation layer to the effective fitting range of the polynomial.

[0112] In this embodiment of the invention, EMA calibration is introduced to keep the input data stable during the polynomial activation process, so that the polynomial activation function can achieve consistent results during both training and inference.

[0113] For each input node EMA calibration normalizes the data by calculating the mean and standard deviation of the nodes, thereby avoiding the impact of fluctuations in the distribution of input data on the model inference results.

[0114] The mean of the nodes and standard deviation The update formula is shown in formula (9):

[0115]

[0116] In this embodiment of the invention, the graph convolutional layers in the optimized spatiotemporal graph convolutional network are configured with learnable gating variables at the node level. Step S2 performs activation processing on the features after ciphertext rotation, including the following step S24:

[0117] S24. Determine and control whether nodes in the graph convolutional layer need to be activated using polynomials based on gating variables.

[0118] Understandably, in encrypted reasoning scenarios, the complexity of nonlinear computation is significantly higher than that of linear computation. Therefore, reducing the number of redundant nonlinear computations is the key to improving reasoning efficiency.

[0119] In this embodiment of the invention, by introducing a structured gating mechanism, each layer of the network can selectively perform nonlinear calculations or directly perform linear transfers, thereby avoiding redundant nonlinear operations.

[0120] By training the gating variables Whether adaptive control uses a multinomial activation function, where the gated variable... The value is used to determine whether non-linear activation is performed on each node in each layer. When, a multinomial activation is applied to the features of the current node; when In this case, the nonlinear activation step is skipped using a linear pass-through method. This process employs a pass-through estimator (STE) for gradient approximation, thereby achieving efficient training of the gated variables.

[0121] The specific gating calculation formula is shown in formula (10):

[0122]

[0123] in, It can be ReLU activation or polynomial activation, when When indicates that nonlinear calculations are retained, The time indicates that linear transfer is performed.

[0124] In this embodiment of the invention, to ensure the consistency of computation at each layer and avoid differences in nonlinear computation between different nodes, a synchronization constraint mechanism is introduced. This constraint enables gated variables within the same network layer to... Maintaining consistency avoids multiple rescale alignment operations, further reducing inference latency and preventing additional computational overhead caused by unbalanced computation paths.

[0125] The formula for the synchronization constraint is formula (11):

[0126]

[0127] In this embodiment of the invention, the optimized spatiotemporal graph convolutional network is obtained by optimizing the original spatiotemporal convolutional network.

[0128] In this embodiment of the invention, the privacy inference method for spatiotemporal graph convolutional networks further includes step S4:

[0129] S4. The optimized spatiotemporal graph convolutional network is trained using a knowledge distillation method.

[0130] In this embodiment of the invention, step S4 includes the following steps S41 and S42:

[0131] S41. Use the original spatiotemporal graph convolutional network as the teacher model and the optimized spatiotemporal graph convolutional network as the student model. Train multiple student models with knowledge distillation based on the teacher model.

[0132] Understandably, after completing the rotation mask optimization and network structure linearization, although the student model has achieved a lower homomorphic encryption execution overhead, its network structure, activation function form, and intermediate feature distribution have deviated significantly from the original baseline model. If only conventional supervised learning methods are used for training, it is often difficult to effectively restore the model's discriminative ability and recognition accuracy.

[0133] For example, this method uses the original baseline ReLU model as the teacher network and the model after rotation constraint and structural linearization as the student network for initialization.

[0134] S42. During the restorative optimization phase of distillation training, freeze the learnable rotation mask and learnable gating variables.

[0135] In this embodiment of the invention, based on the frozen rotation mask and gating structure, restorative distillation optimization is performed on the polynomial coefficients and related trainable weights in the student model, so that the student model can maintain a low homomorphic encryption execution overhead while being as close as possible to the discriminative performance of the teacher network.

[0136] In terms of loss function design, this invention adopts supervised loss, dynamic temperature distillation loss and feature alignment loss as a joint optimization objective.

[0137] The basic supervision term is defined by formula (12):

[0138]

[0139] in, Output for the student model. Output for the teacher model, This is a real label.

[0140] Considering that the output logit values ​​of the teacher network and the student network may have significant differences in scale, this invention first performs standardization and alignment processing on the outputs of the two networks.

[0141] The standardization alignment formula is shown in formula (13):

[0142]

[0143] Based on this, in this embodiment of the invention, an adaptive distillation temperature is dynamically assigned to each input sample according to the entropy and scale information output by the teacher model, rather than using a uniform and fixed temperature parameter.

[0144] The formula for the uncertainty of the teacher sample is formula (14):

[0145]

[0146] And normalize the set corresponding to all samples;

[0147] Constructing a scaling factor using the standard deviation of teacher output The dynamic temperature is then defined by formula (15):

[0148]

[0149] This design enables the knowledge distillation process to adaptively adjust the distribution softening level based on the sample difficulty. For samples with ambiguous class boundaries and high uncertainty in the teacher model's predictions, a higher distillation temperature is used to improve the smoothness of probability distribution matching; for simple samples with high confidence in the teacher model, strong class discrimination is maintained. Compared to traditional knowledge distillation methods using a fixed temperature, this dynamic distillation mechanism is more suitable for heterogeneous model transfer scenarios where teacher and student networks differ in network structure, activation methods, and other aspects.

[0150] In terms of distillation, this invention uses the symmetric KL divergence formula as shown in formula (16) for the outputs of the teacher model and the student model:

[0151]

[0152] The formula for distillation loss is formula (17):

[0153]

[0154] in, The course-based sample weights are used to control the gradual transition from easy to difficult samples during the distillation process.

[0155] In this embodiment of the invention, feature alignment terms are also introduced in several intermediate layers. Let the teacher network and the student network be in the... The features of each alignment layer are as follows: and Then the characteristic constraint can be expressed as formula (18):

[0156]

[0157] This constraint allows the student network to maintain a similar hierarchical feature representation structure to the teacher network even after rotation offset compression, gating structure freezing, and Chebyshev polynomial activation replacement, thus effectively alleviating the representation drift problem caused by differences in model structure.

[0158] It is understood that the knowledge distillation process of this invention is not performed independently of the homomorphic encryption execution cost, but rather an explicit cost-aware mechanism is introduced. The model defines ciphertext rotation proxy, Galois key set size proxy, and multinomial noise proxy to characterize the potential execution overhead of the student model during CKKS homomorphic encryption inference. Although the rotation mask and node-level gating structure are frozen during the distillation recovery optimization phase, and the number of ciphertext rotations and key set size no longer participate in structure search, the distillation process still continues the HE cost-aware design, ensuring that while the model recovers performance, it remains constrained within a predetermined low-overhead structure, avoiding a regression to a high-execution-cost network form.

[0159] The final joint loss function can be written as formula (19):

[0160]

[0161] in, Indicates distillation weight. Indicates feature alignment weights. This represents the cost-aware term. and The progressive scheduling strategy is adopted to enhance the model step by step. This means that in the early stage of training, more reliance is placed on real labels to ensure the stable convergence of the student model, while in the later stage of training, teacher constraints are gradually strengthened to restore a more granular decision boundary.

[0162] The distillation strategy of this invention is not a simple knowledge transfer of the original high-complexity spatiotemporal graph convolution model, but rather a targeted restorative learning of the lightweight student model after Tapmask rotation optimization, gating constraints and polynomial activation adaptation, so that the final optimized model has both low homomorphic encryption execution overhead and plaintext inference accuracy close to that of the original model.

[0163] In this embodiment of the invention, the original spatiotemporal graph convolutional network is used as the teacher network, and a restorative distillation training is performed on the student network, which has been structurally optimized and adapted for ciphertext computation. During the distillation process, mechanisms such as adaptive dynamic temperature, feature distribution alignment, and homomorphic encryption computation cost-aware constraints are comprehensively utilized. This method differs significantly from existing methods that only use ordinary knowledge distillation to achieve accuracy compensation. It can effectively restore the model's feature discrimination capability without destroying the lightweight, low-cost network structure or adding additional ciphertext rotation and key overhead, thereby achieving a balance between model accuracy and inference efficiency in homomorphic encryption privacy inference scenarios.

[0164] Example 2

[0165] Example 2 presents a privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption, applied to a client-side application. Figure 2 A flowchart illustrating a privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption, provided in this embodiment of the invention. Figure 2 ,like Figure 2 As shown, the method includes steps S10 to S50:

[0166] S10. Receive the raw time-series data input by the user, encrypt the raw time-series data using the public key, and generate ciphertext data.

[0167] S20. Generate the set of rotating keys required for homomorphic encryption.

[0168] S30. Send the ciphertext data and the set of rotation keys to the server so that the server can perform ciphertext inference and selective ciphertext rotation operations on the ciphertext data based on the optimized spatiotemporal graph convolutional network.

[0169] S40. Receive the encrypted reasoning result returned by the server. The encrypted reasoning result is obtained by the server through ciphertext reasoning.

[0170] S50. Using the private key corresponding to the public key, decrypt the encrypted reasoning result to obtain the plaintext prediction result and output it.

[0171] Example 3

[0172] Example 3 is an interaction diagram of a privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption provided by the present invention. Figure 3 This is an interactive schematic diagram of a privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption, provided in an embodiment of the present invention. Figure 3 As shown:

[0173] The spatiotemporal convolutional network privacy inference method based on homomorphic encryption described in this invention is mainly implemented collaboratively by the data owner (client) and the model service provider (server); wherein, the client holds the original skeleton temporal plaintext data, and the server deploys the pre-trained ST-GCN model parameters.

[0174] First, the client generates a public-private key pair and a corresponding evaluation key set in a local trusted environment using the CKKS key generation module. Then, the client uses the public key to encode and encrypt the original input data using CKKS, generating ciphertext data and transmitting it securely to the server.

[0175] Secondly, without ever touching the original plaintext data, the server inputs the received ciphertext data into the ST-GCN model based on CKKS homomorphic encryption to perform ciphertext inference and returns the encrypted prediction result ciphertext to the client.

[0176] Finally, the client receives the encrypted inference result returned by the server and decrypts it using the private key stored locally to obtain the corresponding plaintext prediction result.

[0177] For example, this patent uses action recognition datasets such as NTU-XVIEW as examples, and the shape of the model's input tensor can be represented as follows: .in, Represents batch size. Represents the number of coordinate channels. Represents the number of frames in time. Represents the number of joint nodes. This represents the number of instances in the image. In terms of network structure, ST-GCN is used as the backbone network, which is composed of multiple ST-GCN units stacked together. The core computation modules of each unit include: graph convolutional layers for spatial dimension feature aggregation, temporal convolutional layers (Temporal Conv2D) for temporal dimension feature extraction, and nonlinear activation layers using second-order Chebyshev polynomials with input calibration.

[0178] Example 4

[0179] Example 4 is an example of a simulation experiment provided by the present invention.

[0180] This patent is validated through simulation using the NTU-RGB+D xview dataset. The simulation employs a spatial-temporal graph convolutional network (ST-GCN) with three core modules as the backbone network; detailed simulation parameters are shown in Table 1. To adapt the model to homomorphic encrypted computing environments, a multi-stage optimization mechanism is introduced during the training phase. Both training and testing utilize FP16 mixed precision with a batch size of 64. The optimization process uses the SGD optimizer, with fixed configurations for parameters such as weight decay. Training initialization parameters are set according to the baseline model. The overall model training is completed sequentially through structured gating learning and cost-aware knowledge distillation.

[0181] Table 1 Simulation parameter settings

[0182] Optimizer type SGD Initial learning rate / weight decay coefficient 0.01 / 0.001 Learning rate decay step [40,80] Training batch size 64 Allowed rotation offset set [0,1,3]

[0183] Example 5

[0184] Example 5 presents an optimized architecture for a spatiotemporal graph convolutional network provided by this invention. Figure 4 An architecture diagram of an optimized spatiotemporal graph convolutional network provided in an embodiment of the present invention is shown below. Figure 4 As shown, it includes a convolutional layer 110, an activation layer 120, a pooling layer 130, and a fully connected layer 140, wherein the convolutional layer 110 includes a graph convolutional layer 111 and a spatiotemporal convolutional layer 112.

[0185] The graph convolutional layer 111 is used to aggregate spatial features of nodes and extract local spatial correlation features. The graph convolutional layer 111 is equipped with node-level learnable binary gating variables.

[0186] The spatiotemporal convolutional layer 112 is used to model temporal features and fit complex spatiotemporal correlations. A learnable rotation mask is integrated in the spatiotemporal convolutional layer 112.

[0187] Activation layer 120 is used to perform nonlinear transformations on the features, employing a second-order orthogonal polynomial activation function with an EMA calibration mechanism.

[0188] Pooling layer 130 is used to reduce the dimensionality of the feature map and compress key information.

[0189] The fully connected layer 140 is used to map the aggregated spatiotemporal features to the output category space to complete the final action recognition result.

[0190] In this embodiment of the invention, the encrypted original temporal ciphertext data is first input to the convolutional layer 110, where spatial dimension features are aggregated by the graph convolutional layer 111 to filter effective joint node features. Then, the temporal features are modeled by the spatiotemporal convolutional layer 112 and a selective ciphertext rotation operation is performed. Subsequently, the ciphertext features enter the activation layer 120, where a second-order orthogonal polynomial activation function calibrated by EMA is used to complete the nonlinear transformation. Next, the pooling layer 130 performs dimensionality reduction and key information compression on the ciphertext features to reduce subsequent computational overhead. Finally, the action recognition prediction result in the encrypted state is mapped by the fully connected layer 140, completing the overall ciphertext inference process.

[0191] In this embodiment of the invention, to achieve underlying compatibility with homomorphic encryption operations, each nonlinear layer in the network is uniformly replaced with a second-order polynomial form during the polynomial replacement stage, enabling the model to support inference computation in the ciphertext domain. By adopting a polynomialized activation structure, the characteristics of homomorphic encryption computation can be effectively adapted, providing a foundational support for subsequent ciphertext inference.

[0192] In this embodiment of the invention, by introducing a structural linearization gating mechanism and a cost-aware knowledge distillation strategy, the scheme can identify and eliminate redundant nonlinear activation structures in the network, effectively reducing the computation depth of homomorphic multiplication, thereby significantly reducing hardware computation latency in the homomorphic encryption inference process. Simultaneously, the efficient knowledge transfer achieved by the distillation mechanism can fully restore and guarantee the model's discrimination accuracy while optimizing ciphertext computation efficiency, enabling the overall scheme to achieve a balance between computational efficiency and recognition performance in homomorphic encryption inference scenarios.

[0193] In this embodiment of the invention, the computational overhead of ciphertext under homomorphic encryption can be effectively reduced, and the discriminative ability of graph convolutional networks in action recognition tasks can be stably maintained, thereby achieving a synergistic balance between security, inference efficiency and model accuracy in privacy inference scenarios.

[0194] In this embodiment of the invention, the simplification of the ciphertext rotation operator and the selection of low-order polynomial activation can be adaptively realized based on the node feature distribution state and computation graph structure. This method does not require separate constraints on the effective number of nonlinear layers during training, nor does it require a single fixed polynomial activation form, thus effectively improving the computational efficiency and convergence stability of the spatiotemporal graph convolutional network in homomorphic encryption privacy inference scenarios.

[0195] In this embodiment of the invention, the spatiotemporal graph convolutional network is synergistically optimized from three dimensions: ciphertext rotation operation, nonlinear activation calculation, and network structure complexity. This can specifically reduce ciphertext rotation operations, simplify the size of the rotation key set, and control the depth of ciphertext computation multiplication. Compared with conventional network design schemes, it is more compatible with the CKKS homomorphic encryption mechanism and can improve model execution efficiency while ensuring privacy inference security.

[0196] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the aforementioned specific details.

[0197] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0198] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0199] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A privacy-preserving inference method for spatiotemporal graph convolutional networks based on homomorphic encryption, characterized in that, Applied to the server side, the method includes: Receive encrypted data uploaded by the client; the encrypted data is obtained by encrypting the original time-series data based on the client's public key; Ciphertext inference is performed on the ciphertext data based on an optimized spatiotemporal graph convolutional network. During ciphertext inference, selective ciphertext rotation is performed on the ciphertext data based on the spatiotemporal convolutional layers in the optimized spatiotemporal graph convolutional network, and the rotated features are activated based on the activation layers in the optimized spatiotemporal graph convolutional network to obtain the encrypted inference result. The encrypted inference result is returned to the client, so that the client can decrypt the ciphertext data based on the private key to obtain the plaintext prediction result.

2. The privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption according to claim 1, characterized in that, When the spatiotemporal convolutional layer of the spatiotemporal graph convolutional network performs selective ciphertext rotation, the following steps are included: Receive a set of rotation keys generated by the client; the set of rotation keys corresponds to different time offsets; The optimized spatiotemporal graph convolutional network is trained to obtain the learnable rotation mask and the corresponding effective time offset. During the ciphertext inference phase, a corresponding rotation key is selected from the rotation key set based on the effective time offset, and selective ciphertext rotation is performed on the ciphertext data.

3. The privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption according to claim 2, characterized in that, The training of the rotating mask includes the following steps: Features at different time steps are input into the spatiotemporal convolutional layer, and the temporal convolution output is calculated based on the convolution kernel and ciphertext rotation operation of the spatiotemporal convolutional layer. During model training, the parameters of the rotation mask and convolution kernel are optimized through backpropagation; Based on the optimized rotation mask, determine and retain the effective time offsets that contribute to the temporal convolution output and meet the preset conditions.

4. The privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption according to claim 1, characterized in that, The optimized spatiotemporal graph convolutional network uses a second-order orthogonal polynomial as the activation function in its activation layer to perform activation processing on the features after ciphertext rotation.

5. The privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption according to claim 4, characterized in that, Before performing activation processing on the features after ciphertext rotation, the following steps are included: An exponential moving average (EMA) input calibration mechanism is introduced into the activation layer of the spatiotemporal graph convolutional network to normalize the input data of the activation layer to the effective fitting range of the polynomial.

6. The privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption according to claim 5, characterized in that, The optimized spatiotemporal graph convolutional network has graph convolutional layers with node-level learnable gating variables. Activation processing is performed on the features after ciphertext rotation, including the following steps: Based on the gating variables, determine and control whether nodes in the graph convolutional layer need to be activated using polynomials.

7. The privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption according to claim 6, characterized in that, The optimized spatiotemporal graph convolutional network is obtained by optimizing the original spatiotemporal convolutional network.

8. The privacy inference method for spatiotemporal graph convolutional networks based on homomorphic encryption according to claim 7, characterized in that, The method further includes training the optimized spatiotemporal graph convolutional network using knowledge distillation. The method for training the optimized spatiotemporal graph convolutional network using knowledge distillation includes the following steps: The original spatiotemporal graph convolutional network is used as the teacher model, and the optimized spatiotemporal graph convolutional network is used as the student model. Knowledge distillation training is performed on multiple student models based on the teacher model. During the restorative optimization phase of distillation training, the learnable rotation mask and learnable gating variables are frozen.

9. A privacy data processing method based on homomorphic encryption, characterized in that, Applied to a client, the method includes: Receive raw time-series data input by the user, encrypt the raw time-series data using a public key, and generate ciphertext data; Generate the set of rotation keys required for homomorphic encryption; The encrypted data and the set of rotation keys are sent to the server so that the server can perform encrypted inference and selective encrypted rotation operations on the encrypted data based on an optimized spatiotemporal graph convolutional network. Receive the encrypted reasoning result returned by the server, which is obtained by the server through ciphertext reasoning; Using the private key corresponding to the public key, the encrypted reasoning result is decrypted to obtain the plaintext prediction result, which is then output.

Images