Collaborative analysis method for power user privacy data based on homomorphic encryption and distillation

Abstract

The application discloses a power user privacy data collaborative analysis method based on homomorphic encryption and distillation. The method aims to solve the problems of low precision, slow convergence and unstable encrypted training caused by static polynomial approximation activation function in the prior art. The method comprises the following steps: in the encrypted training process, periodically calculating the first-order and second-order power sum of the pre-activation value of the network layer under the homomorphic encryption as a statistic, and submitting the statistic to the model owner for decryption to obtain the mean and variance of the current data distribution; then, a new set of polynomial coefficients is solved, so that the corresponding polynomial function optimally approximates the target nonlinear activation function under the distribution, and the new coefficients are updated to the student model of the central server. The application combines mixed knowledge distillation and noise perception update protocol, improves the precision, convergence speed and training stability of the encrypted collaborative analysis model, and ensures the end-to-end privacy security of the power user data.

Description

Technical Field

[0001] This invention relates to the field of data security and privacy protection technology, and in particular to a collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation. Background Technology

[0002] With the development of smart grids and big data technologies, in-depth analysis of massive amounts of electricity user data has become crucial for improving grid operation efficiency, achieving accurate load forecasting, and optimizing demand-side management. However, electricity user data often contains highly sensitive personal or commercial privacy information. Centralized analysis of multi-party data directly faces serious data leakage risks and legal compliance challenges. To address this issue, privacy-preserving computing technologies, such as homomorphic encryption, have emerged. Homomorphic encryption allows computation on ciphertext without decryption, achieving "data usable but not visible," providing a theoretical basis for secure collaborative analysis of multi-party data. Currently, in the field of privacy-preserving machine learning, homomorphic encryption is often combined with knowledge distillation techniques. This involves transferring the knowledge of a high-performance teacher model trained in a plaintext domain to a simplified, homomorphically encryption-friendly student model, enabling the student model to be trained and inference on encrypted data.

[0003] However, existing collaborative analysis methods based on homomorphic encryption still face significant technical bottlenecks. First, homomorphic encryption itself cannot directly support commonly used nonlinear activation functions in neural networks (such as ReLU and GELU), and existing techniques generally employ fixed low-order polynomials for static approximation. This "one-size-fits-all" approximation strategy fails to consider the dynamic changes in the internal data distribution of the neural network during training. Furthermore, the distribution of pre-activation values ​​in network layers differs greatly between the early and later stages of training, and the fixed polynomial function introduces non-negligible approximation errors in some training phases. These errors accumulate and amplify layer by layer, severely restricting the model's convergence speed and final accuracy. Second, traditional knowledge distillation processes rely heavily on aligning the final outputs of teacher and student models. This single supervisory signal struggles to effectively transfer the rich intermediate layer features of the teacher model to the student model, resulting in insufficient knowledge transfer. Moreover, homomorphic encryption operations (especially multiplication) are accompanied by increasing noise. In the multi-layer iterative computation of deep networks, without a sophisticated noise management mechanism, ciphertext noise can easily overflow, leading to incorrect calculation results and prematurely interrupting the entire encryption training process. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation to solve the problems mentioned in the background art.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation, comprising:

[0007] Obtain encrypted user data by acquiring electricity user privacy data encrypted by at least one data provider using a homomorphic encryption public key;

[0008] On the central server, a student model is trained using the encrypted user data. The encrypted training includes encrypted forward propagation, encrypted loss calculation, encrypted backpropagation, and encrypted parameter update.

[0009] In the encrypted forward propagation, the following dynamic activation policy is applied to at least one network layer of the student model:

[0010] a: Apply a polynomial activation function to the encrypted pre-activation value of the network layer to obtain the encrypted activation value;

[0011] b: During training, one or more cryptographic statistics of the encrypted pre-activation values ​​are periodically calculated under homomorphic encryption, and the cryptographic statistics are used to characterize the data distribution characteristics of the network layer pre-activation values;

[0012] c: The encrypted statistics are given to a model owner who holds a homomorphic encryption private key for decryption to obtain the plaintext statistics;

[0013] d: The model owner solves for a new set of polynomial coefficients based on the data distribution represented by the plaintext statistics and a preset target nonlinear activation function. The new polynomial coefficients aim to make the corresponding polynomial function optimally approximate the target nonlinear activation function under the data distribution.

[0014] e: The central server receives the new polynomial coefficients and uses the new polynomial coefficients to define a new polynomial function as the activation function of the network layer in subsequent training.

[0015] As a preferred embodiment of the collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation described in this invention, the encryption training is a training process based on knowledge distillation, and further includes:

[0016] The model owner provides a pre-trained teacher model, which contains a non-polynomial activation function.

[0017] As a preferred embodiment of the collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation described in this invention, wherein: the training process of the knowledge distillation is a hybrid distillation, and the calculation of the encryption loss includes:

[0018] Under homomorphic encryption, calculate the response distillation loss between the encrypted output of the student model and the encrypted output of the teacher model;

[0019] Under homomorphic encryption, the feature distillation loss between at least one encrypted intermediate layer feature of the student model and the corresponding encrypted intermediate layer feature of the teacher model is calculated.

[0020] The weighted sum of the response distillation loss and the feature distillation loss is used as the total encryption loss.

[0021] As a preferred embodiment of the collaborative analysis method for power user privacy data based on homomorphic encryption and distillation described in this invention, the response distillation loss and the feature distillation loss are both obtained by calculating the mean square error between two encryption vectors under homomorphic encryption.

[0022] As a preferred embodiment of the collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation described in this invention, wherein: the encryption backpropagation includes targeting a network layer for which the polynomial activation function is applied, and the gradient of the network layer includes:

[0023] Calculate the derivative of the polynomial activation function to obtain a derivative polynomial;

[0024] In homomorphic encryption, the encryption gradient from the next layer is homomorphically multiplied with the derivative polynomial that has applied the encryption pre-activation value of the current layer.

[0025] As a preferred embodiment of the collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation described in this invention, the encryption parameter update applies a noise-aware update protocol to manage the noise introduced by the homomorphic encryption operation when updating the encryption weights of the student model.

[0026] As a preferred embodiment of the collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation described in this invention, the noise-aware update protocol includes:

[0027] Before performing parameter updates based on the cryptographic gradient, the noise level of the cryptographic gradient is evaluated;

[0028] When the noise level exceeds a preset threshold, the learning rate used for this update is adjusted or the parameter update is skipped to prevent excessive noise accumulation from causing calculation errors.

[0029] As a preferred embodiment of the collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation described in this invention, wherein: in the initial stage of the encryption training, the polynomial activation function applied in step a is a preset initial polynomial with fixed coefficients; the dynamic adjustment strategy is initiated after training has undergone at least one iteration cycle, and is updated for the first time based on the initial polynomial.

[0030] As a preferred embodiment of the collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation described in this invention, wherein: the encrypted statistics calculated in step b include the ciphertext of the first power sum and the second power sum of the encrypted pre-activation value.

[0031] As a preferred embodiment of the collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation described in this invention, wherein: solving for a new set of polynomial coefficients in d includes:

[0032] The mean and variance of the pre-activated value data distribution are calculated based on the first and second power sums after decryption.

[0033] Construct an optimization problem with the objective of minimizing the weighted integral error between the polynomial function and the target nonlinear activation function under a probability distribution with the mean and variance as parameters;

[0034] Solve the optimization problem to obtain the new set of polynomial coefficients.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] 1. By using a dynamically adjusted multinomial activation strategy, the activation function can fit the changing distribution of pre-activated values ​​within the network in real time, reducing the accumulated error caused by static approximation, thereby improving the convergence speed and final co-analysis accuracy of the student model under encrypted training.

[0037] 2. By adopting a hybrid distillation strategy, not only are the final outputs of the teacher and student models aligned, but also the key intermediate layer features are aligned, providing richer and deeper supervisory signals for the student model, thus enabling a more complete transfer of knowledge from the high-performance teacher model to the encrypted student model.

[0038] 3. By introducing a noise-aware update protocol, the noise level in the encrypted gradient is monitored and managed in real time, which effectively avoids training interruption caused by noise overflow and enhances the stability and feasibility of long-term, multi-round encrypted training on deep networks.

[0039] 4. In addition, the entire collaborative analysis process is carried out under homomorphic encryption. The original power user data does not leave the local machine, the central server only processes the ciphertext, and the interaction of the dynamic activation strategy only leaks the aggregated statistical information, ensuring that the data can be valued under the premise of "usable but not visible". Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0041] Figure 1 This is a flowchart illustrating the overall process of a collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation, as described in one embodiment of the present invention.

[0042] Figure 2 This is a flowchart illustrating the dynamic activation strategy of a collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation, as described in one embodiment of the present invention.

[0043] Figure 3 This is a dynamic polynomial approximation error analysis diagram under a non-stationary distribution of the collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation, as described in an embodiment of the present invention.

[0044] Figure 4 This is a simulation diagram of the convergence of power load prediction based on knowledge distillation in a collaborative analysis method for power user privacy data based on homomorphic encryption and distillation, as described in an embodiment of the present invention. Detailed Implementation

[0045] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0046] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0047] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0048] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0049] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0050] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0051] Example 1

[0052] Reference Figure 1 and Figure 2 This is the first embodiment of the present invention, which provides a collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation, including:

[0053] S1. Obtain electricity user privacy data encrypted by at least one data provider using a homomorphic encryption public key to obtain encrypted user data.

[0054] It should be noted that the task of this step is to securely and properly process the raw electricity user data distributed across various data providers (which could be power companies in different regions) locally, and then transform it into an encrypted form that can be used for privacy-preserving computations on the central server. This process must ensure that the raw data does not leave the local machine.

[0055] Furthermore, before any data encryption operation begins, the model owner with the highest authority (e.g., a power grid dispatch center or research institution) is responsible for generating and distributing the homomorphic encryption key system required for the entire system. In this embodiment, the CKKS (Cheon-Kim-Kim-Song) homomorphic encryption scheme, which supports approximate real number operations, is adopted because it can efficiently handle floating-point operations involved in machine learning models.

[0056] Furthermore, the model owner first selects a set of cryptographic parameters based on the estimated complexity of the student model and the required security level. These cryptographic parameters include:

[0057] Polynomial cyclicity This parameter is directly related to the security level and the length of the data vector that a single ciphertext can encapsulate. For example, choosing... This means that each ciphertext operation can be processed in parallel on a vector. These floating-point values ​​are crucial for processing long-sequence data such as daily load curves sampled in minutes or hours.

[0058] Ciphertext Modular Chain The ciphertext modulus chain is composed of a series of prime numbers. The sequence constitutes the multiplication depth. The total bit length of this sequence determines the multiplication depth that the system can support. The network depth of the student model, the calculation of the loss function, and the order of the dynamic polynomial activation function all consume this multiplication depth. Therefore, the configuration of the ciphertext modulus chain must reserve sufficient computational budget for the entire encrypted training process (including forward propagation, loss calculation, and backpropagation).

[0059] Initial scaling factor The initial scaling factor is a key parameter that determines the accuracy of plaintext encoding. In our CKKS scheme, the plaintext message... Encoded as a polynomial Its coefficient is approximately A larger initial scaling factor can provide higher computational accuracy, but it also leads to a faster increase in noise after homomorphic multiplication. Based on this, we set... For a global It also dynamically manages accuracy and noise through rescaling in subsequent operations.

[0060] Furthermore, based on the above parameters, the model owner can generate a complete key:

[0061] Public-private key pairs : Generated using the KeyGen key generation algorithm.

[0062] Relinearized key : Used to reduce the dimension of a higher-order ciphertext resulting from the multiplication of two ciphertexts back to the standard ciphertext.

[0063] Galois Key : Used to support efficient rotation operations on the internal data of ciphertext, it is an indispensable key when calculating convolution or vector inner product.

[0064] Furthermore, the model owner will store the non-sensitive public key portion, i.e. The private key is securely distributed to all data providers and the central server. These statistics are kept strictly confidential by the model owner and are used only for decrypting the statistics required for the dynamic activation strategy.

[0065] Furthermore, each data provider preprocesses its held electricity user privacy data locally. Assuming the input data is the user's time-series load curve, represented as a vector... ,in, for The load value at a given time. The preprocessing steps are as follows:

[0066] First, handle missing or outlier values, for example, by imputing missing load data points using interpolation. Then, perform data normalization, unifying data of different units and ranges into a fixed interval, typically... It is important to note that in this invention, data normalization is primarily used to approximate the polynomial of the nonlinear activation function. Since its optimal approximation interval is finite, constraining all pre-activated values ​​within a controllable range is a prerequisite for the effective implementation of the dynamic activation strategy. For this normalization, max-min normalization can be used.

[0067] Furthermore, once preprocessing is complete, the data provider uses the public key obtained from the model owner. And related parameters, for the normalized data vector Encryption is performed. The encryption process is as follows:

[0068] First, normalize the data vector. Encoded as a plaintext polynomial CKKS encoders will Mapping to the polynomial ring One of the elements, multiplied by the scaling factor. Preserve precision. Then, use the public key. For the plaintext polynomial Encryption is performed to generate the final encrypted user data, i.e., ciphertext. This process can be represented as:

[0069]

[0070] in, The function's purpose is to use the public key to transform the encoded plaintext polynomial into ciphertext that cannot be directly read. The function transforms a plaintext message from its original domain (e.g., a vector of real or complex numbers) to a plaintext space that the CKKS homomorphic encryption scheme can directly process, namely a specific polynomial ring.

[0071] It should be noted that, due to the vectorization (SIMD) characteristics of CKKS, the normalized data vector... It can be encapsulated in a single ciphertext, thus greatly improving communication and computing efficiency.

[0072] Furthermore, the data provider will generate ciphertext. The data is transmitted to the central server. During this process, the central server only has access to ciphertext data that cannot be decrypted, thus ensuring the privacy and security of electricity user data. Simultaneously, the central server aggregates encrypted data from all data providers to form an encrypted dataset used for training student models.

[0073] S2. On the central server, a student model is trained using encrypted user data. The encrypted training includes encrypted forward propagation, encrypted loss calculation, encrypted backpropagation, and encrypted parameter update.

[0074] Furthermore, the central server, without touching any plaintext data, utilizes encrypted user data collected from various data providers, i.e., ciphertext... An end-to-end cryptographic training process is performed on a pre-defined student model. This process is supervised by the model owner and securely transfers knowledge from a high-performance teacher model to the student model via knowledge distillation. The entire cryptographic training process is an iterative loop, with each iteration including cryptographic forward propagation, cryptographic loss calculation, cryptographic backpropagation, and cryptographic parameter updates for all training batches.

[0075] It should be noted that before training begins, it is necessary to clarify the structure and role of the student model and the teacher model, which is crucial for understanding the subsequent mixed distillation.

[0076] Specifically, for the student model The model is deployed on a central server, and all its parameters (weights and biases) are stored and updated in encrypted form. Its design leans towards a simple structure to accommodate the computational overhead and multiplication depth limitations under homomorphic encryption. In this embodiment, for time-series tasks such as electricity load forecasting, the student model is a simplified Transformer encoder structure specifically adapted for time-series forecasting. The encoder's structural input is the original vector of electricity user time-series load curves. (For example, 96 sampling points per day) First, it passes through an encrypted linear embedding layer (essentially an encrypted fully connected layer), mapping it to the model's internal representation dimension. The input sequence is obtained by adding encrypted position encoding. Where CTXT represents the ciphertext. The student model consists of L=2 identical Transformer encoder blocks stacked together. Each encoder block contains two core sub-layers: a Multi-Head Self-Attention (MHSA) sub-layer and a Feed-Forward Network (FFN) sub-layer. The MHSA sub-layer is configured with two attention heads to reduce computational complexity. The FFN sub-layer has the following structure: linear layer → activation function → linear layer. The activation function here is the dynamic multinomial activation function in this invention. Each sub-layer is followed by a residual connection and layer normalization. After processing by L Transformer encoder blocks, the final encrypted output sequence is obtained. The sequence is then passed through an encrypted linear output layer, reducing its dimension from... Mapping to the dimension required for the prediction target (e.g., if predicting the load at 24 points in the future, the output dimension is 24) yields the final encrypted prediction result. It is important to note that in the initial training phase of the student model, these polynomial functions use a pre-defined initial polynomial with fixed coefficients (e.g., a quadratic or cubic polynomial that approximates GELU in the interval [-1,1]).

[0077] Specifically, regarding teacher models The teacher model is owned by the model owner and runs in their local secure environment. It is important to emphasize that the teacher model is a pre-trained, high-performance large model with a more complex structure than the student model (possessing more layers of Transformer encoders) and uses standard, computationally superior nonlinear activation functions (such as GELU, ReLU, etc.). Furthermore, in this invention, the teacher model's role is to provide high-quality, rich supervisory signals for the student model's learning in the cryptographic domain, not just the final prediction result. In this embodiment, the teacher model is one with L=6 encoder blocks, 8 attention heads, and an internal representation dimension of... The standard Transformer encoder model is used. Furthermore, the input and output layer settings of the teacher model are consistent with those of the student model to ensure dimensional alignment during feature distillation.

[0078] Furthermore, perform encrypted forward propagation. Given a batch of encrypted input data... Calculate the student model Corresponding encrypted prediction results and encrypted intermediate layer features used for feature distillation .

[0079] Specifically, for any fully connected or convolutional layer in the student model, its computation process can be abstracted as one encrypted matrix-vector multiplication and one encrypted bias addition. Taking a fully connected layer as an example, the input is an encrypted vector. The weights are from the encryption matrix. The bias is used as an encryption vector. Then the preactivation value of this layer The calculation is as follows:

[0080]

[0081] It should be noted that the multiplication and addition here are operations supported by homomorphic encryption. After the linear transformation is completed, the activation function is processed. According to the dynamic activation strategy, the central server applies the polynomial activation function corresponding to the current layer. To encrypted pre-activation value Above. Let the polynomial be... The encryption activation value is then calculated as follows:

[0082]

[0083] It is important to note that the calculation of this encrypted activation value is performed entirely within the encrypted domain, through a series of homomorphic multiplications (calculating exponentiation terms) and homomorphic additions. Each homomorphic multiplication requires a relinearization and rescaling operation to control the growth of the ciphertext size and noise.

[0084] Specifically, the encrypted forward propagation process starts from the input layer and propagates forward layer by layer until the final encrypted output of the student model is obtained. Simultaneously, we select one or more intermediate layers of the student model (e.g., the output of the last Transformer encoder block) and save their encrypted activation values ​​for subsequent feature distillation loss calculations.

[0085] Furthermore, the encryption loss is calculated. The difference between the student model's prediction and the teacher model's "guidance" is calculated as the encryption loss signal driving model optimization. This invention employs a hybrid distillation strategy in this process as follows:

[0086] S201, The central server will send the encrypted input data for the current batch. Send to the model owner. The model owner uses their private key. Decryption yields the plaintext data, i.e. Input it into the local teacher model. A plaintext forward propagation is performed. Furthermore, during the propagation, the model owner obtains not only the final plaintext output of the teacher model, but also the output of its last Transformer encoder block, as plaintext intermediate layer features for feature distillation. Subsequently, the model owner uses the public key... The plaintext output and its intermediate features are re-encrypted to obtain the encrypted plaintext output (denoted as...). ) and the encrypted plaintext intermediate layer features (denoted as This process of "decryption-computation-re-encryption" allows for the provision of high-quality encrypted monitoring signals to the central server while ensuring data privacy.

[0087] S202, When the central server obtains... and Then, two types of losses are calculated under homomorphic encryption:

[0088] The first type of loss is the response distillation loss. This loss is calculated by measuring the final output of the student model. The final output of the teacher model The mean square error (MSE) between the encryption methods is obtained as follows:

[0089]

[0090] in, It is the dimension of the output vector. and These are the first two encrypted output vectors. Each element.

[0091] The second type of loss is characteristic distillation loss. Similarly, calculate the encrypted output features of the last Transformer encoder block of the student model. Intermediate layer features corresponding to the teacher model The mean square error of encryption between them:

[0092]

[0093] in, It is the dimension of the intermediate layer feature vectors.

[0094] S203. Sum the two losses with weights to obtain the total encryption loss. :

[0095]

[0096] in, It is a hyperparameter used to balance the importance of final output alignment and intermediate process mimicry.

[0097] Furthermore, perform encrypted backpropagation. Following the chain rule, calculate the total encrypted loss... We begin by calculating the encrypted gradient of the model parameters with respect to the loss layer by layer. Since the starting point for gradient calculation is the loss function applied to the student model output... The derivative of , therefore for MSE loss, the gradient is:

[0098]

[0099] Specifically, when the gradient is backpropagated to a layer that applies a multinomial activation function, its gradient is calculated as follows: Let the activation function of this layer be... The gradient from the next layer is Then the gradient of the pre-activation value of this layer is:

[0100]

[0101] in, This represents element-wise multiplication (Hadamard product). It is a polynomial The derivative of a polynomial. A key advantage is that the derivative of a polynomial is still a polynomial, therefore The computation can also be performed within the cryptographic domain through a series of homomorphic multiplications and additions. This ensures homomorphic compatibility throughout the entire backpropagation process. Subsequently, using... Continue calculating the gradients of the weights and biases of this layer and pass them to the previous layer.

[0102] Furthermore, the encryption parameters are updated. This is done after calculating the encryption gradient for all parameters (…). After that, update the encryption weights of the student model. The traditional gradient descent update rule is as follows:

[0103]

[0104] in, It is the learning rate. It's the updated version. However, because homomorphic operations accumulate noise, unmanaged gradient updates can lead to noise overflow, causing training failure. Therefore, this invention employs a noise-aware update protocol:

[0105] First, before performing parameter updates, the central server evaluates the encryption gradient. The noise level can be assessed. This process can be based on the fundamental principles of homomorphic encryption. It's important to explain that in the CKKS scheme, each ciphertext is associated with a chain of moduli, and its current modulus represents the remaining noise capacity. A newly generated ciphertext typically has the largest initial modulus; after homomorphic multiplication, its modulus decreases while the noise increases. Therefore, the noise level can be evaluated by examining the encryption gradient ciphertext. This is implemented based on the current module level. A lower module level means a greater depth of multiplication and correspondingly higher accumulated noise. A preset noise threshold is then set, which is specifically represented by a minimum module level, denoted as . If the encryption gradient ciphertext The current modulus level is greater than If the noise is within a controllable range, then gradient descent updates will proceed normally. If the modulus of the current gradient ciphertext is less than or equal to... This means the gradient value may have been severely contaminated by noise. In this case, to prevent noise from being incorrectly added to the model weights, the system will take avoidance measures. Specifically, the system will dynamically reduce the learning rate. (For example, This could mitigate the impact of the update; or, in a more serious case, skip the parameter update altogether and wait for the next batch of data to generate potentially "cleaner" gradients.

[0106] It should be noted that, through this noise perception mechanism, the present invention can achieve a better balance between the stability and convergence of encrypted training, thereby improving the feasibility of long-term, multi-round encrypted training on deep networks.

[0107] S3. During the encrypted forward propagation, for at least one network layer of the student model, the following dynamic activation strategy is executed, refer to... Figure 2 :

[0108] a: Apply a polynomial activation function to the encrypted pre-activation value of the network layer to obtain the encrypted activation value;

[0109] b: During training, one or more cryptographic statistics of the encrypted pre-activation values ​​are periodically calculated under homomorphic encryption. The cryptographic statistics are used to characterize the data distribution characteristics of the network layer pre-activation values.

[0110] c: The encrypted statistics are given to a model owner who holds a homomorphic encryption private key to decrypt them, thus obtaining the plaintext statistics;

[0111] d: The model owner solves for a new set of polynomial coefficients based on the data distribution represented by the plaintext statistics and a pre-defined target nonlinear activation function. The new polynomial coefficients aim to make the corresponding polynomial function optimally approximate the target nonlinear activation function under the data distribution.

[0112] e: The central server receives the new polynomial coefficients and uses the new polynomial function defined by the new polynomial coefficients as the activation function of the network layers in subsequent training.

[0113] It should be noted that this step aims to address the bottlenecks in model accuracy and convergence caused by static polynomial approximation. By employing a data-driven adaptive effect, the polynomial used to replace the nonlinear activation function can dynamically fit the ever-changing distribution of pre-activated values ​​during training. This strategy is not executed completely in every forward propagation, but rather triggered periodically, thereby achieving a balance between computational cost and approximation accuracy.

[0114] Furthermore, regarding step a in the above strategy, this step is an application of the dynamic activation strategy in regular forward propagation. Specifically, during the encrypted forward propagation process of the aforementioned step S2, when the calculated... Layer's encrypted pre-activation value vector Then, the central server will call the polynomial activation function currently configured for this layer. It is then processed. This polynomial activation function consists of a set of coefficient vectors. Define, where, Let be the order of the polynomial. And, in the initial phase of encrypted training (e.g., the first training cycle), here... It is a pre-defined initial polynomial with fixed coefficients. These initial coefficients are typically derived from a standard, assumed data distribution (such as a standard normal distribution). This initial polynomial is obtained by offline least-squares fitting of the target nonlinear activation function (such as GELU). This initial polynomial provides a reasonable starting point for the entire dynamic adjustment process.

[0115] Specifically, in the homomorphic encryption environment of this invention, the process of applying this polynomial function can be represented as follows:

[0116]

[0117] in, It is the calculated encrypted activation value vector. These are the plaintext polynomial coefficients, which can be directly multiplied with the ciphertext using a "plaintext-ciphertext" multiplication operation. Through The encrypted preactivation value obtained by subhomomorphic multiplication The power of exponentiation is important. It is important to note that each homomorphic multiplication must be followed by relinearization and rescaling operations to manage the ciphertext size and noise.

[0118] Furthermore, based on step b above, the central server will periodically (e.g., every time it completes...) (During each training cycle, or when the model validation set loss reaches a plateau) the encrypted pre-activation value for a specific network layer. Statistical analysis is performed to capture the current data distribution characteristics. For this process, this invention selects ciphertexts for calculating the first and second power sums of the pre-activation values ​​because these ciphertexts are sufficient statistics for calculating the sample mean and variance. The calculation process is as follows:

[0119] Assuming that the central server processed [number] cases within a statistical period... Data in batches, each batch containing Each sample, and CKKS's vectorization processing capability encapsulates each ciphertext. The nth value. Then for the nth... From the pre-activation values ​​of the layer, we can obtain a series of encryption vectors. ,in, All batches and vectors were traversed.

[0120] b1: Calculate the encrypted first-order power sum For each encryption vector (It is internally encapsulated) (a number of floating-point numbers), the central server first performs a summation within the ciphertext. The specific steps of this summation are as follows, assuming... Powers of 2:

[0121] Command part and ciphertext .

[0122] For loop variables From 1 to Using Galois keys, for Perform a homomorphic rotation operation with a rotation step size of . The rotated ciphertext is obtained. , The function rotates and shifts the encrypted ciphertext. It then performs a homomorphic addition between the rotated ciphertext and the original ciphertext. .

[0123] After the iteration ends, Each slot stores the original vector. The sum of all elements. We take the value of the first slot as the sum of the vector, denoted as . Then, by homomorphically summing the sums of all batches, the total first-order power sum can be obtained:

[0124]

[0125] b2: Calculate the encrypted second power sum First, for each encryption vector Perform element-wise homomorphic square operations:

[0126]

[0127] Next, the obtained squared vector Performing the same homomorphic summation operation as described above will yield the total second-order power sum.

[0128] Finally, the central server received two ciphertexts, namely and These two ciphertexts respectively encrypt the first and second power sums of all pre-activated values ​​for that layer within the statistical period. It is important to emphasize that this process is entirely completed within the encrypted domain; the central server has no way of knowing any specific pre-activated values.

[0129] Furthermore, based on step c above, the central server will calculate the two encrypted statistics, i.e., the ciphertext. and Together with the total number of data points processed within that statistical period (As public metadata), it is sent along with the model owner. The model owner then uses the private key they hold. Decrypt these two ciphertexts:

[0130]

[0131]

[0132] in, The function's purpose is to convert ciphertext back to plaintext using a private key. In short, when a plaintext polynomial is encrypted into ciphertext, decryption requires using the private key to eliminate random elements in the ciphertext, thus recovering an approximate plaintext polynomial. Once the plaintext polynomial is obtained, it still needs to be decoded to convert it back into an approximate real number or complex vector.

[0133] It should be noted that, due to and The aggregated statistical values ​​are so complex that even if the model owner decrypts them, they cannot deduce the original data or intermediate state of any individual user, thus achieving dynamic adjustments while strictly adhering to privacy protection principles.

[0134] Furthermore, based on step d above, the model owner uses the plaintext statistics obtained through decryption. Total number of data points Perform the following operations:

[0135] d1: Calculate the mean and variance of the data distribution. Based on statistical principles, directly calculate the sample mean of the pre-activation value data distribution from the power sum. and sample variance :

[0136]

[0137]

[0138] It should be noted that the sample mean and sample variance are intended to indicate in which region the values ​​of the input activation function of the network layer are mainly concentrated during the current training phase, and the degree of dispersion of its distribution.

[0139] d2: Construct a weighted least squares optimization problem. Since the model owner's goal is to find a new set of polynomial coefficients... The polynomial defined by it. Given the current data distribution, it can optimally approximate the preset target nonlinear activation function. (e.g., GELU). Optimal distribution under a given data distribution means that the weight of the approximation error should be larger in regions with dense data points, and smaller in regions with sparse data points. Based on this, this invention models the problem as a weighted least squares optimization problem, where the weight function is defined by a probability distribution with the calculated sample mean and variance as parameters. A general and efficient choice is a Gaussian distribution. Therefore, the optimization objective is to minimize the following weighted integral error. :

[0140]

[0141] in, .

[0142] d3: Solving the optimization problem. Since the objective problem is a convex optimization problem, an analytical solution can be obtained by solving its corresponding normal equations. Regarding each coefficient Taking the partial derivative and setting it to zero, we get:

[0143]

[0144] It should be noted that this formula can form a system of linear equations. ,in: It is the new coefficient vector to be solved. It is A matrix whose elements It should be explained that this matrix follows a Gaussian distribution. The order moment has a known analytical expression and can be calculated from the sample mean and sample variance. It is A dimensional vector whose elements It should be explained that this term usually does not have an analytical solution, but it can be efficiently calculated using numerical integration methods such as the Gauss-Hermite Quadrature.

[0145] It should be noted that the model owner solved this system of linear equations, that is, through calculation. This will yield the optimal set of new polynomial coefficients.

[0146] Furthermore, based on step e above, the model owner will solve for a new solution containing only... The new polynomial coefficients of the floating-point numbers Simply send it back to the central server. It's important to emphasize that this process involves very low communication overhead. Once the central server receives it, it directly uses... Replace the original stored activation function coefficients of this network layer Starting from the next training iteration, when the cryptographic forward propagation reaches this network layer again, step a will apply this new polynomial function based on the latest data distribution. By repeating this process, the activation function of the student model can keep up with the changes in its internal data manifold in real time, thereby significantly reducing approximation errors, accelerating model convergence, and ultimately improving the accuracy of collaborative analysis on encrypted data.

[0147] Example 2

[0148] Reference Figure 3 and Figure 4 This is the second embodiment of the present invention. In order to verify the effectiveness of the collaborative analysis method described in the present invention, this embodiment evaluates the dynamic polynomial approximation effect and the overall convergence of the model through simulation experiments.

[0149] like Figure 3 As shown, the experiment compared the performance of the traditional static polynomial and the dynamic update strategy of this invention at different iteration cycles. As training progressed (from 1 to 50 iterations), the distribution of pre-activated values ​​within the network underwent significant shifts and scaling. Simultaneously, because the traditional "one-size-fits-all" static polynomial could not adapt to the distribution changes, its mean squared error (MSE) increased exponentially with the number of iterations. By adopting the dynamic activation strategy of this invention, through periodic calculation of encrypted statistics and resolving the polynomial coefficients, the MSE was kept at a consistently low level (approximately...). to (between). When the input data distribution shifts significantly with iteration, the dynamic update strategy keeps the approximation error low, improving numerical stability by approximately two orders of magnitude compared to static polynomials. Furthermore, as... Figure 4 As shown in the figure, the experiment demonstrates the performance of the student model in the power load forecasting task under the hybrid distillation framework. The traditional static baseline scheme plateaus after approximately 50 training epochs, with the loss function decreasing slowly. In contrast, the proposed solution, through hybrid distillation (simultaneously aligning response and intermediate layer features) and a dynamic activation function, achieves a continuous and stable decrease in the total encryption loss. Particularly at the same training epochs (e.g., 200 epochs), the model's accuracy in capturing load peaks and troughs is significantly better than the traditional encryption model, with an overall prediction accuracy improvement of approximately 15%–25%. Furthermore, based on the prediction curve on the right, the prediction curve of the proposed solution closely matches the fluctuation trend of the correct data samples, proving that under homomorphic encryption, the proposed solution still maintains extremely high collaborative analysis accuracy, realizing the value mining of power user data that is "usable but not visible."

[0150] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0151] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0152] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0154] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0155] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation, characterized in that, include: Obtain encrypted user data by acquiring electricity user privacy data encrypted by at least one data provider using a homomorphic encryption public key; On the central server, a student model is trained using the encrypted user data. The encrypted training includes encrypted forward propagation, encrypted loss calculation, encrypted backpropagation, and encrypted parameter update. The encryption parameter update employs a noise-aware update protocol to manage noise introduced by homomorphic encryption operations when updating the encryption weights of the student model. The noise-aware update protocol includes: Before performing parameter updates based on the cryptographic gradient, the noise level of the cryptographic gradient is evaluated; When the noise level exceeds a preset threshold, the learning rate used for this update is adjusted or the parameter update is skipped to prevent excessive noise accumulation from causing calculation errors. In the encrypted forward propagation, the following dynamic activation policy is applied to at least one network layer of the student model: a: Apply a polynomial activation function to the encrypted pre-activation value of the network layer to obtain the encrypted activation value; b: During training, one or more cryptographic statistics of the encrypted pre-activation values ​​are periodically calculated under homomorphic encryption, and the cryptographic statistics are used to characterize the data distribution characteristics of the network layer pre-activation values; c: The encrypted statistics are given to a model owner who holds a homomorphic encryption private key for decryption to obtain the plaintext statistics; d: The model owner solves for a new set of polynomial coefficients based on the data distribution represented by the plaintext statistics and a preset target nonlinear activation function. The new polynomial coefficients aim to make the corresponding polynomial function optimally approximate the target nonlinear activation function under the data distribution. e: The central server receives the new polynomial coefficients and uses the new polynomial coefficients to define a new polynomial function as the activation function of the network layer in subsequent training.

2. The collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation as described in claim 1, characterized in that, The encrypted training is a knowledge distillation-based training process, which also includes: The model owner provides a pre-trained teacher model, which contains a non-polynomial activation function.

3. The collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation as described in claim 2, characterized in that, The training process for the knowledge distillation is a hybrid distillation process, and the calculation of the encryption loss includes: Under homomorphic encryption, calculate the response distillation loss between the encrypted output of the student model and the encrypted output of the teacher model; Under homomorphic encryption, the feature distillation loss between at least one encrypted intermediate layer feature of the student model and the corresponding encrypted intermediate layer feature of the teacher model is calculated. The weighted sum of the response distillation loss and the feature distillation loss is taken as the total encryption loss.

4. The collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation as described in claim 3, characterized in that, Both the response distillation loss and the feature distillation loss are obtained by calculating the mean square error between two encryption vectors under homomorphic encryption.

5. The collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation as described in claim 1, characterized in that, The encrypted backpropagation includes applying the multinomial activation function to a network layer, the gradient of which includes: Calculate the derivative of the polynomial activation function to obtain a derivative polynomial; In homomorphic encryption, the encryption gradient from the next layer is homomorphically multiplied with the derivative polynomial that has applied the encryption pre-activation value of the current layer.

6. The collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation as described in claim 1, characterized in that, In the initial stage of the encrypted training, the polynomial activation function applied in step a is a preset initial polynomial with fixed coefficients; the dynamic adjustment strategy is initiated after training has completed at least one iteration cycle, and the first update is performed based on the initial polynomial.

7. The collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation as described in claim 1, characterized in that, The cryptographic statistics calculated in b include the ciphertext of the first power sum and the second power sum of the cryptographic pre-activation value.

8. The collaborative analysis method for electricity user privacy data based on homomorphic encryption and distillation as described in claim 7, characterized in that, Solving for a new set of polynomial coefficients in d includes: The mean and variance of the pre-activated value data distribution are calculated based on the first and second power sums after decryption. Construct an optimization problem with the objective of minimizing the weighted integral error between the polynomial function and the target nonlinear activation function under a probability distribution with the mean and variance as parameters; Solve the optimization problem to obtain the new set of polynomial coefficients.

Images