An ubiquitous network multi-modal privacy data protection method

Abstract

The application relates to the technical field of privacy data protection, and discloses a ubiquitous network multi-modal privacy data protection method, which comprises the following steps: a client performs band pass filtering, standardization and sliding window slicing pretreatment on multi-modal physiological signals such as electroencephalogram, electrooculogram and electromyogram; frequency band discriminant features are extracted and fused from the obtained time sequence data window according to a preset frequency band; model training is performed locally, and a parameter update difference value is calculated, and differential privacy noise is injected; a non-interactive zero-knowledge proof is generated based on the Groth16 protocol, and the noisy difference value and the proof are uploaded to a trusted third party; after verification by the trusted third party, the difference value is confused and forwarded to a central server; and the central server aggregates and updates a global model according to sample weights and distributes the global model to the client. The method realizes efficient cooperation of multi-modal signal end-side adaptive fusion and distributed privacy protection, and guarantees the safety and credibility of parameter interaction through a non-interactive proof mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of privacy data protection, in particular to a ubiquitous network multi-modal privacy data protection method. BACKGROUND

[0002] In recent years, with the rapid evolution of ubiquitous network technologies such as Internet of Things (IoT), 5G / 6G, multi-modal physiological signal collection and analysis as an important research direction in the field of human-computer interaction and health monitoring, gradually showing a high-dimensional, multi-source, real-time development trend. Traditional single modal research of electroencephalogram (EEG), electrocardiogram (ECG), electromyogram (EMG) and other signals has made remarkable achievements in motor imagery, attention detection and emotion recognition applications; while multi-modal fusion technology further improves the recognition accuracy and robustness by processing data from different signal sources in time-frequency domain or deep feature space. At the same time, federated learning (Federated Learning) introduces a distributed training paradigm, enabling models to be optimized collaboratively while ensuring local data retention; differential privacy (Differential Privacy) and homomorphic encryption and other privacy protection technologies also provide mathematical provable security for participating nodes. In addition, the maturity of non-interactive zero-knowledge proof (Zero-Knowledge Proof, ZKP) technology makes it possible to verify the legality of the calculation result without revealing the original data.

[0003] However, the existing technology still has some deficiencies in practical application: first, existing multi-modal fusion mostly relies on central servers to aggregate raw data or only cross-verify single modal, making it difficult to balance the privacy of device-end data with wide temporal and spatial distribution and fusion efficiency; second, although federated learning framework can reduce data transmission, parameter updates are often not accompanied by mandatory privacy protection in the process of model parameter interaction, making them vulnerable to reverse inference attacks; relying solely on differential privacy to add noise cannot balance model accuracy and privacy budget, and lacks a trusted proof of the legality of noise injection; third, existing zero-knowledge proof schemes are mostly used in interactive scenarios or have complex computing circuit design and high communication overhead, making it difficult to efficiently deploy in ubiquitous network multi-node heterogeneous environments. Based on the above deficiencies, the present application combines multi-modal frequency band features with adaptive fusion, federated learning with differential privacy, and non-interactive zero-knowledge proof mechanism based on Groth16 protocol, which can generate high-discriminative comprehensive feature vectors on the client side and perform local training, and also can perform verifiable privacy protection on parameter updates, significantly improving the privacy security and collaborative learning efficiency of the system. SUMMARY

[0004] (I) The technical problems solved: In view of the shortcomings of the prior art, the present application provides a ubiquitous network multi-modal privacy data protection method, which has the advantages of multi-modal feature end-side adaptive fusion, distributed differential privacy noise injection and non-interactive zero-knowledge proof verification, and solves the problems that the model performance and data privacy protection cannot be considered, and the parameter update security and legality cannot be verified in the prior art.

[0005] (II) Technical scheme: In order to realize the combination of multi-modal feature end-side adaptive fusion, distributed differential privacy noise injection and non-interactive zero-knowledge proof verification, the present application provides the following technical scheme: a ubiquitous network multi-modal privacy data protection method, characterized in that it comprises:

[0006] S1: Collecting multi-modal physiological signals and performing band-pass filtering, standardization and sliding window slicing processing to generate a unified scale time series data window;

[0007] S2: Frequency band feature extraction and fusion of the time series data window to construct comprehensive discriminant features;

[0008] S3: Local model training on the comprehensive discriminant features, calculating the difference between the local model training parameters and the current global model parameters and adding noise to the difference;

[0009] S4: Based on the Groth16 protocol and the public reference string, generating a non-interactive zero-knowledge proof for the noise-added difference, and uploading the noise-added parameter difference and the corresponding non-interactive zero-knowledge proof to a trusted third party;

[0010] S5: The trusted third party verifies the non-interactive zero-knowledge proof, and after verification, randomly obfuscates the noise-added difference and forwards the obfuscated difference to the central server;

[0011] S6: The central server updates the global model parameters after weighted averaging of the difference, and distributes the updated global model parameters to each client to enter the next round of federated training iteration.

[0012] Further, the multi-modal physiological signal data includes electroencephalogram, electrooculogram and electromyogram signal data.

[0013] Further, the S2 comprises:

[0014] S21: For each time series data window of each modality, four sub-band signals are extracted by using a band-pass filter according to a predetermined frequency band;

[0015] S22: Independently calculating the differential entropy of the four sub-band signals to obtain a four-dimensional differential entropy feature vector;

[0016] S23: For each time series data window, map its 4-dimensional differential entropy feature vector into 4 channels respectively, and splice them into a multi-channel 2D feature map according to the electrode spatial layout;

[0017] S24: Input the 2D feature map into an ECA-based ResNet encoder to obtain a high-dimensional feature vector;

[0018] S25: Perform a pooling operation on the high-dimensional feature vectors of all windows under the same modality to obtain the overall feature of the modality, and then fuse the overall features of each modality through an attention mechanism to obtain a comprehensive discriminative feature vector.

[0019] Further, the difference value is added with noise, specifically: random noise is generated based on Laplace distribution and superimposed into the difference value.

[0020] Further, the difference value after adding noise is generated with a non-interactive zero-knowledge proof based on the Groth16 protocol and the public reference string, including:

[0021] The client loads the public reference string of the Groth16 protocol, and generates a commitment for the difference value after adding noise using a preset proof key, and binds the client identity and the current timestamp together;

[0022] Perform a hash operation on the commitment and its associated parameters to generate a unique commitment identifier;

[0023] Take the parameter difference value after adding noise and the commitment identifier as input, and complete polynomial commitment calculation on the proof circuit according to the Groth16 algorithm to generate a zero-knowledge proof.

[0024] Further, the S5 comprises:

[0025] S51: The trusted third party receives the non-interactive zero-knowledge proof and the difference value uploaded by the client, records the receiving timestamp, and compares it with the carried timestamp. Only when the time difference between the two is within a preset threshold, the process can continue, otherwise it is rejected;

[0026] S52: Verify the zero-knowledge proof using a preset verification key, and verify the correctness of each component of the proof according to the Groth16 protocol;

[0027] S53: After verification, the trusted third party generates a random challenge message and sends it to the client. After the client returns the confirmation code, it is confirmed that the proof is generated in real time within the trusted time window;

[0028] S54: Randomly confuse the difference value after adding noise through random permutation and forward it to the central server.

[0029] (Three) beneficial effects: compared with the prior art, the present application provides a ubiquitous network multi-modal privacy data protection method, which has the following beneficial effects:

[0030] 1. The ubiquitous network multi-modal privacy data protection method provides consistent time sequence data basis for different sensing channels by band-pass filtering, standardizing and sliding window slicing of multi-modal physiological signals; then, the frequency band differential entropy features in each signal window are encoded and attention weighted fused by the ResNet with the ECA module, which not only realizes adaptive weight distribution of discriminative frequency bands and modes, but also implicitly smooths the noise difference between channels, improving the robustness and discriminability of feature representation. The effect of this method is to convert multi-source data into high-dimensional and compact fusion vectors at the local end, which not only reduces the computational burden of the backend, but also avoids the risk of exporting raw data, thereby achieving the effect of improving model training efficiency while enhancing privacy security.

[0031] 2. The ubiquitous network multi-modal privacy data protection method effectively masks the influence of single client data by injecting Laplace noise into the backward parameter difference of the local model after training; then, based on the Groth16 protocol, non-interactive zero-knowledge proof is generated for the noisy difference value to prove its update legality without revealing the noise component; finally, after the trusted third party verifies the timeliness and legality of the proof, the noisy difference value is randomly scrambled and securely forwarded to the central server. The effect and purpose of this process is to provide mathematically provable privacy protection and security assurance for each parameter interaction, while avoiding the communication overhead of interactive protocols. Ultimately, this combined scheme achieves the effect of preventing data reverse attack and providing independent audit for the third party in the parameter update process while maintaining the performance of the federated learning model. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The overall flowchart of the ubiquitous network multi-modal privacy data protection method provided by the present application;

[0033] Figure 2 The principle diagram of the ubiquitous network multi-modal privacy data protection method provided by the present application;

[0034] Figure 3 The S2 sub-step flowchart of the ubiquitous network multi-modal privacy data protection method provided by the present application;

[0035] Figure 4 The S5 sub-step flowchart of the ubiquitous network multi-modal privacy data protection method provided by the present application;

[0036] Figure 5 The experimental result comparison chart of the ubiquitous network multi-modal privacy data protection method provided by the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be apparently and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without any creative work belong to the protection scope of the present application.

[0038] Please refer to Figures 1-2 A ubiquitous network multi-modal privacy data protection method, characterized in that, comprising:

[0039] S1: Collecting multi-modal physiological signals and performing band-pass filtering, standardization and sliding window slicing processing thereon to generate time series data windows of uniform scale;

[0040] S2: Performing frequency band feature extraction and fusion on the time series data windows to construct comprehensive discriminative features;

[0041] S3: Performing local model training on the comprehensive discriminative features, calculating the difference between the local model training parameters and the current global model parameters, and adding noise to the difference;

[0042] S4: Based on the Groth16 protocol and the public reference string, generating a non-interactive zero-knowledge proof for the noise-added difference, and uploading the noise-added parameter difference and the corresponding non-interactive zero-knowledge proof to a trusted third party;

[0043] S5: The trusted third party verifies the non-interactive zero-knowledge proof, and after verification, randomly obfuscates the noise-added difference and forwards the obfuscated difference to the central server;

[0044] S6: The central server updates the global model parameters after weighted averaging of the difference, and distributes the updated global model parameters to each client to enter the next round of federated training iteration.

[0045] Specifically, in S1, the client synchronously acquires multimodal physiological signals, including but not limited to electroencephalography (EEG), electrooculography (EOG), and electromyography (EMG), through wearable or fixed sensor modules. The analog voltage signals output by each sensor are first subjected to broadband bandpass filtering by the front-end hardware to remove DC components and high-frequency noise. Simultaneously, the buffer can dynamically adjust the filter order according to the application scenario to balance real-time performance and filtering steepness. The filtered continuous signal is sampled by an analog-to-digital converter (ADC) to obtain multi-channel real-valued time-series data. After sampling, batch standardization is performed on each channel at the software level, i.e., subtracting the mean and dividing by the standard deviation within a sliding window to eliminate amplitude differences and accelerate subsequent model convergence. Subsequently, the standardized multi-channel time series is sliced ​​into sliding windows of fixed length (e.g., 1 second or 2 seconds, corresponding to 250 or 500 samples) with 50% overlap, forming a matrix structure of [number of channels × window length]. Each matrix is ​​a "uniform-scale time-series data window," providing uniform-scale input data for frequency band feature extraction and fusion in subsequent S2.

[0046] For further details, please refer to Figure 3 S2 includes:

[0047] S21: For each timing data window of each mode, extract four sub-frequency band signals using a bandpass filter according to a preset frequency band;

[0048] Specifically, for each temporal data window of each modality, the system first decomposes the wideband (0.5–30Hz) of the original signal into frequency bands conforming to international EEG frequency classification. , , and .

[0049] S22: Calculate the differential entropy independently for the four sub-band signals to obtain a four-dimensional differential entropy feature vector;

[0050] Specifically, the system targets the data extracted in step S21. For the four sub-frequency band signals, the sample variance of all sampling points in the corresponding time-series data window is first calculated. Then, based on the analytical expression of the differential entropy of the Gaussian distribution: ; Calculate the differential entropy value of each sub-band separately, according to The fixed order of these elements forms a four-dimensional differential entropy feature vector. This method calculates entropy values ​​by relying solely on variance information, which reduces computational complexity and effectively characterizes the randomness and information content of signals in each frequency band.

[0051] S23: For each time series data window, map its 4-dimensional differential entropy feature vector into 4 channels respectively, and splice them into a multi-channel 2D feature map according to the electrode spatial layout;

[0052] Specifically, the system first regards the four-dimensional differential entropy feature vector corresponding to each time series data window as the initial pixel value of four channels. For the first window, its entropy values correspond to four frequency bands respectively. The system fills in the entropy values of each frequency band channel at the corresponding electrode position on the pre-defined electrode spatial grid (such as a 32x32 or 64x64 two-dimensional matrix). For the blank areas between the electrodes in the grid, bilinear interpolation or Gaussian kernel interpolation is used to smoothly expand the entropy value information to maintain spatial continuity. After completing the interpolation, four single-channel matrices with the same grid size are obtained. Finally, the four matrices are stacked in the channel dimension to form a 4-channel multi-channel 2D feature map. The feature map not only retains the entropy value characteristics of each frequency band, but also reflects the relative spatial relationship between different channels through electrode spatial mapping.

[0053] S24: Input the 2D feature map into the ECA-based ResNet encoder to obtain a high-dimensional feature vector;

[0054] Specifically, the system inputs the four-channel 2D feature map obtained in step S23 into an encoder based on residual network (ResNet) and integrates an efficient channel attention (ECA) module after each residual unit to adaptively enhance important channel features. The encoder first extracts spatial patterns through a series of convolution, batch normalization and ReLU activation layers, and at the end of each residual block, the ECA module first performs global average pooling on the current feature map to generate a channel descriptor, then uses one-dimensional convolution to capture the interaction between channels, outputs channel attention weights and multiplies them with the original feature map element by element in the channel dimension, achieving dynamic gain for discriminative frequency band channels. With the deepening of network levels, residual connection ensures stable flow of gradients, and ECA improves feature selectivity; after the last residual stage, the encoder performs global average pooling on the feature map to aggregate the spatial dimension into a high-dimensional vector as the final feature representation of the window.

[0055] S25: Pool the high-dimensional feature vectors of all windows under the same modality to obtain the overall feature of the modality, and then fuse the overall features of each modality through an attention mechanism to obtain a comprehensive discriminative feature vector.

[0056] ​Specifically, the system first performs a pooling operation on the high-dimensional feature vectors of all windows under the same modality to obtain the overall feature representation of the modality. Pooling can be performed in two ways: one is global average pooling, that is, the average vector is obtained by averaging all vectors dimension by dimension; the other is attention weighted pooling, that is, a small attention network is used to generate a scalar weight for each vector, and after softmax normalization, all vectors are fused in the form of weighted sum to ensure that more discriminative window fragments get higher weights. The overall feature of the modality generated in this way not only converges the key information of the signal source in the time dimension, but also enhances the influence of important windows through the attention mechanism. Subsequently, the system first calculates the similarity matrix of the feature vectors between modalities, then applies the corresponding attention weight to each modality vector, and finally adds the weighted modality vectors to generate a unified comprehensive discriminative feature vector.

[0057] Further, in S3, the client first receives and loads the latest global model parameters from the central server to initialize the local model; then, using the comprehensive discriminative feature vector obtained in step S2 as input, the cross-entropy loss function and the Adam optimizer are used to perform multiple rounds of iterative training on the local private data. After completing the predetermined training rounds, the client calculates the difference between the parameters before and after updating the local model, that is, subtracts the global model parameters loaded at the time from the model parameters after training to obtain the original parameter difference vector In the present application, in order to meet the requirement of differential privacy, the system injects Laplace noise into each component in , and the noise is generated according to the Laplace distribution, thereby obtaining the noisy update vector: .

[0058] In the formula, represents random noise sampled from a zero-mean Laplace distribution; is the update sensitivity defined as the maximum norm change of a single client model update under any circumstances, used to determine the noise scale; is the privacy budget, which controls the noise intensity; the smaller the value, the larger the noise, the stronger the privacy protection, but at the same time, the greater the impact on the model performance.

[0059] Further, in S4, based on the Groth16 protocol and the public reference string, a non-interactive zero-knowledge proof is generated for the noisy difference, including:

[0060] The client loads the public reference string of the Groth16 protocol and generates a commitment for the noisy difference using the preset proof key, while binding the client identity and the current timestamp together;

[0061] Perform a hash operation on the commitment and its associated parameters to generate a unique commitment identifier;

[0062] Using the noised parameter difference and commitment identifier as input, the Groth16 algorithm is used to perform polynomial commitment calculation on the proof circuit to generate a zero-knowledge proof.

[0063] Specifically, the client first loads the Groth16 protocol's Public Reference String (CRS) and pre-built proof key. Then, based on the client identity and timestamp Together, generate polynomial commitments. : ;In the formula, It is a group generator and the base point of the elliptic curve used in the Groth16 protocol; These are the system parameters used to prove the key circuit when constructing the polynomial, derived from CRS; It is the proof key, a key fragment disclosed by the client to the system for commitment computation; It is a client identity identifier used to bind a commitment to a specific client; It's a timestamp, used to verify the validity of the proof and prevent replay attacks. Next, regarding this commitment... The hash of the associated circuit parameters is calculated to obtain a unique commitment identifier. : Finally, following the Groth16 algorithm framework, random numbers and As a witness input proof circuit, it performs polynomial commitment calculations and generates proof elements A, B, and C, which are combined to form a non-interactive zero-knowledge proof. : .

[0064] Next, the parameter difference after adding noise and the corresponding non-interactive zero-knowledge proof are uploaded to a trusted third party.

[0065] Furthermore, such as Figure 4 As shown, S5 includes:

[0066] S51: The trusted third party receives the non-interactive zero-knowledge proof and difference uploaded by the client, records the receiving timestamp, and compares it with the carried timestamp. It can only continue if the time difference between the two does not exceed a preset threshold; otherwise, it is rejected.

[0067] Specifically, the trusted third party first receives the non-interactive zero-knowledge proof uploaded by the client. Update the corresponding noisy difference vector and immediately record the local reception timestamp. Subsequently, the system obtained proof. Extract the timestamp bound by the client. , and calculate the time difference between the two .

[0068] Only when is less than or equal to a pre-set maximum time offset threshold , the subsequent verification process is continued; otherwise, the trusted third party will consider the upload request as a possible replay attack and reject the processing, while recording abnormal logs for auditing.

[0069] S52: Check the zero-knowledge proof using the preset verification key, and verify the correctness of each component of the proof according to the Groth16 protocol;

[0070] Specifically, first, load the verification key from the public reference string (CRS) generation stage ; then, perform bilinear mapping calculation on the received proof elements A, B, C and their corresponding commitment values, that is, verify that the following relationship holds: ; in the formula, represents the bilinear mapping operation, and are the group generators in the CRS, and are the components in the verification key corresponding to and in the circuit polynomial, respectively. Only when the above mapping equations are simultaneously established, the trusted third party considers the proof valid; otherwise, it is judged as invalid proof and the upload request is rejected. In addition, the verification process also includes integrity checking of the commitment identifier and the timestamp, to ensure that the proof is consistent with the binding information generated in S4 previously, further excluding the risk of tampering or replay attacks.

[0071] S53: After verification, the trusted third party generates a random challenge message and sends it to the client. After the client returns the confirmation code, it confirms that the proof is generated in real time within the trusted time window;

[0072] Specifically, after completing the legality verification of the zero-knowledge proof, the trusted third party will use a secure random number generator to generate a random challenge value of sufficient length, and record the challenge time ; then, the challenge is constructed together with the digital signature of the trusted third party as a challenge message, which is sent to the client through an encrypted channel. After receiving the challenge, the client first verifies the signature legality, and then within the specified response time window, signs or calculates the message authentication code using its local private key or pre-shared key on the challenge value and the timestamp , generates the confirmation code ACK, and sends it together with the client identity and response timestamp to a trusted third party. After receiving the ACK, the trusted third party first verifies whether the signature or message authentication code matches the original challenge value, and then checks whether the difference between the response time and the challenge time is less than the preset maximum response delay threshold; only when both checks pass, can it be determined that the proof activity is generated in real time within the trusted time window, thereby further preventing delay replay or network relay attack means. and the challenge time The difference between the response time and the challenge time is less than the preset maximum response delay threshold; only when both checks pass, can it be determined that the proof activity is generated in real time within the trusted time window, thereby further preventing delay replay or network relay attack means.

[0073] S54: Randomly obfuscate the noisy difference value through random permutation and forward to the central server;

[0074] Specifically, after completing the challenge-response confirmation, the trusted third party will perform random permutation on the noisy update difference vector to obfuscate the element order, thereby disrupting the correspondence between parameters and reducing the risk of reverse analysis of a single update component. In implementation, the system first uses a secure random number generator to generate a seed , and constructs a random permutation index sequence with the same length as the vector. Then, rearrange the components in the vector according to the index to obtain the obfuscated vector. After completing the permutation, the trusted third party forwards the obfuscated vector to the central server, and records the permutation seed and the mapping relationship before and after the permutation for audit. This method not only maintains the differential privacy protection effect of the original update, but also further improves the anti-reverse security of the parameter upload process, ensuring that the server side cannot directly associate back to the detailed update information of any single client when aggregating.

[0075] Further, in S6, the central server first receives the obfuscated difference vector forwarded by the trusted third party through a secure channel, as well as the corresponding client identifier and sample number. In order to update the global model, the server calculates the weighted average difference value according to the sample size of each client and the total sample number ; in the formula, represents the overall representation of the server's model parameter increment after this round of aggregation; is the client index, representing the th client participating in this round of federated training; is the data sample number used by the th client for local training; is the sum of the sample numbers of all participating clients; is the noisy update difference vector calculated and uploaded by the th client. Then apply this weighted average result to the current global model parameter ​To balance the system flexibility and communication efficiency, the server can randomly select a part of all available clients to participate in each round of aggregation, and the clients not selected can continue to use the last issued global model for local training. Finally, the updated global model is distributed to each client through an encrypted distribution mechanism to start the next round of federated training iteration.

[0076] The embodiments of the present application have achieved some positive effects during research and development or use, and indeed have great advantages compared with the prior art, which are described below in combination with data, graphs and the like in the test process.

[0077] Please refer to Figure 5 The technical scheme of constructing a multi-channel 2D feature map based on frequency band differential entropy and combining ECA-ResNet adaptive attention fusion fully demonstrates its innovation and practical value. In the figure, the orange and green columns respectively represent the classification accuracy when using electroencephalogram (EEG) only and when fusing electroencephalogram, electrooculogram and electromyogram (EEG+EOG+EMG) three modal input, the horizontal axis is the arousal and valence two emotional dimensions, and the vertical axis is the accuracy percentage. It can be seen that in the arousal recognition task, the accuracy of the traditional EEG single mode method is about 97%, and after the three modal fusion of the present application, the precision is improved to 99%; in the valence recognition, the EEG single mode is about 98%, and after the fusion, it is close to 100%. More importantly, the error fluctuation of the multi-modal fusion is very small, indicating that the present application has achieved remarkable results in enhancing the feature expression richness, improving the discrimination ability and system robustness, and also verifies the innovative contribution of the end-side adaptive fusion and multi-band differential entropy modeling strategy to the performance of emotional state recognition.

[0078] At this point, the entire process is complete.

[0079] In summary, the ubiquitous network multi-modal privacy data protection method provides a consistent time series data basis for different sensing channels by band-pass filtering, standardizing and sliding window slicing of multi-modal physiological signals; then, the ResNet with an ECA module is used to encode and attention-weighted fuse the frequency band differential entropy features in each signal window, which not only realizes adaptive weight distribution of discriminative frequency bands and modalities, but also implicitly smooths the noise difference between channels, improving the robustness and discrimination ability of feature representation. The role of this method is to convert multi-source data into a high-dimensional and compact fusion vector at the local end, which not only reduces the computational burden of the back end, but also avoids the risk of exporting the original data, thereby achieving the effect of improving the model training efficiency while enhancing the privacy security.

[0080] The ubiquitous network multi-modal privacy data protection method effectively masks the influence of single client data by injecting Laplace noise into the backward parameter difference after local model training; then, based on the Groth16 protocol, non-interactive zero-knowledge proof is generated for the noise-added difference to prove its update legality without leaking the noise component; finally, after the trusted third party verifies the timeliness and legality of the proof, the noise-added difference is randomly scrambled and securely forwarded to the central server. The role and purpose of this process is to provide mathematically provable privacy protection and security assurance for each parameter interaction, while avoiding the communication overhead of interactive protocols. Ultimately, this combined scheme achieves the effect of preventing data reverse attack and providing independent third-party audit in the parameter update process while maintaining the performance of federated learning models.

[0081] It should be noted that, in this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0082] While embodiments of the application have been shown and described, it is to be understood that the application is not limited to the details of the embodiments described, since numerous changes can be made to the embodiments without departing from the spirit and scope of the application as defined by the appended claims and their equivalents.

Claims

1. A method for protecting multimodal privacy data in ubiquitous networks, characterized in that, include: S1: Collect multimodal physiological signals and perform bandpass filtering, standardization, and sliding window slicing to generate a time-series data window of uniform scale; S2: Extract and fuse frequency band features from the time-series data window to construct comprehensive discrimination features; S3: Perform local model training on the comprehensive discriminative features, calculate the difference between the parameters after local model training and the current global model parameters, and add noise to the difference; S4: Based on the Groth16 protocol and public reference strings, generate a non-interactive zero-knowledge proof for the noise-added difference, and upload the noise-added parameter difference and the corresponding non-interactive zero-knowledge proof to a trusted third party. S5: The trusted third party verifies the non-interactive zero-knowledge proof. After the verification is passed, the difference after adding noise is randomly obfuscated, and the obfuscated difference is forwarded to the central server. S6: The central server performs a weighted average of the differences, updates the global model parameters, and distributes the updated global model parameters to each client to enter the next round of federated training iteration; S2 includes: S21: For each timing data window of each mode, extract four sub-frequency band signals using a bandpass filter according to a preset frequency band; S22: Calculate the differential entropy independently for the four sub-band signals to obtain a four-dimensional differential entropy feature vector; S23: For each time series data window, its 4-dimensional differential entropy feature vector is mapped to 4 channels respectively, and then stitched together according to the electrode space layout to form a multi-channel 2D feature map. S24: Input the 2D feature map into an ECA-based ResNet encoder to obtain a high-dimensional feature vector; S25: Perform pooling operation on the high-dimensional feature vectors of all windows under the same modality to obtain the overall features of the modality, and then fuse the overall features of each modality through an attention mechanism to obtain a comprehensive discriminative feature vector; The S5 includes: S51: The trusted third party receives the non-interactive zero-knowledge proof and difference uploaded by the client, records the receiving timestamp, and compares it with the carried timestamp. It can only continue if the time difference between the two does not exceed a preset threshold; otherwise, it is rejected. S52: Verify the zero-knowledge proof using a pre-set verification key, and verify the correctness of each component of the proof in accordance with the Groth16 protocol requirements; S53: After successful verification, the trusted third party generates a random challenge message and sends it to the client. After the client returns a confirmation code, it confirms that the proof was generated in real time within the trusted time window. S54: The noise-added difference is randomly obfuscated by random permutation and then forwarded to the central server.

2. The method for protecting multimodal privacy data in ubiquitous networks according to claim 1, characterized in that: The multimodal physiological signals include electroencephalogram (EEG), electrooculogram (EOG), and electromyogram (EMG) signals.

3. The method for protecting multimodal privacy data in ubiquitous networks according to claim 1, characterized in that, The addition of noise to the difference specifically involves generating random noise based on a Laplace distribution and superimposing it onto the difference.

4. The method for protecting multimodal privacy data in ubiquitous networks according to claim 3, characterized in that, The process of generating a non-interactive zero-knowledge proof for the noisy difference based on the Groth16 protocol and public reference strings includes: The client loads the public reference string of the Groth16 protocol and uses a pre-set proof key to generate a commitment to the noisy difference, while binding the client's identity identifier with the current timestamp; Perform a hash operation on the commitment and its associated parameters to generate a unique commitment identifier; Using the noised parameter difference and commitment identifier as input, the Groth16 algorithm is used to perform polynomial commitment calculation on the proof circuit to generate a zero-knowledge proof.

Images