Ubiquitous network multi-mode privacy data protection method

Abstract

The invention relates to the technical field of privacy data protection, and discloses a ubiquitous network multi-mode privacy data protection method, which comprises the following steps that: a client performs band-pass filtering, standardization and sliding window slice preprocessing on multi-mode physiological signals such as electroencephalogram, electro-oculogram and myoelectricity; extracting and fusing frequency band discrimination features from the obtained time sequence data window according to a preset frequency band; model training is executed locally, a parameter updating difference value is calculated, and differential privacy noise is injected; generating a non-interactive zero-knowledge proof based on a Groth16 protocol, and uploading the noise-containing difference value and the proof to a trusted third party; after the credible third party passes verification, the difference values are confused and forwarded to the central server; and the central server updates the global model according to the sample weight aggregation and issues the global model to the client. According to the method, efficient cooperation of multi-modal signal end-side adaptive fusion and distributed privacy protection is realized, and security and credibility of parameter interaction are guaranteed through a non-interactive proving mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of privacy data protection, and in particular to a ubiquitous network multimodal privacy data protection method. Background Art

[0002] In recent years, with the rapid evolution of ubiquitous network technologies such as the Internet of Things (IoT) and 5G / 6G, multimodal physiological signal acquisition and analysis, as a key research direction in human-computer interaction and health monitoring, has gradually shown a trend towards high-dimensional, multi-source, and real-time development. Traditional single-modality research using signals such as electroencephalogram (EEG), electrocardiogram (ECG), and electromyography (EMG) has achieved remarkable results in applications such as motor imagery, attention detection, and emotion recognition. Multimodal fusion technology, by collaboratively processing data from different signal sources in the time-frequency domain or deep feature space, further improves recognition accuracy and robustness. Federated learning, meanwhile, introduces a distributed training paradigm, enabling collaborative model optimization while preserving local data. Privacy-preserving techniques such as differential privacy and homomorphic encryption provide mathematically provable security guarantees for participating nodes. Furthermore, the maturity of non-interactive zero-knowledge proof (ZKP) technology makes it possible to verify the legitimacy of computational results without leaking the original data.

[0003] However, the existing technology still has several deficiencies in practical applications: First, the existing multimodal fusion relies on a central server to aggregate raw data or only cross-validates a single modality, which makes it difficult to balance the privacy of device-side data with wide temporal and spatial distribution and the fusion efficiency; Second, although the federated learning framework can reduce data transmission, in the process of model parameter interaction, parameter updates are often not accompanied by mandatory privacy protection and are vulnerable to reverse inference attacks; relying solely on differential privacy to add noise cannot simultaneously balance model accuracy and privacy budget, and lacks credible proof of the legitimacy of noise injection; Third, the existing zero-knowledge proof schemes are mostly used in interactive scenarios or have complex computing circuit design and high communication overhead, making them difficult to deploy efficiently in a ubiquitous network multi-node heterogeneous environment. Based on the above deficiencies, the present invention combines the fine division of multimodal frequency band features with adaptive fusion, the organic combination of federated learning and differential privacy, and a non-interactive zero-knowledge proof mechanism based on the Groth16 protocol, which can not only generate highly discriminative comprehensive feature vectors and perform local training on the client side, but also perform verifiable privacy protection on parameter updates, significantly improving the privacy security and collaborative learning efficiency of the system. Summary of the Invention

[0004] (1) Technical problems solved: In response to the shortcomings of the existing technology, the present invention provides a ubiquitous network multimodal privacy data protection method, which has the advantages of combining end-side adaptive fusion of multimodal features, distributed differential privacy noise injection, and non-interactive zero-knowledge proof verification. It solves the problems in the existing technology that it is difficult to balance model performance and data privacy protection, and the security and legality of parameter updates cannot be verified.

[0005] (II) Technical Solution: To achieve the aforementioned combination of adaptive multimodal feature fusion on the device side, distributed differential privacy noise injection, and non-interactive zero-knowledge proof verification, the present invention provides the following technical solution: a method for protecting multimodal privacy data in a ubiquitous network, characterized by comprising: S1: Collect multimodal physiological signals and perform bandpass filtering, normalization, and sliding window slicing on them to generate a time series data window of uniform scale; S2: extracting and fusing frequency band features of the time series data window to construct comprehensive discriminant features; S3: performing local model training on the comprehensive discriminant features, calculating the difference between the trained local model parameters and the current global model parameters, and adding noise to the difference; S4: Based on the Groth16 protocol and the public reference string, generate a non-interactive zero-knowledge proof for the noised difference, and upload the noised parameter difference and the corresponding non-interactive zero-knowledge proof to a trusted third party. S5: The trusted third party verifies the non-interactive zero-knowledge proof. After verification, the trusted third party randomly obfuscates the noise-added difference and forwards the obfuscated difference to the central server. S6: The central server updates the global model parameters after performing weighted averaging on the differences, and distributes the updated global model parameters to each client to enter the next round of federated training iteration.

[0006] Furthermore, the multimodal physiological signal data includes electroencephalogram (EEG), electrooculogram (EOG), and electromyography (EMG) signal data.

[0007] Furthermore, the S2 includes: S21: For each time series data window of each modality, extract four sub-band signals respectively using a bandpass filter according to a preset frequency band; S22: Calculate differential entropy independently for the four sub-band signals to obtain a four-dimensional differential entropy feature vector; S23: For each time series data window, its 4-dimensional differential entropy feature vector is mapped into 4 channels respectively, and then spliced ​​into a multi-channel 2D feature map according to the electrode spatial layout; S24: Input the 2D feature map into an ECA-based ResNet encoder to obtain a high-dimensional feature vector; S25: Pooling operation is performed on the high-dimensional feature vectors of all windows under the same modality to obtain the overall features of the modality, and then the overall features of each modality are fused through the attention mechanism to obtain a comprehensive discriminant feature vector.

[0008] Furthermore, the adding noise to the difference is specifically: generating random noise based on Laplace distribution and superimposing the noise onto the difference.

[0009] Furthermore, generating a non-interactive zero-knowledge proof for the noised difference based on the Groth16 protocol and the public reference string includes: The client loads the public reference string of the Groth16 protocol and uses the preset proof key to generate a commitment to the noised difference, while binding the client identity to the current timestamp; Performing a hash operation on the commitment and its associated parameters to generate a unique commitment identifier; The noisy parameter difference and commitment identifier are used as input, and the polynomial commitment calculation is completed on the proof circuit according to the Groth16 algorithm to generate a zero-knowledge proof.

[0010] Furthermore, the S5 includes: S51: The trusted third party receives the non-interactive zero-knowledge proof and the difference uploaded by the client, records the receiving timestamp, and compares it with the carried timestamp. The process is allowed only if the time difference between the two does not exceed a preset threshold; otherwise, the process is rejected. S52: Use the preset verification key to verify the zero-knowledge proof and verify the correctness of each component according to the Groth16 protocol requirements; S53: After verification, the trusted third party generates a random challenge message and sends it to the client. After the client returns a confirmation code, it confirms that the proof was generated in real time within the trusted time window; S54: The noisy difference is randomly obfuscated by random permutation and forwarded to the central server.

[0011] (III) Beneficial Effects: Compared with the prior art, the present invention provides a ubiquitous network multimodal privacy data protection method with the following beneficial effects: 1. This ubiquitous network multimodal privacy data protection method provides a consistent time series data foundation for different sensor channels by bandpass filtering, normalizing, and sliding window slicing multimodal physiological signals. Subsequently, a ResNet with an ECA module encodes the frequency band differential entropy features within each signal window and fuses them with attention weighting. This not only achieves adaptive weight allocation for discriminative frequency bands and modalities, but also implicitly smooths noise differences between channels, improving the robustness and discriminative ability of feature representation. This method converts multi-source data into high-dimensional and compact fusion vectors locally, reducing the backend computational burden and avoiding the risk of external transmission of the original data, thereby improving model training efficiency while enhancing privacy and security.

[0012] 2. This ubiquitous network multimodal privacy data protection method effectively masks the influence of individual client data by injecting Laplace noise into the parameter differences after local model training. Next, a non-interactive zero-knowledge proof is generated for the noisy differences based on the Groth16 protocol, proving the legitimacy of the update without leaking the noise component. Finally, a trusted third party verifies the timeliness and legitimacy of the proof before randomly obfuscating the noisy differences and securely forwarding them to a central server. The purpose of this process is to provide mathematically provable privacy protection and security and reliability guarantees for each parameter interaction, while avoiding the communication overhead of the interactive protocol. Ultimately, this combined solution achieves the goal of ensuring that the parameter update process is both resistant to data inversion attacks and accessible to independent third-party audits, while maintaining the performance of the federated learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The overall flow chart of a ubiquitous network multimodal privacy data protection method provided by the present invention; Figure 2 A schematic diagram of a ubiquitous network multimodal privacy data protection method provided by the present invention; Figure 3 A flowchart of sub-step S2 of a ubiquitous network multimodal privacy data protection method provided by the present invention; Figure 4 A flowchart of sub-step S5 of a ubiquitous network multimodal privacy data protection method provided by the present invention; Figure 5 A comparison chart of experimental results of a ubiquitous network multimodal privacy data protection method provided by the present invention. DETAILED DESCRIPTION

[0014] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0015] See also Figure 1-2 A method for protecting multimodal privacy data in a ubiquitous network, comprising: S1: Collect multimodal physiological signals and perform bandpass filtering, normalization, and sliding window slicing on them to generate a time series data window of uniform scale; S2: extracting and fusing frequency band features of the time series data window to construct comprehensive discriminant features; S3: performing local model training on the comprehensive discriminant features, calculating the difference between the trained local model parameters and the current global model parameters, and adding noise to the difference; S4: Based on the Groth16 protocol and the public reference string, generate a non-interactive zero-knowledge proof for the noised difference, and upload the noised parameter difference and the corresponding non-interactive zero-knowledge proof to a trusted third party. S5: The trusted third party verifies the non-interactive zero-knowledge proof. After verification, the trusted third party randomly obfuscates the noise-added difference and forwards the obfuscated difference to the central server. S6: The central server updates the global model parameters after performing weighted averaging on the differences, and distributes the updated global model parameters to each client to enter the next round of federated training iteration.

[0016] Specifically, in S1, the client uses wearable or fixed sensor modules to synchronously collect multimodal physiological signals, including but not limited to electroencephalogram (EEG), electrooculogram (EOG), and electromyography (EMG). The analog voltage signals output by each sensor are first subjected to wideband bandpass filtering by front-end hardware to remove DC components and high-frequency noise. The buffer dynamically adjusts the filter order based on the application scenario to balance real-time performance and filter steepness. The filtered continuous signal is sampled by an analog-to-digital converter (ADC) to produce multi-channel real-valued time series data. After sampling, batch normalization is performed on each channel at the software level. This involves subtracting the mean and dividing by the standard deviation within a sliding window to eliminate amplitude variations and accelerate subsequent model convergence. The normalized multi-channel time series is then sliced ​​into sliding windows of fixed length (e.g., 1 or 2 seconds, corresponding to 250 or 500 samples) with 50% overlap, forming a matrix structure of [number of channels × window length]. Each matrix represents a "uniformly scaled time series data window," providing uniformly scaled input data for frequency band feature extraction and fusion in S2.

[0017] For further information, see Figure 3 , said S2, comprising: S21: For each time series data window of each modality, extract four sub-band signals respectively using a bandpass filter according to a preset frequency band; Specifically, for each time series data window of each modality, the system first decomposes the wide frequency band (0.5–30 Hz) of the original signal into 、 、 and .

[0018] S22: Calculate differential entropy independently for the four sub-band signals to obtain a four-dimensional differential entropy feature vector; Specifically, the system extracts the For the four sub-band signals, first calculate the sample variance of all sampling points in the corresponding time series data window , and then according to the analytical formula of Gaussian distribution differential entropy: ; Obtain the differential entropy value of each sub-band respectively, according to The fixed order of the two signals is arranged to form a four-dimensional differential entropy feature vector. This method relies solely on variance information to complete the entropy calculation, which not only reduces the computational complexity but also effectively characterizes the randomness and information content of the signals in each frequency band.

[0019] S23: For each time series data window, its 4-dimensional differential entropy feature vector is mapped into 4 channels respectively, and then spliced ​​into a multi-channel 2D feature map according to the electrode spatial layout; Specifically, the system first considers the four-dimensional differential entropy feature vector corresponding to each time series data window as the initial pixel value of the four channels. windows, whose entropy Corresponding respectively Four frequency bands. The system fills the entropy value of each frequency band channel at the corresponding electrode position on a pre-defined electrode space grid (such as a 32×32 or 64×64 two-dimensional matrix); for the blank areas between the electrodes in the grid, bilinear interpolation or Gaussian kernel interpolation is used to smoothly expand the entropy information to maintain spatial continuity. After interpolation, four single-channel matrices with the same grid size are obtained. Finally, these four matrices are stacked in the channel dimension to form a 4-channel multi-channel 2D feature map. This feature map not only retains the entropy value characteristics of each frequency band, but also reflects the relative spatial relationship between different channels through electrode space mapping.

[0020] S24: Input the 2D feature map into an ECA-based ResNet encoder to obtain a high-dimensional feature vector; Specifically, the system takes the four-channel 2D feature map obtained in step S23 as input and feeds it into an encoder based on a residual network (ResNet). An efficient channel attention (ECA) module is integrated after each residual unit to adaptively enhance important channel features. The encoder first extracts spatial patterns through a series of convolution, batch normalization, and ReLU activation layers. After each residual block, the ECA module first performs global average pooling on the current feature map to generate a channel descriptor. It then uses one-dimensional convolution to capture the interaction between channels, outputs channel attention weights, and multiplies them element-wise with the original feature map to achieve dynamic gain of discriminative frequency band channels. As the network layer deepens, the residual connection ensures the stable flow of gradients, and ECA improves feature selectivity. After the last residual stage, the encoder performs global average pooling on the feature map, aggregating the spatial dimensions into a high-dimensional vector as the final feature representation of the window.

[0021] S25: Pooling operation is performed on the high-dimensional feature vectors of all windows under the same modality to obtain the overall features of the modality, and then the overall features of each modality are fused through the attention mechanism to obtain a comprehensive discriminant feature vector.

[0022] Specifically, the system first performs a pooling operation on the high-dimensional feature vectors of all windows under the same modality to obtain the overall feature representation of the modality. Pooling can be done in two ways: one is global average pooling, which is to obtain the average vector by averaging all vectors dimension by dimension; the other is attention-weighted pooling, which is to generate a scalar weight for each vector through a small attention network, and after softmax normalization, all vectors are fused in the form of a weighted sum to ensure that more discriminative window fragments receive higher weights. The resulting overall modal features not only gather the key information of the signal source in the time dimension, but also enhance the influence of important windows through the attention mechanism. Subsequently, the system first calculates the similarity matrix of the feature vectors between the modalities, then applies the corresponding attention weight to each modal vector, and finally adds the weighted modal vectors to generate a unified comprehensive discriminant feature vector.

[0023] Furthermore, in S3, the client first receives and loads the latest global model parameters from the central server to initialize the local model; then, using the comprehensive discriminant feature vector obtained in step S2 as input, it performs multiple rounds of iterative training on the local private data using the cross entropy loss function and the Adam optimizer. After completing the scheduled training rounds, the client calculates the difference between the parameters before and after the local model update, that is, subtracting the global model parameters at the time of loading from the trained model parameters to obtain the original parameter difference vector In the present invention, in order to satisfy Differential privacy requires that the system Each component in is injected with Laplace noise, and the noise is generated according to the Laplace distribution, so as to obtain the noisy update vector: .

[0024] In the formula, represents random noise sampled from a zero-mean Laplace distribution; The update sensitivity is defined as the maximum norm change of a single client model update in any case, which is used to determine the noise scale; It is the privacy budget that controls the noise intensity. The smaller the value, the greater the noise and the stronger the privacy protection, but at the same time the greater the impact on model performance.

[0025] Furthermore, in S4, generating a non-interactive zero-knowledge proof for the noise-added difference based on the Groth16 protocol and the public reference string includes: The client loads the public reference string of the Groth16 protocol and uses the preset proof key to generate a commitment to the noised difference, while binding the client identity to the current timestamp; Performing a hash operation on the commitment and its associated parameters to generate a unique commitment identifier; The noisy parameter difference and commitment identifier are used as input, and the polynomial commitment calculation is completed on the proof circuit according to the Groth16 algorithm to generate a zero-knowledge proof.

[0026] Specifically, the client first loads the Groth16 protocol's public reference string (CRS) and the pre-set proof key , then for the client identity and timestamp , and generate polynomial commitments together : ;In the formula, It is the group generator and the base point of the elliptic curve used in the Groth16 protocol; They are the system parameters used to construct the polynomials of the key circuit, which are derived from CRS; Is the proof key, the key fragment disclosed by the client to the system for commitment calculation; It is the client identity, used to bind the commitment to a specific client; It is a timestamp used to prove the timeliness of the binding and prevent replay attacks. Calculate the hash of its associated circuit parameters to obtain a unique commitment identifier : ; Finally, according to the Groth16 algorithm framework, Random Numbers and As a witness input proof circuit, perform polynomial commitment calculation and generate proof elements A, B, C, combined into a non-interactive zero-knowledge proof : .

[0027] Then the noisy parameter difference and the corresponding non-interactive zero-knowledge proof are uploaded to a trusted third party.

[0028] Further, such as Figure 4 As shown, the S5 includes: S51: The trusted third party receives the non-interactive zero-knowledge proof and the difference uploaded by the client, records the receiving timestamp, and compares it with the carried timestamp. The process is allowed only if the time difference between the two does not exceed a preset threshold; otherwise, the process is rejected. Specifically, the trusted third party first receives the non-interactive zero-knowledge proof uploaded by the client Update the difference vector with the corresponding noise and immediately record the local reception timestamp . Subsequently, the system proves Extract the timestamp bound by the client , and calculate the time difference between the two .

[0029] Only when Less than or equal to the preset maximum time offset threshold Otherwise, the trusted third party will regard the upload request as a possible replay attack and refuse to process it, while recording the exception log for auditing.

[0030] S52: Use the preset verification key to verify the zero-knowledge proof and verify the correctness of each component according to the Groth16 protocol requirements; Specifically, first, load the verification key issued during the Common Reference String (CRS) generation phase Next, perform a bilinear mapping check on the received proof elements A, B, and C and their corresponding commitment values, verifying that the following relationship holds: ;In the formula, represents the bilinear mapping operation, and is the group generator in CRS, and To verify the key, the corresponding circuit polynomials are and Only when the above mapping equations are simultaneously true does the trusted third party deem the proof valid; otherwise, it is deemed invalid and the upload request is rejected. Furthermore, the verification process includes an integrity check of the commitment identifier and timestamp to ensure that the proof is consistent with the binding information previously generated in S4, further eliminating the risk of tampering or replay attacks.

[0031] S53: After verification, the trusted third party generates a random challenge message and sends it to the client. After the client returns a confirmation code, it confirms that the proof was generated in real time within the trusted time window; Specifically, after completing the zero-knowledge proof legitimacy verification, the trusted third party will use a secure random number generator to generate a random challenge value of sufficient length. , and record the challenge time at this time ; Then challenge the Together with the digital signature of a trusted third party, it is constructed into a challenge message and sent to the client through an encrypted channel. After receiving the challenge, the client first verifies the legitimacy of the signature and then uses its local private key or pre-shared key to respond to the challenge value within the specified response window. and timestamp Sign or calculate the message authentication code, generate the confirmation code ACK, and send it together with the client identity and response timestamp After receiving the ACK, the trusted third party first verifies whether its signature or message authentication code matches the original challenge value, and then checks the response time. and challenge time The difference between the two is less than the preset maximum response delay threshold; only when both checks are passed can the attestation activity be deemed to be generated in real time within the trusted time window, thereby further preventing attacks such as delayed replay or network relay.

[0032] S54: randomly obfuscate the noisy difference by random permutation and forward it to the central server; Specifically, after completing the challenge-response confirmation, the trusted third party will perform random permutations on the noisy update difference vector to confuse the order of its elements, thereby disrupting the correspondence between the parameters and reducing the risk of reverse analysis of a single update component. When implementing, the system first uses a secure random number generator to generate a seed , and based on this, a random permutation index sequence with the same length as the vector is constructed. Subsequently, the components in the vector are rearranged according to the index to obtain the obfuscation vector. After the permutation is completed, the trusted third party forwards the obfuscation vector to the central server and records the permutation seed locally. This method not only maintains the differential privacy protection of the original update, but also further improves the anti-reversal security of the parameter upload process, ensuring that the server cannot directly link back to the detailed update information of any single client during aggregation.

[0033] Furthermore, in S6, the central server first receives the obfuscated difference vector and its corresponding client identifier and sample size forwarded by the trusted third party through a secure channel. In order to update the global model, the server calculates the sample size of each client. and the total number of samples Calculate the weighted mean difference: ;In the formula, Represents the server's overall representation of the model parameter increment after this round of aggregation; Is the client index, indicating the number of clients participating in this round of federated training. clients; It is The number of data samples used by a client for local training; is the sum of the number of samples of all participating clients; It is The noisy update difference vector is calculated and uploaded by each client. The weighted average result is then applied to the current global model parameters. To balance system flexibility and communication efficiency, the server randomly selects a subset of all available clients to participate in each round of aggregation. Unselected clients can continue to use the previously distributed global model for local training. Finally, the updated global model is distributed to each client via an encrypted distribution mechanism, initiating the next round of federated training iterations.

[0034] The embodiments of the present invention have achieved some positive results during the development or use process, and indeed have great advantages over the existing technology. The following content describes them in conjunction with data, charts, etc. from the experimental process.

[0035] See also Figure 5 The proposed technical solution, which constructs multi-channel 2D feature maps based on band-wise differential entropy and combines it with ECA-ResNet adaptive attention fusion, fully demonstrates its innovation and practical value. The orange and green bars in the figure represent the classification accuracy using only EEG and a trimodal fusion of EEG, EOG, and EMG input, respectively. The horizontal axis represents the emotional dimensions of arousal and valence, while the vertical axis represents the accuracy percentage. As can be seen, in the arousal recognition task, the accuracy of the traditional EEG single-modality method is approximately 97%, while the accuracy is improved to 99% after the trimodal fusion of the present invention. For valence recognition, the accuracy of the EEG single-modality method is approximately 98%, while the fusion method approaches 100%. More importantly, the error fluctuation after multimodal fusion is extremely minimal, demonstrating the significant achievements of the present invention in enhancing the richness of feature representation, improving discriminative ability, and enhancing system robustness. This also verifies the innovative contribution of the on-device adaptive fusion and multi-band differential entropy modeling strategy to the performance of emotional state recognition.

[0036] At this point, the entire process ends.

[0037] In summary, this ubiquitous network multimodal privacy data protection method provides a consistent time series data foundation for different sensor channels by bandpass filtering, normalizing, and sliding window slicing multimodal physiological signals. Subsequently, the frequency band differential entropy features within each signal window are encoded and fused with attention weighting through a ResNet with an ECA module. This not only achieves adaptive weight allocation for discriminative frequency bands and modalities, but also implicitly smooths noise differences between channels, improving the robustness and discriminative ability of feature representation. This method converts multi-source data into high-dimensional and compact fusion vectors locally, reducing the backend computational burden and avoiding the risk of external transmission of the original data, thereby achieving the effect of improving model training efficiency while enhancing privacy and security.

[0038] This ubiquitous network multimodal privacy data protection method effectively masks the influence of individual client data by injecting Laplace noise into the parameter differences after local model training. Next, a non-interactive zero-knowledge proof is generated for the noisy differences based on the Groth16 protocol, proving the legitimacy of the update without leaking the noise component. Finally, a trusted third party verifies the timeliness and legitimacy of the proof before randomly obfuscating the noisy differences and securely forwarding them to a central server. The purpose of this process is to provide mathematically provable privacy protection and security and credibility guarantees for each parameter interaction, while avoiding the communication overhead of the interactive protocol. Ultimately, this combined solution achieves the goal of ensuring that the parameter update process is both protected against data inversion attacks and can be independently audited by a third party, while maintaining the performance of the federated learning model.

[0039] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0040] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A ubiquitous network multimodal privacy data protection method, characterized in that: include: S1: Collect multimodal physiological signals and perform bandpass filtering, normalization, and sliding window slicing on them to generate a time series data window of uniform scale; S2: extracting and fusing frequency band features of the time series data window to construct comprehensive discriminant features; S3: performing local model training on the comprehensive discriminant features, calculating the difference between the trained local model parameters and the current global model parameters, and adding noise to the difference; S4: Based on the Groth16 protocol and the public reference string, generate a non-interactive zero-knowledge proof for the noised difference, and upload the noised parameter difference and the corresponding non-interactive zero-knowledge proof to a trusted third party. S5: The trusted third party verifies the non-interactive zero-knowledge proof. After verification, the trusted third party randomly obfuscates the noise-added difference and forwards the obfuscated difference to the central server. S6: The central server updates the global model parameters after performing weighted averaging on the differences, and distributes the updated global model parameters to each client to enter the next round of federated training iteration.

2. The method for protecting multimodal privacy data in a ubiquitous network according to claim 1, characterized in that: The multimodal physiological signal data includes electroencephalogram (EEG), electrooculogram (EOG) and electromyography (EMG) signal data.

3. A ubiquitous network multimodal privacy data protection method according to claim 2, characterized in that: Said S2 comprises: S21: For each time series data window of each modality, extract four sub-band signals respectively using a bandpass filter according to a preset frequency band; S22: Calculate differential entropy independently for the four sub-band signals to obtain a four-dimensional differential entropy feature vector; S23: For each time series data window, its 4-dimensional differential entropy feature vector is mapped into 4 channels respectively, and then spliced ​​into a multi-channel 2D feature map according to the electrode spatial layout; S24: Input the 2D feature map into an ECA-based ResNet encoder to obtain a high-dimensional feature vector; S25: Pooling operation is performed on the high-dimensional feature vectors of all windows under the same modality to obtain the overall features of the modality, and then the overall features of each modality are fused through the attention mechanism to obtain a comprehensive discriminant feature vector.

4. A ubiquitous network multimodal privacy data protection method according to claim 3, characterized in that: The adding noise to the difference is specifically: generating random noise based on Laplace distribution and superimposing the noise onto the difference.

5. A ubiquitous network multimodal privacy data protection method according to claim 4, characterized in that: Generating a non-interactive zero-knowledge proof for the noised difference based on the Groth16 protocol and the public reference string includes: The client loads the public reference string of the Groth16 protocol and uses the preset proof key to generate a commitment to the noised difference, while binding the client identity to the current timestamp; Performing a hash operation on the commitment and its associated parameters to generate a unique commitment identifier; The noisy parameter difference and commitment identifier are used as input, and the polynomial commitment calculation is completed on the proof circuit according to the Groth16 algorithm to generate a zero-knowledge proof.

6. The method for protecting multimodal privacy data in a ubiquitous network according to claim 5, characterized in that: Said S5 comprises: S51: The trusted third party receives the non-interactive zero-knowledge proof and the difference uploaded by the client, records the receiving timestamp, and compares it with the carried timestamp. The process is allowed only if the time difference between the two does not exceed a preset threshold; otherwise, the process is rejected. S52: Use the preset verification key to verify the zero-knowledge proof and verify the correctness of each component according to the Groth16 protocol requirements; S53: After verification, the trusted third party generates a random challenge message and sends it to the client. After the client returns a confirmation code, it confirms that the proof was generated in real time within the trusted time window; S54: The noisy difference is randomly obfuscated by random permutation and forwarded to the central server.

Images