A privacy protection model lightweight training method and system in a multi-modal missing scene

Abstract

This invention discloses a lightweight training method and system for privacy-preserving models in multimodal missing scenarios, belonging to the fields of artificial intelligence and information security technology. The method first collects raw multimodal data and stores it on a local client using a federated learning framework. To address the modality missing problem, a learnable virtual modality generator based on a cross-modal attention mechanism is designed. This generator dynamically generates pseudo-feature representations of missing modalities using features from existing modalities, rather than simple data imputation. A differential privacy mechanism is introduced, with noise intensity dynamically adjusted based on the current training epoch and gradient sensitivity. This invention solves the problems of severe performance degradation in multimodal missing scenarios, privacy leakage risks in centralized training, and difficulties in deployment on edge devices. While ensuring user privacy and model accuracy, it significantly reduces the model's storage and computational overhead, making it suitable for privacy-sensitive and resource-constrained scenarios such as medical diagnosis and intelligent vehicle systems.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and information security technology, specifically to a lightweight training method and system for privacy protection models in multimodal missing scenarios. Background Technology

[0002] With the deepening application of artificial intelligence technology in fields such as autonomous driving, smart healthcare, and industrial IoT, multimodal learning has become the mainstream paradigm for improving the accuracy of model decision-making due to its significant advantages in information complementarity. Ideally, multimodal models, by fusing heterogeneous data from multiple sources such as vision, hearing, text, and sensors, can achieve more robust inference performance than single-modal models. However, real-world application scenarios are highly uncertain and complex, and modality loss is extremely common. Specifically, in mobile edge computing scenarios, due to hardware cost constraints, low-end devices are often equipped with only a single type of sensor; in healthcare scenarios, patients may lose facial expression or voice data due to privacy concerns or operational errors; in autonomous driving scenarios, severe weather (such as heavy rain or dense fog) can cause camera visual information to fail. Traditional multimodal fusion models (such as architectures based on tensor fusion or late fusion) are usually built on the strong assumption of "modal completeness." Once a key modality is missing, the model's input dimension will shift, causing the internal feature alignment mechanism to fail, leading to a precipitous drop in performance. Although existing solutions attempt to fill in missing data using mean padding, zero padding, or simple generative adversarial networks, these methods either ignore the deep semantic relationships between modalities, leading to distorted feature distribution, or introduce excessive computational overhead, failing to meet the real-time requirements of edge devices.

[0003] To train high-performance multimodal models, massive amounts of user data are typically collected for centralized training. However, multimodal data often carries strong personal biometrics and sensitive behavioral information (e.g., medical images combined with electronic medical records, or driver facial recognition combined with voice commands in smart cockpits). This centralized data processing model faces severe privacy compliance challenges and struggles to meet the requirements of regulations such as the General Data Protection Regulation (GDPR). Federated learning, as a distributed training framework that ensures "data does not leave the domain," has alleviated the problem of raw data leakage to some extent, but recent research indicates that it still has serious privacy vulnerabilities. Attackers can analyze model gradients or updated parameters uploaded by clients to launch gradient inversion attacks or model inversion attacks, thereby reconstructing the user's original training samples (such as facial images or private conversations) with high precision. Especially in multimodal scenarios, the correlation between different modalities may provide attackers with more dimensions of reasoning clues, multiplying the risk of privacy leakage. Therefore, how to further enhance the privacy protection strength in the federated learning process and prevent gradient leakage in complex environments where modalities are missing is another major bottleneck currently facing the technology.

[0004] With the exponential growth of deep learning model parameters (such as large-scale vision-language pre-trained models), the storage requirements and computational complexity of multimodal models have reached alarming levels. However, the terminals where multimodal applications are deployed are mostly resource-constrained edge devices, such as smartphones, wearable medical devices, and embedded vehicle controllers. These devices typically have strict memory limitations, low computing power, and limited battery life. Directly deploying large multimodal models from the server to the terminal not only leads to excessively high inference latency and slow response speed, but also significantly increases energy consumption and may even cause the device to overheat and crash. Existing model compression techniques (such as pruning, quantization, and knowledge distillation), while aiming to reduce model complexity, often perform poorly in scenarios with missing multimodalities. This is because traditional compression methods are usually optimized for complete modal inputs, and when faced with dynamically changing missing modalities, the compressed lightweight model is prone to overfitting or decreased generalization ability. Therefore, how to efficiently train and deploy lightweight models while ensuring the protection of privacy and the availability of missing modalities is a pressing problem that needs to be solved by both academia and industry.

[0005] To address this, we propose a lightweight training method and system for privacy-preserving models in multimodal missing scenarios. Summary of the Invention

[0006] To achieve the above objectives, the present invention provides the following technical solution: a lightweight training method for privacy-preserving models in multimodal missing scenarios, comprising the following steps:

[0007] Step S1: System initialization and parameter configuration: Construct a federated learning system containing a central server and multiple edge clients. Initialize a global teacher model on the central server. The global teacher model contains multiple modal encoders and cross-modal fusion modules. Set the differential privacy budget parameters (ϵ,δ) and model compression hyperparameters.

[0008] Step S2: Local Modal Integrity Detection and Feature Extraction: Each client obtains a local multimodal dataset. For each training batch, it checks whether there are missing modalities in the input sample's modal set. If there are missing modalities, it activates the cross-modal virtual feature generator to dynamically generate virtual feature vectors for the missing modalities using existing modal features. If the modalities are complete, it directly extracts the real feature vectors of each modality.

[0009] Step S3: Privacy-preserving local model training: Input the fused feature vector into the local model for forward propagation and calculate the task loss function; perform backpropagation to obtain the model gradient, and perform adaptive differential privacy noise injection on the gradient to generate a noisy gradient;

[0010] Step S4: Secure gradient aggregation and global update: Each client uploads the processed noisy gradient to the central server; the central server uses the federated averaging algorithm to perform weighted aggregation of gradients from all clients, updates the global teacher model parameters, and distributes the updated parameters to each client.

[0011] Step S5: Iterative training and convergence determination: Repeat steps S2 to S4 until the global teacher model converges or reaches the preset maximum number of communication rounds;

[0012] Step S6: Knowledge Distillation and Lightweight Model Conversion: Using the converged global teacher model as the knowledge source, the lightweight student model is trained by knowledge distillation online or offline; then, structured pruning and quantization-aware training are performed on the student model to generate the final deployable model.

[0013] Preferably, the cross-modal virtual feature generator in step S2 is constructed using a multi-head cross-attention mechanism, and its specific calculation process is as follows:

[0014] Let the set of feature maps for available modalities be Xavail={x1,x2,...,xm}, and the set of missing modalities be Ymiss={y1,y2,...,yn};

[0015] For any missing mode yj∈Ymiss, select one or more available modes xi∈Xavail as auxiliary modes, and generate a query matrix Qj=WQ⋅xi, a key matrix Kj=WK⋅xi and a value matrix Vj=WV⋅xi;

[0016] Calculate the attention score and generate a virtual feature y^j:

[0017] y^j=softmax(dkQjKjT)Vj

[0018] Wherein, WQ, WK, and WV are learnable linear projection parameters, and dk is the feature dimension; the virtual feature y^j is consistent with the real modal feature in terms of dimension and participates in subsequent fusion operations.

[0019] Preferably, the adaptive differential privacy noise injection operation in step S3 specifically includes:

[0020] Gradient clipping: The gradient gt calculated by the local model is clipped using the L2 norm to limit it to within a preset threshold C, i.e., gtclip=gt⋅min(1,|gt||2C);

[0021] Adaptive noise addition: Gaussian-distributed noise is added to the clipped gradient to generate the upload gradient gtupload.

[0022] gtupload=gtclip+N(0,(σt⋅C)2⋅I)

[0023] Where σt is the adaptive noise scaling factor for the t-th training round, and its value decreases monotonically with the increase of the training round t, satisfying σt=t+βσ0, where σ0 is the initial noise intensity and β is a small constant to prevent division by zero.

[0024] Preferably, the knowledge distillation process in step S6 employs a hybrid distillation strategy combining soft label matching and intermediate layer feature matching, and its total loss function is defined as:

[0025] Ltotal = αLtask + βLKL + γLHint

[0026] in:

[0027] Ltask is the cross-entropy loss of the student model for the target task;

[0028] LKL is the soft-label distillation loss. It calculates the KL divergence between the teacher model output zt and the student model output zs at the temperature coefficient T, i.e., LKL=T2⋅KL(softmax(zs / T),softmax(zt / T)).

[0029] LHint is the intermediate layer feature distillation loss, used to align the feature representations of the teacher model and the student model in the cross-modal fusion layer;

[0030] α, β, γ are the weighting coefficients for balancing the losses of each item.

[0031] Preferably, the quantization perception training step in step S6 specifically includes:

[0032] A pseudo-quantization operator is inserted into the model training graph to simulate the quantization error during inference.

[0033] For weight w and activation value a, the quantization function Q(⋅) is defined as:

[0034] Q(x)=sround(clip(x,xmin,xmax) / s),s=2b−1xmax−xmin

[0035] Where b is the target number of quantization bits (e.g., 8 bits) and s is the scaling factor; during training, the non-differentiable rounding function is bypassed by the pass-through estimator, and the floating-point parameters before quantization are updated by backpropagation.

[0036] Preferably, before step S2, a modal importance assessment step is included:

[0037] A modal contribution analysis model is pre-trained on the server side to calculate the Shapley value or mutual information of each modality under a specific task.

[0038] Based on the evaluation results, the generation weights of the cross-modal virtual feature generator are dynamically adjusted. For missing modalities that contribute significantly to the task, more generation resources are allocated or the confidence threshold of the generated features is increased.

[0039] Preferably, the method supports an asynchronous federated update mechanism:

[0040] It allows different clients to upload gradients at different frequencies based on their network conditions and computing power;

[0041] After receiving updates from a subset of clients, the central server immediately aggregates and distributes the model, without waiting for all clients to synchronize, thus reducing system stagnation caused by "lagging out" clients.

[0042] A lightweight training system for privacy-preserving models in multimodal missing scenarios includes:

[0043] The data management module, deployed on the client side, is used to collect, clean, and standardize multimodal raw data, and detect modality missing status.

[0044] The privacy computing engine, deployed on the client side, has a built-in cross-modal virtual feature generator, differential privacy noise generator, and local model trainer for performing privacy-preserving local model updates.

[0045] A federated aggregation server, deployed in the cloud or at the edge, is used to receive and aggregate encrypted gradients from multiple clients and maintain global model parameters.

[0046] The model compression and distillation unit receives the trained global teacher model and generates a lightweight student model through knowledge distillation, pruning, and quantization operations.

[0047] A secure communication bus is used to achieve two-way authentication and encrypted data transmission between clients and servers, preventing eavesdropping and tampering on the transmission link.

[0048] Preferably, the privacy computing engine adopts a heterogeneous computing architecture, which can dynamically schedule computing tasks according to the client hardware type, and prioritize the use of hardware acceleration units to perform cross-modal attention computing and convolution operations to reduce local training power consumption.

[0049] Compared with existing technologies, this invention provides a lightweight training method and system for privacy-preserving models in multimodal missing scenarios, which has the following beneficial effects:

[0050] 1. This invention presents a lightweight training method and system for privacy-preserving models in multimodal missing scenarios. By introducing a learnable "virtual modality generator" based on a multi-head cross-attention mechanism, it fundamentally overcomes the strong dependence of traditional multimodal models on input completeness. Unlike existing technologies that use static mean padding or zero padding leading to feature distribution shifts, this invention utilizes the inherent deep semantic relationships between modalities to dynamically generate high-dimensional feature representations of missing modalities. This "feature-level" completion method enables the model to capture complementary cross-modal information even in extreme scenarios where some sensors fail or data is selectively turned off by the user, thus maintaining the stability of inference performance. Experiments show that even with a random modality missing rate as high as 30%, the model determined by this invention still significantly improves key indicators such as classification accuracy compared to traditional baseline methods, greatly enhancing the system's practicality and robustness in complex open environments.

[0051] 2. This invention presents a lightweight training method and system for privacy-preserving models in multimodal missing scenarios. It innovatively integrates a federated learning architecture with an adaptive differential privacy mechanism, constructing a comprehensive privacy protection system from data acquisition and transmission to model updates. By performing gradient pruning and adaptive noise injection locally on the client side, it effectively resists gradient inversion attacks and membership inference attacks. In particular, the noise intensity attenuation strategy designed in this invention ensures a high level of privacy protection in the early stages of training while avoiding the negative impact of fixed high-intensity noise on the model's later convergence accuracy. Furthermore, the optional communication protocol, combining secure multi-party computation or homomorphic encryption, further ensures the confidentiality of gradients during transmission. This design perfectly aligns with the compliance requirements of privacy-sensitive fields such as healthcare and financial risk control, achieving the security goal of "data usable but not visible."

[0052] 3. This lightweight training method and system for privacy-preserving models in multimodal missing scenarios successfully addresses the industry pain point of deploying large multimodal models on resource-constrained devices by employing a "large model teacher - small model student" distillation architecture combined with structured pruning and QAT training. Through knowledge distillation, the "dark knowledge" fused across modalities in the teacher model is efficiently transferred to the lightweight student model, preserving the model's expressive power while significantly reducing the number of parameters. Subsequent pruning and QAT steps further optimize the model's computational graph and memory footprint, making it compatible with low-bit hardware accelerators. Compared to the original uncompressed model, the final model generated by this invention can be reduced in size by more than 70%, inference latency by more than 50%, and accuracy loss is kept within an acceptable range. This enables complex multimodal AI applications to run smoothly on edge devices such as mobile phones and embedded MCUs, significantly reducing bandwidth dependence and cloud inference costs. Detailed Implementation

[0053] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Example

[0055] An embodiment of a lightweight training method and system for privacy-preserving models in multimodal missing scenarios.

[0056] A lightweight training method for privacy-preserving models in multimodal missing scenarios includes the following steps:

[0057] Step S1: System initialization and parameter configuration: Construct a federated learning system containing a central server and multiple edge clients. Initialize a global teacher model on the central server. The global teacher model contains multiple modal encoders and cross-modal fusion modules. Set the differential privacy budget parameters (ϵ,δ) and model compression hyperparameters.

[0058] Step S2: Local Modal Integrity Detection and Feature Extraction: Each client obtains a local multimodal dataset. For each training batch, it checks whether there are missing modalities in the input sample's modal set. If there are missing modalities, it activates the cross-modal virtual feature generator to dynamically generate virtual feature vectors for the missing modalities using existing modal features. If the modalities are complete, it directly extracts the real feature vectors of each modality.

[0059] Step S3: Privacy-preserving local model training: Input the fused feature vector into the local model for forward propagation and calculate the task loss function; perform backpropagation to obtain the model gradient, and perform adaptive differential privacy noise injection on the gradient to generate a noisy gradient;

[0060] Step S4: Secure gradient aggregation and global update: Each client uploads the processed noisy gradient to the central server; the central server uses the federated averaging algorithm to perform weighted aggregation of gradients from all clients, updates the global teacher model parameters, and distributes the updated parameters to each client.

[0061] Step S5: Iterative training and convergence determination: Repeat steps S2 to S4 until the global teacher model converges or reaches the preset maximum number of communication rounds;

[0062] Step S6: Knowledge Distillation and Lightweight Model Conversion: Using the converged global teacher model as the knowledge source, the lightweight student model is trained by knowledge distillation online or offline; then, structured pruning and quantization-aware training are performed on the student model to generate the final deployable model.

[0063] Specifically, the cross-modal virtual feature generator in step S2 is constructed using a multi-head cross-attention mechanism, and its specific calculation process is as follows:

[0064] Let the set of feature maps for available modalities be Xavail={x1,x2,...,xm}, and the set of missing modalities be Ymiss={y1,y2,...,yn};

[0065] For any missing mode yj∈Ymiss, select one or more available modes xi∈Xavail as auxiliary modes, and generate a query matrix Qj=WQ⋅xi, a key matrix Kj=WK⋅xi and a value matrix Vj=WV⋅xi;

[0066] Calculate the attention score and generate a virtual feature y^j:

[0067] y^j=softmax(dkQjKjT)Vj

[0068] Where WQ, WK, and WV are learnable linear projection parameters, and dk is the feature dimension; the virtual feature y^j maintains the same dimension as the real modal feature and participates in subsequent fusion operations.

[0069] Specifically, the adaptive differential privacy noise injection operation in step S3 includes:

[0070] Gradient clipping: The gradient gt calculated by the local model is clipped using the L2 norm to limit it to within a preset threshold C, i.e., gtclip=gt⋅min(1,|gt||2C);

[0071] Adaptive noise addition: Gaussian-distributed noise is added to the clipped gradient to generate the upload gradient gtupload.

[0072] gtupload=gtclip+N(0,(σt⋅C)2⋅I)

[0073] Where σt is the adaptive noise scaling factor for the t-th training round, and its value decreases monotonically with the increase of the training round t, satisfying σt=t+βσ0, where σ0 is the initial noise intensity and β is a small constant to prevent division by zero.

[0074] Specifically, the knowledge distillation process in step S6 employs a hybrid distillation strategy combining soft label matching and intermediate layer feature matching, and its total loss function is defined as:

[0075] Ltotal = αLtask + βLKL + γLHint

[0076] in:

[0077] Ltask is the cross-entropy loss of the student model for the target task;

[0078] LKL is the soft-label distillation loss. It calculates the KL divergence between the teacher model output zt and the student model output zs at the temperature coefficient T, i.e., LKL=T2⋅KL(softmax(zs / T),softmax(zt / T)).

[0079] LHint is the intermediate layer feature distillation loss, used to align the feature representations of the teacher model and the student model in the cross-modal fusion layer;

[0080] α, β, γ are the weighting coefficients for balancing the losses of each item.

[0081] Specifically, the quantization perception training step in step S6 includes:

[0082] A pseudo-quantization operator is inserted into the model training graph to simulate the quantization error during inference.

[0083] For weight w and activation value a, the quantization function Q(⋅) is defined as:

[0084] Q(x)=sround(clip(x,xmin,xmax) / s),s=2b−1xmax−xmin

[0085] Where b is the target number of quantization bits (e.g., 8 bits) and s is the scaling factor; during training, the non-differentiable rounding function is bypassed by the pass-through estimator, and the floating-point parameters before quantization are updated by backpropagation.

[0086] Specifically, before step S2, there is also a modal importance assessment step:

[0087] A modal contribution analysis model is pre-trained on the server side to calculate the Shapley value or mutual information of each modality under a specific task.

[0088] Based on the evaluation results, the generation weights of the cross-modal virtual feature generator are dynamically adjusted. For missing modalities that contribute significantly to the task, more generation resources are allocated or the confidence threshold of the generated features is increased.

[0089] Specifically, the method supports an asynchronous federated update mechanism:

[0090] It allows different clients to upload gradients at different frequencies based on their network conditions and computing power;

[0091] After receiving updates from a subset of clients, the central server immediately aggregates and distributes the model, without waiting for all clients to synchronize, thus reducing system stagnation caused by "lagging out" clients.

[0092] A lightweight training system for privacy-preserving models in multimodal missing scenarios includes:

[0093] The data management module, deployed on the client side, is used to collect, clean, and standardize multimodal raw data, and detect modality missing status.

[0094] The privacy computing engine, deployed on the client side, has a built-in cross-modal virtual feature generator, differential privacy noise generator, and local model trainer for performing privacy-preserving local model updates.

[0095] A federated aggregation server, deployed in the cloud or at the edge, is used to receive and aggregate encrypted gradients from multiple clients and maintain global model parameters.

[0096] The model compression and distillation unit receives the trained global teacher model and generates a lightweight student model through knowledge distillation, pruning, and quantization operations.

[0097] A secure communication bus is used to achieve two-way authentication and encrypted data transmission between clients and servers, preventing eavesdropping and tampering on the transmission link.

[0098] Specifically, the privacy computing engine adopts a heterogeneous computing architecture, which can dynamically schedule computing tasks according to the client hardware type, and prioritize the use of hardware acceleration units to perform cross-modal attention computing and convolution operations to reduce local training power consumption.

[0099] Through the above technical solution, this invention fundamentally overcomes the strong dependence of traditional multimodal models on input completeness by introducing a learnable "virtual modality generator" based on a multi-head cross-attention mechanism. Unlike the feature distribution shift caused by static mean padding or zero padding in existing technologies, this invention utilizes the inherent deep semantic relationships between modalities to dynamically generate high-dimensional feature representations of missing modalities. This "feature-level" completion method enables the model to capture complementary information across modalities even in extreme scenarios where some sensors fail or data is selectively turned off by the user, thereby maintaining the stability of inference performance. Experiments show that even with a random modality missing rate as high as 30%, the model determined by this invention still significantly improves key indicators such as classification accuracy compared to traditional baseline methods, greatly enhancing the system's practicality and robustness in complex open environments. This invention innovatively integrates a federated learning architecture and an adaptive differential privacy mechanism to construct a full-process privacy protection system from data acquisition and transmission to model updates. By performing gradient pruning and adaptive noise injection locally on the client side, it effectively resists gradient inversion attacks and membership inference attacks. In particular, the noise intensity attenuation strategy designed in this invention ensures high-level privacy protection in the early stages of training while avoiding the negative impact of fixed high-intensity noise on the model's convergence accuracy in the later stages. Furthermore, the optional communication protocol, combining secure multi-party computation or homomorphic encryption, further ensures the confidentiality of gradients during transmission. This design perfectly aligns with compliance requirements in privacy-sensitive fields such as healthcare and financial risk control, achieving the security goal of "data usable but not visible." By employing a "large model teacher - small model student" distillation architecture, combined with structured pruning and quantized perceptual training, it successfully addresses the industry pain point of deploying multimodal large models on resource-constrained devices. Through knowledge distillation, the "dark knowledge" fused across modalities in the teacher model is efficiently transferred to the lightweight student model, preserving the model's expressive power while significantly reducing the number of parameters. Subsequent pruning and QAT steps further optimize the model's computation graph and memory footprint, making it adaptable to low-bit hardware accelerators. Compared to the original uncompressed model, the final model generated by this invention can reduce its size by more than 70%, inference latency by more than 50%, and accuracy loss is controlled within an acceptable range. This enables complex multimodal AI applications to run smoothly on edge devices such as mobile phones and embedded MCUs, significantly reducing bandwidth dependence and cloud inference costs.

[0100] Example 1: Detailed Explanation of the Method Flow

[0101] Step S1: System Initialization and Parameter Configuration

[0102] Environment Setup: Construct a federated learning system. A high-performance GPU cluster is configured on the central server, and a global teacher model (MT) is initialized. This model adopts a dual-tower structure: one side is a multimodal encoder, containing a VisionTransformer (ViT) for images, a TransformerEncoder for audio, and an LSTM for sensor time-series data; the other side is a cross-modal fusion Transformer.

[0103] Parameter settings: The system administrator sets the privacy budget (ϵ=2.0, δ=10−5), initial noise intensity σ0=0.5, and gradient clipping threshold C=1.0. The maximum number of communication rounds Tmax=200, and the number of local training epochs is E=5.

[0104] Step S2: Local Modal Integrity Detection and Feature Extraction

[0105] Data Acquisition: User Alice's mobile client collected a set of multimodal data samples D={I,A,S}. Wherein:

[0106] I: User selfie image (the image is blurry due to being in night mode, and is therefore considered missing / unavailable).

[0107] A: The user reads a 30-second audio waveform of the specified text.

[0108] S: User's typing speed, pause intervals, and other interaction logs on the mobile phone.

[0109] Modality detection: The client calls the ModalityChecker module and outputs the detection results: Mavail={A,S}, Mmiss={I}.

[0110] Virtual feature generation:

[0111] First, audio features fA∈R512 are extracted using AudioEncoderEA, and sensor features fS∈R256 are extracted using SensorEncoderES.

[0112] Since image I is missing, the system activates the cross-modal virtual feature generator Gvirtual.

[0113] In this embodiment, a multi-head cross-attention mechanism is employed:

[0114] The Query(Q) is set to be derived from fA (audio emotion is often related to facial expression).

[0115] The Key(K) and Value(V) are set from fS (typing rhythm reflects cognitive state and is indirectly related to micro-expressions).

[0116] The calculation formula is as follows:

[0117] f^I=MHA(Q=fA,K=fS,V=fS)

[0118] The output f^I∈R768 is the virtual image feature, and its dimension is consistent with that of the real image feature.

[0119] Step S3: Privacy-preserving local model training

[0120] Feature fusion: {f^I,fA,fS} are concatenated and a joint representation hfusion is generated through a modality-gating fusion mechanism.

[0121] Forward propagation: Input hfusion into the classification head to predict the probability of developing Alzheimer's disease (high risk / low risk).

[0122] Loss calculation: Calculate the cross-entropy loss Ltask.

[0123] Backpropagation and gradient processing:

[0124] Perform backpropagation to calculate the gradient g = ∇θLtask of the model parameters θ.

[0125] Gradient clipping: Calculate the L2 norm of the gradient. If ||g||2>C, then scale: gclip=g×||g||2C. Assuming C=1.0 and the current gradient norm is 1.5, the scaling factor is 2 / 3.

[0126] Adaptive noise addition: Assuming the current training is in the t=10th round, according to the formula σt=tσ0, we can calculate σ10=100.5≈0.158.

[0127] Generate Gaussian noise η∼N(0,(σt⋅C)2I), and add a gradient: gdp=gclip+η.

[0128] Step S4: Safe gradient aggregation and global update

[0129] Upload: The client Alice uploads the noisy gradient gdp to the central server via a TLS 1.3 encrypted channel.

[0130] Aggregation: The server simultaneously receives gradients {g1dp, g2dp, ...} from 100 clients (including Alice, Bob, etc.).

[0131] The server executes the FedAvg algorithm:

[0132] θglobalt+1=θglobalt−η⋅N1i=1∑Ngidp

[0133] Where N=100, and η is the learning rate.

[0134] Distribution: The server distributes the updated global model parameter θglobalt+1 to all participating clients.

[0135] Step S5: Iterative Training and Convergence Determination

[0136] Repeat steps S2-S4. As the number of rounds t increases, the noise coefficient σt gradually decreases (for example, σ100≈0.05 in the 100th round), and the model gradually converges.

[0137] The loss function on the monitoring validation set is stopped when the loss no longer decreases or reaches Tmax after 5 consecutive rounds, thus obtaining the final global teacher model MT∗.

[0138] Step S6: Knowledge Distillation and Lightweight Model Conversion

[0139] Distillation training:

[0140] Design a lightweight student model MS (e.g., replace ViT with MobileNetV3 and Transformer with GRU).

[0141] Using MT* as the teacher, distillation is performed on unlabeled or poorly labeled public datasets.

[0142] The total loss function is set as: Ltotal = 0.4LCE + 0.4LKL + 0.2LHint, where LHint forces the intermediate layer features of the student model to approximate the output of the virtual feature generator of the teacher model.

[0143] Structured pruning:

[0144] Amplitude pruning is performed on the convolutional and fully connected layers of the MS network. A threshold τ = 0.01 is set to remove all connections with an absolute weight less than 0.01, forming a sparse network.

[0145] Quantitative perception training:

[0146] Insert a pseudo-quantization node in MS.

[0147] Simulate the INT8 quantization process: wfake=round(w / sw)⋅sw.

[0148] Continue fine-tuning the model parameters for 1-2 epochs to recover the accuracy loss caused by quantization.

[0149] Deployment: The final generated INT8 quantization model, Mfinal, is packaged into a mobile app. This model can run offline locally, completing risk assessments without an internet connection, and its size is less than 15MB, resulting in extremely low memory usage.

[0150] Example 2: Specific Composition of System Architecture

[0151] This system specifically includes the following modules:

[0152] Data Management Module:

[0153] Implemented using native Android / iOS APIs.

[0154] Function: Collect raw data by calling the microphone, camera and system log interfaces; Built-in ModalityChecker logic to determine image clarity (using the Laplacian variance algorithm) or user authorization status.

[0155] Privacy Computing Engine (ClientSide):

[0156] Built on TensorFlowLite or PyTorchMobile.

[0157] Core components:

[0158] Cross-modal virtual feature generator: implemented as an independent sub-graph, with both input and output being tensors.

[0159] Differential Privacy Library: Integrates Google's TensorFlow Privacy library, wraps DPGradientDescentOptimizer, and automatically performs gradient pruning and noise injection.

[0160] Federated Aggregator Server (ServerSide):

[0161] Built using Python Flask or gRPC frameworks.

[0162] Functions: Maintain a model version number; handle concurrent upload requests; execute a secure model averaging algorithm; have a fault tolerance mechanism to handle client disconnections.

[0163] Model compression and distillation unit (ServerSide / Offline):

[0164] It runs on a high-performance workstation.

[0165] Toolchain: Distillation, pruning, and QAT are performed using Distiller (Intel), NNCF (NVIDIA), or custom-developed tools.

[0166] Secure communication bus:

[0167] Two-way TLS certificate authentication is used to ensure that only authorized clients can upload gradients.

[0168] For extremely high security requirements, homomorphic encryption (such as the Microsoft SEAL library) can be superimposed to achieve gradient aggregation (SecureAggregation) in the ciphertext state.

[0169] Example 3: Dynamic Application of Modal Importance Assessment

[0170] To further optimize performance, this embodiment introduces the modal importance assessment described in claim 6:

[0171] Pre-training analysis: On the server side, a modality contribution analysis model is pre-trained using complete historical data.

[0172] Dynamic weight adjustment:

[0173] The analysis results show that, in nighttime scenes, the contribution of the "audio" modality to the sentiment analysis task (Shapley value) is 0.6, the "sensor" modality is 0.3, and the "image" modality is only 0.1.

[0174] When the system detects that the current time is night (via the system time API) and the image is missing, the virtual feature generator automatically reduces the confidence weight of the generated features or reduces the number of attention heads within the generator, relying more on audio features instead. This dynamic adjustment mechanism avoids forcibly generating misleading features when the image quality is already poor, thereby improving the system's stability.

[0175] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A privacy protection model lightweight training method in a multi-modal missing scene, characterized in that: Includes the following steps: Step S1: System initialization and parameter configuration: Construct a federated learning system containing a central server and multiple edge clients. Initialize a global teacher model on the central server. The global teacher model contains multiple modal encoders and cross-modal fusion modules. Set the differential privacy budget parameters (ϵ,δ) and model compression hyperparameters. Step S2: Local Modal Integrity Detection and Feature Extraction: Each client obtains a local multimodal dataset. For each training batch, it checks whether there are missing modalities in the input sample's modal set. If there are missing modalities, it activates the cross-modal virtual feature generator to dynamically generate virtual feature vectors for the missing modalities using existing modal features. If the modalities are complete, it directly extracts the real feature vectors of each modality. Step S3: Privacy-preserving local model training: Input the fused feature vector into the local model for forward propagation and calculate the task loss function; perform backpropagation to obtain the model gradient, and perform adaptive differential privacy noise injection on the gradient to generate a noisy gradient; Step S4: Secure gradient aggregation and global update: Each client uploads the processed noisy gradient to the central server; the central server uses the federated averaging algorithm to perform weighted aggregation of gradients from all clients, updates the global teacher model parameters, and distributes the updated parameters to each client. Step S5: Iterative training and convergence determination: Repeat steps S2 to S4 until the global teacher model converges or reaches the preset maximum number of communication rounds; Step S6: Knowledge Distillation and Lightweight Model Conversion: Using the converged global teacher model as the knowledge source, the lightweight student model is trained by knowledge distillation online or offline; then, structured pruning and quantization-aware training are performed on the student model to generate the final deployable model.

2. The lightweight training method and system for privacy-preserving models in multimodal missing scenarios according to claim 1, characterized in that: The cross-modal virtual feature generator mentioned in step S2 is constructed using a multi-head cross-attention mechanism, and its specific calculation process is as follows: Let the set of feature maps for available modalities be Xavail={x1,x2,...,xm}, and the set of missing modalities be Ymiss={y1,y2,...,yn}; For any missing mode yj∈Ymiss, select one or more available modes xi∈Xavail as auxiliary modes, and generate a query matrix Qj=WQ⋅xi, a key matrix Kj=WK⋅xi and a value matrix Vj=WV⋅xi; Calculate the attention score and generate a virtual feature y^j: y^j=softmax(dkQjKjT)Vj Wherein, WQ, WK, and WV are learnable linear projection parameters, and dk is the feature dimension; the virtual feature y^j is consistent with the real modal feature in terms of dimension and participates in subsequent fusion operations.

3. The lightweight training method and system for privacy-preserving models in multimodal missing scenarios according to claim 1, characterized in that: The adaptive differential privacy noise injection operation in step S3 specifically includes: Gradient clipping: The gradient gt calculated by the local model is clipped using the L2 norm to limit it to within a preset threshold C, i.e., gtclip=gt⋅min(1,|gt||2C); Adaptive noise addition: Gaussian-distributed noise is added to the clipped gradient to generate the upload gradient gtupload. gtupload=gtclip+N(0,(σt⋅C)2⋅I) Where σt is the adaptive noise scaling factor for the t-th training round, and its value decreases monotonically with the increase of the training round t, satisfying σt=t+βσ0, where σ0 is the initial noise intensity and β is a small constant to prevent division by zero.

4. The lightweight training method and system for privacy-preserving models in multimodal missing scenarios according to claim 1, characterized in that: The knowledge distillation process in step S6 employs a hybrid distillation strategy combining soft label matching and intermediate layer feature matching, and its total loss function is defined as: Ltotal = αLtask + βLKL + γLHint in: Ltask is the cross-entropy loss of the student model for the target task; LKL is the soft-label distillation loss. It calculates the KL divergence between the teacher model output zt and the student model output zs at the temperature coefficient T, i.e., LKL=T2⋅KL(softmax(zs / T),softmax(zt / T)). LHint is the intermediate layer feature distillation loss, used to align the feature representations of the teacher model and the student model in the cross-modal fusion layer; α, β, γ are the weighting coefficients for balancing the losses of each item.

5. The lightweight training method and system for privacy-preserving models in multimodal missing scenarios according to claim 1, characterized in that: The quantization perception training step in step S6 specifically includes: A pseudo-quantization operator is inserted into the model training graph to simulate the quantization error during inference. For weight w and activation value a, the quantization function Q(⋅) is defined as: Q(x)=sround(clip(x,xmin,xmax) / s),s=2b−1xmax−xmin Where b is the target number of quantization bits (e.g., 8 bits) and s is the scaling factor; during training, the non-differentiable rounding function is bypassed by the pass-through estimator, and the floating-point parameters before quantization are updated by backpropagation.

6. The lightweight training method and system for privacy-preserving models in multimodal missing scenarios according to claim 1, characterized in that: Before step S2, a modal importance assessment step is also included: A modal contribution analysis model is pre-trained on the server side to calculate the Shapley value or mutual information of each modality under a specific task. Based on the evaluation results, the generation weights of the cross-modal virtual feature generator are dynamically adjusted. For missing modalities that contribute significantly to the task, more generation resources are allocated or the confidence threshold of the generated features is increased.

7. The lightweight training method and system for privacy-preserving models in multimodal missing scenarios according to claim 1, characterized in that: The method supports an asynchronous federated update mechanism: It allows different clients to upload gradients at different frequencies based on their network conditions and computing power; After receiving updates from a subset of clients, the central server immediately aggregates and distributes the model, without waiting for all clients to synchronize, thus reducing system stagnation caused by "lagging out" clients.

8. A lightweight training system for privacy-preserving models in multimodal missing scenarios, characterized in that: include: The data management module, deployed on the client side, is used to collect, clean, and standardize multimodal raw data, and detect modality missing status. The privacy computing engine, deployed on the client side, has a built-in cross-modal virtual feature generator, differential privacy noise generator, and local model trainer for performing privacy-preserving local model updates. A federated aggregation server, deployed in the cloud or at the edge, is used to receive and aggregate encrypted gradients from multiple clients and maintain global model parameters. The model compression and distillation unit receives the trained global teacher model and generates a lightweight student model through knowledge distillation, pruning, and quantization operations. A secure communication bus is used to achieve two-way authentication and encrypted data transmission between clients and servers, preventing eavesdropping and tampering on the transmission link.

9. A lightweight training system for privacy-preserving models in multimodal missing scenarios according to claim 8, characterized in that: The privacy computing engine adopts a heterogeneous computing architecture, which can dynamically schedule computing tasks according to the client hardware type, and prioritize the use of hardware acceleration units to perform cross-modal attention computing and convolution operations to reduce local training power consumption.