Adaptive differential privacy budget adjustment method for multi-modal learning

Abstract

The invention discloses a multi-modal learning-oriented adaptive differential privacy budget adjustment method, which dynamically allocates a differential privacy budget in a training process by evaluating the feature distribution difference, task contribution degree and noise sensitivity of multi-modal data, and specifically comprises the following steps of: extracting a multi-modal feature vector and calculating modal correlation; fusing features and predicting downstream task results; determining a contribution ratio and a sensitivity difference ratio based on the prediction accuracy and the noise sensitivity; the privacy budget of each round is finely adjusted in combination with the correlation, so that the high-contribution mode distributes the large budget to guarantee the performance, and the low-contribution mode distributes the small budget to strengthen privacy protection; gaussian noise is injected according to a budget in gradient back propagation.

Description

Technical Field

[0001] The present invention relates to the fields of multimodal learning and privacy protection, and specifically provides an adaptive differential privacy budget adjustment method for multimodal learning. Background Art

[0002] Multimodal learning, due to its adaptability to complex tasks, has been widely applied in fields such as autonomous driving, medical diagnosis, emotion recognition, and robotics, becoming a key focus in the development of artificial intelligence. However, in the actual training of multimodal learning models, differences exist in feature scales, representation methods, and data distributions among different modalities, often leading to challenges in dealing with these differences during the multimodal learning process.

[0003] Although multimodal learning offers significant advantages, it also inherits privacy risks associated with individual modalities, such as membership inference attacks and attribute inference attacks. Researchers have found that multimodal models performing sentiment analysis may inadvertently expose users' privacy information, such as gender, age, and race. Additionally, attackers can exploit the relationships between modalities to link vulnerabilities in one modality to other modalities, thereby extracting more privacy information. Therefore, effective multimodal learning privacy protection measures are crucial for reducing these potential information leakage risks.

[0004] Current multimodal learning privacy protection methods mainly include two categories: one is the multimodal privacy protection method based on adversarial learning, which generates desensitized data to replace the original data, thereby protecting the privacy of the original data. The other is the multimodal privacy protection based on differential privacy, which fuses modalities into a shared subspace through multimodal representation learning and then applies the differential privacy mechanism for unified noisy perturbation to achieve privacy protection. However, this will destroy the correlation between modalities and affect the task accuracy of the fusion model. By adjusting the differential privacy budgets of different modalities through a post-feedback mechanism, the overall accuracy of the fusion model is enhanced. However, this method requires exploring a wide parameter space, resulting in significant computational overhead during the optimization process.

[0005] Therefore, there is a need for an adaptive differential privacy budget adjustment method for multimodal learning. Summary of the Invention

[0006] To solve the above technical problems, the present application is proposed. Embodiments of the present application provide an adaptive differential privacy budget adjustment method for multimodal learning, which solves the problem of difficulty in balancing privacy protection and model performance in the prior art, and has the advantages of adaptive adjustment, low computational overhead, and strong generality, and is applicable to multimodal scenarios such as intelligent driving and medical diagnosis.

[0007] According to one aspect of the present application, an adaptive differential privacy budget adjustment method for multi-modal learning is provided, which includes: S101, extracting feature vectors of multi-modal data, and respectively using encoders adapted to the modalities to extract features from the data of the first modality X A and the second modality X B to obtain corresponding feature vectors E A and E B ; S102, calculating modal correlation, based on the maximum mean discrepancy (MMD) to measure the distribution difference between the feature vectors E A and E B to obtain the inter-modal correlation p; S103, multi-modal feature fusion, fusing the feature vectors E A and E B into a multi-modal feature vector E M through concatenation, addition and gating mechanisms; S104, downstream task prediction, respectively making predictions based on E A , E B and E M to obtain prediction labels and S105, calculating contribution ratios, determining the contribution ratios R A and R B of the first modality X A and the second modality X B according to the accuracy of the prediction labels; S106, evaluating noise sensitivity, adding differential privacy noise to the feature vectors, calculating the sensitivities S A and S B of the two modalities to obtain the sensitive difference ratios W A and W B ; S107, dynamically allocating privacy budgets, calculating adjustment factors λ A , λ B based on the contribution ratios, sensitivities and modal correlation, and fine-tuning through the correlation p to allocate the privacy budgets ε A and ε B of the first modality X A and the second modality X B ; S108, model training and noise injection, calculating the model loss and backpropagating, adding Gaussian noise to the gradients according to the allocated privacy budgets to achieve dynamic adjustment of the privacy budgets during one model convergence process.

[0008] Preferably, in step S101, the feature encoders f A and f B of the first modality X A and the second modality X B are respectively used to encode the data of their respective modalities to extract the corresponding feature vectors E A = f A(X A ) and E B = f B (X B ).

[0009] Preferably, in the step S102, the calculation formula of the modal correlation p is: p = 1 / MMD(E A , E B ), where MMD is calculated by a Gaussian kernel function.

[0010] Preferably, in the step 104, the first modality X A , the second modality X B and the fused multi-modalities are respectively used to predict downstream tasks, and prediction labels and are obtained, where g A and g B are classifiers for downstream tasks.

[0011] Preferably, in the step S105, the calculation formulas of the contribution ratios R A and R B are as follows:[[]] R B = 1 - R A .

[0012] Preferably, in the step S106, the sensitivities S A and S B are calculated by the cosine similarity of the feature vectors before and after adding noise, and the formula is as follows: S A = H(E A , E' A ), S B = H(E B , E' B ), where E' A and E' B are the feature vectors after adding noise to E A and E B .

[0013] Preferably, in the step S107, the allocation formula of the privacy budget is as follows:[[]] Among them, W B = 1 - W A .

[0014] Preferably, the fine-tuning of the privacy budget is achieved by an adjustment factor , where α is a hyperparameter, and the final privacy budget is: ε A = λ A ·λ·ε global , ε B= λ B · ε global 。

[0015] Preferably, the smaller the value of MMD, the smaller the distribution difference between the two modalities and the greater the correlation between the modalities; the larger the value of MMD, the greater the distribution difference between the two modalities and the smaller the correlation between the modalities.

[0016] Preferably, it further includes step S109 of iterative optimization, and the above steps are repeated until the model converges.

[0017] The present invention designs an adaptive differential privacy budget adjustment method for multi-modal learning. In each training iteration, appropriate differential privacy noise is dynamically added to each modality without the need for external human control, realizing adaptive differential privacy budget allocation. By considering the differences between modalities, different modalities play different degrees of privacy protection roles and task performance guarantees, and on the premise of protecting privacy requirements, the optimization of modal task performance is realized. There is no additional deep neural network and parameter search process, and compared with the differential privacy budget allocation method of the post-feedback mechanism, it has lower computational overhead. And it is only analyzed and optimized on the basis of the original task model, so it can be flexibly embedded into other multi-modal learning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present application will become more obvious. The accompanying drawings are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification, and are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation to the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 It is a flowchart of an adaptive differential privacy budget adjustment method for multi-modal learning according to an embodiment of the present application.

[0020] Figure 2 It is a flowchart of differential privacy in an adaptive differential privacy budget adjustment method for multi-modal learning according to an embodiment of the present application.

[0021] Figure 3 It is a flowchart of multi-modal learning in an adaptive differential privacy budget adjustment method for multi-modal learning according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The following will detail various exemplary embodiments, features, and aspects of the present application with reference to the accompanying drawings. The same reference numerals in the drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0023] As used herein, the term "exemplary" means "serving as an example, instance, or illustration". Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or better than other embodiments.

[0024] In addition, for the purpose of better explaining the present application, numerous specific details are given in the following specific implementation manners. Those skilled in the art should understand that the present application can also be implemented without some specific details. In some instances, methods, means, elements, and circuits well-known to those skilled in the art are not described in detail so as to highlight the gist of the present application.

[0025] Furthermore, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present application, "a plurality" means two or more unless otherwise specifically defined.

[0026] Current multi-modal learning privacy protection methods mainly include two categories: one is the multi-modal privacy protection method based on adversarial learning, which generates desensitized data to replace the original data, thereby protecting the privacy of the original data. An adversarial learning framework is established for text, acoustic, and multi-modal data to eliminate the hidden privacy information in the training data. However, this method requires identifying the opponent's attack strategy before targeted training to reduce leakage, making the complexity of the model high, the training cost large, and it is difficult to effectively balance the data availability. The other is the multi-modal privacy protection based on differential privacy. Through multi-modal representation learning, the modalities are fused into a shared subspace, and then the differential privacy mechanism is applied for unified noise perturbation to achieve privacy protection. However, this will destroy the correlation between modalities and affect the task accuracy of the fusion model. The overall accuracy of the fusion model is enhanced by adjusting the differential privacy budget of different modalities through a post-feedback mechanism. However, this method requires exploring a wide parameter space, resulting in significant computational overhead during the optimization process.

[0027] Therefore, the present invention jointly determines the size of the privacy budget allocated to each modality by evaluating the contribution of different modalities to the fusion model, the correlation between modalities, and the sensitivity of each modality to differential privacy noise, so that each modality assumes different degrees of privacy protection and task performance guarantee, and optimizes the performance of modal tasks under the premise of protecting privacy requirements. An adaptive differential privacy budget adjustment method for multimodal learning is designed. During the training process of the multimodal fusion model, the adjustment factor is calculated to dynamically allocate the privacy budget, so that the optimal privacy budget can be allocated to each modality in each training iteration, thereby adding appropriate differential privacy noise to achieve adaptive differential privacy budget adjustment. On the basis of the original task model, no additional deep neural network and parameter search process are added. Compared with the differential privacy budget adjustment method of the post-feedback mechanism, it has lower computational overhead and can be flexibly embedded in other multimodal learning tasks.

[0028] Specifically, the present invention designs an adaptive differential privacy budget adjustment method for multimodal learning. The scheme framework is as follows: Figure 1 As shown. This method uses the interpretability analysis of the multimodal learning model to evaluate the correlation of different modalities and their contribution to the fusion model, and combines it with the sensitivity of the modality to differential privacy noise to jointly calculate the adjustment factor. During the training process of the multimodal fusion model, the privacy budget is dynamically allocated, so that the optimal privacy budget can be adaptively allocated to each modality in each training iteration, thereby adding appropriate differential privacy noise, and optimizing the performance of the modal task while protecting privacy requirements.

[0029] Among them, differential privacy (such as Figure 2 ) is a privacy protection technology. Its application in deep learning is mainly to protect the privacy of data and prevent the leakage of sensitive information during model training and use. Its core idea is to cover up the influence of individual data by adding noise, thereby preventing attackers from inferring individual information.

[0030] Multimodal learning (e.g. Figure 3 ) Through the fusion and correlation analysis of different modal data, it provides a more comprehensive and richer perspective for machine learning models, and can mine more valuable information, thereby significantly improving the model's perception, decision-making and generalization capabilities.

[0031] Accordingly, according to an embodiment of the present application, an adaptive differential privacy budget adjustment method for multimodal learning specifically includes:

[0032] The first step is to design different feature extraction methods for different types of multimodal data, using modality X A and modality X B The feature encoder f A and fB , encode the data of each modality, and extract the corresponding feature vector E A = f A (X A ) and E B = f B (X B ).

[0033] Second, the maximum mean discrepancy (MMD) is used to measure the difference between two distributions. Here, the MMD is used to calculate the correlation p between the feature vectors of modality X A and modality X B : p = 1 / MMD(E A , E B ). The smaller the value of MMD, the smaller the distribution difference between the two modalities, and the larger the correlation p between the modalities; the larger the value of MMD, the larger the distribution difference between the two modalities, and the smaller the correlation p between the modalities.

[0034] It should be understood that when designing a feature extraction method for data of different modalities and obtaining feature vectors using an encoder, the data types of different modalities need to be considered, such as images, texts, audios, etc. Each requires a specific encoder, such as a CNN for processing images and an RNN for processing texts. Then, the MMD is used to measure the distribution difference between two modalities. The smaller the value of MMD, the larger the correlation. At this time, it is necessary to confirm whether the calculation of MMD is correctly applied and whether it is suitable for measuring the correlation between modalities.

[0035] Third, through a multi-modal fusion method f M , such as mechanisms like concatenation, addition, gating, etc., fuse the feature vectors E A of modality X B and the feature vector E A and E B of modality X M = f M (E A , E B ) into a multi-modal feature vector E

[0036] Fourth, respectively predict the downstream tasks for modality X A , modality X B and the fused multi-modal to obtain the prediction labels and where g A and g B are classifiers or regressors for the downstream tasks.

[0037] It should be understood that for multi-modal fusion, methods include splicing, addition, gating mechanisms, etc. The impact of different fusion methods on subsequent tasks needs to be considered. For example, splicing may retain more information but increase the dimension, while the gating mechanism may be more flexible but complex. Predictions for downstream tasks are made for each modality and the fused features, resulting in three prediction results. Here, the specific structure of the downstream task may need to be clarified, as well as whether the designs of the classifier or regressor are consistent.

[0038] Step 5, according to the predicted labels and 's accuracy rates, calculate the contribution ratios A of modality X B and modality X and R B = 1 - R A .

[0039] Step 6, add differential privacy noise of the same size to the feature vectors E B and E B respectively to obtain the noise-added feature vectors E' A and E' B , and then calculate the sensitivities S A of modality X B to the differential privacy noise and S A = H(E A , E' A ) and S B = H(E B , E' B ) through the cosine similarity H, obtaining the sensitive difference ratio W B = 1 - W a .

[0040] Step 7, based on the difference degree of the modal feature vectors, the contribution ratios of the predicted labels, and the sensitivities to the differential privacy noise, comprehensively consider the relationships among these influencing factors and allocate different sizes of privacy budgets to each modality. Specifically, first allocate the total privacy budget ε global to each modality as the pre-allocated privacy budget according to the contribution degrees and sensitivities of each modality. Relatively larger privacy budgets are allocated to the modalities with larger contribution degrees and larger sensitivities to ensure the performance of the model task, and relatively smaller privacy budgets are allocated to the modalities with smaller contribution degrees and smaller sensitivities to ensure the effect of privacy protection. Calculate the adjustment factors and Then, further fine-tune the privacy budget through the correlation p, reducing the privacy budget of the modality with a small contribution degree so that it plays a greater role in privacy protection, while keeping the privacy budget of the other modality unchanged to ensure that it obtains sufficient training to guarantee the performance of the model task. Suppose here, modality X Ahas a relatively small contribution degree, i.e., R A <1 / 2, modality X B has a relatively large contribution degree, i.e., R B ≥1 / 2, and then calculate the adjustment factor according to the difference degree p where α is a hyperparameter that controls the adjustment degree. Then modality X A 's privacy budget is adjusted to ε A = λ A ·λ·ε global while modality X B 's privacy budget is adjusted to ε B = λ B ·ε global , and under the premise of protecting privacy requirements, the performance of the modality task is optimized.

[0041] Step 8, calculate the loss of the model Update the model parameters by backpropagation through minimizing the loss to optimize the performance of the model.

[0042] Step 9, during the model backpropagation process, for the gradients of each modality, add differential privacy based on the Gaussian mechanism according to the privacy budgets ε A and ε B allocated in Step 7, where the Gaussian mechanism uses a Gaussian distribution to generate noise to achieve the adjustment of the adaptive differential privacy budget and the injection of noise.

[0043] Step 10, repeat Steps 1 to 9. The model is gradually optimized and converges until the model reaches the preset number of iterations and then ends the training.

[0044] In summary, the present invention provides an adaptive differential privacy budget adjustment method for multi-modal learning. In each training iteration, appropriate differential privacy noise is dynamically added to each modality without the need for artificial external control participation, realizing the adaptive allocation of the differential privacy budget. By considering the differences of each modality, different modalities play different degrees of privacy protection roles and task performance guarantees, and under the premise of protecting privacy requirements, the performance of the modality task is optimized. There is no additional deep neural network and parameter search process, and compared with the differential privacy budget allocation method of the post-feedback mechanism, it has a lower computational overhead. And it is only analyzed and optimized on the basis of the original task model, so it can be flexibly embedded into other multi-modal learning tasks.

[0045] Specifically, the current in-vehicle intelligent driving system can monitor the driver's fatigue state in real time to reduce traffic accidents caused by fatigue driving and ensure driving safety. In the process of judging the driver's fatigue state, it is necessary to first collect the driver's audio, image and other data by using devices such as cameras and microphones. However, a large amount of the user's privacy information is often involved in these multi-modal data. For example, audio data may contain privacy information such as personal identity, health status, and finances, while image data may involve privacy information such as behavior habits and location. If there is no effective privacy protection mechanism in the learning process of the multi-modal fatigue detection model, attackers may obtain more privacy information of the user through means such as mining, analyzing, and correlating between multi-modal data, thus threatening the user's life and property safety. Therefore, in order to reduce these potential privacy information leakage risks, we adopt an adaptive differential privacy budget adjustment method for multi-modal learning to perform privacy protection on the multi-modal fatigue detection model. Specifically, in the multi-modal fatigue detection model, the audio encoder and the image encoder are respectively used to extract the features of audio and image data to obtain the corresponding audio feature vectors and image feature vectors; the maximum mean discrepancy is used to calculate the correlation between the audio feature vector and the image feature vector to obtain the correlation between the two modalities of audio and image; the audio feature vector and the image feature vector are fused in a splicing manner to obtain the fused multi-modal feature vector; the audio feature vector, the image feature vector, and the multi-modal feature vector are respectively used to perform the multi-modal fatigue detection task to obtain their respective task prediction labels; according to the task prediction labels of audio and image, calculate the contribution degrees of the two to the prediction of the multi-modal fatigue detection model; add differential privacy noise of the same size to the audio feature vector and the image feature vector, and calculate the sensitivity of audio and image to the noise respectively; according to the modality correlation, task contribution degree, and noise sensitivity of audio and image, calculate the adjustment factor, and comprehensively determine the privacy budget sizes of audio and image respectively; use the task prediction label and the true label of the multi-modal feature vector to calculate the loss function of the multi-modal fatigue detection model, and backpropagate to update the model parameters to continuously optimize the model; when backpropagating during the training of the multi-modal fatigue detection model, add differential privacy noise based on the Gaussian mechanism to the corresponding gradients according to the privacy budget sizes of audio and image respectively; repeat the above steps until the model reaches the preset number of training rounds and then ends the training. Finally, a multi-modal fatigue detection model with privacy protection effect is obtained.

[0046] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of technologies in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An adaptive differential privacy budget adjustment method for multimodal learning, characterized in that Including: S101, Extract the feature vectors of the multimodal data. Respectively perform feature extraction on the data of the first modality X A and the second modality X B through the encoders adapted to the modalities, and obtain the corresponding feature vectors E A and E B ; S102, calculate the modal correlation, and measure the distribution difference between the eigenvectors E A and E B to obtain the inter-modal correlation p; S103, Multimodal feature fusion, fusing the feature vectors E A and E B into a multimodal feature vector E M through concatenation, addition, and gating mechanisms; S104, downstream task prediction, respectively based on E A , E B and E M for prediction to obtain prediction labels and S105, Calculate the contribution ratio, and determine the contribution ratio R of the first modality X and the second modality X according to the accuracy of the prediction label A and the second modality X B of the contribution ratio R A and R B ; S106, evaluate the noise sensitivity, add differential privacy noise to the feature vector, and calculate the sensitivities F A and S B of the two modalities, and obtain the sensitive difference ratio W A and W B ; S107. Dynamically allocate the privacy budget, calculate the adjustment factors λ and λ based on the contribution ratio, sensitivity, and modal correlation, and fine-tune through the correlation p to allocate the privacy budgets ε and ε for the first modality X and the second modality X; A and λ B , and fine-tune through the correlation p to allocate the privacy budget ε A for the first modality X B and the second modality X A and ε B ; S108, model training and noise injection, calculating the model loss and backpropagating, adding Gaussian noise to the gradient according to the allocated privacy budget, realizing dynamic adjustment of the privacy budget during one model convergence process.

2. The adaptive differential privacy budget adjustment method for multi-modal learning according to claim 1, characterized in that In the said step S101, respectively using the feature encoders f A and f B of the first modality X A and the second modality X B , encode the data of their respective modalities, and extract the corresponding feature vectors E A = f A (X A ) and E B = f B (X B ).

3. The adaptive differential privacy budget adjustment method for multi-modal learning according to claim 2, wherein In the step S102, the calculation formula of the modality correlation p is: p = 1 / MMD(E A , E B ) wherein, MMD is calculated through a Gaussian kernel function.

4. The adaptive differential privacy budget adjustment method for multi-modal learning according to claim 3, characterized in that In step 104, downstream task predictions are respectively made for the first modality X A , the second modality X B and the fused multi-modalities, to obtain prediction labels and where g A and g B are classifiers for the downstream tasks.

5. The adaptive differential privacy budget adjustment method for multi-modal learning according to claim 4, wherein The step S105, the contribution ratio R A and R B The calculation formula is as follows:

6. The adaptive differential privacy budget adjustment method for multi-modal learning according to claim 5, wherein The step S106, the sensitivity S A and S B are calculated by the cosine similarity of the feature vectors before and after adding noise, and the formula is as follows: S A = H(E A , E′ A ), S B = H(E B , E′ B ) Among them, E' A and E' B are the feature vectors after adding noise to E A and E B .

7. The adaptive differential privacy budget adjustment method for multi-modal learning according to claim 6, characterized in that In the step S107, the allocation formula of the privacy budget is: Among them, W B = 1 - W A .

8. The adaptive differential privacy budget adjustment method for multi-modal learning according to claim 7, characterized in that The fine-tuning of the privacy budget is achieved by an adjustment factor where α is a hyperparameter, and the final privacy budget is: ε A = λ A · λ · ε g l o b a l , ε B = λ B · ε g l o b a l .

9. The adaptive differential privacy budget adjustment method for multi-modal learning according to claim 8, characterized in that The smaller the value of MMD, the smaller the distribution difference between the two modalities, and the greater the correlation between the modalities; the larger the value of MMD, the greater the distribution difference between the two modalities, and the smaller the correlation between the modalities.

10. The adaptive differential privacy budget adjustment method for multi-modal learning according to claim 1, characterized in that, It further includes step S109, iterative optimization, repeating the above steps until the model converges.

Images