Multimodal learning and its use of modal level adaptive differential privacy pruning method

Abstract

The application discloses a multi-modal learning and a modal-level adaptive differential privacy clipping method used by the multi-modal learning, and comprises the following steps: a clipping method using gradient normalization scaling is used instead of a traditional clipping method of scaling the gradient exceeding the clipping value; different clipping values are set for each mode to control the optimization speed of each mode; then, expected mode contribution is set according to the quality of the mode data; in the multi-modal model updating process, the mode clipping value is adjusted according to the difference between the current mode contribution and the expected contribution, so as to balance the optimization speed of each mode. In addition, the application also comprises an adaptive clipping value updating method, which adjusts the mode clipping value through the difference between the current and past accumulated mode contribution, and can appropriately accelerate the weak mode which cannot be fully optimized for a long time. The application can realize differential privacy protection while effectively relieving the imbalance problem of mode optimization through the adaptive adjustment of the mode clipping value.

Description

Technical Field

[0001] This invention belongs to the field of machine learning, and mainly relates to a multimodal learning method and the modality-level adaptive differential privacy pruning method used therein. Background Technology

[0002] With the rapid development of IoT technology and the widespread application of smart devices, the modern information environment is undergoing a profound transformation in data formats. Traditional single-modal data processing models can no longer meet the increasingly complex application needs, and multimodal data processing capabilities are gradually becoming a core requirement of modern machine learning systems. Multimodal data not only exhibits high heterogeneity in terms of dimensionality, feature space, and semantic representation, but also poses more stringent systemic requirements at the engineering implementation level, such as computational complexity. Multimodal learning models exhibit an inherently greedy characteristic in their optimization mechanisms. This characteristic stems from the inherent differences in learning difficulty and convergence rates among different modalities. During training, optimization algorithms often tend to update parameters along the path with the fastest gradient descent, typically corresponding to the dominant modalities that are richer in information and have clearer feature representations. While this optimization bias can quickly reduce the overall loss function, it systematically ignores non-dominant modalities that are more difficult to learn, thus severely hindering the effective optimization of these modalities.

[0003] Differential privacy, as a rigorous mathematical privacy protection framework, provides theoretically provable security for machine learning systems by introducing a carefully calibrated random noise mechanism. This protection mechanism can effectively resist various complex privacy attacks, including model inversion attacks that attempt to reconstruct the original training data from model parameters, and membership inference attacks that analyze model behavior to infer the participation of specific data samples. It is precisely because of the strong security provided by differential privacy that it has achieved deep integration and widespread application in various machine learning methods. However, introducing differential privacy into multimodal learning to protect data privacy often exacerbates the optimization imbalance phenomenon in multimodal learning. This is because differential privacy requires gradient pruning, and in multimodal data, samples that favor optimizing weaker modalities often have larger gradient norms, making them more susceptible to gradient pruning. This suppression cannot be solved by simply adjusting the pruning value, because the gradient differences between samples favoring different modalities are large, and a larger pruning value will introduce greater differential privacy noise into the model, causing a decline in model performance. Furthermore, traditional pruning methods cannot strictly control the speed of modality optimization because they only constrain the upper bound of the gradient norm. Moreover, pruning the gradient distorts the gradient update direction, making it difficult to achieve better results when adding other multimodal balancing learning methods later. Summary of the Invention

[0004] The technical problem to be solved by this invention is to design a method that can alleviate the imbalance between modal optimizations while achieving differential privacy protection in the existing deep learning architecture based on multimodal fusion, in order to meet the privacy protection requirements of multimodal data. This is to solve the problem that the current differential privacy method exacerbates the modal imbalance caused by the uniform pruning value, resulting in a significant reduction in system performance and a decrease in generalization ability.

[0005] To achieve the above objectives, the solution of the present invention is:

[0006] A modal-level adaptive differential privacy pruning method, the pruning method comprising:

[0007] The overall pruning value of the multimodal model is assigned to each modality; where the input of the multimodal model is multimodal data;

[0008] Each mode uses a normalized modality-level pruning mechanism for gradient pruning to control the pruning value on the speed of modality optimization.

[0009] Specifically, the normalized modal-level pruning mechanism involves normalizing the modal gradients that need to be pruned, and then scaling them proportionally according to the pruning values ​​assigned to those modalities.

[0010] As a further optimization of the present invention, the method for gradient normalization is as follows:

[0011]

[0012] in, Indicates the uncut version The mode in the th ... The gradient of the training round, express The normalized result is given by r, which is a constant parameter.

[0013] As a further optimization of the present invention, the method for allocating the overall clipping value of the multimodal model to each modality includes:

[0014] Select a mode as the baseline mode, and the modality clipping value share of the baseline mode is 1 in each training round;

[0015] Based on the real-time acquired modal contribution of each modality, the modal pruning value share of the remaining modalities other than the baseline mode is updated:

[0016] ,

[0017] Where p represents the reference mode number, Indicates the first The mode in the th ... The modal clipping value share of the training round. Indicates the first The mode in the th ... The modal clipping value share of the training round. , Indicates the number obtained in real time The mode in the th ... Modal contribution during round training This indicates that the real-time acquired reference mode is in the 1st... Modal contribution during round training Indicates the first The expected modal contribution of each modality. This represents the expected modal contribution of the baseline mode. and Both represent parameters used to control the magnitude of changes in the trimming value. Indicates the total number of training rounds;

[0018] Based on the modal clipping value share of each modality, the overall clipping value of the multimodal model is allocated to each modality.

[0019] As a further optimization of the present invention, the method for obtaining the modality contribution of each modality in each training round includes:

[0020] In a certain training round, using only data from a specific modality as input, the reciprocal of the calculated model loss is the modal contribution of that modality in that training round.

[0021] As a further optimization of the present invention, the method for obtaining the modality contribution of each modality in each training round includes:

[0022] In a certain training round, when multimodal fusion is performed, only one modality is retained and the other modalities are masked. The reciprocal of the calculated model loss is the modal contribution of that modality in that training round.

[0023] As a further optimization of the present invention, the multimodal data includes at least two of the following: text data, image data, and sound data.

[0024] The present invention also provides a multimodal learning method, wherein the multimodal learning method uses the pruning method described above to perform gradient pruning in order to update the modal model parameters of each modality.

[0025] As a further optimization of the present invention, the model parameter update expressions for each modality are as follows:

[0026] ,

[0027] in, Indicates the first The mode in the th ... Modal model parameters during round training, Indicates the first The mode in the th ... Modal model parameters during round training, It's the learning rate. Indicates the first The noise variance of each mode, Indicates the first The mode in the th ... The modality clipping values ​​assigned during each training round. Indicates the uncut version The mode in the th ... Gradient of training rounds The normalization result, This indicates that the mean is 0 and the covariance matrix is ​​the identity matrix. It follows a multivariate standard normal distribution.

[0028] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the trimming method described above.

[0029] The present invention also provides an electronic device, comprising:

[0030] Memory, used to store computer programs;

[0031] A processor for implementing the trimming method as described above when executing the computer program.

[0032] Compared with existing technologies, the significant advantages of this invention are as follows: Traditional differential privacy methods are not suitable for multimodal models because they do not consider the different gradient magnitudes of samples with different optimization preferences. Using a uniform pruning value will suppress the optimization of already weaker modalities, making the optimization of the multimodal model more unbalanced and reducing its performance and generalization ability. This invention, by using a modality-level pruning method based on gradient normalization, achieves differential privacy protection while controlling the optimization speed of different modalities through modality pruning values. Furthermore, by comparing the current modality contribution with the preset modality contribution in real time during training, it adaptively adjusts the pruning values ​​of different modalities to achieve balanced optimization among different modalities in the multimodal model. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method of the present invention.

[0034] Figure 2 This is a performance chart of the present invention on a multimodal emotion recognition dataset. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0036] This invention proposes a multimodal learning method and its modality-level adaptive differential privacy pruning method. The core of this method is to allocate the overall pruning value of the model to each modality, and to control the optimization speed of the modality by applying a normalization-based pruning method to different modal models. This approach, based on adaptive modality pruning values, mitigates the modality imbalance problem in multimodal learning to the greatest extent possible while ensuring privacy.

[0037] like Figure 1 As shown, the specific implementation process of the modal-level adaptive differential privacy pruning method is described below:

[0038] I. Modal-level clipping based on normalization

[0039] This invention first implements a normalized modal-level pruning mechanism to control the pruning value on the modal optimization speed.

[0040] Step 1: Normalization-based cropping method

[0041] The initial step of this method is to replace the traditional clipping method with a normalization-based clipping method. The normalization-based clipping method first normalizes the gradient to be clipped, and then scales it proportionally according to the clipping value. The gradient normalization method used in this invention is as follows:

[0042]

[0043] in Indicates uncropped The mode in the first The gradient of the training round, This represents the normalized gradient, where r is a constant value. This normalization method, while normalizing the gradient, can mitigate the excessive amplification of the gradient when the gradient decreases, thus reducing its impact on model convergence.

[0044] Step 2: Implementation of the Modal-Level Clipping Method

[0045] To simultaneously prune modalities and control the optimization speed using pruning values, thereby balancing optimization speed by selecting appropriate pruning values, this invention proposes a modality-level pruning method. This method applies different modality pruning values ​​to the gradients of different modal models using a normalization-based pruning approach, thereby updating the modal model parameters of each modality in multimodal learning.

[0046]

[0047] in Indicates the first The mode in the th ... Modal model parameters during round training, Indicates the first The mode in the th ... Modal model parameters during round training, It's the learning rate. Indicates the first The noise variance of each mode, Indicates the first The mode in the th ... The modality clipping values ​​assigned during each training round. Indicates the uncut version The mode in the th ... Gradient of training rounds The normalization result, This indicates that the mean is 0 and the covariance matrix is ​​the identity matrix. It follows a multivariate standard normal distribution.

[0048] The above formula shows that setting different clipping values ​​for each modality can control the optimization speed of the modal model. Furthermore, by setting appropriate clipping values ​​at the modality level, this method can alleviate modality imbalance in multimodal learning by adjusting the modality clipping values.

[0049] II. Implementation of the Adaptive Modal Clipping Value Algorithm

[0050] After determining the normalization-based modality-level pruning method, precise pruning values ​​need to be assigned to each modality during training to achieve balanced optimization among modalities while maintaining the sensitivity of the multimodal model. If the pruning value of a weak modality is too small, its optimization will still be suppressed by the dominant modality; if the pruning value of the dominant modality is too small, it will not optimize the dominant modality and will instead severely degrade the performance of the multimodal model. Therefore, this invention addresses this problem by guiding the adjustment of pruning values ​​during the multimodal model training process based on the data quality of the modalities.

[0051] Step 1: Set the expected modal contribution using modal data quality.

[0052] In multimodal learning, the contribution of different modalities to the final result is often related to the data quality of each modality. However, in multimodal learning, the dominant modality with high data quality often inhibits the optimization of other modalities. Therefore, before training, we train each modality separately and score it according to the training difficulty of the samples to determine the data quality of the modality. Based on the modality data quality, this invention first sets the expected modality contribution for multimodal learning. When the modality contribution of multimodal learning deviates from the preset contribution, the modality pruning value is readjusted for each modality to balance the optimization of different modalities.

[0053] Step 2: Adaptive Dynamic Update of Modal Clipping Values

[0054] In the process of multimodal learning, a mode is selected as the baseline mode, and the modality clipping value share of the baseline mode is 1 in each round of training.

[0055] The modal pruning values ​​of modes other than the baseline mode are updated by using the real-time acquired current modal contribution values ​​of each mode:

[0056] ,

[0057] Where p represents the reference mode number, Indicates the first The mode in the th ... The modal clipping value share of the training round. Indicates the first The mode in the th ... The modal clipping value share of the training round. , Indicates the number obtained in real time The mode in the th ... Modal contribution during round training This indicates that the real-time acquired reference mode is in the 1st... Modal contribution during round training Indicates the first The expected modal contribution of each modality. This represents the expected modal contribution of the baseline mode. and Both represent parameters used to control the magnitude of changes in the trimming value. This indicates the total number of training rounds.

[0058] Based on the modality clipping value share of each modality, the overall clipping value of the multimodal model is allocated to each modality. Taking three modalities as an example, the first modality is selected as the baseline modality. In the second round of training, the modality clipping value share of the second modality is 0.8, and the modality clipping value share of the third modality is 1.3. Then, the overall clipping value is allocated to each modality in the ratio of 1:0.8:1.3.

[0059] This invention adjusts the modality pruning value by considering the difference between the current and past accumulated modal contributions, which can appropriately accelerate weak modes that have not been fully optimized in the long term, thereby achieving balanced optimization among individual modes.

[0060] It should be noted here that, as a further optimization of the present invention, there are two methods for obtaining the modality contribution of each modality in each training round:

[0061] (1) In a certain round of training, only one modality of data is used as input, and the reciprocal of the model loss is the modal contribution of that modality in that round of training. The reciprocal is taken here because the smaller the loss, the greater the contribution.

[0062] (2) In a certain round of training, only one mode is retained during multimodal fusion, and other modes are masked. The reciprocal of the calculated model loss is the modal contribution of that mode in that round of training. The reciprocal is taken here because the smaller the loss, the greater the contribution.

[0063] Example 1: Application on an action recognition multimodal dataset

[0064] In action recognition multimodal datasets, each sample contains both audio and video data. During training, feature extraction networks are used to extract features from each modality, and these feature vectors are then fused as input to the decision network model. The results are then used for backpropagation to update the model parameters. Under this architecture, differential privacy protection mechanisms for model parameters are difficult to apply. Traditional methods use a uniform pruning value for multimodal models, which exacerbates modality imbalance in multimodal learning. To address this issue, this invention designs a normalized modality-level pruning method. In this method, the optimization speed of each modality is controlled by the pruning value. By setting the pruning value appropriately, balanced optimization among modalities can be achieved. Furthermore, by adaptively adjusting the modality pruning value by comparing the difference between the current modality contribution and the preset modality contribution in real time, differential privacy protection can be achieved while maintaining balanced optimization among modalities, improving the performance and generalization ability of the multimodal model.

[0065] Figure 2In this study, all experiments were conducted under the same experimental settings, and all pruning methods met the gradient pruning requirements for differential privacy. The diamond-marked lines represent traditional uniform pruning of the multimodal model; the triangle-marked lines represent methods that balance modal optimization speed after traditional uniform pruning; the square-marked lines represent static pruning values ​​set according to preset modal contributions; and the circle-marked lines represent the dynamic pruning method based on modal data quality proposed in this invention. The results show that this method balances intermodal optimization, improves the performance of the multimodal model, and has very broad application prospects.

[0066] Based on the same technical solution, the present invention also proposes an electronic device, comprising:

[0067] Memory, used to store computer programs;

[0068] A processor is configured to implement the steps of the modality-level adaptive differential privacy pruning method in the above-described multimodal learning when executing the computer program.

[0069] Based on the same technical solution, this invention also proposes a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the modality-level adaptive differential privacy pruning method in multimodal learning described above. This computer-readable storage medium may include various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0070] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Claims

1. A modal-level adaptive differential privacy pruning method, characterized in that, The cutting method includes: The overall clipping value of the multimodal model is assigned to each modality; wherein the input of the multimodal model is multimodal data, which includes at least two of the following: text data, image data, and audio data; Each mode uses a normalized modality-level pruning mechanism for gradient pruning to control the pruning value on the speed of modality optimization. The methods for assigning the overall clipping value of a multimodal model to each modality include: Select a mode as the baseline mode, and the modality clipping value share of the baseline mode is 1 in each training round; Based on the real-time acquired modal contribution of each modality, the modal pruning value share of the remaining modalities other than the baseline mode is updated: ， Where p represents the reference mode number, Indicates the first The mode in the th ... The modal clipping value share of the training round. Indicates the first The mode in the th ... The modal clipping value share of the training round. , Indicates the number obtained in real time The modal contribution of each modality in the training round 1. This indicates that the real-time acquired reference mode is in the 1st... Modal contribution during round training Indicates the first The expected modal contribution of each modality. This represents the expected modal contribution of the baseline mode. and Both represent parameters used to control the magnitude of changes in the trimming value. Indicates the total number of training rounds; Based on the modal clipping value share of each modality, the overall clipping value of the multimodal model is allocated to each modality; The methods for obtaining the modality contribution of each modality in each training round include: In a training round, using only one modality's data as input to the multimodal model, the reciprocal of the calculated model loss represents the modal contribution of that modality in that training round. In a certain training round, when multimodal fusion is performed, only one modality is retained and the other modalities are masked. The reciprocal of the calculated model loss is the modal contribution of that modality in that training round. The normalization-based modal-level clipping mechanism works as follows: The gradient of the modality to be clipped is normalized, and then scaled proportionally according to the clipping value assigned to that modality. The method for gradient normalization is as follows: ， in, Indicates the uncut version The mode in the th ... The gradient of the training round, express The normalized result is given by r, which is a constant parameter.

2. A multimodal learning method, characterized in that, The multimodal learning method employs the pruning method described in claim 1 to perform gradient pruning in order to update the modal model parameters of each modality.

3. The multimodal learning method as described in claim 2, characterized in that, The model parameter update expressions for each modality are as follows: ， in, Indicates the first The mode in the th ... Modal model parameters during round training, Indicates the first The modal model parameters of each modality in the round of training. It's the learning rate. Indicates the first The noise variance of each mode, Indicates the first The mode in the th ... The modality clipping values ​​assigned during each training round. Indicates the uncut version The mode in the th ... Gradient of training rounds The normalization result, This represents a multivariate standard normal distribution with a mean of 0 and a covariance matrix that is the identity matrix.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the trimming method as described in claim 1.

5. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for implementing the steps of the trimming method as described in claim 1 when executing the computer program.

Images