Missing modal robustness enhancement training method based on multi-modal probability shape-preserving distillation

Abstract

The invention relates to a missing mode robustness enhancement training method based on multi-mode probability shape-preserving distillation, and the method comprises the following steps: obtaining a first multi-mode training set with a complete mode, defining an auxiliary indication vector to represent whether the mode is missing or not, and constructing a second multi-mode training set with mode missing; respectively mapping each input mode in the second multi-mode training set into independent D-dimensional normal distribution, and fusing to obtain fused D-dimensional normal distribution; end-to-end distillation is performed based on a self-distillation framework, in a training stage, a teacher model processes a first multi-modal training set and provides complete modal representation, and a student model processes a second multi-modal training set and provides complete modal representation supervision. And on the basis of the independent D-dimensional normal distribution of each input mode, the D-dimensional normal distribution and the complete mode representation are fused to calculate probability extremum optimization loss and geometric consistency optimization loss, and a loss function is constructed to carry out model training. Compared with the prior art, the method has the advantages that the effectiveness of information migration can be improved, and the processing capacity of the model to the missing mode is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal learning, and in particular to a missing modality robustness enhancement training method based on multimodal probability conformal distillation. Background Art

[0002] Traditional multimodal learning usually presupposes that all modalities of data are complete during training and testing. However, such perfect conditions are usually difficult to achieve due to data acquisition constraints such as equipment limitations, budget constraints, and restricted working conditions. When only some modalities are available, multimodal models trained on modally complete data often experience significant performance degradation. Given the important role of multimodal learning in the real world, how to improve the robustness of multimodal learning in the face of missing modal data has attracted a series of explorations recently.

[0003] Existing methods to solve this problem are usually divided into two paradigms, namely independent modeling for different combinations of modal missingness and unified modeling, of which the latter is more popular due to its advantages of low storage cost and strong flexibility. As a popular method for unified modeling, cross-modal knowledge distillation (KD) has made continuous progress in recent years. It attempts to guide the alignment of modality missing representations with the corresponding modality complete representations, and promote the training of modality missing representations under the supervision of modality complete representations with privileged information. The paper "Multimodal Learning with Incomplete Modalities by KnowledgeDistillation" discloses a single-modal information supplementation framework based on knowledge distillation, which uses a single-modal model trained independently on all available data as a teacher and trains the student model with samples with complete modalities. The paper "MMANet: Margin-aware Distillation and Modality-aware Regularization for IncompleteMultimodal Learning" discloses a missing modality algorithm based on edge-aware distillation and modality-aware regularization, which uses edge-aware distillation to assist in aligning the complete modality representation with the missing modality representation by weighing sample contribution and classification uncertainty. However, the above methods do not take into account the inherent information asymmetry problem, that is, once one modality is missing, it is impossible to recover its personalized information by forced alignment. Ignoring the inherent information asymmetry in such alignment may cause multimodal models to incorrectly fit false information.

[0004] In summary, the prior art has the following defects:

[0005] 1. High storage cost and poor flexibility: Existing work on independent modeling of different modal missing combinations trains a specific model for each missing case, which is often not flexible enough for actual scenarios and has a huge number of parameters.

[0006] 2. Weak information capture capability: Existing methods that attempt to extract redundant information across modalities by designing different fusion networks often ignore the complementary information of each modality, resulting in the poor performance of the unified multimodal model relative to specific models.

[0007] 3. Low efficiency of information transfer: Existing knowledge distillation-based methods usually emphasize the precise alignment of modality-missing representations with their complete corresponding representations, which may lead to low information transfer due to overfitting on spurious features due to inherent information asymmetry. Summary of the invention

[0008] The purpose of the present invention is to provide a missing modality robustness enhancement training method based on multimodal probability conformal distillation to address the problem of modality missingness that is prevalent in multimodal learning scenarios. As a result, multimodal models trained on modality complete data will suffer serious performance degradation when encountering modality missing data.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] Existing methods based on cross-modal knowledge distillation usually accurately align the representations of modal missing data and their corresponding modal complete data to enhance multimodal robustness. However, due to the inherent information asymmetry between the two, this deterministic alignment is too difficult and can easily lead to the modal missing features mistakenly capturing false information. The present invention proposes a missing modality robustness enhancement training method based on multimodal probabilistic conformal distillation. This method takes into account the inherent uncertainty in the alignment of modal missing and complete modalities, and aims to learn the unknown probability density function of the mapping variables in the modal complete space, rather than relying on forced point alignment.

[0011] A missing modality robustness enhancement training method based on multimodal probability shape-preserving distillation, the method comprising the following steps:

[0012] S1, obtaining a first multimodal training set with complete modalities, defining an auxiliary indicator vector to indicate whether a modality is missing, and using the auxiliary indicator vector to construct a second multimodal training set with missing modalities;

[0013] S2, mapping each input modality in the second multimodal training set into a separate D-dimensional normal distribution;

[0014] S3, fusing the individual D-dimensional normal distributions of each input modality to obtain a fused D-dimensional normal distribution;

[0015] S4, end-to-end distillation is performed based on the self-distillation framework, wherein the parameters of the fixed teacher model are obtained through pre-training. In the training phase, the teacher model processes the first multimodal training set to provide a complete modal representation, and the student model processes the second multimodal training set to provide complete modal representation supervision. The probability extreme value optimization loss and the geometric consistency optimization loss are calculated based on the separate D-dimensional normal distribution of each input modality, the fused D-dimensional normal distribution and the complete modal representation, and the model is trained using the probability extreme value optimization loss and the geometric consistency optimization loss.

[0016] As a preferred technical solution, the input modalities in the multimodal training set include at least two of RGB images, depth images, text, voice and video.

[0017] As a preferred technical solution, the step S1 is specifically as follows: given a first multimodal training set with complete modalities Among them, each input It consists of multimodal data and is represented as y i Represents the corresponding true label, N represents the dataset size, and M represents the number of modalities; define the modality missing data and for x i Define an auxiliary indicator vector δ i Represents x i Is the modality in missing, where in, Indicates that the corresponding mode is missing. Indicates the existence of the corresponding mode, based on the auxiliary indicator vector δ i Construct a second multimodal training set with missing modalities using the first multimodal training set

[0018] As a preferred technical solution, the step S2 is specifically as follows: for each input mode Extract the network f using the given feature mean m and variance extraction head h m , the input modal data Mapped to a separate D-dimensional normal distribution

[0019]

[0020] in, are the characteristic mean and variance of the D-dimensional normal distribution, D is the dimension; Represents the representation of the missing mode m.

[0021] As a preferred technical solution, the step S3 is specifically as follows: using a given multimodal feature fusion network f and a fusion variance extraction head h, the feature mean of each input modality is fused to obtain the feature mean corresponding to the input x i The fusion D-dimensional normal distribution q(z i |x i ):

[0022]

[0023] in, σ i =h(μ i ),μ i ∈R D ,σ i ∈R D are the fused mean and variance of the D-dimensional normal distribution, D is the dimension; z i Represents fused modality representation.

[0024] As a preferred technical solution, for the fusion D-dimensional normal distribution q(z i |x i ), the calculation formula of probability extreme value optimization loss is:

[0025]

[0026] For input modality The corresponding single D-dimensional normal distribution The calculation formula of probability extreme value optimization loss is:

[0027]

[0028] Among them, z ＊ represents the D-dimensional complete modal representation; the positive set Z p by and Complete representation of D-dimensional modalities with the same label Composition, that is Label y p Satisfy p =y i ; Negative set Z n By all and Full modality representation with different labels Composition, that is Label y n Satisfy n ≠y i ; {p|y p =y i} represents the corresponding complete modal representation Belongs to positive set Z p ,{n|yn ≠y i} represents the corresponding complete modal representation Belongs to negative set Z m .

[0029] As a preferred technical solution, the calculation method of the geometric consistency optimization loss is:

[0030] Compute a complete modal characterization The corresponding geometric vector And each missing mode representation z i , The corresponding geometric vector g i ,

[0031]

[0032] Among them, g i , is a |B|-dimensional vector, |B| is the batch size, represents the complete modal representation of the bth complete modal sample in the batch, μ b are the feature mean and fusion mean of modality m of the b-th missing modality sample, respectively, and α(x,y) represents the distance between x and y;

[0033] Constructs a vector containing positive geometry The positive set G p , its core With X i Share the same class, i.e. and the negative set containing the remaining geometric vectors The similarity function is calculated using the following learning-based contrast form

[0034]

[0035] Among them, β(g ＊ ,g) indicates the calculation of g ＊ The cosine similarity between and g, τ is the temperature coefficient;

[0036] Align g separately i , With g ＊ ∈G p , achieving the geometric consistency between the missing modal representation and the complete modal representation, and obtaining the geometrically consistent optimization loss:

[0037]

[0038] in, represents the geometrically consistent optimization loss of the fused D-dimensional normal distribution, Indicates the input mode Geometrically consistent optimization loss for a D-dimensional normal distribution.

[0039] As a preferred technical solution, the overall loss function of the model is constructed based on probability extreme value optimization loss, geometric consistency optimization loss and task loss.

[0040] As a preferred technical solution, the calculation formula of the overall loss function of the model is:

[0041]

[0042] Among them, λ is a hyperparameter used to balance different losses. Indicates mission loss,

[0043] As a preferred technical solution, when the model is used for downstream classification tasks, the task loss is classification cross entropy loss; when the model is used for downstream regression tasks, the task loss is regression loss; when the model is used for downstream segmentation tasks, the task loss is segmentation loss.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] (1) Compared with the traditional distillation method based only on point-to-point precise alignment, the present invention considers the uncertainty in the mapping from incomplete to complete to transfer the privileged information of the complete modal representation. This is a more tolerant alignment method that can more effectively transfer the complete modal information, greatly improving the effectiveness of information transfer.

[0046] (2) The present invention considers parameterizing the missing representations of different modalities into different distributions to fit their unknown probability density functions in the complete modal space. This multimodal probability conformal distillation is achieved by considering the probability of extreme points and ensuring the geometric consistency between the peak points of different probability density functions and the modeled distribution.

[0047] (3) The present invention designs a special loss function for the multimodal probabilistic shape-preserving distillation method, which fully utilizes the probability extrema and geometric consistency of the probability density function and can be easily integrated into any existing multimodal fusion system.

[0048] (4) The present invention further improves the training process and framework. The proposed self-distillation framework stabilizes the training effect and reduces the dependence on parameter adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1is a flow chart of the method of the present invention;

[0050] Figure 2 Schematic diagram of the loss function construction process of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0052] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0053] This embodiment provides a missing modality robustness enhancement training method based on multimodal probability shape-preserving distillation. Figure 1 As shown, the method comprises the following steps:

[0054] S1, obtain a first multimodal training set with complete modalities, define an auxiliary indicator vector to indicate whether a modality is missing, and use the auxiliary indicator vector to construct a second multimodal training set with missing modalities.

[0055] The input modalities in the multimodal training set include at least two of RGB images, depth images, text, speech, and video.

[0056] Given a first multimodal training set with complete modality Among them, each input It consists of multimodal data and is represented as y i Represents the corresponding true label, N represents the dataset size, and M represents the number of modalities; define the modality missing data and for x i Define an auxiliary indicator vector δ i Represents x i Is the modality in missing, where in, Indicates that the corresponding mode is missing. Indicates the existence of the corresponding mode, based on the auxiliary indicator vector δ i Construct a second multimodal training set with missing modalities using the first multimodal training set

[0057] This embodiment uses RBG image and depth image as input for detailed description: Given a complete RGB-depth first multimodal training set Each of the input It consists of RGB and depth image modalities and is represented as y i Denotes the corresponding true label. Define the missing data of the modality and for x i Using an auxiliary indicator vector δ i ,in Represents x i Is the modality in missing, so as to construct the second multimodal training set

[0058] S2, mapping each input modality in the second multimodal training set into a separate D-dimensional normal distribution.

[0059] For each input mode Extract the network f using the given feature mean m and variance extraction head h m , the input modal data Mapped to a separate D-dimensional normal distribution

[0060]

[0061] in, are the characteristic mean and variance of the D-dimensional normal distribution, D is the dimension; Represents the representation of the missing mode m.

[0062] In this embodiment, the given RGB image feature mean extraction network f 1 and variance extraction head h 1 , the input RGB image Mapped to a separate D-dimensional normal distribution:

[0063]

[0064] in, are the D-dimensional mean and variance, D is the dimension; Indicates missing representation of the RGB modality.

[0065] Using the given deep image feature mean extraction network f 2 and variance extraction head h 2 , the input depth image Mapped to a separate D-dimensional normal distribution:

[0066]

[0067] in, are the D-dimensional mean and variance, D is the dimension; Representation of missing deep modality.

[0068] S3, fusing the separate D-dimensional normal distribution of each input modality to obtain a fused D-dimensional normal distribution.

[0069] Using the given multimodal feature fusion network f and fusion variance extraction head h, the feature mean of each input modality is fused to obtain the corresponding input x i The fusion D-dimensional normal distribution q(z i |x i ):

[0070]

[0071] in, σ i =h(μ i ),μ i ∈R D ,σ i ∈R D are the fused mean and variance of the D-dimensional normal distribution, D is the dimension; z i represents the fusion modality representation. In this embodiment,

[0072] S4, end-to-end distillation is performed based on the self-distillation framework, wherein the parameters of the fixed teacher model are obtained through pre-training. In the training phase, the teacher model processes the first multimodal training set to provide a complete modal representation, and the student model processes the second multimodal training set to provide complete modal representation supervision. The probability extreme value optimization loss and the geometric consistency optimization loss are calculated based on the separate D-dimensional normal distribution of each input modality, the fused D-dimensional normal distribution and the complete modal representation, and the model is trained using the probability extreme value optimization loss and the geometric consistency optimization loss.

[0073] S41, calculate the probability extreme value optimization loss.

[0074] Using the mean and anti-scratch obtained in the above steps S2 and S3, combined with the complete modal representation z ＊ Perform probability extreme value optimization.

[0075] For the fused D-dimensional normal distribution q(z i |x i ), the calculation formula of probability extreme value optimization loss is:

[0076]

[0077] For input modality The corresponding single D-dimensional normal distribution The calculation formula of probability extreme value optimization loss is:

[0078]

[0079] Among them, z ＊ represents the D-dimensional complete modal representation; the positive set Z p by and Complete representation of D-dimensional modalities with the same label Composition, that is Label y p Satisfy p =y i ; Negative set Z n By all and Full modality representation with different labels Composition, that is Label y n Satisfy n ≠y i ; {p|y p =y i} represents the corresponding complete modal representation Belongs to positive set Z p ,{n|y n ≠y i} represents the corresponding complete modal representation Belongs to negative set Z n .

[0080] Using the above method, we can get q(z i |x i )’s probability extreme value optimization loss as well as The probability extreme value optimization loss

[0081] S42, compute geometrically consistent optimization loss.

[0082] Compute a complete modal characterization The corresponding geometric vector And each missing mode representation z i , The corresponding geometric vector g i ,

[0083]

[0084] Among them, g i , is a |B|-dimensional vector, |B| is the batch size, represents the complete modal representation of the bth complete modal sample in the batch, μ b are the feature mean and fusion mean of modality m of the b-th missing modality sample, and α(x,y) represents the distance between x and y.

[0085] Similar to positive set Z p and negative set Z n , constructs a positive geometric vector The positive set G p , its core With X i Share the same class, i.e. and the negative set containing the remaining geometric vectors The similarity function is calculated using the following learning-based contrast form

[0086]

[0087] Among them, β(g ＊ ,g) indicates the calculation of g ＊ and g, where τ is the temperature coefficient.

[0088] Align g separately i , With g ＊ ∈G p , achieving the geometric consistency between the missing modal representation and the complete modal representation, and obtaining the geometrically consistent optimization loss:

[0089]

[0090] in, represents the geometrically consistent optimization loss of the fused D-dimensional normal distribution, Indicates the input mode Geometrically consistent optimization loss for a D-dimensional normal distribution.

[0091] From the above steps, we can get q(z i |x i ) is a geometrically consistent optimization loss as well as Geometrically consistent optimization loss

[0092] S43, calculate the task loss.

[0093] When the model is used for downstream classification tasks, the task loss is the classification cross entropy loss; when the model is used for downstream regression tasks, the task loss is the regression loss; when the model is used for downstream segmentation tasks, the task loss is the segmentation loss.

[0094] S44, calculate the overall loss function of the model.

[0095] like Figure 2 As shown in Figure 1, the overall loss function of the model is constructed based on probability extreme value optimization loss, geometric consistency optimization loss and task loss, and its calculation formula is:

[0096]

[0097] Among them, λ is a hyperparameter used to balance different losses. Indicates mission loss,

[0098] This embodiment takes the classification task as an example. represents the classification cross entropy loss,

[0099] This embodiment has been tested on two trimodal classification benchmark datasets and two bimodal segmentation benchmark datasets, and has achieved the most advanced performance compared to a variety of existing missing modality robust methods. In particular, in the seven modality missing scenarios on the classification dataset CeFA, the present invention has achieved an average performance improvement of about 5%. In addition, from the perspective of ablation, this embodiment has conducted experiments from multiple perspectives to prove the effectiveness of each component of the present invention.

[0100] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A missing modality robustness enhancement training method based on multimodal probabilistic shape-preserving distillation, characterized in that: The method comprises the following steps: S1, obtaining a first multimodal training set with complete modalities, defining an auxiliary indicator vector to indicate whether a modality is missing, and using the auxiliary indicator vector to construct a second multimodal training set with missing modalities; S2, mapping each input modality in the second multimodal training set into a separate D-dimensional normal distribution; S3, fusing the individual D-dimensional normal distributions of each input modality to obtain a fused D-dimensional normal distribution; S4, end-to-end distillation is performed based on the self-distillation framework, wherein the parameters of the fixed teacher model are obtained through pre-training. In the training phase, the teacher model processes the first multimodal training set to provide a complete modal representation, and the student model processes the second multimodal training set to provide complete modal representation supervision. The probability extreme value optimization loss and the geometric consistency optimization loss are calculated based on the separate D-dimensional normal distribution of each input modality, the fused D-dimensional normal distribution and the complete modal representation, and the model is trained using the probability extreme value optimization loss and the geometric consistency optimization loss.

2. According to claim 1, a missing modality robustness enhancement training method based on multimodal probability shape-preserving distillation is characterized in that: The input modalities in the multimodal training set include at least two of RGB images, depth images, text, voice and video.

3. The missing modality robustness enhancement training method based on multimodal probability conformal distillation according to claim 1, characterized in that: The step S1 is specifically as follows: given a first multimodal training set with complete modality Among them, each input It consists of multimodal data and is represented as y i Represents the corresponding true label, N represents the dataset size, and M represents the number of modalities; define the modality missing data and for x i Define an auxiliary indicator vector δ i Represents x i Is the modality in missing, where in, Indicates that the corresponding mode is missing. Indicates the existence of the corresponding mode, based on the auxiliary indicator vector δ i Construct a second multimodal training set with missing modalities using the first multimodal training set 4. The missing modality robustness enhancement training method based on multimodal probability conformal distillation according to claim 1, characterized in that: The step S2 is specifically as follows: for each input mode Extract the network f using the given feature mean m and variance extraction head h m , the input modal data Mapped to a separate D-dimensional normal distribution in, are the characteristic mean and variance of the D-dimensional normal distribution, D is the dimension; Represents the representation of the missing mode m.

5. The missing modality robustness enhancement training method based on multimodal probability shape-preserving distillation according to claim 4, characterized in that: The step S3 is specifically as follows: using the given multimodal feature fusion network f and fusion variance extraction head h, the feature mean of each input modality is fused to obtain the feature mean corresponding to the input x i The fusion D-dimensional normal distribution q(z i |x i ): in, σ i =h(μ i ),μ i ∈R D ,σ i ∈R D are the fused mean and variance of the D-dimensional normal distribution, D is the dimension; z i Represents fused modality representation.

6. The missing modality robustness enhancement training method based on multimodal probability shape-preserving distillation according to claim 5, characterized in that: For the fused D-dimensional normal distribution q(z i |x i ), the calculation formula of probability extreme value optimization loss is: For input modality The corresponding single D-dimensional normal distribution The calculation formula of probability extreme value optimization loss is: Among them, z ＊ represents the D-dimensional complete modal representation; the positive set Z p by and Complete representation of D-dimensional modalities with the same label Composition, that is Label y p Satisfy p =y i ; Negative set Z n By all and Full modality representation with different labels Composition, that is Label y n Satisfy n ≠y i ; {p|y p =y i } represents the corresponding complete modal representation Belongs to positive set Z p ,{n|y n ≠y i } represents the corresponding complete modal representation Belongs to negative set Z n .

7. The missing modality robustness enhancement training method based on multimodal probability shape-preserving distillation according to claim 6, characterized in that: The calculation method of the geometric consistency optimization loss is: Compute a complete modal characterization The corresponding geometric vector And each missing modal representation z i , The corresponding geometric vector g i , Among them, g i , is a |B|-dimensional vector, |B| is the batch size, represents the complete modal representation of the bth complete modal sample in the batch, μb are the feature mean and fusion mean of modality m of the b-th missing modality sample, and α(x, y) represents the distance between x and y; Constructs a vector containing positive geometry The positive set G p , its core With X i Share the same class, i.e. and the negative set containing the remaining geometric vectors The similarity function is calculated using the following learning-based contrastive form Where β(g*, g) represents the calculation of the cosine similarity between g* and g, and T is the temperature coefficient; Align g separately i , With g ＊ ∈G p , achieving the geometric consistency between the missing modal representation and the complete modal representation, and obtaining the geometrically consistent optimization loss: in, represents the geometrically consistent optimization loss of the fused D-dimensional normal distribution, Indicates the input mode Geometrically consistent optimization loss for a D-dimensional normal distribution.

8. The missing modality robustness enhancement training method based on multimodal probability shape-preserving distillation according to claim 7, characterized in that: The overall loss function of the model is constructed based on probability extremum optimization loss, geometric consistency optimization loss and task loss.

9. The missing modality robustness enhancement training method based on multimodal probability shape-preserving distillation according to claim 8, characterized in that: The calculation formula of the overall loss function of the model is: Among them, λ is a hyperparameter used to balance different losses. Indicates mission loss, 10. The missing modality robustness enhancement training method based on multimodal probability shape-preserving distillation according to claim 8, characterized in that: When the model is used for downstream classification tasks, the task loss is the classification cross entropy loss; when the model is used for downstream regression tasks, the task loss is the regression loss; when the model is used for downstream segmentation tasks, the task loss is the segmentation loss.

Images