Residual direction disambiguation and state calibration method for incomplete multi-modal learning

Abstract

The application provides a residual direction disambiguation and state calibration method for incomplete multi-modal learning, and belongs to the technical field of artificial intelligence and multi-modal information processing. The method comprises the following steps: constructing a quality consistency anchor point state of a multi-modal sample, and generating a preliminary prediction value; maintaining a global state-label memory bank, and constructing a candidate reference sample set; bipolar reference perception calibration: constructing a bipolar reference sample set based on the candidate sample reference set, calibrating the quality consistency anchor point state of the multi-modal sample based on the bipolar reference sample set, and obtaining a final calibration prediction value; training a model to obtain a trained residual direction disambiguation and state calibration model; and calibrating the state of a sample by using the trained residual direction disambiguation and state calibration model, and finally obtaining a calibrated prediction value. The application can reduce the probability of residual direction judgment error, and improve the prediction stability and robustness of the model in a high missing rate, low information amount and multi-modal observation incomplete scene.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal information processing technology, and in particular relates to a residual direction disambiguation and state calibration method for incomplete multimodal learning. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal learning has made significant progress in areas such as sentiment analysis and fake news detection. However, in real-world multimodal data applications, the data is often incomplete. For example, social media posts may lose images during the crawling process, short videos may lack audio, and user records may be partially unavailable due to privacy controls, transmission failures, and platform-specific preprocessing. Therefore, incomplete multimodal learning aims to make reliable predictions from only a subset of available modalities, which has become an important research topic in the field of multimodal information processing.

[0003] Existing techniques for incomplete multimodal learning can be mainly divided into the following three technical approaches:

[0004] 1. Modality Reconstruction and Evidence Completion Methods: These methods treat modality loss as a form of evidence attenuation, attempting to reconstruct missing modal information through generative models or translation networks. Core techniques include learning robust joint representations using recurrent translation, recovering missing modal features using imaginative networks or diffusion models, and employing graph completion-based techniques for modality reconstruction. The core idea behind these methods is that different subsets of observed modalities contain varying amounts of label-related information; reconstructing missing modalities or completing evidence improves predictive performance.

[0005] 2. Robust Representation and Regularization Methods: These methods do not explicitly reconstruct missing modalities. Instead, they improve the quality of shared representations by designing robust fusion mechanisms, cross-modal regularization, knowledge distillation, or modality enhancement. Their technical approaches encompass using mask fusion, cross-modal regularization, and distillation techniques to enhance model robustness, as well as employing relative advantage-aware cross-modal learning, surrogate regularization, or modality enhancement strategies to optimize the latent space structure. The core objective of these methods is to internally improve the quality of available evidence or enrich its representation quantity, thereby addressing the information decay caused by modality loss.

[0006] 3. Retrieval Enhancement and Contextual Hints: These methods enhance predictive performance by retrieving auxiliary samples similar to the current sample from memory or external knowledge sources and introducing them into the model as context for demonstrations, dynamic hints, or associative reasoning. Techniques include context-based multimodal completion, dynamic hint tuning for retrieval enhancement, and cross-sample associative reasoning. The core of these methods lies in transforming information from other historical samples into demonstrations, hints, or auxiliary supervision signals, thereby compensating for the lack of information due to missing modalities.

[0007] However, the aforementioned existing technologies still have significant limitations when dealing with incomplete multimodal data. Existing methods primarily focus on increasing or improving the quantity and quality of available evidence, but overlook a crucial failure mode: after the model derives a rough preliminary prediction using existing modes, it cannot determine whether the latent state should be corrected in the positive or negative direction to reduce prediction error. This phenomenon is known as "residual orientation ambiguity."

[0008] While existing similarity-based retrieval augmentation methods can find auxiliary samples similar to the current state, simple similarity measurements do not contain information about the order or geometric orientation of the labels. Neighboring reference samples may contain instances with both large and small label values, making it impossible for them to provide a directional calibration signal with clear sign guidance for correcting the current state. Therefore, existing techniques struggle to effectively resolve the residual orientation ambiguity caused by modality loss, limiting the prediction accuracy and robustness of the model in scenarios with high loss of information or low information content.

[0009] In summary, while existing technologies can generate preliminary predictions under conditions of modality missing, they fail to address the following technical requirement: how to extract signed implicit correction directions from historical state-label pairs without relying on test labels, ensuring that the calibrated state meets gradient-related technical conditions. This allows for adjustment towards lower labels when predictions are too high and towards higher labels when predictions are too low. This technical requirement has long existed in the field of incomplete multimodal learning, but current technologies have not yet provided an effective solution. Summary of the Invention

[0010] The present invention aims to provide a residual orientation disambiguation and state calibration method for incomplete multimodal learning. This method is executed by a computer device. For multimodal input samples containing one or more missing modalities, it first generates quality-consistent anchor states based on an availability mask and a reliability-weighted fusion mechanism. Then, it constructs a global state-label memory based on the historical state-label pairs of the training samples. Next, based on the preliminary prediction value of the current sample, the retrieved historical reference samples are divided into a high-label reference sample set and a low-label reference sample set, and a bipolar reference orientation is constructed based on the difference in state centroids and label means between the high-label and low-label reference sample sets. Finally, using a gated calibration network, a signed latent state update is generated based on the relative position of the current anchor state in the bipolar reference frame, resulting in the calibrated task state and the final prediction result. This addresses the technical problem of existing incomplete multimodal learning methods that can only generate coarse predictions under modality-missing conditions, but struggle to determine the latent state correction direction.

[0011] Unlike methods that rely solely on similarity to retrieve auxiliary samples, this invention constructs historical state-label pairs as a local reference coordinate system with label order relationships. This transforms the retrieved information from merely undirected context into a signed calibration signal that indicates whether the latent state should be adjusted towards a higher or lower label direction. Consequently, without relying on the true labels of the current test samples, the probability of incorrect residual direction determination can be reduced, improving the model's predictive stability and robustness in scenarios with high missing rates, low information content, and incomplete multimodal observations.

[0012] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0013] A residual orientation disambiguation and state calibration method for incomplete multimodal learning, the method comprising the following steps:

[0014] Step S1: Obtain multimodal samples, construct quality consistency anchor states for multimodal samples, and generate preliminary prediction values.

[0015] In this step, the system receives incomplete multimodal input data and constructs multimodal samples. The multimodal input data primarily originates from practical application scenarios such as social media content analysis, short video understanding, or human-computer interaction recordings. It typically includes one or more combinations of visual modalities (image frames and video clips), text modalities (post text, comments, and subtitles), and audio modalities (speech signals and background music). Through availability masking and reliability-weighted fusion mechanisms, a smooth potential task state anchor point free from missing modal human contributions is constructed. Specifically:

[0016] Step S11: Obtain multimodal samples, perform modal encoding on the multimodal samples to obtain slot sequences for each modality, and perform mean pooling on the slot sequences to obtain modal summary vectors.

[0017] For the A multimodal sample, let Represents the set of all modalities. Modality , Indicates the first Multimodal sample modes Availability indicator of whether it has been observed, where Representing modes Available Representing modes Missing. The available modal set is defined as follows: The observed input is , Indicates the first A multimodal sample in modality The input features are as follows. This represents the set of observation inputs for a multimodal sample, consisting of all available modal inputs.

[0018] Each multimodal sample corresponds to an ordered scalar label. This label is obtained during the data collection phase through manual annotation, expert review, or based on the results of fact-checking organizations. For example, in sentiment analysis, It represents a continuous emotional intensity value; in rumor detection, This is the mapped ordered scalar. For any sample, all its available modalities share the same true label, i.e., the ordered scalar label. The multimodal dataset containing labels is divided into a training set, a validation set, and a test set. The training set is used to perform model parameter optimization as described in step S4, and to build and update the global state-label memory in step S2; the validation set and the test set are used to evaluate model performance. During the validation and testing phases, the global state-label memory remains frozen, and the state of the current samples is calibrated only using the memory built during the training phase. Label information from the validation set or test set is not written to the memory to prevent data leakage.

[0019] For each available mode First, through the feature encoder of the corresponding mode. Extract each available modality feature, then pass it through a one-dimensional convolution. and pooling operations And add learnable slot position embedding vectors This yields a compact slot sequence. :

[0020] (1)

[0021] in Indicates the first A multimodal sample in modality The slot sequence below, Indicates modality Feature encoder, Indicates modality One-dimensional convolution, Indicates modality Pooling operations, This represents the learnable slot position embedding vector.

[0022] Next, mean pooling is performed on the slot sequence to obtain the modality summary vector. :

[0023] (2)

[0024] in, Indicates the first A multimodal sample in modality The modal summarization vector below, This indicates that the average pooling operation is performed on the slot sequence.

[0025] Step S12: Calculate the reliability score and fusion weight of each modality based on the modality summary vector.

[0026] Based on the modality summary vector obtained in step S11, a reliability score is calculated for each modality. And calculate the fusion weight under availability mask constraint based on the reliability score.

[0027] The reliability score The calculation formula is:

[0028] (3)

[0029] in, Indicates the first A multimodal sample in modality The reliability score below, This represents a learnable weight vector. This represents the learnable weight matrix. This represents a learnable bias vector. This represents the hyperbolic tangent activation function. This represents the transpose of a vector.

[0030] Based on the reliability score and availability indicator Calculate the fusion weights The calculation method is as follows:

[0031] (4)

[0032] in, Indicates the first A multimodal sample in modality The final fusion weight, Indicates the first Multimodal sample modes Availability indicator as to whether it has been observed. This represents an exponential function, where the summation operation in the denominator iterates through all modes. , Indicates the first Multimodal sample modes Availability indicator as to whether it has been observed. Indicates the first A multimodal sample in modality The reliability score is as follows.

[0033] Because the molecule contains availability indicators When the mode When missing, This makes the fusion weights corresponding to the missing mode... Strictly zero; meanwhile, the denominator is normalized only for the available modes, so that the sum of the fusion weights corresponding to all available modes is 1.

[0034] Step S13: Initialize the joint sequence, and iteratively update the joint sequence through a multi-layer mask self-attention processing module to obtain the first... Layer joint sequence, based on the first The layer-joint sequence generates quality-consistent anchor point states.

[0035] Based on the fusion weights obtained in step S12, the summation vectors of each modality are weighted and summed, and a learnable embedding of the global seed type is added to obtain a global seed vector. For each modality, the corresponding slot sequence is added to the learnable embedding of the modality type, and processed using an availability indicator to obtain a modality slot sequence processed by the availability indicator. The global seed vector and the modality slot sequences processed by the availability indicator are concatenated to construct an initial joint sequence. The interaction relationships between available modalities are refined through a multi-layer mask self-attention processing module to obtain the first... Layer joint sequence, finally extracting the first layer The first position vector of the layer joint sequence serves as the quality consistency anchor state. .

[0036] Initial joint sequence The construction formula is:

[0037] (5)

[0038] in, This represents the initial joint sequence for initialization, containing a global seed vector ( ) and the sequence of modal slots processed by availability indicators ( ), Indicates the first A multimodal sample in modality The final fusion weight, Indicates the first A multimodal sample in modality The modal summarization vector below, Learnable embeddings representing global seed types Indicates the first A multimodal sample in modality The slot sequence below, Indicates the first Multimodal sample modes Availability indicator as to whether it has been observed. Representing modes Type of learnable embeddings, It represents the set of all modalities.

[0039] Subsequently, the initial joint sequence was passed through The layer mask self-attention processing module JointBlock performs iterative updates:

[0040] (6)

[0041] in, , , Indicates the first Layer and first The layer mask is the joint sequence after processing by the attention module. Indicates the first Layer mask self-attention processing module.

[0042] through The layer mask self-attention processing module processes the data to obtain the first layer mask. Layer joint sequence .

[0043] Final extraction of the first The first position vector of the layer joint sequence serves as the quality consistency anchor state:

[0044] (7)

[0045] in, Indicates the final extracted number The quality consistency anchor state of a multimodal sample. Indicates taking the first Layer joint sequence (i.e.) The first position vector of the final layer output sequence of the layer mask self-attention processing module.

[0046] Step S14: Input the quality consistency anchor point status into the prediction head function to obtain the preliminary prediction value.

[0047] Based on the quality consistency anchor point status obtained in step S13 Preliminary prediction values ​​are generated through the prediction head function. :

[0048] (8)

[0049] in, Indicates the prediction head function. Indicates the first Preliminary predicted values ​​for a multimodal sample.

[0050] Step S2: Maintain the global state-label memory, construct the candidate reference sample set, and maintain the high-label tail prototype vector and the low-label tail prototype vector.

[0051] In this step, the system maintains a global state-label memory to store the historical quality consistency anchor states and their corresponding labels for training samples, and to provide local reference samples for the current sample. The global state-label memory is used to construct a local reference coordinate system with label order relationships, enabling subsequent calibration processes to generate signed potential state correction directions based on historical state-label pairs. During the training phase, the global state-label memory is constructed or updated by the quality consistency anchor states and their true labels for the training set samples; during the validation or testing phase, the global state-label memory remains frozen and serves as a read-only retrieval source, storing only historical state-label pairs accumulated during the training phase. The true labels of the samples to be predicted from the validation or test set are not involved in memory writing, neighborhood partitioning, bipolar reference direction construction, or gated calibration calculations; they are only used for the final performance evaluation.

[0052] Step S21: Construct a global state-label memory, calculate the multimodal sample query key vector and the global state-label memory key vector; select the multimodal sample query key vector with the highest cosine similarity to all global state-label memory key vectors. A set of 10 samples constitutes the candidate reference sample set. .

[0053] Construct a global state-label memory using anchor states of quality consistency from training set samples and their true labels. ,in Indicates the first The quality consistency anchor state is saved when each historical training sample is written to the global state-label memory. The quality consistency anchor state is obtained by the method described in step S1. Indicates the first The ordered scalar labels corresponding to each historical training sample. Indicates the size of the memory bank. This serves as an index for historical training samples in the memory bank. ; Indicates the first A set of memory key vectors used for retrieval.

[0054] The first A set of memory key vectors used for retrieval The calculation formula is:

[0055] (9)

[0056] in, express Normalization operation, Let represent the learnable linear projection matrix. For the Each historical training sample is used as an anchor point for its quality consistency. Projected through the linear projection matrix and then... Normalization yields the memory key vector. Through this projection and normalization operation, the quality consistency anchor states of historical training samples are mapped into a key space suitable for similarity retrieval.

[0057] For the current number The quality consistency anchor state of a multimodal sample Calculate its query key vector :

[0058] (10)

[0059] in, Indicates the first The query key vector of a multimodal sample; It is a learnable projection matrix.

[0060] Subsequently, the query key vector is calculated. With all global state-label memory key vectors The cosine similarity between them is calculated using the following formula:

[0061] (11)

[0062] in, Indicates the first The query key vector of the multimodal sample and the first Cosine similarity between the memory key vectors used for retrieval; Let Euclidean norm represent the vector. and Both have been normalized, and the cosine similarity is simplified to the dot product of the two: .

[0063] Next, from the global state-tag memory Select the top cosine similarity values. A set of 10 samples constitutes the candidate reference sample set. .

[0064] Step S22: Maintain the high-label tail prototype vector and the low-label tail prototype vector based on the exponential moving average (EMA).

[0065] To avoid the candidate reference sample set of the current sample After bipolar partitioning, if either the high-label reference sample set or the low-label reference sample set is empty, a stable bipolar reference direction cannot be constructed. Therefore, the high-label tail prototype vector is maintained separately. and low-label tail prototype vector The high-label tail prototype vector represents the representative quality consistency anchor state corresponding to samples with higher true labels in the training set, while the low-label tail prototype vector represents the representative quality consistency anchor state corresponding to samples with lower true labels in the training set. Given the tail quantization quantile... momentum coefficient and the true ordered scalar label range of the training set samples. Define a high label threshold and low label threshold :

[0066] (12)

[0067] (13)

[0068] in, Indicates a high label threshold. This indicates a low label threshold; This represents the minimum value of the true ordered scalar label of a sample in the training set; This represents the maximum value of the true ordered scalar labels of the training set samples. The true ordered scalar labels satisfy... The training samples constitute a high-label tail sample set, and the true ordered scalar labels satisfy... The training samples constitute the low-label tail sample set.

[0069] Based on the high label threshold and low label threshold The average value of the quality consistency anchor point states in the high-label tail sample set and the low-label tail sample set is calculated respectively to obtain the average state of the high-label tail. and low label tail average state The high-label tail prototype vector is updated using an exponential moving average method. and low-label tail prototype vector The update rules are as follows:

[0070] (14)

[0071] (15)

[0072] in, This represents the high-label tail prototype vector. This represents the low-label tail prototype vector. This represents the average state at the high-label tail, where the true label in the training samples currently written to memory is not less than the high-label threshold. The average value of the quality consistency anchor state of the training samples; This represents the average state at the low-label tail, where the true label in the training samples currently written to memory is no greater than the low-label threshold. The average value of the quality consistency anchor state of the training samples. This represents the momentum coefficient.

[0073] Step S3: Bipolar reference sensing calibration: Construct a bipolar reference sample set based on the candidate sample reference set, and calibrate the quality consistency anchor state of the multimodal samples based on the bipolar reference sample set to obtain the final calibration prediction value.

[0074] In this step, the system transforms the retrieved labeled historical neighbors into signed latent state correction signals, thereby resolving the residual orientation ambiguity problem in incomplete multimodal learning. Specifically, based on the preliminary prediction value of the current sample, the system divides the candidate sample reference set into high-label reference neighborhoods and low-label reference neighborhoods, and constructs a bipolar reference orientation using the difference in state centroids and label mean differences between the two neighborhoods. Subsequently, it generates the calibrated task state and the final prediction value through gating symbol updates.

[0075] Step S31: Construct a high-label reference sample set and a low-label reference sample set based on the candidate sample reference set and the preliminary predicted values.

[0076] Based on the candidate sample reference set retrieved in step S2 Based on the preliminary predicted value obtained in step S14 The candidate sample reference set is divided into a high-label reference sample set. and low-label reference sample set :

[0077] (16)

[0078] (17)

[0079] in, For the first A set of multimodal high-label reference samples, representing the true ordered scalar labels of the candidate reference samples. Compared to the initial forecast value The set of neighbors with significantly higher label values For the first A set of low-label reference samples representing multimodal samples, indicating the true ordered scalar labels of the candidate reference samples. Compared to the initial forecast value The set of neighbors with significantly lower label values This indicates the preset label interval threshold; Indicates the first The true ordered scalar labels of the candidate reference samples.

[0080] No. A set of multimodal sample high-label reference samples With low-label reference sample set Constituting the first A bipolar reference sample set of multimodal samples .

[0081] Step S32: Based on the high-label reference sample set and the low-label reference sample set, calculate the similarity weight, weighted state centroid, and weighted label mean of the multimodal samples in the high-label reference sample set and the low-label reference sample set, respectively.

[0082] For neighbors in the high-label reference sample set or the low-label reference sample set Its similarity weight and The calculation formula is:

[0083] (18)

[0084] (19)

[0085] in, Indicates the first The neighbors of a multimodal sample and a set of high-label reference samples. Similarity weights, Indicates the first The neighbors of a multimodal sample and a low-label reference sample set Similarity weights; Indicates the temperature coefficient. This represents the function for calculating cosine similarity. Indicates the first The query key vector of a multimodal sample. Indicates the first A set of memory key vectors used for retrieval. Represents the bipolar reference sample set The summation index is used to iterate through each reference sample. Indicates the first A set of memory key vectors used for retrieval.

[0086] Based on the similarity weights of multimodal samples in high / low label reference sample sets, the weighted state centroid and weighted label mean of multimodal samples in high / low label reference sample sets are calculated:

[0087] (20)

[0088] (twenty one)

[0089] (twenty two)

[0090] (twenty three)

[0091] in, Indicates the first The weighted state centroids of a multimodal sample in a high-label reference sample set. Indicates the first The weighted state centroids of a multimodal sample in a low-label reference sample set. Indicates the first The weighted label mean of a multimodal sample in a high-label reference sample set. Indicates the first The weighted label mean of a multimodal sample in a low-label reference sample set. Indicates the first in the memory bank The quality consistency anchor state of each historical training sample.

[0092] Furthermore, the bipolar reference direction is calculated by combining the weighted state centroid and the weighted label mean of the multimodal samples in the high / low label reference sample set. The calculation method is as follows:

[0093] (twenty four)

[0094] in, Indicates the first Each multimodal sample bipolar reference direction represents a direction vector in the local latent space pointing from the set of low-label reference samples to the set of high-label reference samples. It is a tiny positive constant used to prevent the denominator from being zero.

[0095] The bipolar reference direction expresses the direction of local state change through the difference between the high and low label reference centroids, and is scaled by the difference in label mean values, thereby forming a directed reference signal that can be used for state calibration.

[0096] Step S33: Construct the gated input vector of the multimodal sample by using the weighted state centroids of the multimodal sample in the high-label reference sample set and the low-label reference sample set.

[0097] Weighted state centroids based on the high-label reference sample set obtained in step S32 Weighted state centroid of the low-label reference sample set Construct gated input vectors And predict the signed shift. The gated input vector The construction formula is:

[0098] (25)

[0099] in, Indicates the first A gated input vector of multimodal samples; This represents a vector concatenation operation. The gating input vector... It also includes the current quality consistency anchor point status. Weighted state centroids of the high-label reference sample set Weighted state centroid of the low-label reference sample set And the difference in weighted state centroids between the high and low label reference sample sets. Therefore, it can characterize the relative position of the current multimodal sample in the bipolar reference frame and the direction of local label change.

[0100] Step S34: Predict the step size gating and signed displacement of the multimodal sample from the gated input vector of the multimodal sample, and calculate the potential state update of the multimodal sample.

[0101] Gated input vector Input gated calibration network. The gated calibration network includes a gated branch for predicting the step size of each dimension update (predicting the step size gate for each dimension). ) and the displacement branch used to predict signed displacements (predicting signed displacements) ).

[0102] Based on the gated input vector, predict the step size gate for each dimension. and signed displacement :

[0103] (26)

[0104] (27)

[0105] in, This represents the Sigmoid activation function. and Represents a learnable neural network mapping. Indicates the first The step size gating of each multimodal sample is a gating vector that controls the update amplitude. Indicates the first The signed displacements of the multimodal samples are the signed directional displacements predicted by the network.

[0106] Furthermore, calculate the final potential state update. :

[0107] (28)

[0108] in, Indicates the first Potential state update quantity for a multimodal sample; This indicates element-wise multiplication.

[0109] Step S35: Based on the potential state update of the multimodal samples, calibrate the quality consistency anchor state of the multimodal samples to obtain the calibrated task state of the multimodal samples, and predict the final calibration prediction value from the calibrated task state.

[0110] Based on the potential state update amount, the calibrated task state is obtained. :

[0111] (29)

[0112] in, Indicates the first The task status after calibration of multiple modal samples; Indicates the final extracted number The quality consistency anchor state of a multimodal sample.

[0113] Then, the final calibrated prediction value is output through the prediction head function. :

[0114] (30)

[0115] in, Indicate the first The final calibration prediction value for each multimodal sample; This represents the prediction head function.

[0116] Step S4: Train the model using the loss function to obtain the trained residual direction disambiguation and state calibration model.

[0117] The training process is divided into a warm-up phase and a full calibration training phase. The warm-up phase is used to stabilize the quality consistency anchor state and the global state-label memory, while the full calibration training phase is used to learn the signed calibration updates under the bipolar reference frame.

[0118] Step S41: Warm-up Training Phase: Optimize the model parameters for generating quality consistency anchor states and preliminary prediction values ​​using the loss function of the warm-up phase.

[0119] in front During each training cycle, the bipolar reference calibration path in step S3 is disabled, and only the model parameters used in step S1 to generate the quality consistency anchor state and preliminary prediction values ​​are optimized. The loss function for the warm-up phase is:

[0120] (31)

[0121] in, This represents the loss function during the preheating stage; This indicates the number of training samples used in the current preheating loss calculation. Indicates the first The true ordered scalar labels of each training sample. It represents the absolute value.

[0122] Step S42: Complete calibration training phase: Use the global loss consisting of orientation consistency loss and prediction task loss to train and obtain the trained residual orientation disambiguation and state calibration model.

[0123] After the warm-up phase, the global state-label memory is reset and rebuilt. The quality consistency anchor states and their labels of the training samples are written into the global state-label memory, and the bipolar reference calibration path is activated. During this phase, the final prediction task loss and orientation consistency loss are optimized simultaneously. The orientation consistency loss is defined as follows: .

[0124] For a set of positive residual samples, It is a set of negative residual samples. This represents the set of training samples whose true ordered scalar labels are higher than the initial predicted values. This represents the set of training samples whose true ordered scalar labels are lower than the initial predicted values. The directional loss is:

[0125] (32)

[0126] in, Indicates the first Orientation loss for each training sample; Indicates directional margin. Represents the rectified linear unit function. The sign function is represented. This directional loss is used to constrain the direction of movement of the calibration predictions, when the true ordered scalar label... Higher than the initial forecast At that time, the calibrated predicted values ​​are encouraged. Compared to the initial forecast value Move up; when the true ordered scalar label Lower than the initial forecast At that time, the calibrated predicted values ​​are encouraged. Compared to the initial forecast value Move downwards.

[0127] Directional consistency loss for:

[0128] (33)

[0129] in, This indicates the loss of directional consistency. This represents the number of samples in the positive residual sample set; This represents the number of samples in the negative residual sample set.

[0130] The task prediction loss is calculated as follows:

[0131] (34)

[0132] in, This represents the task prediction loss, which is the final predicted value. With real ordered scalar labels The predicted loss between tasks.

[0133] The final global loss is:

[0134] (35)

[0135] in, Indicates global loss; Indicates the predicted loss of the task; and These are the weighting coefficients for task prediction loss and direction consistency loss, respectively.

[0136] Through the two-stage training method described above, the model first learns stable and comparable quality consistency anchor states, then uses a global state-label memory to learn the signed calibration rules under the bipolar reference frame, and finally obtains a trained residual direction disambiguation and state calibration model. This model includes fixed modal encoder and quality consistency anchor state parameters, prediction head function parameters, gating calibration network parameters, and a global state-label memory built during the training phase.

[0137] Step S5: Use the trained residual direction disambiguation and state calibration model to perform state calibration on the validation set samples, and finally obtain the calibrated prediction values.

[0138] During the validation phase, the trained residual orientation disambiguation and state calibration model is used to calibrate the state of the validation set samples, ultimately obtaining the calibrated predicted values. And by comparing the calibrated predicted values With real ordered scalar labels The accuracy and other metrics were calculated to verify the beneficial effects of the invention. This improves the stability and robustness of prediction results under incomplete multimodal input conditions.

[0139] Compared with the prior art, the present invention has the following beneficial technical effects:

[0140] 1. Reduce the probability of incorrect residual direction determination

[0141] Based on the preliminary prediction values ​​of the current multimodal samples, this invention divides the candidate sample reference set into a high-label reference sample set and a low-label reference sample set. Based on the weighted state centroid and weighted label mean in the two reference sample sets, a bipolar reference direction is constructed. This transforms the directionless similar sample retrieval results into a signed calibration signal with a label order relationship, enabling the quality consistency anchor point state to be calibrated in the direction of reducing prediction error. This helps to reduce the probability of incorrect residual direction judgment and improve the accuracy of the final prediction result.

[0142] 2. Improve the stability of the quality consistency anchor point state.

[0143] This invention calculates reliability scores based on modality summary vectors of each modality and calculates fusion weights in conjunction with availability indicators, ensuring that the fusion weights corresponding to missing modalities are strictly 0 and the sum of the fusion weights corresponding to all available modalities is 1. Simultaneously, a multi-layer mask self-attention processing module iteratively updates the interaction relationships between available modalities, avoiding artificial contributions from missing modalities to the quality consistency anchor state, thereby providing a stable and consistent task state representation for subsequent bipolar reference sensing calibration.

[0144] 3. Improve calibration robustness when the high-label reference sample set or the low-label reference sample set is empty.

[0145] This invention maintains the high-label tail prototype vector and the low-label tail prototype vector using an exponential moving average. When either the high-label reference sample set or the low-label reference sample set is empty, the corresponding high-label tail prototype vector or low-label tail prototype vector is used as a backoff reference. When the corresponding tail prototype vector has not yet been initialized, the current quality consistency anchor state of the multimodal samples is used as the backoff state, making the potential state update tend to shrink and avoiding arbitrary state updates due to the emptiness of the high-label reference sample set or the low-label reference sample set. This improves the stability and robustness of bipolar reference sensing calibration in high-missing-rate scenarios.

[0146] 4. Implement theory-driven signed calibration to reduce the probability of incorrect direction updates.

[0147] This invention constructs a bipolar reference direction based on the state differences between the weighted state centroids in the high-label and low-label reference sample sets, as well as the differences between the corresponding weighted label means. A gated input vector is then constructed based on the quality consistency anchor state, the two weighted state centroids, and their unnormalized displacements. The potential state update is calculated by predicting the step size gating and signed displacement using this gated input vector. Combined with a direction consistency loss, the adjustment direction of the final calibration prediction relative to the initial prediction is kept consistent with the true residual direction, thereby reducing the probability of incorrect direction updates.

[0148] 5. Improve the stability of model training and prediction.

[0149] This invention employs a two-stage training approach consisting of a preheating training phase and a full calibration training phase. During the preheating training phase, the bipolar reference calibration path is disabled, and the model parameters for generating quality consistency anchor states and preliminary predictions are optimized using the loss function developed during the preheating phase. After preheating training, the global state-label memory is reset, and the model is initialized using the quality consistency anchor states and their corresponding labels from the training set samples. During the full calibration training phase, the bipolar reference calibration path is enabled, and the model is trained using a global loss comprised of directional consistency loss and prediction task loss. This approach avoids the unstable state representation in the early stages of training affecting the construction of the candidate sample reference set and subsequent calibration processes, thereby improving the model's prediction stability in scenarios with low information content and high missing rate. Attached Figure Description

[0150] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0151] Figure 1 This is a general flowchart of a specific embodiment of the present invention.

[0152] Figure 2 This is a flowchart illustrating the construction of the quality consistency anchor point state in step 1 of a specific embodiment of the present invention.

[0153] Figure 3 This is a flowchart illustrating the maintenance of the global state-tag memory in step 2 of a specific embodiment of the present invention.

[0154] Figure 4 This is a flowchart of the bipolar reference sensing calibration in step 3 of a specific embodiment of the present invention.

[0155] Figure 5 This is a flowchart illustrating the model training and optimization process in step 4 of a specific embodiment of the present invention. Detailed Implementation

[0156] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0157] This invention provides a residual orientation disambiguation and state calibration method for incomplete multimodal learning. For example... Figure 1 As shown, the method can be executed by a computer device, which may include a processor, a memory, and program instructions stored in the memory and executable by the processor. When the program instructions are executed, they implement the construction of quality consistency anchors for incomplete multimodal input samples, maintenance of a global state-label memory, bipolar reference sensing calibration, and model training optimization.

[0158] In this embodiment, let the multimodal input sample be the first... Let there be a sample, and the complete set of modes be denoted as . Modality Whether it is observed is determined by the availability indicator. It indicates. Among them, Representing modes Available Representing modes Missing. (Number) The set of available modes for a sample is denoted as . The observed input is denoted as The ordered scalar label corresponding to the sample is denoted as . For sentiment analysis tasks, It can represent the intensity of emotion or an ordered scalar obtained by mapping emotion categories; for binary or multi-class classification tasks, the category labels can be mapped to ordered scalar forms as needed for the task.

[0159] The overall process of this invention includes the following steps.

[0160] Step 1: Construct quality consistency anchor state

[0161] like Figure 2 As shown, under incomplete multimodal input conditions, a quality-consistent anchor state is first generated based on the available modalities. This quality-consistent anchor state represents the approximate position of the current sample in the potential task state space and serves as the basis for subsequent reference retrieval and state calibration. To avoid missing modalities artificially contributing to the anchor state through default values, padding values, or unmasked network channels, this embodiment employs an availability mask and reliability-weighted fusion mechanism.

[0162] For each available mode First, through the encoder of the corresponding mode. For input features Encode the data, then perform a one-dimensional convolution. and pooling operations And add learnable slot embedding The modal slot sequence is obtained. :

[0163] (36)

[0164] in, Representing modes Feature encoder, Representing modes One-dimensional convolution mapping, Representing modes Pooling operations, This represents a learnable slot embedding.

[0165] Subsequently, mean pooling along the slot dimension is performed on the modal slot sequence to obtain the modal summary vector. :

[0166] (37)

[0167] After obtaining the mode summary vectors for each available mode, the reliability score for each mode is calculated. :

[0168] (38)

[0169] in, , and For learnable parameters, This represents the hyperbolic tangent activation function. Subsequently, it is combined with an availability indicator. The reliability scores of each modality are normalized to obtain the fusion weights under the availability mask constraint. :

[0170] (39)

[0171] From the above formula, it can be seen that when the mode When missing, The fusion weights corresponding to this mode The value is strictly 0; when a mode is available, its fusion weight is determined by the reliability score, and the sum of the fusion weights of all available modes is 1. This ensures that missing modes do not contribute to the anchor state.

[0172] Then, an initial joint sequence is constructed using the fused global seed vector and the slot sequences of each modality. :

[0173] (40)

[0174] in, Learnable embeddings representing global seed types Representing modes Learnable embeddings of various types. For missing modalities, due to The corresponding slot is set to zero and is masked during the mask self-attention process.

[0175] Then, input the initial joint sequence. The layer mask self-attention processing module JointBlock performs iterative updates:

[0176] (41)

[0177] Finally, the first position vector of the last layer's output sequence is taken as the first position vector. The quality consistency anchor state of each sample :

[0178] (42)

[0179] Based on the quality consistency anchor point status Scalar prediction head Preliminary predicted values ​​were obtained. :

[0180] (43)

[0181] in, This represents the prediction head function that maps potential task states to scalar predicted values. Indicates the first Preliminary predicted values ​​for a sample before calibration.

[0182] Step 2: Maintain global state - tag memory

[0183] like Figure 3 As shown, after obtaining the quality consistency anchor state of the training samples, this embodiment maintains a global state-label memory. It is used to store the quality consistency anchor states and ordered scalar labels of historical training samples. The global state-label memory provides local reference samples with label order relationships for the current sample, enabling the subsequent calibration process to construct a set of high-label reference samples and a set of low-label reference samples in the latent state space.

[0184] The global state-tag memory is defined as follows: .

[0185] in, Indicates the first The quality consistency anchor state of each historical training sample Indicates the first Ordered scalar labels for a number of historical training samples Indicates the first A set of memory key vectors used for retrieval. This indicates the memory capacity. It also represents the historical quality consistency anchor state. Its key vector Calculated using the following formula:

[0186] (44)

[0187] in, It is a learnable linear projection matrix. This indicates a normalization operation.

[0188] For the current number The quality consistency anchor state of a multimodal sample Calculate its query key vector :

[0189] (45)

[0190] in, This is a learnable projection matrix. Then, based on the query key vector... With all global state-label memory key vectors Cosine similarity from global memory Searching for the top with the highest similarity A set of historical samples constitutes the candidate sample reference set. Similarity can be expressed as the dot product of normalized vectors:

[0191] (46)

[0192] In this embodiment, a global state-label memory is used instead of maintaining separate memories for different combinations of missing modes. This allows the reference geometry to become an attribute of the learned task state manifold, rather than an attribute of a specific missing mode, thereby improving calibration consistency between different missing modes.

[0193] To avoid instability in the bipolar reference direction due to empty high-label or low-label sides in the candidate sample reference set, this embodiment further maintains the high-label tail prototype vector. and low-label tail prototype vector Let the range of values ​​for the label be... Tail quantization quantile is The momentum coefficient is High label threshold and low label threshold Defined as:

[0194] (47)

[0195] (48)

[0196] During training, the tail prototype is updated based on the state of the training samples written to the memory. If the training samples written to the memory contain labels... If the state is such that the true label in the training samples currently written to the memory is not less than the high label threshold, then... The average value of the anchor state for the quality consistency of the training samples And updated:

[0197] (49)

[0198] If the training samples written to the memory contain labels If the state is such that the true label in the training sample currently written to the memory is not greater than the low label threshold, then... The average value of the anchor state for the quality consistency of the training samples And updated:

[0199] (50)

[0200] When the high-label reference sample set and the low-label reference sample set of the current sample are empty, the corresponding tail prototype vector is used as the backoff reference; if the tail prototype has not been initialized, the current quality consistency anchor state is used. As a fallback state, the calibration update tends to shrink, rather than producing state updates in arbitrary directions.

[0201] During the training phase, the global state-label memory is only written with the quality consistency anchor state and ordered scalar labels of the training samples. During the validation and testing phases, the memory remains frozen and read-only, and no label information of validation or test samples is written to it, thus avoiding test label leakage.

[0202] Step 3: Bipolar Reference Sensing Calibration

[0203] like Figure 4 As shown, in obtaining the candidate sample reference set for the current sample Then, this embodiment uses the preliminary predicted values ​​of the current sample. Candidate reference samples are divided into high-label reference sample sets and low-label reference sample sets based on their label order. Let the preset label interval threshold be... High-label reference sample set and low-label reference sample set They are defined as follows:

[0204] (51)

[0205] (52)

[0206] in, This represents the ordered scalar label value relative to the current preliminary prediction value. A significantly higher reference sample set, This represents the ordered scalar label value relative to the current preliminary prediction value. A significantly lower set of reference samples. This partitioning method allows the system to organize retrieved samples into a bipolar reference frame with label order, instead of relying solely on undirected similar samples.

[0207] For samples in the high-label reference sample set and the low-label reference sample set, the similarity weights are... and The calculation formula is:

[0208] (53)

[0209] (54)

[0210] in, Represents the temperature coefficient of similarity. Represents the bipolar reference sample set The summation index is obtained by iterating through each reference sample.

[0211] Based on the aforementioned similarity weights, the reference centroids of high-label tags are calculated respectively. Low label reference centroid and the corresponding high label mean and low label mean :

[0212] (55)

[0213] (56)

[0214] (57)

[0215] (58)

[0216] Then, based on the state differences between the high-label reference centroid and the low-label reference centroid, and the differences between the corresponding label means, a bipolar reference direction is constructed. :

[0217] (59)

[0218] in, This is a positive, stable term used to avoid numerical instability caused by an excessively small denominator. The bipolar reference direction represents the local direction in the latent state space from the low-label reference region to the high-label reference region. Unlike ordinary similarity retrieval, this direction not only describes which historical samples the current sample is similar to, but also describes the direction in which the latent state changes towards the high-label direction in the local reference coordinate system.

[0219] To further determine whether the current sample should be corrected along the bipolar reference direction or in the opposite direction, this embodiment constructs a gated calibration input. The gated calibration input is obtained by concatenating the current quality consistency anchor state, the high-label reference centroid, the low-label reference centroid, and the unnormalized displacement between them:

[0220] (60)

[0221] in, This indicates a vector concatenation operation. The input includes both the relative position of the current sample within the bipolar reference frame and local directional information from the low-label region to the high-label region.

[0222] Then Input the gated calibration network. The gated calibration network consists of a gated branch for predicting the dimension-wise update magnitude and a displacement branch for predicting the signed displacement. Its calculation process is as follows:

[0223] (61)

[0224] (62)

[0225] (63)

[0226] in, and For learnable neural network mapping, This represents the Sigmoid activation function. Represents a dimension-wise gated vector. This represents the signed shift vector predicted by the network. This represents element-wise multiplication. This represents the final potential state update. (Gated vector) The displacement vector is used to control the magnitude of the state update in each dimension. Used to provide a signed state correction direction.

[0227] Based on potential state update amount Obtain the calibrated task status :

[0228] (64)

[0229] Then, the calibrated task status Input the same prediction head function To obtain the final predicted value :

[0230] (65)

[0231] Therefore, the prediction process for the current sample is expanded from "obtaining a preliminary prediction based solely on available modes" to "first obtaining the quality consistency anchor state, and then performing signed calibration using the historical state-label reference frame." During the testing phase, the above calibration process does not use the real label of the current test sample, but instead infers the correction direction from the relative geometric relationship between the current anchor state and the bipolar reference frame through the gated calibration network learned during the training phase.

[0232] Step 4: Model Training and Optimization

[0233] like Figure 5 As shown, this embodiment adopts a two-stage training strategy, including a warm-up training stage and a full calibration training stage.

[0234] During the warm-up training phase, the bipolar reference calibration path is disabled, and only model parameters based on quality-consistent anchor states and preliminary predictions are generated for optimization. This phase is used to stabilize the anchor state space and prevent the instability of state representations in the memory bank from compromising the effectiveness of local reference retrieval in the early stages of training. Let the number of warm-up training rounds be... The loss function during the preheating stage is:

[0235] (66)

[0236] in, Indicates the number of training samples. Indicates the first Preliminary predicted values ​​for each training sample. Indicates the first The true ordered scalar labels of the training samples.

[0237] After the warm-up phase, the global state-label memory is reset, and initialized using the anchor states and labels from the training set. During subsequent training, the anchor states and corresponding labels from the training mini-batch are continuously written to or updated in the memory. For verification and testing, the memory is frozen in a read-only state.

[0238] During the full calibration training phase, a bipolar reference calibration path is enabled, allowing the model to simultaneously optimize both the final prediction accuracy and the consistency of the state correction direction. For each training sample, the final prediction value is obtained after bipolar reference sensing calibration. The task prediction loss is defined as:

[0239] (67)

[0240] To align the calibrated predicted values ​​towards a direction that reduces residuals, this embodiment introduces a directional consistency loss. Let... This represents the set of samples whose initial predictions are lower than the true labels. This represents the sample set whose initial predictions are higher than the true labels. This represents the directional consistency margin. For the... For each sample, the directional loss is defined as:

[0241] (68)

[0242] in, Represents a symbolic function. This represents a linear rectified function. When... At that time, the expected calibrated predicted value Compared to the initial forecast value Adjust upwards; when At that time, the expected calibrated predicted value Compared to the initial forecast value Adjust downwards. The orientation consistency loss is used to penalize calibration results that are inconsistent with the true residual orientation or whose adjustment magnitude is insufficient.

[0243] Combining the positive and negative residual sample sets, the directional consistency loss is:

[0244] (69)

[0245] when or When a set is empty, the corresponding item is omitted. The complete training objective is:

[0246] (70)

[0247] in, and These are the weighting coefficients for task prediction loss and direction consistency loss, respectively.

[0248] Through the training method described above, the model first learns stable and comparable quality consistency anchor states during the warm-up phase. Then, during the full training phase, it constructs a bipolar reference frame using a global state-label memory and learns signed latent state update rules through orientation consistency loss. Thus, under incomplete multimodal input conditions, the model can transform the undirected context obtained from ordinary retrieval into locally calibrated signals with labeled order, thereby alleviating the problem of residual orientation ambiguity.

[0249] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A residual orientation disambiguation and state calibration method for incomplete multimodal learning, characterized in that, The method includes the following steps: Step S1: Obtain multimodal samples, construct quality consistency anchor point states for multimodal samples, and generate preliminary prediction values; Step S2: Maintain the global state-label memory, construct the candidate reference sample set, and maintain the high-label tail prototype vector and the low-label tail prototype vector; Step S3: Bipolar reference sensing calibration: Construct a bipolar reference sample set based on the candidate sample reference set, and calibrate the quality consistency anchor point state of the multimodal samples based on the bipolar reference sample set to obtain the final calibration prediction value. Step S4: Train the model using the loss function to obtain the trained residual direction disambiguation and state calibration model; Step S5: Use the trained residual direction disambiguation and state calibration model to perform state calibration on the validation set samples, and finally obtain the calibrated prediction values.

2. The residual orientation disambiguation and state calibration method for incomplete multimodal learning according to claim 1, characterized in that, Step S1 includes the following steps: Step S11: Obtain multimodal samples, perform modal encoding on the multimodal samples to obtain slot sequences for each modality, and perform mean pooling on the slot sequences to obtain modal summary vectors; Step S12: Calculate the reliability score and fusion weight of each modality based on the modality summary vector; Step S13: Initialize the joint sequence, and iteratively update the joint sequence through a multi-layer mask self-attention processing module to obtain the first... Layer joint sequence, based on the first The layer-joined sequence generates quality-consistent anchor point states; Step S14: Input the quality consistency anchor point status into the prediction head function to obtain the preliminary prediction value.

3. The residual orientation disambiguation and state calibration method for incomplete multimodal learning according to claim 1, characterized in that, Step S2 includes the following steps: Step S21: Construct a global state-label memory, calculate the multimodal sample query key vector and the global state-label memory key vector; select the multimodal sample query key vector with the highest cosine similarity to all global state-label memory key vectors. A set of 10 samples constitutes a candidate reference sample set; Step S22: Maintain the high-label tail prototype vector and the low-label tail prototype vector based on the exponential moving average.

4. The residual orientation disambiguation and state calibration method for incomplete multimodal learning according to claim 1, characterized in that, Step S3 includes the following steps: Step S31: Construct a high-label reference sample set and a low-label reference sample set based on the candidate sample reference set and the preliminary predicted values; Step S32: Based on the high-label reference sample set and the low-label reference sample set, calculate the similarity weight, weighted state centroid and weighted label mean of the multimodal sample in the high-label reference sample set and the low-label reference sample set respectively; Step S33: Construct the gated input vector of the multimodal sample by using the weighted state centroids of the multimodal sample in the high-label reference sample set and the low-label reference sample set; Step S34: Predict the step size gating and signed displacement of the multimodal sample from the gated input vector of the multimodal sample, and calculate the potential state update of the multimodal sample; Step S35: Based on the potential state update of the multimodal samples, calibrate the quality consistency anchor state of the multimodal samples to obtain the calibrated task state of the multimodal samples, and predict the final calibration prediction value from the calibrated task state.

5. The residual orientation disambiguation and state calibration method for incomplete multimodal learning according to claim 1, characterized in that, Step S4 includes the following steps: Step S41: Warm-up training phase: Optimize the model parameters for generating quality consistency anchor states and preliminary prediction values ​​using the loss function of the warm-up phase; Step S42: Complete calibration training phase: Use the global loss consisting of orientation consistency loss and prediction task loss to train the residual orientation disambiguation and state calibration model.

6. The residual orientation disambiguation and state calibration method for incomplete multimodal learning according to claim 4, characterized in that, The high-label tail prototype vector and the low-label tail prototype vector are maintained based on exponential moving averages. When either the high-label reference sample set or the low-label reference sample set is empty, the corresponding high-label tail prototype vector or low-label tail prototype vector is used as a backoff reference. When the corresponding tail prototype vector has not yet been initialized, the current quality consistency anchor point state of the multimodal sample is used as the backoff state.

7. The residual orientation disambiguation and state calibration method for incomplete multimodal learning according to claim 4, characterized in that, In step S32, a bipolar reference direction is constructed based on the state difference between the weighted state centroids in the high-label reference sample set and the weighted state centroids in the low-label reference sample set, as well as the difference between the corresponding weighted label means. In step S33, the quality consistency anchor state of the multimodal sample, the weighted state centroids in the high-label reference sample set, the weighted state centroids in the low-label reference sample set, and the unnormalized displacement between the two weighted state centroids are concatenated to construct the gated input vector of the multimodal sample. In step S34, the step-size gating and signed displacement of the multimodal sample are predicted based on the gated input vector, and the potential state update amount of the multimodal sample is calculated based on the element-wise product of the step-size gating and the signed displacement.

8. The residual orientation disambiguation and state calibration method for incomplete multimodal learning according to claim 5, characterized in that, In step S42, a positive residual sample set and a negative residual sample set are constructed based on the relationship between the preliminary predicted value and the true ordered scalar label. For training samples whose preliminary predicted value is lower than the true ordered scalar label, the final calibration predicted value is adjusted upward relative to the preliminary predicted value. For training samples whose preliminary predicted value is higher than the true ordered scalar label, the final calibration predicted value is adjusted downward relative to the preliminary predicted value. The direction consistency loss is used to penalize calibration results that are inconsistent with the true residual direction or whose adjustment magnitude is insufficient. When either the positive residual sample set or the negative residual sample set is empty, the corresponding item in the direction consistency loss is omitted.

Images