A multimodal collaborative competition optimization method and device and a task reasoning method

Abstract

This invention discloses a multimodal cooperative competitive optimization method, apparatus, and task inference method, relating to the field of artificial intelligence, to enhance cooperative information representation. It involves acquiring multimodal data samples, encoding and perturbing these samples to obtain latent and perturbed representations for each modality; performing single-modal prediction, joint prediction, and perturbed prediction based on the latent and perturbed representations respectively; calculating the total loss based on the prediction results of each type; and performing multimodal learning based on the calculated total loss. This application guides the fusion network to suppress dependence on any single-modal data through an anti-dominant objective, and guides the fusion network to reveal enhanced cooperative information through a cooperative objective. It utilizes an adaptive gating mechanism to dynamically activate cooperative enhancement regularization terms based on the single-modal proficiency of the multimodal data samples. This application coordinately interweaves and balances modality utilization and cooperative information discovery to significantly improve the gain of inference tasks that rely on cooperative information.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal cooperative competitive optimization method, apparatus, and task reasoning method. Background Technology

[0002] Multimodal learning, by integrating heterogeneous signals such as vision, audio, and language, has achieved superior performance in inference tasks compared to single-modal learning. Its core advantage lies in its ability to combine complementary information from different modalities. However, jointly trained multimodal models are often affected by the modality imbalance problem, where the model becomes overly reliant on a dominant modality, thus hindering effective cross-modal fusion.

[0003] To address the aforementioned issues, one known solution proposed is the Multimodal Competition Regularizer (MCR) to mitigate the imbalance problem in multimodal training. Specifically, MCR evaluates and enhances single-modal information by perturbing modal features, where each modality encoder is treated as a greedy player attempting to maximize its own importance.

[0004] However, while the competitive game mechanism of MCR can effectively balance the value of multimodal utilization, it enhances the information of each individual modality, neglecting the most important collaborative and complementary information in multimodal learning. The collaborative gains in multimodal learning essentially originate from this complex, disordered, and collaborative interactive information. Therefore, MCR offers extremely limited performance improvement for tasks relying on multimodal collaborative information for reasoning. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal cooperative competitive optimization method, apparatus, and task reasoning method to address all or part of the aforementioned problems. This method aims to alleviate the imbalance of modal data in multimodal learning while enhancing the representation of cooperative information and improving the prediction gain of downstream multimodal cooperative reasoning tasks.

[0006] The technical solution adopted in this invention is as follows: In a first aspect, this application provides a multimodal cooperative competitive optimization method for extracting multimodal representations from multimodal data for multimodal reasoning tasks; the multimodal data includes at least two of text, image, audio, time series, or tabular data; the multimodal representation includes redundant information, unique information, and cooperative information of the multimodal data; the unique information represents information unique to each modality of data; the redundant information represents information that can be independently extracted from any modality of data; the cooperative information represents evidence that can only be obtained when more than one modality of data is jointly interpreted; The methods include: Acquire multimodal data samples, encode and perturb the multimodal data samples to obtain the latent representation and perturbation representation of each modality; Single-modal prediction, joint prediction, and perturbation prediction are performed based on the latent representation and the perturbation representation, respectively. The total loss is calculated based on the prediction results of each type, and multimodal learning is performed based on the calculated total loss to optimize the fusion network. The total loss includes task loss, MCR loss, and collaborative enhancement regularization term; wherein, the collaborative enhancement regularization term guides the fusion network to suppress dependence on any single-modal data through anti-dominant objective, guides the fusion network to display enhanced collaborative information through collaborative objective, and dynamically activates the collaborative enhancement regularization term based on the single-modal proficiency of multimodal data samples using an adaptive gating mechanism.

[0007] Optionally, the fusion network can be guided to suppress dependence on any single modality of data by means of an anti-dominant objective, including: The sensitivity of the fusion network to each modality of data is quantitatively evaluated based on the joint prediction results and the perturbation prediction results. Based on the aforementioned sensitivity, an anti-dominant objective is constructed, and minimizing the anti-dominant objective is taken as the optimization objective.

[0008] Optionally, the sensitivity is quantified using the JS divergence of the joint prediction results and the perturbation prediction results; the anti-dominant objective is constructed by applying a smoothmax penalty to the sensitivity.

[0009] Optionally, enhanced collaborative information can be displayed on the fusion network through collaborative goals, including: An additive single-mode reference distribution is constructed based on the prediction results of each single mode; Calculate the task loss of the joint prediction result and the task loss of the reference distribution, respectively. A collaborative objective is constructed based on the computational task loss, and the optimization objective is to minimize the collaborative objective.

[0010] Optionally, the reference distribution is constructed as the mean of the prediction results of each single mode; the cooperative objective is constructed as: ; in, Indicate collaborative goals, This represents a pre-defined marginal parameter; These represent the task loss of the joint prediction result and the task loss of the reference distribution, respectively.

[0011] Optionally, an adaptive gating mechanism is used to dynamically activate the collaborative enhancement regularization term based on the unimodal proficiency of the multimodal data samples, including: Calculate the task loss for each single-modal prediction result, and quantitatively evaluate the single-modal proficiency of the multimodal data samples based on the calculated task loss; Based on the single-modal proficiency, a gating function is constructed and activated, and an adaptive gating factor is calculated. The adaptive gating factor is used as the optimization coefficient of the collaborative enhancement regularization term.

[0012] Optionally, the adaptive gating factor is calculated as follows: ; in, Indicates the adaptive gating factor; This represents the sigmoid activation function; hyperparameters These represent the sharpness parameter and activation threshold of the gating function, respectively. This indicates single-modal proficiency.

[0013] Optionally, the collaborative enhancement regularization term is constructed as follows: ; in, This indicates a cooperative enhancement regularization term; They represent the anti-dominant targets respectively. and collaborative goals The weights; It is an adaptive gating factor.

[0014] In a second aspect, this application also provides a multimodal cooperative competition optimization apparatus, which includes a processor and a storage medium, wherein the storage medium stores a computer program; the processor runs the computer program to execute the above-described multimodal cooperative competition optimization method.

[0015] In a third aspect, this application also provides a multimodal task reasoning method for performing task reasoning based on multimodal data; the method includes: Encode multimodal data; The encoded latent representation is input into a fusion network optimized using the multimodal cooperative competitive optimization method described above, for joint prediction. Output the joint prediction results.

[0016] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: The proposed multimodal cooperative competitive optimization scheme transforms multimodal learning from simple weight balancing into proactive cooperative information discovery through a unified framework. This application breaks the monopoly of dominant modes through an anti-dominance objective. By perturbing the modal latent representation in the latent space to evaluate the strength of single-modal information, the over-reliance of the fusion network on any single modality is explicitly penalized, effectively suppressing the excessive occupation of the fusion space by single-modal and redundant information, thereby creating the necessary "representation space" for cross-modal interaction (i.e., multimodal cooperation). Furthermore, considering that simply clearing this feature extraction space does not guarantee meaningful cooperative information fusion, this application introduces a cooperative objective to proactively fill this capacity with cooperative gains. This objective constructs an additive reference baseline derived from independent single-modal predictions. By forcing the fusion network to acquire discriminative gains exceeding this ensemble baseline, the fusion network is driven to discover higher-order cooperative information, rather than merely aggregating isolated unique information. Finally, to ensure optimization stability and avoid over-regularization, this application also introduces an instance-level, single-modal proficiency-aware gating mechanism. This gating mechanism monitors the single-modal explanation capability in real time and dynamically activates the co-enhancement operation only on difficult samples where single-modal evidence is insufficient. This coordinately interweaves and balances modality utilization and co-information discovery to significantly improve the gain of reasoning tasks that rely on co-information. Attached Figure Description

[0017] The present invention will be described by way of example and with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart of the stages of the multimodal cooperative competition optimization method.

[0018] Figure 2 This is a flowchart for calculating total loss.

[0019] Figure 3 This is a network architecture diagram of the collaborative enhancement regularization term. Detailed Implementation

[0020] All features disclosed in this specification, or steps in all methods or processes disclosed herein, may be combined in any way, except for mutually exclusive features and / or steps.

[0021] Any feature disclosed in this specification (including any appended claims and abstract) may be replaced by other equivalent or similar features, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0022] According to the Partial Information Decomposition (PID) theory, the task-related information carried by multimodal data can be decomposed into three information theory components: redundant information, unique information, and collaborative information. Unique information represents information unique to each modality and can only be obtained from the individual modal data. Redundant information represents information that can be independently extracted from any modality, is shared by all individual modal data, and can be recovered from more than one modality. Collaborative information represents evidence that can only be obtained when more than one modality is jointly interpreted (i.e., simultaneously involved in task execution).

[0023] Collaborative information is a crucial source of advantage in multimodal learning, offering not only predictive gains beyond unimodal evidence but also capturing collaborative patterns that truly make fusion valuable. When a model over-relies on a dominant modality, it tends to focus on redundant information while neglecting collaborative information. This makes modality imbalance not just a performance bias but also an information loss. Conversely, discovering collaborative information forces the model to recognize that even a stronger modality is insufficient for complex reasoning tasks. For example, recognizing sarcasm often requires jointly understanding facial expressions and speech tone. Collaborative information discovery also fundamentally alleviates modality imbalance: by uncovering necessary collaborative points, weaker modalities can occupy more meaningful space in gradient updates. Therefore, promoting collaborative information discovery not only suppresses unimodal shortcuts and encourages more refined cross-modal alignment but also ensures that a balanced model truly possesses stronger collaborative capabilities.

[0024] However, the imbalance problem in traditional multimodal learning significantly hinders the discovery of collaborative information. MCR, on the other hand, focuses on enhancing the gain of single-modal data, neglecting the more important collaborative information component.

[0025] To alleviate the modal imbalance problem, MCR introduces a framework for adjusting the contributions of different modal data. Based on PID theory, the mutual information of multimodal data (for simplicity, an example containing two modal data points is used; the same principle applies to more modal data) is decomposed into unique information, redundant information, and cooperative information: Equation (1): ; in, These represent the first modality data, the second modality data, and the sample labels, respectively. Represents mutual information; These represent unique information from the first modal data and the second modal data, respectively. This indicates redundant information between the first modality data and the second modality data; Represents collaborative information.

[0026] MCR uses three key regularization terms to represent the multimodal learning loss: Equation (2): ; in: This indicates MCR loss.

[0027] The perturbation loss introduced by introducing Modality-Specific Information Perturbed Difference (MIPD) is defined as the Jensen-Shannon (JS) divergence between the original joint prediction and the perturbed joint prediction, used to estimate the unique information of each single-modal data point. Contribution, through single-modal data Potential representation (by the corresponding mode encoder) Encoding obtained) Apply a permutation To achieve the perturbation, the corresponding latent representation after perturbation is as follows: .

[0028] Equation (3): ; in, This indicates the calculation of the expected value. This indicates the calculation of the Jensen–Shannon Divergence (JSD). This represents the predicted probability of Y.

[0029] When multimodal data contains M modes The calculation can be extended according to the modal perturbation method. Specifically, assume that the latent representation is obtained after encoding each modal data. The original joint prediction distribution output by the fusion network is For the first Each modality, and its potential representation By performing a permutation perturbation, we obtain the perturbation representation. And keeping the latent representations of other modes unchanged, we obtain the perturbation prediction distribution. Subsequently, the following was adopted. and JS divergence between The perturbation difference corresponding to each mode, i.e. The disturbance loss is obtained by summing or weighted summing the disturbance differences corresponding to all modes. ,For example .

[0030] To capture redundant information, a supervised contrastive loss is used. This loss aligns representations of different modalities from the same sample while pushing representations from different classes apart, thereby maximizing the lower bound of mutual information between modalities.

[0031] Equation (4): = ; in, This represents the set of sample indices in the current batch; i Indicates the anchor sample index; Indicates the relationship with the first i A set of positive sample indices with the same label; Indicates the number of positive samples; Indicates the first modal data i The latent representation of each sample; This indicates that the second modal data is related to the first modality. i Potential representations of positive samples with the same sample label; Indicating the second modal data, the first... j Potential representations of candidate samples; This represents a discriminant function or similarity function used to measure the similarity between two latent representations. In one alternative implementation, for two latent representations to be discriminated... a , b , ,in, This represents the dot product similarity or cosine similarity. This indicates the temperature parameter for comparison and learning.

[0032] This represents the noise suppression loss, expressed as a Conditional Entropy Bottleneck (CEB), used to suppress cooperative information. .

[0033] Equation (5): ; in, This represents the bottleneck loss due to conditional entropy. These represent the first modal data and the second modal data, respectively. Indicates sample label; and These represent the first mode encoder and the second mode encoder, respectively. and These represent the latent representations obtained after encoding the first modal data and the second modal data, respectively. This represents the joint latent representation obtained by splicing or combining the latent representations of two modalities; Indicates parameters A parameterized reconstruction head is used to predict the corresponding joint latent representation of the target based on the sample label Y. The reconstruction head... The input can be a one-hot encoding of the sample label Y, a label embedding representation, or a continuous label representation, and the output dimension is the joint latent representation. The dimensions are consistent. In one alternative implementation, the reconstruction head... It consists of linear mapping layers or small multilayer perceptrons, and is jointly optimized with modal encoders and fusion networks during training. Represents square Norm, used to measure the reconstruction error between the actual joint latent representation and the reconstruction head predicted representation.

[0034] MCR treats each modal encoder as a player in a game. By default, it employs a greedy strategy, where each modality attempts to maximize its own importance. This mechanism can effectively balance modality utilization, but the discovery of collaborative information that can only emerge through complex intermodal interactions remains largely unresolved.

[0035] To address this situation, this application proposes a multimodal cooperative competition optimization method, which aims to alleviate the imbalance of modal data in multimodal learning while enhancing the representation of cooperative information, thereby improving the prediction gain of downstream multimodal cooperative reasoning tasks.

[0036] The reasoning task in this application is applicable to emotion perception scenarios, such as intelligent customer service or companion robot systems, which require judging a user's true emotions based on text, facial expressions, and tone of voice. The multimodal data includes the user's text data (which can be manually entered text or text transcribed from speech), facial expression frames (i.e., image data) during user interaction, and speech (i.e., audio data) generated during user interaction. For example, the text data, image data, and audio data generated when a user says "Great, this is exactly what I wanted." For unimodal data, text data can be encoded using a BERT-based text encoder, image data can be encoded using a ResNet-50 or ViT visual encoder, and audio data can be encoded using a wav2vec 2.0 audio encoder. Subsequently, unique information, redundant information, and collaborative information are extracted and dynamically recombined layer by layer. From the text alone, the above statements contain obvious positive vocabulary and are easily judged as positive emotions. However, if the user frowns and rolls their eyes in the visual modality, and the tone is flat, with abnormal emphasis and impatience in the audio modality, the true label may be sarcasm or negative emotions. In this sample, the positive vocabulary in the text, the negative facial expressions in the image, and the impatient tone in the audio each constitute unique information for each single modality. In ordinary non-sarcasm samples, when multiple modalities consistently express the same emotion, redundant information is formed. The sarcastic meaning generated by the cross-modal conflict between "positive text" and "negative facial expressions / tones" belongs to collaborative information that cannot be recovered from single modality data and can only be obtained through joint understanding.

[0037] Alternatively, for robot operation and perception tasks, such as grasping, pushing, or assembling tasks with a robotic arm, the robot needs to use multiple sensors to determine whether it will make stable contact with the target object in the next moment, or to predict the position of the end effector. A multimodal data sample can consist of the current RGB image (2D image data), depth map (3D image data), force sensor readings (numerical data), and proprioceptive state (tabular data), where the proprioceptive state includes joint angles, end effector position, and velocity. For RGB images or depth maps, ResNet-18 or ViT visual encoders can be used for encoding; for force sensor readings, MLP (Multilayer Perceptron) force encoders can be used; and for proprioceptive state, MLP proprioceptive encoders can be used. Subsequently, unique information, redundant information, and collaborative information are extracted and dynamically recombined layer by layer. For example, in a task of pushing a cup, the RGB image shows the robotic gripper approaching the cup, the depth map shows the distance is very close, the force signal shows a slight increase, and the proprioceptive state shows the end effector moving towards the cup. Relying solely on vision can only determine spatial proximity, but cannot confirm actual contact; relying solely on force perception may fail to distinguish the object being contacted; and relying solely on proprioception cannot understand the positional relationships of external objects. Only by combining visual geometry, depth distance, force changes, and end-effector motion can the model determine whether contact is imminent and whether it is stable. In this scenario, the object's appearance and geometry are unique to vision, changes in contact pressure are unique to force perception, and end-effector motion is unique to proprioception; multiple sensors collectively reflect "the robotic gripper is approaching the object," creating redundant information; and the stable contact state comes from the combined relationship between vision, force perception, and proprioception, which is collaborative information.

[0038] In one alternative implementation, such as Figure 1 As shown, the multimodal cooperative competition optimization method provided in this application includes the following stages: I. Sample Coding Stage.

[0039] In the sample encoding stage, the multimodal data samples are encoded and perturbed to obtain the latent representation and perturbation representation of each modality.

[0040] Obtain a set D of multimodal data samples, where each multimodal data sample consists of multimodal data... It consists of the modalities and their corresponding sample labels Y, where M represents the number of modalities and i represents the modal index. Assuming set D contains B multimodal data samples, then set D is represented as... Here, the superscript j denotes the sample index of the multimodal data. For ease of explanation later, we will still use the example of multimodal data containing two modalities, where the sample is represented as... .

[0041] For each batch of multimodal data samples, a modal encoder corresponding to each modal data (see the example scenario above) is used to encode each modal data in the original sample into a corresponding latent representation. For example, ... Encoding as a latent representation , .

[0042] In addition, they also separately addressed By performing a permutation perturbation, we obtain the perturbation representation. .

[0043] II. Loss Calculation Stage.

[0044] In the loss calculation phase, single-mode prediction, joint prediction, and perturbation prediction are performed based on the latent representation and the perturbation representation, respectively. The total loss is calculated based on the prediction results of each type, such as... Figure 2 , Figure 3 As shown.

[0045] As one of the key improvements of this application, the total loss includes task loss, MCR loss, and a collaborative enhancement regularization term. Specifically, the collaborative enhancement regularization term guides the fusion network to suppress dependence on any single-modal data through an anti-dominant objective, guides the fusion network to display enhanced collaborative information through a collaborative objective, and dynamically activates the collaborative enhancement regularization term based on the single-modal proficiency of the multimodal data samples using an adaptive gating mechanism.

[0046] 1. Prediction steps.

[0047] (1) Single-mode prediction.

[0048] For each single-modal data, the corresponding single-modal predictor is used to perform single-modal prediction on its latent representation.

[0049] Similar to the previous embodiment, two single-modal data The corresponding single-modal predictors (for reasoning tasks, such as emotion perception tasks, robot operation perception tasks, etc.) are respectively (The same applies to other modalities), so the prediction results of two single-mode predictions are... Represented as: .

[0050] (2) Joint forecasting.

[0051] Joint prediction is performed simultaneously on the latent representations of all modalities. This is achieved using a fusion network. Combine the latent representations (after alignment) of the modal data: and based on fusion representation Joint forecasting was conducted, and the joint forecasting results were obtained. Represented as: .

[0052] (3) Disturbance prediction.

[0053] Perturbation prediction also targets multimodal data, and therefore also utilizes a fusion network for prediction. The difference lies in the fact that the features fused by the fusion network include perturbation representations.

[0054] For the perturbation of the first mode data The situation: Converged Network The first disturbance is represented Second latent representation Combine: ; and represent fusion Joint forecasting was performed, and the first disturbance forecast result was obtained. Represented as: .

[0055] Similarly, for the perturbation of the second mode data In this case, the second perturbation prediction structure is obtained. Represented as: .

[0056] 2. Steps for calculating task loss.

[0057] According to the joint forecast results and two-way single-mode prediction results The standard task loss is calculated using the standard cross-entropy loss calculation method. ; in, These represent the weights of the first single-modal prediction loss, the second single-modal prediction loss, and the joint prediction task loss, respectively. This represents the standard cross-entropy loss.

[0058] As a key improvement of this application, a cooperative enhancement regularization term is proposed to achieve cooperative competitive optimization. This term suppresses dependence on any single-modal data by opposing the dominant objective, reveals enhanced cooperative information through the cooperative objective, and utilizes an adaptive gating mechanism to dynamically activate the term based on the single-modal proficiency of the multimodal data samples. This dynamic activation means dynamically adjusting the cooperative competitive optimization strength of the term between completely off and completely on, thereby achieving an instance-level, single-modal proficiency-aware adaptive gating mechanism.

[0059] 3. Calculation steps for the anti-dominant target.

[0060] The anti-dominant objective represents the suppression loss for dependence on any single-modal data, denoted as: .

[0061] A fundamental challenge in multimodal learning is the modality dominance problem, where fusion networks tend to over-rely on specific single-modal data, typically those with more information or easier optimization, while neglecting other modalities. Even with dynamic balancing methods like MCR, the model may still greedily exploit the unique or redundant information of the single-modal data. This over-reliance severely limits the ability of fusion networks to discover and utilize complex cross-modal cooperative interactions. Therefore, the first objective of this application is to suppress this excessive single-modal occupancy, thereby creating the necessary "representation space" for cooperative information.

[0062] To formally characterize this dominant effect, this application uses the permutation perturbation strategy described above, applying random permutations to the latent representations of each modality along the batch dimension to perturb the latent representations of each modality, thereby obtaining the corresponding perturbation representations and the corresponding perturbation prediction results.

[0063] The sensitivity of the fusion network to each modality of data is quantitatively evaluated based on both joint prediction results and perturbation prediction results. Standard joint prediction results. Compared with any disturbance prediction result , The difference reflects the model's sensitivity to specific modal data. If the first modal data is perturbed... If the perturbation prediction results change drastically, it indicates that the model is not accurately predicting the first mode data. There is a strong dependency (the same applies to other modalities). To quantify this sensitivity to perturbations and to consider training stability, this application uses JS divergence to quantify the sensitivity.

[0064] Equation (6): ; in, express Sensitivity; Indicates KL divergence; Indicating targeting The average distribution.

[0065] In contrast to MCR, which aims to maximize this difference to uncover modality-specific features (i.e., unique information), this application reverses this approach by utilizing the aforementioned sensitivity to suppress severe modality dominance. This application constructs an anti-dominance objective based on sensitivity and optimizes by minimizing this anti-dominance objective. To this end, this application uses a differentiable smooth maximization operator on the sensitivity. Perform aggregation and define the anti-dominant objective as: Equation (7): ; in, This is a temperature hyperparameter used to control the sharpness of the approximation in the equation. It is achieved by minimizing... To optimize for the objective, the model penalizes the highest sensitivity, thereby forcing the fusion network to reduce its disproportionate dependence on any single modality of data. This mechanism effectively suppresses single-modality data and redundant information that excessively occupy the fusion space, prompting the network to capture / discover collaborative information that cannot be established from independent modality data.

[0066] 4. Collaborative target calculation steps.

[0067] The collaborative objective represents the loss that enhances collaborative information, denoted as... .

[0068] Suppressing the excessive influence of unimodal data is necessary, but not sufficient, for the effective fusion of multimodal features. Even if the fusion bottleneck has been balanced, the fusion network... It may still degenerate into a simple unimodal evidence aggregator, unable to learn predictive gains derived solely from complex multimodal interactions. To explicitly encourage the capture of collaborative information, this application introduces a marginal collaborative objective. .

[0069] To quantify the cooperative gain, this application first needs to establish a conservative, non-cooperative baseline. The additive single-mode reference distribution is defined as follows: When more modal data is included, the averaging calculation is performed similarly. This reference distribution represents the performance achievable by integrating independent single-modal decisions without any feature-level cross-modal interactions. If the fusion network extracts meaningful collaborative information, then its joint prediction results... It should be significantly better than this reference distribution.

[0070] The task loss of the joint prediction result and the task loss of the reference distribution are defined as follows: Equation (8): .

[0071] Define the collaborative goal as: Equation (9): ; in, This represents a pre-defined marginal parameter. It is achieved by minimizing... To achieve this objective, this application is required to meet certain conditions. Therefore, strict requirements are placed on converged networks. Compared to the additive baseline (i.e.) This achieves a lower prediction error. The collaborative objective... With the anti-dominant target Mutual complementarity: The representation space is cleaned by squeezing out redundant information from single modes, while This explicitly utilizes higher-order cooperative gains to fill this space.

[0072] 5. Adaptive gating calculation steps.

[0073] Adaptive gating of computational perception proficiency is used to dynamically adjust the intensity of collaborative competition optimization.

[0074] Not all samples require the same level of collaborative reinforcement. From a learning perspective, when at least one independent unimodal predictor can correctly classify a sample with high confidence, further strengthening collaborative regularization (i.e., collaborative competitive optimization) may be redundant and could perturb stable unimodal latent representations. Conversely, when both unimodal predictions are ambiguous or incorrect, the model must actively mine collaborative information to reduce task uncertainty. Based on this requirement, this application employs an instance-level, unimodal proficiency-aware adaptive gating strategy to dynamically adjust the strength of collaborative competitive optimization.

[0075] First, the task loss for each single-modal prediction result is calculated separately. Based on the calculated task loss, the single-modal proficiency of the multimodal data samples is quantitatively evaluated. For a given sample, let the first modality data... Second mode data The single-mode prediction losses are as follows: The current best single-modal proficiency is quantified by evaluating the minimum single-modal prediction loss: .

[0076] A gating function is constructed and activated based on unimodal proficiency, and an adaptive gating factor is calculated. Adaptive gating factor. The calculation is performed using a smoothly parameterized step function: Equation (10): ; in, This represents the sigmoid activation function; hyperparameters represents the sharpness parameter and activation threshold of the gating function (i.e., equation (10)).

[0077] An adaptive gating factor is used as the optimization coefficient for the cooperative enhancement regularization term. Adaptive gating achieves instance-level adaptive dynamic intensity variation. This dynamic gating mechanism acts as a soft switch for cooperative competitive optimization, ensuring that optimization only occurs when single-modal evidence is insufficient. Only through collaborative competition can optimization be significantly enhanced. This naturally guides the model's focus to discovering collaborative information in difficult samples. Conversely, for simple samples that can already be well reasoned from single-modality data (…), the focus shifts to… The gate will close smoothly. This approach maintains the integrity of the unimodal latent representation. This sample-adaptive method enables the application to coordinately interweave balanced modality utilization and collaborative information discovery while avoiding the risk of over-regularization.

[0078] 6. Steps for calculating the collaborative enhancement regularization term.

[0079] Collaborative enhancement regularization terms Represented as: Equation (11): ; in, They represent the anti-dominant targets respectively. and collaborative goals The weight. As mentioned earlier, the anti-dominant target. To prevent a converged network from being monopolized by any single modality, a collaborative objective. This is used to incentivize fusion networks to achieve prediction gains that surpass those of simply relying on single-modal aggregation. As an adaptive gating factor for perceiving single-modal proficiency, it is responsible for adjusting the intensity of collaborative information discovery (i.e., the intensity of collaborative competition optimization) based on single instance samples.

[0080] 7. MCR loss calculation steps.

[0081] MCR loss The calculation method is described in the previous formula (2), and will not be repeated here.

[0082] 8. Steps for calculating total loss.

[0083] Based on the calculated task loss MCR loss and collaborative enhancement regularization terms Calculate the total loss : Equation (12): .

[0084] III. Model Training Phase.

[0085] Based on the acquired set D, and according to a preset batch size, the fusion network is trained by sequentially inputting multimodal data samples (including sample labels) in batches, with the goal of minimizing the total loss. This updates all trainable model parameters of the fusion network until the network converges or reaches the maximum number of iterations, at which point training ends and the model parameters are saved. The trained fusion network, based on the single-modal backbone network, can then achieve adaptive collaborative information enhancement and joint prediction of the input multimodal data.

[0086] The fusion network is connected after each modal encoder to receive the aligned latent representations output by each modal encoder and fuses the latent representations of each modality to obtain a fused representation and a joint prediction result. In one optional embodiment, the fusion network may include one or more of the following: a feature splicing layer, a fully connected layer, a gated fusion layer, an attention layer, a cross-attention layer, or a Transformer encoding layer. This application does not limit the specific network structure of the fusion network, which can be flexibly selected according to the specific reasoning task. For example, for emotion perception tasks, a multimodal classifier model is used; for robot operation perception tasks, a multilingual sensory-action model is used. Of course, the specific model structure / type of the fusion network in this application is not limited to this, as long as it can generate a fused representation based on the latent representations of multiple modalities and output a joint prediction result.

[0087] During the training phase, this application optimizes the trainable parameters in the task prediction heads of the fusion network and its connections with the goal of minimizing the total loss. In one optional implementation, each modal encoder can use pre-trained parameters and keep them fixed. The single-modal prediction results, joint prediction results, and perturbation prediction results generated during the training phase are mainly used to calculate the task loss, MCR loss, and collaborative enhancement regularization term. The optimization result is the trained fusion network and its model parameters. After training is completed, in the task inference phase, the latent representations obtained by the multimodal data to be inferred through each modal encoder are input into the fusion network, and the fusion network outputs the fused representation and joint prediction results. The joint prediction results are the prediction outputs of the target multimodal inference task, such as classification probability, category label, sentiment recognition result, behavior recognition result, or regression prediction value.

[0088] The optimization method proposed in this application applies only to the fusion network and does not interfere with the pre-trained modal encoder (i.e., the single-modal backbone network). This application first actively cleans the representation space by suppressing redundant information in single-modal models, and then forces the fusion network to capture joint gains beyond simple additive aggregation. These two parts are strictly complementary, jointly enhancing the fusion network's ability to learn collaborative information. This application also uses a proficiency-aware adaptive gating mechanism for dynamic adjustment to avoid over-regularization. Extensive experimental results covering different benchmark datasets demonstrate that this application can continuously improve the performance of downstream tasks and truly enhance the model's collaborative capabilities, rather than merely achieving numerical modal balance.

[0089] IV. Experimental verification.

[0090] To verify the technical effectiveness of the proposed multimodal cooperative competitive optimization method, comparative and ablation experiments were conducted on multiple multimodal benchmark datasets. The experiments covered task scenarios such as audio-visual emotion recognition, audio-visual event recognition, action recognition, and multimodal sentiment analysis. For classification tasks, classification accuracy was used as the evaluation metric; for multimodal sentiment analysis tasks, binary classification accuracy was used. Experimental results are expressed as mean ± standard deviation; higher values ​​indicate better model prediction performance, corresponding to the invention's objective, representing higher prediction gain for multimodal cooperative inference tasks.

[0091] (a) Comparative experiment.

[0092] In the comparative experiment, the method of this application (referred to as MSCR) was compared with existing multimodal competitive regularization methods such as MCR. The experimental results are shown in Table 1 below.

[0093] Table 1 Comparative Experiment Table

[0094] The relevant units in Table 1 are explained as follows: (1) Each cell in the first row of the table header.

[0095] CREMA-D, short for Crowd-sourced Emotional Multimodal Actors Dataset, is an audiovisual emotion recognition dataset containing video clips of 91 actors. Each actor expresses one of six emotions: happiness, sadness, anger, fear, disgust, and neutrality. It is used for audio-visual emotion recognition tasks.

[0096] AVE, short for Audio-Visual Event dataset, contains 4,143 videos across 28 event categories. Each video contains time-stamped audio-visual events lasting at least 2 seconds and is used for audio-visual event recognition tasks.

[0097] UCF refers to the UCF101 dataset, short for UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild (where UCF is an abbreviation for the University of Central Florida). This application selects a subset of 51 action categories that simultaneously contain audio and video streams, according to a multimodal evaluation protocol, for action recognition tasks.

[0098] MOSI, or CMU-MOSI dataset, stands for CMU Multimodal Opinion-level Sentiment Intensity corpus. Each segment is labeled with a sentiment score ranging from -3 to 3, and it is used for multimodal sentiment analysis tasks.

[0099] MOSEI refers to the CMU-MOSEI dataset, which stands for CMU Multimodal Opinion Sentiment and Emotion Intensity dataset. It is larger than the CMU-MOSI dataset and is also labeled with sentiment scores ranging from -3 to 3. It is used for multimodal sentiment analysis tasks.

[0100] (2) Each cell in the second row of the table header.

[0101] ResNet, short for Residual Network, is a deep residual network. This application uses randomly initialized ResNet-18 as a single-modal encoder for both visual and auditory modalities on the CREMA-D, AVE, and UCF datasets.

[0102] Conformer, short for Convolution-augmented Transformer, is a high-performance experimental configuration using a large-capacity pre-trained single-modal encoder, as shown in Table 1. On the CREMA-D dataset, the auditory branch uses the first 12 layers of the Wav2Vec2 model, and the visual branch uses the MTCNN face detection algorithm combined with EfficientNet-B2 to extract facial features. On the AVE dataset, the visual branch uses the ViViT model pre-trained on the Kinetics dataset, and the auditory branch uses the HuBERT model pre-trained on the AudioSet dataset. The features of the above modalities are further refined by a 5-layer Conformer network, hence this experimental configuration is collectively referred to as the "Conformer configuration".

[0103] Transformer, also known as "transformer network", is a neural network based on the self-attention mechanism. This application uses a 5-layer Transformer as a single-modal encoder for each modality on the MOSI and MOSEI datasets.

[0104] (3) The cells in the third row of the table header.

[0105] V represents the visual modality (i.e., video frames or user facial expression frames), A represents the auditory modality (i.e., speech or audio), and T represents the text modality (i.e., text). Correspondingly, "VA" represents visual-auditory bimodal input, "VT" represents visual-text bimodal input, and "VAT" represents visual-auditory-text trimodal input. Therefore, the header of Table 1 corresponds to nine experimental settings: ResNet / VA and Conformer / VA on the CREMA-D dataset, ResNet / VA and Conformer / VA on the AVE dataset, ResNet / VA on the UCF dataset, Transformer / VT and Transformer / VAT on the MOSI dataset, and Transformer / VT and Transformer / VAT on the MOSEI dataset.

[0106] (4) First column baseline method.

[0107] Unimodals, also known as "single-modal baselines," refer to models that are independently trained and used for prediction using only a single modality of data.

[0108] Ensemble, also known as "single-modal ensemble baseline" in Chinese, refers to the process of independently training each single-modal model and then performing a late-stage additive ensemble (i.e., decision-level fusion) on the prediction results of each single-modal model to obtain the final prediction result.

[0109] Joint Training, also known as "Joint Training Baseline," refers to directly feeding the features of each modality's data into a fusion network and performing end-to-end joint training using only a single joint task loss.

[0110] Multi-Loss, also known as "multi-loss joint training method", refers to adding independent task losses to each single-modal branch on the basis of joint training, and optimizing them together with the joint task loss.

[0111] Uni-Pre Frozen, short for Unimodal Pre-trained encoders Frozen, refers to the process of first pre-training each unimodal encoder separately, then freezing the parameters of each unimodal encoder, and training only the fusion network and the task prediction head.

[0112] Uni-Pre Finetuned, short for Unimodal Pre-trained encoders Finetuned, refers to the joint fine-tuning of each single-modal encoder and the fusion network based on the above-mentioned single-modal pre-training.

[0113] MSLR, short for Modality-Specific Learning Rates, is a method that alleviates modality imbalance by setting different learning rates for different modes.

[0114] MMCosine, short for Multi-Modal Cosine loss, is a method that balances the learning of each modality by cosine normalizing the multimodal features and classification weights.

[0115] OGM, short for On-the-fly Gradient Modulation, is a method that achieves modal balance by monitoring the contribution of each modality in real time and modulating its gradient during training.

[0116] AGM, short for Adaptive Gradient Modulation, is a method that improves the performance of multimodal models by adaptively modulating the gradients of each mode.

[0117] MLB, short for "Multi-loss Gradient Modulation (Balancing) Method", gets its name from the multi-loss gradient modulation balancing strategy it employs, which balances the learning of each mode by modulating the gradients of multiple loss terms.

[0118] ReconBoost, short for Reconcilement Boosting, is a modal harmonization and enhancement method that uses a boosting strategy to alternately update each modality in order to achieve harmonization between modalities.

[0119] MMPareto, short for MultiModal Pareto, is a method that uses Pareto-optimal gradient integration to improve multimodal learning performance by leveraging single-modal auxiliary tasks.

[0120] D&R, short for Diagnosing and Re-learning, is a method that balances multimodal learning by diagnosing the learning state of each single-modal encoder and relearning it.

[0121] MCR, short for Multimodal Competition Regularizer, is the closest prior art described in the background section of this application. It alleviates the modality imbalance problem in multimodal training through game theory-based competitive regularization.

[0122] (5) Table data section.

[0123] This section of data corresponds to the performance metrics obtained by each baseline algorithm under the corresponding dataset, backbone network, and modality combination, expressed as a percentage in the form of "mean ± standard deviation". For the CREMA-D, AVE, and UCF classification task datasets, classification accuracy is used as the performance metric; for the MOSI and MOSEI sentiment regression task datasets, binary accuracy is used as the performance metric according to the MCR evaluation protocol. The statistical methods for mean and standard deviation are as follows: CREMA-D and UCF use three-fold cross-validation, and the mean and standard deviation of the three-fold results are reported; AVE, MOSI, and MOSEI use three independent repeated experiments with three different random seeds, and the mean and standard deviation of the three results are reported. A larger value indicates better predictive performance. The symbol "×" in Table 1 indicates that the method is not applicable to regression tasks, and therefore there are no corresponding results on the MOSI and MOSEI datasets; the symbol "×" indicates that the method is not applicable to regression tasks, and therefore there are no corresponding results on the MOSI and MOSEI datasets. "" indicates that the method is not applicable to trimodal inputs, and therefore there are no corresponding results under the VAT setting. In addition, to ensure the fairness of the comparison, the results of each comparison method in Table 1 are directly taken from the publicly reported results under the strict evaluation settings established by MCR. The MSCR method of this application and each comparison method were experimented with under the same data partitioning, single-modal encoder, fused backbone network, optimizer settings, learning rate strategy, batch size and training epochs.

[0124] As shown in Table 1 above, compared with the closest existing technology MCR, the proposed method MSCR achieves higher prediction performance under different datasets, different modality combinations, and different backbone network settings. Compared with other comparative methods, the proposed method MSCR also achieves higher prediction performance in most experimental settings. These results indicate that the proposed method can not only alleviate the modality imbalance problem in multimodal learning, but also further enhance the fusion network's ability to utilize cross-modal collaborative information through anti-dominant targets, cooperative targets, and adaptive gating mechanisms, thereby improving the prediction performance of target multimodal reasoning tasks.

[0125] The comparative experiments described above are used to demonstrate the advantages of this application over existing technologies. The specific relationship between these advantages and the invention's objective—"to alleviate the imbalance of modal data in multimodal learning while enhancing collaborative information representation and improving the prediction gain of downstream multimodal collaborative reasoning tasks"—is as follows: Firstly, regarding "alleviating modal data imbalance in multimodal learning," MSLR, MMCosine, OGM, AGM, MLB, ReconBoost, MMPareto, D&R, and MCR listed in Table 1 are all existing methods specifically designed to address the modal imbalance problem. Our proposed method, MSCR, outperforms the closest existing method, MCR, in all nine experimental settings, and surpasses all the aforementioned existing methods in eight of the nine settings. Only in the Conformer setting of the AVE dataset does the D&R method achieve a higher accuracy (91.0%) than our proposed method (89.3%). However, our proposed method still outperforms all other existing methods, including MCR, in this setting. In all other settings, the D&R method is inferior to our proposed method and is not applicable to the MOSI and MOSEI regression task datasets. These results demonstrate that our proposed method is generally no less effective than, and even better than, existing modal balancing methods in alleviating modal data imbalance.

[0126] Secondly, regarding "enhanced cooperative information representation," the Unimodals row in Table 1 reflects the performance level provided by each individual modality's data itself, while the Ensemble row reflects the performance level achievable through additive integration of independent single-modality decisions without any feature-level cross-modal interactions. Both can serve as a reference baseline for "no cooperative information." The performance gain of the MSCR of the method in this application relative to the aforementioned baseline represents the cooperative gain brought about by feature-level cross-modal interactions, which cannot be obtained by simply superimposing single-modality evidence. For example, under the Conformer setting of the CREMA-D dataset, the Ensemble is 84.6%, while the MSCR is 86.4%; under the VAT setting of the MOSI dataset, the Ensemble is 67.2%, while the MSCR is 80.1%; and under the VAT setting of the MOSEI dataset, the Ensemble is 77.2%, while the MSCR is 81.8%. This result demonstrates that this application does indeed enhance the ability of the fusion network to represent cooperative information.

[0127] Thirdly, regarding "improving the prediction gain of downstream multimodal collaborative inference tasks," compared with the closest existing technology MCR (which only enhances the information of each single modality and ignores collaborative information), the prediction performance of the MSCR method in this application is improved in all experimental settings: CREMA-D (ResNet, VA) increased from 74.5% to 75.7%, CREMA-D (Conformer, VA) increased from 85.1% to 86.4%, AVE (ResNet, VA) increased from 73.5% to 74.5%, and AVE (Conf... The percentage of Ormer (VA) increased from 87.4% to 89.3%, UCF (ResNet, VA) increased from 52.9% to 54.4%, MOSI (Transformer, VT) increased from 75.3% to 76.8%, MOSI (Transformer, VAT) increased from 78.6% to 80.1%, MOSEI (Transformer, VT) increased from 80.8% to 81.3%, and MOSEI (Transformer, VAT) increased from 81.1% to 81.8%.

[0128] Furthermore, the aforementioned performance improvements are consistently achieved across two task domains: audio-visual classification and multimodal sentiment analysis; two modal scenarios: bimodal (VA, VT) and trimodal (VAT); and encoder capacities: randomly initialized ResNet and large-scale pre-trained Conformer and Transformer. This demonstrates that the prediction gain brought by this application does not depend on specific modal combinations, task types, or network structures, but rather stems from the more thorough discovery and utilization of cross-modal collaborative information through anti-dominant objectives, collaborative objectives, and adaptive gating mechanisms, thus confirming the achievement of the aforementioned inventive objectives.

[0129] (ii) Ablation experiment.

[0130] To further verify the roles of the anti-dominant and cooperative objectives in this application, four experimental configurations were set up: no anti-dominant and cooperative objectives, only anti-dominant objectives, only cooperative objectives, and both anti-dominant and cooperative objectives. The experimental results are shown in Table 2 below.

[0131] Table 2 Ablation Experiment Table

[0132] The cell definitions in Table 2 are as follows: (1) Each cell in the first row of the table header.

[0133] The loss term corresponding to the anti-dominant objective. This represents the loss term corresponding to the collaborative objective. The definitions of the remaining cells are the same as those in Table 1.

[0134] (2) The first two columns.

[0135] In the first two columns, " "" indicates that the corresponding loss item is retained / enabled, while blank indicates that the corresponding loss item is discarded / disabled.

[0136] (3) Table data section.

[0137] The test values ​​are the prediction performance indicators measured under the corresponding experimental settings. The performance indicators tested (classification accuracy for classification tasks and binary classification accuracy for sentiment regression tasks), expressed as percentages (mean ± standard deviation), and their statistical methods are defined in the same way as the previous explanation of Table 1. The larger the value, the better the prediction performance.

[0138] As shown in Table 2, the ablation experiment results indicate that introducing either the anti-dominant objective or the cooperative objective alone can improve the prediction performance of the baseline model. When both the anti-dominant and cooperative objectives are introduced simultaneously, the model achieves optimal or better performance in all experimental settings. These results demonstrate that the anti-dominant objective can suppress the excessive reliance of the fusion network on single-modal information or redundant information, providing a representation space for cross-modal cooperative information. The cooperative objective can further guide the fusion network to learn joint prediction gains beyond the additive aggregation of single modalities. The two objectives work together to enhance the cooperative information discovery capability of the fusion network.

[0139] The purpose of conducting ablation experiments is to verify the independent technical contributions of each of the two key technical features, as well as the complementary effect produced by their combination, by adding and deleting the anti-dominant target and the cooperative target one by one, under the premise that all other experimental conditions (including dataset, data partitioning, single-modal encoder, fusion network, optimizer settings and training hyperparameters) are exactly the same. This will prove that the aforementioned technical effect is indeed brought about by the technical solution designed in this application itself, rather than by other experimental factors.

[0140] Specifically, when only the anti-dominant objective is enabled, the prediction performance is improved relative to the baseline configuration (e.g., from 85.1% to 85.9% in the Conformer setting of CREMA-D and from 87.4% to 88.8% in the Conformer setting of AVE). This result corresponds to the "mitigation of modal data imbalance in multimodal learning" in the purpose of the invention, indicating that suppressing the excessive dependence of the fusion network on the dominant mode can free up representation space for cooperative information. When only the cooperative objective is enabled, the prediction performance is also improved relative to the baseline configuration (e.g., from 85.1% to 86.2% in the Conformer setting of CREMA-D, and from 75.3% to 76.1% in the VT setting of MOSI). This result corresponds to the "enhanced cooperative information representation" in the invention's objective, indicating that the explicit requirement for the prediction error of the fusion network to be lower than the additive single-modal reference distribution can enable the fusion network to capture cooperative gains beyond the simple superposition of single-modal evidence. When both are enabled simultaneously, the model achieves optimal performance in all seven experimental settings (e.g., 86.4% in the Conformer setting of CREMA-D, and 89.3% in the Conformer setting of AVE). This result corresponds to the "improved prediction gain of downstream multimodal cooperative reasoning tasks" in the invention's objective, indicating that the two steps of "cleaning the representation space" and "filling cooperative gains" are complementary and indispensable, and the combination of the two can produce technical effects that are better than using them alone.

[0141] Based on the ideas of this application, this application also proposes a multimodal cooperative competition optimization device, which includes a processor and a storage medium. The storage medium stores a computer program, and the processor runs the computer program to execute the multimodal cooperative competition optimization method of the above embodiments.

[0142] Furthermore, this application also proposes a multimodal task reasoning method for performing task reasoning based on multimodal data; the method includes: Encode multimodal data; The encoded latent representation is input into a fusion network optimized using the multimodal cooperative competitive optimization method described in the above embodiments, for joint prediction; Output the joint prediction results.

[0143] After training, the fusion network can fully discover the collaborative information of different modal data based on modal balance, thereby significantly improving the prediction gain of task reasoning that relies on collaborative information.

[0144] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A multimodal cooperative competitive optimization method for extracting multimodal representations from multimodal data for multimodal reasoning tasks; characterized in that, The multimodal data includes at least two of the following: text, image, audio, time series, or tabular data; the multimodal representation includes redundant information, unique information, and collaborative information of the multimodal data; the unique information represents information unique to each modality; the redundant information represents information that can be independently extracted from any modality. The collaborative information represents evidence that can only be obtained when more than one modality of data is jointly interpreted; The methods include: Acquire multimodal data samples, encode and perturb the multimodal data samples to obtain the latent representation and perturbation representation of each modality; Single-modal prediction, joint prediction, and perturbation prediction are performed based on the latent representation and the perturbation representation, respectively. The total loss is calculated based on the prediction results of each type, and multimodal learning is performed based on the calculated total loss to optimize the fusion network. The total loss includes task loss, MCR loss, and collaborative enhancement regularization term; wherein, the collaborative enhancement regularization term guides the fusion network to suppress dependence on any single-modal data through anti-dominant objective, guides the fusion network to display enhanced collaborative information through collaborative objective, and dynamically activates the collaborative enhancement regularization term based on the single-modal proficiency of multimodal data samples using an adaptive gating mechanism.

2. The multimodal cooperative competition optimization method as described in claim 1, characterized in that, Suppressing dependence on any single modality of data by guiding the fusion network through anti-dominant objectives includes: The sensitivity of the fusion network to each modality of data is quantitatively evaluated based on the joint prediction results and the perturbation prediction results. Based on the aforementioned sensitivity, an anti-dominant objective is constructed, and minimizing the anti-dominant objective is taken as the optimization objective.

3. The multimodal cooperative competition optimization method as described in claim 2, characterized in that, The sensitivity is quantified and evaluated using the JS divergence of the joint prediction results and the perturbation prediction results; the anti-dominant objective is constructed by applying a smoothmax penalty to the sensitivity.

4. The multimodal cooperative competition optimization method as described in claim 1, characterized in that, Enhanced collaborative information is displayed through collaborative goal-oriented fusion networks, including: An additive single-mode reference distribution is constructed based on the prediction results of each single mode; Calculate the task loss of the joint prediction result and the task loss of the reference distribution, respectively. A collaborative objective is constructed based on the computational task loss, and the optimization objective is to minimize the collaborative objective.

5. The multimodal cooperative competition optimization method as described in claim 4, characterized in that, The reference distribution is constructed as the mean of the prediction results of each single mode; the cooperative objective is constructed as: ； in, Indicate collaborative goals, This represents a pre-defined marginal parameter; These represent the task loss of the joint prediction result and the task loss of the reference distribution, respectively.

6. The multimodal cooperative competition optimization method as described in claim 1, characterized in that, Using an adaptive gating mechanism, the collaborative enhancement regularization term is dynamically activated based on the unimodal proficiency of the multimodal data samples, including: Calculate the task loss for each single-modal prediction result, and quantitatively evaluate the single-modal proficiency of the multimodal data samples based on the calculated task loss; Based on the single-modal proficiency, a gating function is constructed and activated, and an adaptive gating factor is calculated. The adaptive gating factor is used as the optimization coefficient of the collaborative enhancement regularization term.

7. The multimodal cooperative competition optimization method as described in claim 6, characterized in that, The adaptive gating factor is calculated as follows: ； in, Indicates the adaptive gating factor; This represents the sigmoid activation function; hyperparameters These represent the sharpness parameter and activation threshold of the gating function, respectively. This indicates single-modal proficiency.

8. The multimodal cooperative competition optimization method as described in claim 6, characterized in that, The collaborative enhancement regularization term is constructed as follows: ； in, This indicates a cooperative enhancement regularization term; They represent the anti-dominant targets respectively. and collaborative goals The weights; It is an adaptive gating factor.

9. A multimodal cooperative competition optimization device, characterized in that, It includes a processor and a storage medium, the storage medium storing a computer program; the processor runs the computer program to perform the multimodal cooperative competition optimization method as described in any one of claims 1-8.

10. A multimodal task reasoning method for performing task reasoning based on multimodal data; characterized in that, The methods include: Encode multimodal data; The encoded latent representation input is used to optimize a fusion network using the multimodal cooperative competitive optimization method as described in any one of claims 1-8, for joint prediction; Output the joint prediction results.

Images