A training method of a multi-modal classification model

Abstract

The application discloses a kind of training methods of multi-modal classification model, it is related to data prediction technical field, method includes: obtaining by original image data and corresponding original text data constitute multi-modal data set, to original multi-modal data in multi-modal data set is executed multi-level mixed intervention processing, obtain multi-modal intervention feature, then based on multi-modal intervention feature determines reconstruction loss and alignment loss, reconstruction loss, alignment loss and independent regular term are fused to obtain the target loss for representing the comprehensive optimization target of initial classification model, finally using multi-modal data set and target loss to initial classification model is trained, obtain target multi-modal classification model;Solved the technical problem that model in complex and changeable real application scene in related art is poor in generalization ability, reached the technical effect of improving the generalization ability of model in complex and changeable real application scene.

Description

Technical Field

[0001] This application relates to the field of model training technology, and in particular to a training method for a multimodal classification model. Background Technology

[0002] With the deepening application of artificial intelligence technology in key areas such as industrial quality inspection, autonomous driving, and medical diagnosis, the robustness of models when the distribution of test data is inconsistent with the distribution of training data has become the core key to determining whether they can be practically deployed.

[0003] However, when such models are fine-tuned to adapt to specific downstream tasks (such as defect detection, object classification, etc.), there are still significant technical bottlenecks: the models tend to over-rely on spurious correlation features in the training data (such as background environment, lighting conditions, material texture and other surface interference factors) rather than learning the causal features of the task itself, which leads to a sharp drop in performance in out-of-distribution scenarios (such as environmental changes, data distribution shifts).

[0004] Therefore, there is a technical problem in related technologies: the models have poor generalization ability in complex and ever-changing real-world application scenarios. Summary of the Invention

[0005] This application provides a training method for a multimodal classification model to at least address the problem of poor generalization ability of models in complex and ever-changing real-world application scenarios in related technologies.

[0006] This application provides a method for training a multimodal classification model, the method comprising: Acquire a multimodal dataset; the multimodal dataset includes raw image data and raw text data belonging to the same acquisition scene as the raw image data; The original multimodal data in the multimodal dataset is subjected to multi-level hybrid intervention processing to obtain multimodal intervention features, which are a cross-modal feature set containing invariant factor features and variable factor features; Based on the aforementioned multimodal intervention features, the reconstruction loss and alignment loss are determined; The reconstruction loss, the alignment loss, and the independent regularization term are fused to obtain the target loss; the target loss is used to characterize the comprehensive optimization objective of the initial classification model. The initial classification model is trained using the multimodal dataset and the target loss to obtain the target multimodal classification model.

[0007] This application also provides a method for predicting data types based on a multimodal classification model, the method comprising: Acquire the image data to be processed; The target multimodal classification model is used to predict the type of the image data to be processed, wherein the target multimodal classification model is obtained by training the multimodal classification model as described above.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of training a multimodal classification model or predicting a data type based on a multimodal classification model.

[0009] Through this application, the training method of the multimodal classification model obtains a multimodal dataset composed of original images and text. It then uses a multi-layered hybrid intervention involving directional mutation generation at the data layer and causal decoupling at the feature layer to obtain multimodal intervention features containing cross-modal shared causal features (invariant factors) and surface interference features (variable factors). Furthermore, it uses reconstruction loss to ensure semantic integrity after intervention, alignment loss to ensure cross-modal semantic consistency in the invariant subspace, and independent regularization terms to strengthen factor independence and fuse these to obtain the target loss. Finally, it trains the model using the dataset and the target loss. This not only allows the model to focus on essential causal features, significantly improving generalization performance and robustness in distribution-biased scenarios, but also enhances the interpretability of model decisions through systematic intervention design and explicit feature decoupling. Simultaneously, the highly automated intervention generation and training process combines creativity and practicality, effectively adapting to the complex real-world scenario requirements of key fields such as industrial quality inspection and autonomous driving. Therefore, it can solve the technical problem of poor generalization ability of models in complex and ever-changing real-world application scenarios in related technologies, achieving the technical effect of improving the generalization ability of models in complex and ever-changing real-world application scenarios. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 One of the flowcharts illustrating a training method for a multimodal classification model provided in this application embodiment; Figure 2 A second schematic flowchart illustrating a training method for a multimodal classification model provided in this application embodiment; Figure 3 A flowchart illustrating a data type prediction method based on a multimodal classification model provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of a training system for a multimodal classification model provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0013] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] This application provides an embodiment of a training method for a multimodal classification model. The method is described in detail below, along with its execution flow. (Refer to...) Figure 1 The diagram shown is one of the flowcharts illustrating a training method for a multimodal classification model provided in this application. The specific steps include the following: S101. Obtain the multimodal dataset.

[0016] The multimodal dataset includes raw image data and raw text data belonging to the same acquisition scene as the raw image data.

[0017] Specifically, obtain a set of labeled samples containing original image data and corresponding original text data. The original image data must clearly present the core visual information of the target object (such as product appearance, road conditions), and the original text data must describe the semantics of the image (such as defect type, environmental conditions).

[0018] S102. Perform multi-level hybrid intervention processing on the original multimodal data to obtain multimodal intervention features.

[0019] Among them, multi-level hybrid intervention processing is used to characterize the collaborative intervention operation of directional mutation generation and causal decoupling on the original multimodal data. The multimodal intervention feature is a cross-modal feature set containing invariant factor features and variable factor features. Invariant factor features are used to characterize the causal features shared across modalities, and variable factor features are used to characterize surface interference features.

[0020] Specifically, the intervention involves a two-tiered approach: directional mutation generation at the data layer and causal decoupling at the feature layer. First, based on the structural anchors and semantic instructions of the original multimodal data, intervention data is generated that retains the core structure and only mutates surface features. Then, the multimodal features of the original multimodal data and the intervention data are decoupled, separating invariant and variable factors. Causal constraints are strengthened through feature exchange, ultimately yielding multimodal intervention features.

[0021] Specifically, the above S102 can be further refined into the following steps 1021 to 1024: Step 1021: Determine the structural data of the original image data and the original text data corresponding to the original image data from the original multimodal data.

[0022] Among them, structural data is used to characterize the foreground and background separation information and object geometry of the original image data.

[0023] Specifically, structural data is extracted from the original image data, and original text data that is strongly correlated with the semantics of the image is acquired simultaneously. The structural data constrains the invariant boundaries of the intervention, while the original text data provides the direction of variation for the intervention.

[0024] For example, pixel-level depth prediction is performed on the original image data I using a pre-trained depth estimation model (such as MiDaS or LeReS), and a normalized depth map is output as structured data D using the following formula: D= (I) in, The depth estimation model is represented by the structure data D. The larger the structure data D, the closer the distance. It can clearly separate the foreground subject (such as the product to be detected) from the background (such as the production line) and completely preserve the geometry of the object. The original text data T is a natural language description that corresponds one-to-one with the original image data I, for example: "black plastic shell, side shot, no scratches, production process: injection molding", which must include key semantics such as object material, shooting angle, core state, etc.

[0025] Step 1022: Generate intervention instructions based on the original text data and the variable factor lexicon.

[0026] The intervention instruction is a text instruction that modifies the surface features of the original image data.

[0027] Specifically, phrases are extracted from the original text data to obtain a current phrase list; at least one phrase is identified from the current phrase list; based on semantic annotation of the phrases, a replacement phrase is determined from a variable factor lexicon; the replacement phrase is then reorganized into a sentence to generate an intervention instruction. In other words, key phrases are extracted from the original text through semantic role annotation, and targeted replacement and reorganization are performed using a dynamic variable factor lexicon to generate semantically coherent and directional intervention instructions, achieving semantically perceptive targeted intervention.

[0028] For example, semantic role labeling models Parsing the raw text data:

[0029] Specifically, noun phrases (adjective-noun combinations: e.g., "black plastic"; noun-noun combinations: e.g., "injection molding process") and adverbs (manner / time / place: e.g., "side view") are extracted to obtain the current phrase list. :

[0030] in, Different phrases are represented, for example, the current phrase list can be [black plastic, side view, injection molding process, no scratches]; 1-2 surface feature phrases (such as "black plastic" and "side view") are randomly selected, avoiding core feature phrases (such as "no scratches"); based on the semantic annotation of the phrase to be replaced (such as "black plastic" belonging to the "theme appearance" category and "side view" belonging to the "shooting angle" category), replacement phrases (such as "white metal" and "front view") are extracted from the corresponding category buckets of the variable factor lexicon; the replacement phrases and the unreplaced phrases are recombined into a complete sentence to generate the intervention instruction: "white metal shell, front view, injection molding process, no scratches"; the variable factor lexicon consists of an initial dynamic lexicon and an LLM extended lexicon: the initial lexicon is obtained by clustering phrases from all texts in the dataset, and the extended lexicon is generated by a large language model based on the initial lexicon to generate novel phrases.

[0031] Step 1023: Based on the intervention instructions and structural data, determine the intervention image data and the corresponding intervention text data.

[0032] Intervention image data is used to characterize image data that retains the structure of the original image data and incorporates variations of the intervention instructions. Intervention text data is a textual description that is semantically bound to the intervention image data.

[0033] Specifically, intervention instructions are encoded as semantic embedding features, and structural data is encoded as geometric constraint features. Together, they guide the controllable diffusion model to denoise and reconstruct the latent spatial features of the original image data, ultimately obtaining an intervention image that meets the constraints.

[0034] Intervention instructions are encoded as textual conditional features, which are used to characterize the semantic embedding vector of the intervention instructions; structural data are encoded as structural conditional features, which are used to characterize the geometric structure embedding vector state of the original image data; based on the textual and structural conditional features, the noise latent variables of the original image data are denoised and reconstructed to obtain the intervention image data, and the noise latent variables are used to characterize the image features with noise interference in the latent space.

[0035] Intervention text data is generated by modifying the original text data. The modification logic is based on the variation logic of the intervention image data synchronized with the intervention instructions. Only the phrases describing the variable features of the surface in the original text data are replaced, while the content describing the core causal features is retained. Finally, a text description that is consistent with the surface feature variation of the intervention image and whose core semantics remain unchanged is formed.

[0036] For example, the intervention instruction can be encoded as a textual conditional feature ctext, which can be represented as: ctext=T (Pdo) Here, the textual conditional feature ctext represents the semantic direction of the intervention, and Pdo represents the intervention operation performed on the variable; the structured data (depth map) is encoded into the structured conditional feature cdepth using the encoder network fc, which can be represented as: cdepth=fc(D) The structural conditional feature cdepth is used to constrain the geometric shape of the image. The original image is encoded into the latent space by a VAE encoder to obtain the initial latent variable z0, and random noise is added to it to generate the noisy latent variable zt, which represents the image features in the latent space containing noise interference. The zt is iteratively denoised by a UNet network. Each denoising step takes zt, time step t, text conditional feature ctext, and structural conditional feature cdepth as inputs to ensure that the generated image follows the intervention instructions without destroying the original structure. Finally, the denoised latent variable is decoded by a decoder to obtain the intervention image data, which can be represented as: Igen=VAE_Decoder (z0) Here, Decoder represents the decoding module of the VAE encoder, whose function is to map the latent spatial features back to the image space.

[0037] Step 1024: Perform mixed intervention processing on the original image data, intervention image data, original text data and intervention text data to obtain multimodal intervention features.

[0038] Specifically, the process first extracts multimodal semantic features from four types of data, then separates invariant and variable factors using a dedicated adapter, and finally severs the original spurious association between the two through cross-sample feature exchange to generate multimodal intervention features. This involves extracting multimodal features from the original image data, intervention image data, original text data, and intervention text data. These multimodal features characterize the original semantic embedding of the original multimodal data and can include both visual and textual features. The multimodal features are then decoupled to determine invariant and variable factor features. Finally, feature exchange operations are performed on the invariant and variable factor features to generate multimodal intervention features. This feature exchange characterizes the causal intervention operation that severs the original spurious association between the invariant and variable factor features.

[0039] For example, the visual features of the original image data I and the intervened image data Igen are extracted using a pre-trained visual encoder Ev, respectively, as follows: Fv=Ev(I), =Ev(Igen) Where Fv represents the visual features of the original image data I, The visual features of the intervention image data Igen are represented by the text features Ft of the original text data T and the text features of the intervention text data Tgen are extracted using a pre-trained text encoder Et. Represented as: Ft=Et(T), =Et(Tgen) The above four types of features are combined to obtain multimodal features, which represent the original semantic embedding of the original multimodal data; visual features are then combined. Input Invariant Adapter Image invariant feature obtained , is represented as:

[0040] Input visual features Fv into the variable adapter (Standard MLP) to obtain image variable factor features , is represented as:

[0041] Similarly, by inputting the text features Ft into the text-invariant adapter and the variable adapter, we obtain the text-invariant factor features. Textual variable factor features , respectively represented as:

[0042] Next, the image invariant factor Text Invariant Factor Features Concatenate and input the shared Transformer network backbone The invariant factor characteristics shared across modes are obtained. , is represented as:

[0043] Secondly, within the same training batch, samples i and j (i≠j) are randomly selected, and the invariant factor features of sample i are... Image variable factor features of sample j Text variable factors Recombination, generating post-intervention image features and post-intervention text features , respectively represented as:

[0044] This operation severs the original spurious correlation between the invariant factor and its own variable factors, forcing the model to rely solely on the invariant factor for judgment; finally, the image features after intervention are merged. and post-intervention text features This yields the final set of multimodal intervention features.

[0045] S103. Based on multimodal intervention features, determine reconstruction loss and alignment loss.

[0046] Among them, the reconstruction loss is used to characterize the difference between the original multimodal data in the multimodal dataset and the reconstructed multimodal data after reconstructing the original multimodal data; the alignment loss is used to characterize the semantic consistency of the multimodal intervention features obtained based on the original multimodal data in the invariant subspace.

[0047] Specifically, the above S103 can be further refined into the following steps 1031 to 1032: Step 1031: Reconstruct the multimodal intervention features and determine the reconstruction loss.

[0048] Specifically, based on multimodal intervention features, image and text data consistent with the original data modality are reconstructed using a dedicated decoder. The intramodal error and cross-modal error between the reconstructed data and the original data are calculated, and the reconstruction loss is obtained by weighted fusion.

[0049] First, based on multimodal intervention features, reconstructed image data and reconstructed text data are generated.

[0050] Specifically, by using a dedicated decoder adapted to multimodal intervention features, the post-intervention image features and post-intervention text features are reverse reconstructed to generate reconstructed data that is semantically consistent with the original data and format compatible.

[0051] For example, a lightweight U-Net deconvolutional network is used as the image decoder. The network structure includes three downsampling layers (3×3 convolution + ReLU activation + 2×2 max pooling) and three upsampling layers (transposed convolution + 3×3 convolution + ReLU activation). After upsampling, each layer is connected to the feature map of the corresponding downsampling layer via skip connections (preserving image details). The input is the post-intervention image features. After inverse processing by the decoder, the output is reconstructed image data Irec with the same resolution as the original image; a Transformer decoder is used as the text decoder, containing two decoding layers, each consisting of a multi-head attention layer, an encoder-decoder attention layer, and a feedforward neural network; the input is the text features after intervention. Cross-entropy loss is used to assist in sequence generation, outputting reconstructed text data that is compatible with the original text in terms of length and semantics. This ensures that the syntax is correct and that the core semantics are not lost.

[0052] Secondly, determine the image reconstruction loss between the reconstructed image data and the original image data.

[0053] Among them, the text reconstruction loss between the reconstructed text data and the original text data, and the cross-modal loss between the reconstructed image data and the reconstructed text data are determined.

[0054] Specifically, image reconstruction loss ensures that the core visual information of the image modality is not lost, text reconstruction loss ensures that the core descriptive information of the text modality is not distorted, and cross-modal loss ensures that the semantics of the reconstructed image and text remain consistent.

[0055] For example, suppose the image reconstruction loss consists of pixel-level L1 loss and feature-level MSE loss; the L1 loss calculates the difference in pixel grayscale values ​​between the reconstructed image and the original image. To ensure visual consistency, the feature-level MSE loss is first applied by encoding Irec and I separately using an encoder, and then the squared difference of the encoded feature vectors is calculated. This ensures deep semantic consistency; the weighted sum of the two results in the image reconstruction loss. , is represented as:

[0056] The weight coefficients are represented by the text reconstruction loss, which consists of sequence-level L1 loss and feature-level MSE loss. The L1 loss calculates the difference between the word embedding vectors of the reconstructed text and the original text. To ensure text sequence coherence, the feature-level MSE loss is first applied by encoding Trec and T separately using an encoder, and then the squared difference of the encoded feature vectors is calculated. This ensures deep semantic consistency of the text; the weighted sum of the two (weight ratio 1:0.1) yields the text reconstruction loss, expressed as... :

[0057] The first step involves representing the text feature encoder. Then, by applying Irec and Trec encodings using different encoders, the squared difference between the two sets of encoded feature vectors is calculated, representing the cross-modal loss. , is represented as: .

[0058] Finally, the image reconstruction loss, text reconstruction loss, and cross-modal loss are weighted to obtain the reconstruction loss.

[0059] Specifically, based on the requirements of modal integrity and cross-modal consistency in different application scenarios, differentiated weights are assigned to the three types of sub-losses, and the weighted sum is used to obtain the final reconstruction loss.

[0060] For example, suppose the image reconstruction loss weights Text reconstruction loss weights Cross-modal loss weights Calculate the reconstruction loss using the following formula. :

[0061] Among them, reconstruction loss The smaller the value, the better the semantic integrity of the intervention feature, indicating that the core causal information has not been lost due to the intervention.

[0062] Step 1032: Align the multimodal intervention features and determine the alignment loss.

[0063] Specifically, the multimodal intervention features are mapped to a unified invariant subspace, and the same semantic positive sample feature pairs and non-matching negative sample feature pairs are divided. The semantic similarity of the feature pairs is quantified by comparison loss to obtain the alignment loss.

[0064] First, the multimodal intervention features are mapped to an invariant subspace to obtain the mapped features.

[0065] Among them, the mapping feature is used to characterize the embedding vector of the multimodal intervention feature in the causal space.

[0066] Specifically, by using linear projection or lightweight neural networks, the post-intervention image features and post-intervention text features are mapped to a causal invariant subspace, so that the mapped features retain only causal-related information and filter out surface interference.

[0067] For example, suppose a linear projection matrix is ​​used as the projection tool. Linear transformation and normalization are performed on the post-intervention image feature I' and post-intervention text feature T', respectively; the output image mapping feature is then obtained. and text mapping features , respectively represented as: .

[0068] Secondly, feature pairs are determined based on the mapping features.

[0069] Among them, feature pairs ( The positive sample feature pairs and the non-matching negative sample feature pairs are used to represent the same semantic positive sample feature pairs and the same original image data and original text data corresponding to the image mapping features and text mapping features, respectively. The negative sample feature pairs are the image mapping features and text mapping features corresponding to different original image data and original text data.

[0070] Specifically, based on the image-text correspondence of the original multimodal data, positive sample feature pairs with the same semantics and non-matching negative sample feature pairs are determined to ensure that the distinction between positive and negative samples focuses on causal semantics rather than surface features.

[0071] For example, for the i-th group of "image Ii-text Ti" in the original multimodal data, its corresponding post-intervention image mapping features i and text mapping features i constitutes a positive sample feature pair; the core semantics of this feature pair are consistent (e.g., both correspond to "scratches on the phone frame"), and only the surface features may differ; for the image mapping features of the i-th group i, and the text mapping features of other groups in the training batch (j≠i). j constitutes a negative sample feature pair; the core semantics of this feature pair are inconsistent (e.g., ... i corresponds to "scratch defect". j corresponds to “dent defect”, ensuring that the semantic differences of negative samples originate from causal features rather than surface features.

[0072] Finally, the alignment loss is determined based on the feature pairs.

[0073] Specifically, InfoNCE or cosine similarity comparison loss can be used to calculate the semantic similarity score of all feature pairs. The proportion of positive sample scores can be converted into alignment loss through logarithmic operation. The smaller the loss, the stronger the cross-modal semantic consistency.

[0074] For example, using the InfoNCE loss function to calculate the alignment loss, the similarity of feature pairs is first calculated: for each positive sample feature pair, cosine similarity is calculated; for each negative sample feature pair, cosine similarity is calculated; all similarity scores are divided by the temperature hyperparameter t; the ratio of the exponential value of the positive sample score to the sum of the exponential values ​​of all sample scores is calculated, and the natural logarithm is taken as the negative value; finally, the calculation results of all i are summed to obtain the final alignment loss. , is represented as:

[0075] Where N represents the batch size; alignment loss The smaller the value, the stronger the clustering of cross-modal features with the same semantic meaning in the invariant subspace, and the better the semantic consistency.

[0076] S104. The reconstruction loss, alignment loss, and independent regularization term are fused to obtain the target loss.

[0077] The target loss is used to characterize the overall optimization objective of the initial classification model, and the independence regularization term is used to constrain the independence of the invariant factor features and variable factor features decoupled from the original multimodal data of the multimodal dataset.

[0078] Specifically, initial weights are first assigned to the three types of loss based on the training stage and scenario requirements. Then, the weights are dynamically adjusted based on the loss values ​​from each training round. Finally, the target loss is obtained by weighted summation, achieving dynamic balance in multi-objective optimization.

[0079] For example, using the Hilbert-Schmidt Independence Criterion (HSIC), the constrained fusion invariant factor characteristics are calculated. Image variable factor features Text variable factors The statistical independence is determined by the formula:

[0080] in, and This represents the modal weights.

[0081] Specifically, step 104 above can be further refined into steps 1041 to 1043 as follows: Step 1042: Determine the training phase of the initial classification model and the scenario requirements corresponding to the multimodal dataset.

[0082] Specifically, the training phase is divided into three levels according to the model's learning progress: initial stage (relying on simple interventions within the dataset), mid-stage (introducing dynamic vocabulary to expand phrases for intervention), and late stage (using creative interventions generated by LLM). Scenario requirements necessitate defining downstream core tasks (such as industrial quality inspection and autonomous driving), locating essential causal features, identifying surface interference factors, and clarifying key performance indicators. Ultimately, this makes multi-level causal interventions more targeted, avoids blind training, and ensures that model training aligns with actual application goals while balancing model convergence speed and generalization ability.

[0083] Step 1042: Based on the training phase and / or scenario requirements, determine the weights of the reconstruction loss, alignment loss, and independent regularization term.

[0084] Specifically, weights are determined based on the initial classification model training phase or the scenario requirements corresponding to the multimodal dataset; initial weights are determined based on different stages of model training (initial stage focuses on decoupling, mid-stage focuses on alignment, and late stage focuses on semantic integrity) and the core requirements of the application scenario (such as industrial quality inspection focusing on semantic integrity), providing a basis for the calculation of target loss.

[0085] Step 1043: Based on the weights, perform a weighted summation of the reconstruction loss, alignment loss, and independent regularization term to obtain the target loss.

[0086] Specifically, the dynamically adjusted weights are multiplied by their corresponding losses and summed to obtain the target loss. The direction of the decrease in the target loss is the direction of improvement in the overall performance of the model.

[0087] For example, the loss will eventually be reconstructed. Alignment loss and independent regularization terms The target loss is obtained by weighting according to the following formula. :

[0088] in, and The smaller the target loss value, the better the overall performance of the model.

[0089] After each training round, the weight allocation is dynamically optimized based on the loss value feedback of each training round, so that the target loss focuses on the weak links of the current model, thereby improving training efficiency and model performance. That is, after each training round, the reconstruction loss, alignment loss and independent regularization term of the current round are determined. Based on the reconstruction loss, alignment loss, and independent regularization term of the current round, an adjustment strategy is determined; based on the adjustment strategy, a new target loss is determined.

[0090] If the reconstruction loss in the current round is higher than a preset threshold, the adjustment strategy is determined to be the first adjustment strategy, which increases the weight of the reconstruction loss while decreasing the weight of the alignment loss and the independent regularization term. If the alignment loss in the current round is higher than the preset threshold, the adjustment strategy is determined to be the second adjustment strategy, which increases the weight of the alignment loss while decreasing the weight of the reconstruction loss and the independent regularization term. If the independent regularization term in the current round is higher than the preset threshold, the adjustment strategy is determined to be the third adjustment strategy, which increases the weight of the independent regularization term while decreasing the weight of the reconstruction loss and the alignment loss.

[0091] Specifically, after each round of training, it is determined whether the various types of losses exceed the preset threshold. For losses that exceed the threshold, their weights are increased and the weights of other losses are decreased, generating a new target loss to achieve adaptive optimization that compensates for the deficiencies.

[0092] S105. Train the initial classification model using the multimodal dataset and target loss to obtain the target multimodal classification model.

[0093] Specifically, the gradient descent optimization algorithm can be used to calculate the gradient of the model parameters based on the target loss, and iteratively update the parameters until the model converges (the target loss tends to stabilize), thus obtaining a target multimodal classification model with strong generalization ability and high accuracy.

[0094] In this embodiment, the target loss is used as the optimization objective. By iteratively updating the parameters of the initial classification model, the model gradually masters the ability to extract and classify cross-modal causal features, and finally obtains a target model with stable performance.

[0095] In the above scheme, the acquired multimodal dataset of original image data and corresponding original text data provides basic samples with complete modality and strong semantic association for training, solving the problem of one-sided information in single-modal data; the innovative multi-level hybrid intervention of data-layer directed mutation (intervening in image generation) + feature-layer causal decoupling overcomes the limitations of traditional data augmentation random low-level transformations that cannot destroy spurious associations and the indirectness of causal intervention methods, separating cross-modal causal invariant features from superficially variable features; the reconstruction loss is obtained through intra-modal + Cross-modal error constraints ensure that intervention and feature processing do not lose core semantics, preventing the model from deviating from the essence of the task. Alignment loss strengthens cross-modal causal semantic consistency by comparing positive and negative samples through invariant subspace mapping, solving the semantic bias problem in multimodal training. Dynamic weight target loss design based on training stage, scenario requirements, and real-time loss feedback achieves a balance between decoupling, semantics, and alignment objectives, avoiding the one-sidedness of single weight optimization. The target model trained iteratively based on the target loss not only significantly improves the generalization robustness of out-of-distribution scenarios due to the focus on causal features, but also makes the decision-making process more interpretable due to clear feature decoupling. At the same time, improved cross-modal synergy reduces the risk of misjudgment, completely solving the core pain points of traditional multimodal models such as weak generalization, easy loss of semantics, and poor cross-modal synergy.

[0096] This application provides another method for training a multimodal classification model. The method is described in detail below, along with the execution flow of the multimodal classification model training method. (Refer to...) Figure 2 The diagram shown illustrates a training method for a multimodal classification model provided in this application. Before performing multi-level hybrid intervention processing on the original multimodal data to obtain multimodal intervention features, the specific steps include the following: S201. Perform semantic annotation on the original text data corresponding to the original image data in the original multimodal data.

[0097] Specifically, semantic role labeling technology is used to analyze the grammatical structure and semantic relationships of the original text, and to label the semantic components related to variable factor features (such as object attributes, scene conditions, expression methods, etc.), so that key information in the text can be extracted and classified.

[0098] For example, a pre-trained semantic role labeling model is selected to adapt to the semantic characteristics of texts in multiple domains (industrial quality inspection, autonomous driving, medical diagnosis). For the original text, core semantic roles are labeled, including but not limited to components related to surface features such as "subject attributes (e.g., material, color), adverbs (e.g., shooting angle, ambient light, time), and manner complements (e.g., production process)." Scenario examples: Original text in an industrial quality inspection scenario: "black plastic shell, side view, injection molding process, no scratches," labeled as: subject attributes (black plastic, shell), shooting angle (side view), production process (injection molding process), core state (no scratches, non-variable factor, labeled as an exclusion item); Original text in an autonomous driving scenario: "urban road, rainy day, daytime, no obstruction," labeled as: road type (urban road), weather condition (rainy day), time period (daytime), obstruction status (no obstruction). This forms structured data of "text fragments + semantic role labels," providing clear filtering criteria for subsequent phrase extraction.

[0099] In this embodiment, semantic structuring processing is performed on the original text data to provide a semantic classification basis for subsequent phrase extraction, clustering, and lexicon construction. The core purpose is to identify the semantic components in the text that describe the features of variable factors, ensuring that the lexicon focuses on the variation dimensions required for intervention.

[0100] S202. Extract phrases from the semantically annotated original text data.

[0101] Specifically, based on the semantic role labeling results of S201, phrases describing variable factor features are selected (excluding core causal feature phrases). The phrase types include noun phrases (attribute combinations) and adverbial phrases (scene / condition descriptions), ensuring that the extracted phrases have the characteristics of being replaceable and mutable.

[0102] For example, only semantic components labeled as "subject attribute, adverbial, manner complement" and related to surface variable features are retained, while non-mutable causal feature phrases such as "core state, category label" (e.g., "no scratches" "lesions") are excluded; noun phrases (attribute combinations): "black plastic" and "shell" are extracted from "black plastic shell", and "aluminum alloy material" is extracted from "aluminum alloy material"; adverbial phrases (scene / condition): "side shot" is extracted from "side shot", and "rainy day" and "daytime" are extracted from "rainy day, daytime"; extraction is performed using natural language processing tools combined with SRL annotation results, supporting batch processing of all original text in multimodal datasets; output results: forming a "list of variable feature phrases corresponding to a single text", for example: original text "black plastic shell, side shot, injection molding process", extracted phrase list: [black plastic, shell, side shot, injection molding process].

[0103] In this embodiment, core semantic unit phrases are extracted from the structured annotation results. These phrases need to correspond to variable factor features to provide basic materials for subsequent clustering and lexicon construction.

[0104] S203. Cluster the extracted phrases according to semantic annotation to form an initial dynamic variable factor lexicon.

[0105] Specifically, based on the semantic role labeling of S201, a clustering algorithm is used to group phrases with similar semantics and belonging to the same variable dimension into one category (forming category buckets). Each category bucket corresponds to a surface variable factor type, which ultimately constitutes the initial dynamic variable factor lexicon of category buckets + phrases.

[0106] In this embodiment, the extracted phrases are semantically categorized and organized to form a structured initial lexicon. The core purpose is to divide the phrases into category buckets according to the variable factor type, so as to provide a categorized replacement basis for the generation of subsequent intervention instructions.

[0107] S204. Expand the semantic space of the initial dynamic variable factor lexicon using a large language model to generate an expanded lexicon.

[0108] Specifically, using the category buckets and phrases of the initial dynamic variable factor lexicon as context, targeted prompt words are input into the large language model to guide it to generate novel phrases that conform to the semantic style of the corresponding category bucket and do not appear in the initial lexicon, ensuring that the extended phrases still belong to the category of surface variable factor features.

[0109] In this embodiment, the semantic generation capabilities of a large language model are used to expand the phrase diversity of the initial lexicon. The core objective is to generate novel phrases that exceed the experience of existing datasets, providing more creative variation dimensions for targeted interventions.

[0110] S205. Merge the expanded lexicon with the initial dynamic variable factor lexicon to form a variable factor lexicon.

[0111] Specifically, the initial and expanded lexicons are deduplicated and categorized to ensure that phrases within the same category are not repeated and are semantically consistent, ultimately forming a structured lexicon that covers known variants and unknown variants, supporting the implementation of semantic-aware targeted intervention.

[0112] In this embodiment, existing experience phrases (initial lexicon) and innovative variant phrases (extended lexicon) are integrated to form a complete and diverse variable factor lexicon, providing a comprehensive source of replacement phrases for the generation of subsequent targeted intervention instructions.

[0113] In the above scheme, S201 semantic annotation identifies the semantic components of variable factor features in the text and excludes causal core features, laying a targeted foundation for the lexicon to focus on the required variation dimensions for intervention; S202 extracts corresponding phrases to ensure that the lexicon materials are all replaceable and mutable surface feature units, avoiding interference from invalid information; S203 clustering classifies phrases into structured category buckets according to variable factor types, enabling the lexicon to have efficient classification retrieval and providing clear replacement basis for subsequent intervention instruction generation; S204 uses a large language model to generate novel phrases that exceed the experience of existing datasets, breaking through the limitations of traditional lexicons that rely on existing data, and giving intervention more creative variation dimensions; S205 merging and deduplication integrates known experience phrases and innovative variation phrases to form a complete variable factor lexicon that combines semantic consistency, diversity, and structure. The final lexicon not only supports the generation of targeted intervention instructions, ensuring that subsequent multi-level hybrid interventions only affect surface features and do not damage the core structure, but also provides the model with richer causal variation samples due to the innovation and diversity of phrases, improving the model's generalization robustness in out-of-distribution scenarios. At the same time, the structured design greatly improves the efficiency and accuracy of intervention instruction generation, solving the core pain points of ambiguous variation direction and limited materials in traditional interventions.

[0114] This application provides a data type prediction method based on a multimodal classification model. The method is described in detail below, in conjunction with the execution flow of the multimodal classification model training method. (Refer to...) Figure 3 The diagram shown is a flowchart illustrating a data type prediction method based on a multimodal classification model provided in this application. The specific steps include the following: S301. Obtain the image data to be processed.

[0115] Specifically, image data adapted to multimodal classification scenarios should be acquired. This data should reflect the key visual information of the target object (such as the main body shape, surface features, etc.) and be compatible with the image modalities of the multimodal dataset used for training in terms of data format and information dimension, so as to provide visual input support for subsequent prediction based on the causal features learned by the model.

[0116] For example, the image data to be processed can be target image data in specific scenarios such as industrial quality inspection, autonomous driving, and medical diagnosis: in industrial quality inspection scenarios, it is the appearance image of the product to be inspected, such as a high-definition real shot of the aluminum alloy frame of a mobile phone; in autonomous driving scenarios, it is real-time road condition images collected by vehicle cameras, such as road condition maps of urban roads in rainy weather; in medical diagnosis scenarios, it is the medical image data of patients, etc.

[0117] S302. Using a target multimodal classification model, predict the type of image data to be processed.

[0118] The target multimodal classification model is obtained by training the initial classification model using a multimodal dataset and a target loss function. The target loss function is used to calculate the reconstruction loss, alignment loss, and independence regularization term, and the reconstruction loss, alignment loss, and independence regularization term are fused to obtain the loss between the labels in the dataset and the model inference results. That is, using a multimodal dataset containing image data and corresponding text data as training samples, the target loss function, which fuses the reconstruction loss, alignment loss, and independence regularization term, constrains the initial classification model to learn the decoupling of causal invariant features and surface variable features shared across modalities, thereby achieving synergistic optimization of the model's generalization ability and accuracy.

[0119] The independence regularization term is used to constrain the independence of invariant and variable factor features decoupled from the original multimodal data of the multimodal dataset. In other words, as a key component of the objective loss function, the independence regularization term is used to constrain the statistical independence of cross-modal shared causal features (i.e., invariant factor features) and surface interference features (i.e., variable factor features) decoupled from the original image and text data of the multimodal dataset, thereby severing the false association between the two and forcing the model to focus on essential causal information.

[0120] Reconstruction loss is used to characterize the difference between the original multimodal data in a multimodal dataset and the reconstructed multimodal data after reconstructing the original multimodal data. In other words, as a component of the objective loss function, reconstruction loss measures the semantic difference between the reconstructed image and text data and the original multimodal data after intervention processing of the original multimodal data in the multimodal dataset. This ensures that the intervention process does not destroy the core causal semantics and avoids the model learning from deviating from the essence of the task.

[0121] Alignment loss is used to characterize the semantic consistency of multimodal intervention features obtained from the original multimodal data in an invariant subspace. In other words, alignment loss, as a component of the objective loss function, measures the degree of semantic matching of the multimodal intervention features obtained after performing multi-level hybrid interventions on the original multimodal dataset in a shared invariant subspace, ensuring that cross-modal intervention features still maintain the core semantic association with the original data.

[0122] Specifically, the model, trained and optimized using a multimodal dataset and a target loss function, is invoked to extract features and perform classification reasoning on the input data, outputting the category to which the data belongs, thus determining the target object.

[0123] For example, in industrial quality inspection scenarios, the model predicts the quality of the product image based on the learned causal features of defects (such as the geometric shape of scratches) and outputs pass / fail labels and specific defect types (such as scratches, dents, and cracks); in autonomous driving scenarios, the model outputs road condition element types (such as pedestrians, vehicles, and no-entry signs) based on causal features of road conditions (such as the outline of traffic signs and the limb structure of pedestrians); in medical diagnosis scenarios, the model outputs the lesion type (such as benign nodules and malignant tumors) of the image data based on causal features of lesions (such as the size and edge shape of nodules).

[0124] In the above scheme, the objective loss function is designed to integrate reconstruction loss, alignment loss, and independent regularization terms that constrain the independence of causal invariant features and surface variable features. The objective multimodal classification model is trained based on a multimodal dataset, which effectively enables the model to discard spurious associations in the training data and focus on essential causal features shared across modalities. This not only improves the model's generalization robustness and prediction accuracy in out-of-distribution scenarios where the distribution of test data and training data is inconsistent, avoiding performance degradation caused by reliance on statistical associations or indirect intervention, but also makes the model's decision-making process more interpretable due to the explicit causal feature constraints. This provides core technical support for reliable deployment in key areas such as industrial quality inspection, autonomous driving, and medical diagnosis, and effectively solves the core pain point of weak generalization of existing multimodal classification models in real-world applications.

[0125] Figure 4 This is a schematic diagram of the structure of a training system for a multimodal classification model provided in an embodiment of this application. Figure 4 As shown, the training system 400 for the multimodal classification model includes: a multi-level hybrid intervention module 401, a cross-modal reconstruction module 402, and an invariant alignment learning module 403.

[0126] Specifically, using a multimodal dataset as input, the multi-level hybrid intervention module 401 is divided into a data layer intervention module 4011 and a feature layer intervention module 4012. The data layer intervention module 4011 generates intervention images that retain the core structure and incorporate surface variations. The feature layer intervention module 4012 separates causal invariant features from surface variable features and exchanges them across samples to cut off false associations. This not only overcomes the random inefficiency of traditional data augmentation but also provides the model with causal variation features. The subsequent cross-modal reconstruction module 402 calculates the reconstruction loss to ensure that the intervention does not lose the core semantics and avoids the model deviating from the essence of the task. The invariant alignment learning module 403 strengthens the semantic consistency of cross-modal causal features by mapping invariant subspaces and comparing positive and negative samples, solving the semantic bias problem in multimodal training. Finally, the model trained based on this process outputs prediction results, which not only significantly improves the generalization robustness in out-of-distribution scenarios but also makes the decision more interpretable due to the decoupling of causal features. This completely solves the core pain points of traditional multimodal models, such as weak generalization, semantic variability, and poor cross-modal collaboration.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0128] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps of any of the above-described training methods for a multimodal classification model or prediction methods for data types based on a multimodal classification model.

[0129] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute, at runtime, the steps of the training method of any of the above-described multimodal classification models or the prediction method of the data type based on the multimodal classification model.

[0130] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0131] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any of the above-described multimodal classification model training methods or data type prediction methods based on multimodal classification models.

[0132] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the training method for any of the above-described multimodal classification models or the prediction method for data types based on the multimodal classification model.

[0133] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0134] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0135] The training method for a multimodal classification model provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A training method for a multimodal classification model, characterized in that, The method includes: Acquire a multimodal dataset; the multimodal dataset includes raw image data and raw text data belonging to the same acquisition scene as the raw image data; The original multimodal data in the multimodal dataset is subjected to multi-level hybrid intervention processing to obtain multimodal intervention features, which are a cross-modal feature set containing invariant factor features and variable factor features; Based on the aforementioned multimodal intervention features, the reconstruction loss and alignment loss are determined; The reconstruction loss, the alignment loss, and the independent regularization term are fused to obtain the target loss; the target loss is used to characterize the comprehensive optimization objective of the initial classification model. The initial classification model is trained using the multimodal dataset and the target loss to obtain the target multimodal classification model; The step of fusing the reconstruction loss, the alignment loss, and the independent regularization term to obtain the target loss includes: Determine the training phase of the initial classification model and the scenario requirements corresponding to the multimodal dataset; Based on the training phase and / or the scenario requirements, the weights of the reconstruction loss, the alignment loss, and the independent regularization term are determined. The independent regularization term is used to constrain the independence of invariant factor features and variable factor features decoupled from the original multimodal data of the multimodal dataset. Based on the weights, the reconstruction loss, the alignment loss, and the independent regularization term are weighted and summed to obtain the target loss.

2. The method according to claim 1, characterized in that, The original multimodal data includes at least original image data and original text data. The step of performing multi-level hybrid intervention processing on the original multimodal data in the multimodal dataset to obtain multimodal intervention features includes: The structural data of the original image data and the original text data corresponding to the original image data are determined from the original multimodal data. The structural data is used to characterize the foreground and background separation information and object geometry of the original image data. Based on the original text data and the variable factor lexicon, an intervention instruction is generated, which is a text instruction to modify the surface features of the original image data; Based on the intervention command and the structural data, intervention image data and corresponding intervention text data are determined. The intervention image data is used to characterize image data that retains the structure of the original image data and incorporates variations of the intervention command. The original image data, the intervention image data, the original text data, and the intervention text data are subjected to mixed intervention processing to obtain the multimodal intervention features.

3. The method according to claim 2, characterized in that, The step of generating intervention instructions based on the original text data and the variable factor lexicon includes: Extract phrases from the original text data to obtain the current phrase list; Determine at least one phrase from the current phrase list; Based on the semantic annotation of the phrase, a replacement phrase is determined from the variable factor lexicon; The replaced phrases are recombined into sentences to generate the intervention instructions.

4. The method according to claim 2, characterized in that, The process of determining intervention image data based on the intervention instructions and the structural data includes: The intervention instruction is encoded as a textual conditional feature, which is used to characterize the semantic embedding vector of the intervention instruction; The structural data is encoded into structural condition features, which are used to characterize the geometric structure embedding vector state of the original image data. Based on the textual conditional features and the structural conditional features, the noise latent variables of the original image data are denoised and reconstructed to obtain the intervention image data. The noise latent variables are used to characterize the image features containing noise interference in the latent space.

5. The method according to claim 2, characterized in that, The process of performing mixed intervention processing on the original image data, the intervention image data, the original text data, and the intervention text data to obtain the multimodal intervention features includes: Multimodal features are extracted from the original image data, the intervention image data, the original text data, and the intervention text data. These multimodal features are used to characterize the original semantic embedding of the original multimodal data. The multimodal features are decoupled to determine invariant factor features and variable factor features. The invariant factor features are used to characterize causal features shared across modalities, and the variable factor features are used to characterize surface interference features. A feature exchange operation is performed on the invariant factor features and the variable factor features to generate the multimodal intervention features. The feature exchange is used to characterize the causal intervention operation that severs the original spurious association between the invariant factor features and the variable factor features.

6. The method according to claim 1, characterized in that, The determination of reconstruction loss based on the multimodal intervention features includes: Based on the aforementioned multimodal intervention features, reconstructed image data and reconstructed text data are generated; Determine the image reconstruction loss between the reconstructed image data and the original image data, the text reconstruction loss between the reconstructed text data and the original text data, and determine the cross-modal loss between the reconstructed image data and the reconstructed text data; The image reconstruction loss, the text reconstruction loss, and the cross-modal loss are weighted to obtain the reconstruction loss, which is used to characterize the difference between the original multimodal data in the multimodal dataset and the reconstructed multimodal data after reconstructing the original multimodal data.

7. The method according to claim 1, characterized in that, The process of determining the alignment loss based on the multimodal intervention features includes: The multimodal intervention features are mapped to an invariant subspace to obtain mapped features, which are used to characterize the embedding vector of the multimodal intervention features in the causal space. Based on the mapping features, feature pairs are determined. The feature pairs are used to characterize the same semantic positive sample feature pairs and non-matching negative sample feature pairs. The positive sample feature pairs are the image mapping features and text mapping features corresponding to the same original image data and the same original text data. The negative sample feature pairs are the image mapping features and text mapping features corresponding to different original image data and original text data. Based on the feature pairs, the alignment loss is determined, which is used to characterize the semantic consistency of the multimodal intervention features obtained based on the original multimodal data in the invariant subspace.

8. The method according to claim 1, characterized in that, Before performing multi-level hybrid intervention processing on the original multimodal data to obtain multimodal intervention features, the method further includes: Semantic annotation is performed on the original text data corresponding to the original image data in the original multimodal data; Extract phrases from the semantically annotated raw text data; The extracted phrases are clustered according to semantic annotation to form an initial dynamic variable factor lexicon; The initial dynamic variable factor lexicon is semantically expanded using a large language model to generate an expanded lexicon. The expanded lexicon is merged with the initial dynamic variable factor lexicon to form a variable factor lexicon.

9. A method for predicting data types based on a multimodal classification model, the method comprising: Acquire the image data to be processed; The image data to be processed is predicted using a target multimodal classification model to obtain the type of the image data, wherein the target multimodal classification model is obtained using the training method of the multimodal classification model according to any one of claims 1-8.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the training method for the multimodal classification model as described in any one of claims 1-8 or the prediction method for data types based on the multimodal classification model as described in claim 9.

Images