Multimodal interpretable classification method, system, device, and storage medium

Abstract

This invention belongs to the field of artificial intelligence technology and relates to a multimodal interpretable classification method, which includes: S1: acquiring multimodal samples; S2: generating pseudo-labels for key text evidence and key image regions; S3: obtaining token-level text feature representations and patch-level image feature representations; S4: obtaining the distribution of key text evidence and key image regions; S5: performing bidirectional key evidence alignment; S6: obtaining a global representation and obtaining classification probabilities based on the global representation; S7: performing multi-objective joint optimization to obtain a multimodal information classification model; S8: performing multimodal interpretable classification based on the multimodal information classification model. While ensuring classification accuracy, it also possesses high evidence extraction capabilities, strong modal collaboration capabilities, good output interpretability, and strong adaptability to complex scenarios, providing a new technical path for building a highly reliable, interpretable, and scalable disaster multimodal intelligent analysis system.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a multimodal interpretable classification method, system, device and storage medium, especially a multimodal interpretable classification method, system, device and storage medium based on large model weak supervision and bidirectional key evidence alignment. Background Technology

[0002] With the widespread deployment of social media platforms, emergency management systems, disaster awareness terminals, drone inspection platforms, and multi-source disaster information collection systems, a large amount of multimodal data containing both textual descriptions and on-site images is typically generated in a short period after a disaster event. For example, in disaster scenarios such as floods, earthquakes, fires, typhoons, and landslides, social media users, on-site rescue personnel, news media, and monitoring equipment continuously release multimodal information including textual descriptions of the disaster, images of the disaster site, records of rescue operations, information on infrastructure damage, and information on people trapped. How to automatically understand, quickly classify, and effectively filter this multimodal disaster data has become a key technical issue in emergency response, disaster assessment, resource allocation, public opinion monitoring, and decision support.

[0003] Compared to traditional disaster identification methods that rely solely on text or scene recognition methods that rely solely on images, multimodal disaster information classification technology can simultaneously utilize the explicit semantic expression capabilities of text and the intuitive visual evidence in images, thereby more comprehensively identifying disaster categories, disaster states, damaged objects, and rescue stages. For example, text can directly describe semantic information such as "bridge collapse," "road flooding," and "rescue personnel distributing supplies," while images can further provide intuitive evidence such as the disaster scope, damage level, key target locations, and on-site environment. Therefore, multimodal disaster information classification methods demonstrate greater application potential than traditional single-modal methods in complex scene understanding, fine-grained disaster identification, and highly reliable decision support.

[0004] However, disaster multimodal information classification technology still faces a core contradiction in practical application: on the one hand, the system needs to fully utilize the complementary information of text and images to improve the accuracy and stability of disaster category recognition; on the other hand, text descriptions in disaster scenarios often suffer from incomplete expressions, colloquialisms, high noise levels, and semantic ambiguity, while image information may have problems such as complex backgrounds, target occlusion, unclear regional boundaries, and strong interference from irrelevant content. Although existing methods generally adopt a joint text-image modeling strategy, most are still at the coarse-grained modeling stage of "extracting text features, extracting image features, and classifying after fusion," lacking systematic modeling for questions such as "which text content is the key evidence supporting the classification," "which image regions are the key regions supporting the classification," and "whether the key evidence in text and images is consistent, complementary, or mutually corroborative." As the requirements for model interpretability, robustness, and high reliability in emergency management and disaster analysis scenarios increase simultaneously, the limitations of the traditional training framework "optimization based on the final classification result" are becoming increasingly apparent, making it difficult to meet the comprehensive requirements of accuracy, stability, and interpretability for future disaster multimodal intelligent analysis systems.

[0005] 1. General limitations of disaster multimodal classification frameworks trained on whole-sample classification labels.

[0006] Most current mainstream disaster multimodal information classification methods are based on the "text-image-category label" triple. During the training phase, each disaster sample is usually treated as an independent supervised unit, that is, directly minimizing the error between the model's predicted category and the true category, while rarely explicitly considering the fine-grained differences between textual and image evidence within a sample. For traditional multimodal classification models, image-text joint recognition models, and most disaster social media analysis frameworks, the system usually only retains the overall category label during the data organization phase, such as "damaged infrastructure," "disaster victims," ​​"rescue operations," "damaged buildings," or "post-disaster environment," etc. Then, during training, the text is input into the text encoder and the image is input into the visual encoder, and the final classification is performed through feature concatenation, attention fusion, or joint representation learning.

[0007] For example, one type of method encodes disaster text using word vectors, recurrent neural networks, or pre-trained language models, and images using convolutional neural networks or visual transformers, then concatenates the two modal representations before inputting them into a classifier; another type of method enhances the information interaction between text and images through co-attention or cross-modal attention mechanisms, and then outputs the disaster category based on the fused representation; yet another type of method introduces image-text matching or multimodal alignment modules to improve the consistency of text and image features in a unified semantic space.

[0008] Although this type of training method has a clear implementation path and a relatively simple structure, its limitations become increasingly apparent as the complexity of disaster scenarios and application requirements increase. The main limitations are: (1) It lacks fine-grained supervision capabilities for key evidence. The model can usually only learn "which category this sample belongs to", but it is difficult to learn "why it belongs to this category"; (2) The model is prone to relying on surface patterns, background bias, or local high-frequency features in the training data, rather than disaster-related evidence that truly supports classification; (3) Although text and images are input into the model together, they are not explicitly constrained to make collaborative judgments around unified key evidence; (4) It is difficult to ensure reliable evidence extraction, transparent classification basis, and balanced modality utilization in addition to classification accuracy.

[0009] Therefore, the framework for independent training of whole sample category labels is essentially a "result-oriented coarse-grained supervision mechanism," which is difficult to achieve unified modeling of key evidence in disaster texts and key regions in images, and also difficult to support the interpretable classification requirements in high-reliability disaster analysis scenarios.

[0010] 2. A single-layer target optimization scheme with final classification accuracy as the core.

[0011] In existing research on disaster multimodal classification, most methods still use the final category prediction accuracy, minimization of cross-entropy loss, or minimization of other output error functions as the main or even sole optimization objective. For example, minimizing the classification cross-entropy loss can improve the model's ability to distinguish disaster categories, or end-to-end training of the classification head can be performed using a joint loss method.

[0012] These methods are highly comparable in public dataset evaluations and offline experiments, but they reveal significant problems in real-world deployment scenarios. Typical problems include: (1) focusing only on the final classification result and ignoring the intermediate evidence extraction process: even if the model classifies correctly, it may not actually capture the correct text keywords or key image regions; (2) focusing only on the accuracy of the result and ignoring the consistency of evidence: in the same sample, the text end and the image end may rely on different or even contradictory information to complete the classification; (3) difficulty in supporting interpretable disaster analysis: the system outputs category labels, but cannot explain which words or regions support the judgment; (4) lack of explicit constraints on the stability of classification basis: when the text expression changes, the image background changes, or the modal information is incomplete, the model output is prone to instability; (5) lack of collaborative optimization of the evidence layer, alignment layer, and decision layer: existing methods rarely model the distribution of text evidence, the distribution of image regions, the correspondence between modal evidence, and the final classification decision based on evidence at the same time.

[0013] Therefore, although such methods may perform well in category prediction metrics, they often fail to achieve a balance between "correct classification, clear basis, consistency between text and graphics, and stable process" in system-level applications.

[0014] 3. Improved schemes based on one-way critical clue migration or shallow interpretable enhancement.

[0015] In recent years, with the development of multimodal learning and interpretable artificial intelligence, some studies have begun to explore ways to enhance the interpretability of disaster multimodal classification models, such as keyword extraction, salient region detection, text attention-guided image focus, or image-text heatmap visualization. These approaches typically do not change the overall classification task definition, but rather add a layer of "interpretation generation" or "key region hints" mechanism during training or inference.

[0016] For example, some solutions first extract significant keywords, key phrases, or disaster entities from disaster texts, and then use these textual signals to guide the image to focus on the corresponding areas; in some image-text classification frameworks, the system uses attention scores, saliency maps, or gradient heatmaps to show the text and image locations that the model focuses on during the classification process; in other solutions, auxiliary tasks are added to the text and region attention modules are added to the image to improve the model's sensitivity to local disaster evidence.

[0017] Such solutions can improve the interpretability of the model to some extent, especially in certain tasks, which can help the system to intuitively show "where the model seems to be looking and what it is focusing on". However, such solutions still have obvious limitations: (1) The improvement focus is mostly on the interpretation and display level, rather than establishing a truly trainable and constrained key evidence supervision mechanism; (2) Text interpretation is relatively easy to extract, but key regions in the image often lack stable and low-cost supervision sources; (3) Most solutions are still mainly based on the one-way transfer of text to guide the image, and lack the ability of the image to correct the text's focus; (4) Even if some significant words and significant regions can be output, it may not be possible to guarantee that this evidence truly supports the final classification decision; (5) It is more inclined to "explain the model results" rather than solve the problem of "how to build a unified training mechanism around key evidence".

[0018] Therefore, although such solutions are closer in form to the idea of ​​"key evidence extraction and utilization" in this invention, they are essentially still technical routes of "locally interpretable enhancement" or "one-way prompting improvement", and cannot achieve true dual-modal key evidence collaborative learning and bidirectional alignment optimization.

[0019] 4. A text-based key evidence-guided image region selection scheme for multimodal interpretable classification.

[0020] In the fields of disaster multimodal classification and interpretable multimodal learning, there are also some technical approaches that are closer to the ideas of this invention. These approaches typically attempt to improve the interpretability and stability of the model in disaster classification through key evidence learning, attention guidance, text region alignment, image-text matching constraints, or multimodal interpretation generation.

[0021] For example, in some multimodal interpretable classification schemes, the system first manually or semi-automatically annotates key evidence in the text and trains a key text evidence extraction module. Then, using the similarity between key text evidence and image regions, cross-modal attention, or optimal transfer relationships, the key evidence information in the text is mapped to image patches or candidate regions, thereby generating a key region heatmap in the image. Finally, classification is performed based on the extracted key text evidence and key image regions. These schemes typically assume that disaster semantics in text are more direct and easier to annotate; therefore, they regard key text evidence as a key source of interpretation and establish a multimodal interpretation foundation through unidirectional text-to-image transfer.

[0022] Such solutions are somewhat similar to this invention because they both involve the idea of ​​"enhancing multimodal classification capabilities through learning from key evidence" and both focus on fine-grained associations between text and images and improving model interpretability. However, there are still significant gaps compared with the present invention: (1) The source of key evidence supervision is usually biased towards the text end, and the key areas of the image end lack direct, independent and scalable weak supervision sources; (2) The direction of key evidence transfer is usually unidirectional, that is, the text guides the image, and there is a lack of reverse correction and bidirectional constraint mechanism between the image and the text; (3) More attention is paid to "how to find the image area based on the text" rather than "how to make the text evidence and the image evidence mutually verify and jointly optimize"; (4) The key areas of the image often rely on the indirect generation of text transfer. Once the text description is incomplete, vague or has errors, the key areas of the image end are easily skewed; (5) Although evidence-level interpretation is introduced, the "large model weak supervision pseudo-label generation - key evidence prediction - bidirectional evidence alignment - key evidence-driven classification" is usually not constructed into a complete and unified training framework; (6) More emphasis is placed on enhancing local interpretability on the existing interpretation framework, and the low-cost acquisition of bimodal key evidence and bidirectional collaborative modeling are not included as core optimization objects in the unified system.

[0023] Therefore, although such schemes are similar in form to the present invention in terms of "key evidence learning" or "cross-modal evidence mapping", their technical essence is still mainly at the level of "text-dominated unidirectional interpretable classification enhancement", and has not yet solved the core problem that the present invention focuses on: "how to use large models to generate bimodal key evidence supervision at low cost and establish a two-way alignment mechanism between text and image key evidence".

[0024] 5. Common problems of existing solutions.

[0025] In summary, the current disaster multimodal information classification system still suffers from the following three fundamental shortcomings: (1) The supervision granularity is coarse, and there is a lack of low-cost unified modeling capability for key text evidence and key image regions. Existing methods usually rely only on whole sample class labels for training. Even if an interpretation mechanism is introduced, it is mostly limited to the text or local saliency display, making it difficult to build a dual-modal key evidence supervision system of text and image at low cost and on a large scale.

[0026] (2) The collaborative mechanism is insufficient, and it is impossible to achieve two-way mutual verification and consistency optimization of textual evidence and image evidence. Existing solutions usually perform coarse-grained fusion of text and images, or adopt a one-way migration method of text guiding images. It is difficult to correct textual concerns based on strong image evidence, and it is also difficult to ensure stable correspondence and unified expression of the two modalities at the level of key evidence.

[0027] (3) The optimization objective is singular, making it difficult to achieve synergistic improvement in key evidence extraction, modality alignment, and classification decision-making. Current systems generally focus on optimizing the final classification result, lacking joint modeling of the key evidence extraction process, the cross-modal key evidence alignment process, and the classification process based on key evidence. As the complexity of the scenario increases and the data distribution changes, the interpretability, robustness, and generalization ability of the model cannot be continuously enhanced.

[0028] This invention is proposed against this background, aiming to construct a disaster multimodal interpretable information classification method based on large model weak supervision and bidirectional key evidence alignment. Summary of the Invention

[0029] To address the problems existing in current technologies, this invention proposes a multimodal interpretable classification method based on large-scale model weak supervision and bidirectional key evidence alignment. It constructs a large-scale model weak supervision key evidence pseudo-label generation mechanism for disaster text and images, establishes separate distributions of key evidence in text and key regions in images, constructs bidirectional key evidence projection and alignment mechanisms from text to image and from image to text, and establishes a multimodal classification decision mechanism based on weighted representation of key evidence. The model is jointly optimized through classification loss, key evidence supervision loss, and bidirectional alignment loss. This improves classification accuracy while enhancing the model's stable extraction capability of key disaster information, cross-modal consistent modeling capability, and interpretability of classification criteria. It achieves a paradigm upgrade from "coarse-grained image-text fusion classification relying on whole-sample labels" to "integrated training based on large-scale model weak supervision, bidirectional key evidence collaborative modeling, and key evidence-driven decision-making." This enables the disaster multimodal classification model to simultaneously possess high classification accuracy, strong modal complementarity utilization capability, and good interpretability.

[0030] To achieve the above objectives, the present invention provides the following technical solution: A multimodal interpretable classification method, characterized by comprising the following steps: S1: Obtain multimodal samples, each of which includes a set of text, images, and category labels; S2: Use a visual language model to generate pseudo-labels for key text evidence and pseudo-labels for key image regions for the multimodal samples respectively; S3: Encode the text and image of the multimodal samples respectively to obtain token-level text feature representation and patch-level image feature representation; S4: Based on the token-level text feature representation and the patch-level image feature representation, the distribution of key text evidence and the distribution of key image regions are obtained respectively; S5: Based on the token-level text feature representation and patch-level image feature representation, as well as the distribution of key text evidence and the distribution of key image regions, perform bidirectional key evidence alignment, so that the text and image achieve consistency at the key evidence level; S6: Construct a text weighted representation based on the token-level text feature representation and the distribution of key text evidence; construct an image weighted representation based on the patch-level image feature representation and the distribution of key image regions; fuse the text weighted representation and the image weighted representation to obtain a global representation; and use a visual language model to obtain the classification probability based on the global representation. S7: Multi-objective joint optimization of the visual language model is performed based on classification loss, text key evidence supervision loss, image key region supervision loss, global key evidence representation alignment loss and sparsity constraints to obtain a multimodal information classification model. S8: Perform multimodal interpretable classification based on the multimodal information classification model to obtain and output the classification results.

[0031] Preferably, step S2 specifically includes: S21: Input the text and image of the multimodal sample and the target category definition into the visual language model, and prompt the visual language model to output a set of pseudo-labels for key textual evidence that support the target category determination. :

[0032] In the formula, Indicates the first in the text Each token represents the confidence level of key textual evidence. It is the total number of tokens in the text; S22: Input the text and image of the multimodal sample and the target category definition into the visual language model, and prompt the visual language model to output a set of pseudo-labels for key image regions in the image that support target category determination. :

[0033] In the formula, Indicates the first in the image The confidence level that an image patch belongs to a key region of the image. It is the total number of patches in the image; S23: Based on the text pseudo-label retention threshold, analyze the set of pseudo-labels for the key text evidence. Confidence filtering is performed and the set of pseudo-labels for key regions of the image is retained based on the image pseudo-label retention threshold. Confidence filtering is performed to obtain the final pseudo-labels for key textual evidence. and image key region pseudo-labels .

[0034] Preferably, step S4 specifically includes: S41: Score each of the token-level text feature representations to obtain the distribution of key text evidence:

[0035] In the formula, It is the first in the text Distribution of key textual evidence for each token; It is the first in the text Token-level textual feature representation of each token; For the Sigmoid function; and These are learnable parameters; S42: Based on the distribution of each textual key evidence, obtain the textual key evidence vector of the text. :

[0036] In the formula, It is the total number of tokens in the text; S43: Score each patch-level image feature representation to obtain the distribution of key image regions:

[0037] In the formula, It is the first in the image Distribution of key regions in the image of each patch; It is the first in the image Patch-level image feature representation of each patch; For the Sigmoid function; and These are learnable parameters; S44: Based on the distribution of each of the image's key regions, obtain the image key region vector of the image. :

[0038] In the formula, It is the total number of patches in the image.

[0039] Preferably, step S5 specifically includes: S51: Based on the token-level text feature representation and patch-level image feature representation, obtain the attention weight of the text token to the image patch and the attention weight of the image patch to the text token:

[0040]

[0041]

[0042] In the formula, It is the first in the text The token is related to the first token in the image. Relevance scores between patches It is the first in the text Token-level textual feature representation of each token It is the first in the image Patch-level image feature representation of each patch It refers to the dimensions of the token-level text feature representation and the patch-level image feature representation. It is the total number of tokens in the text. It is the total number of patches in the image. It is the first in the text The token is added to the image. Attention weights for each patch It is to make the first part of the text Each token is associated with all of the images in the image. The correlation scores between patches are summed exponentially. It is the first in the image The nth patch is the first one in the text. Attention weight of each token It is to take the first one in the image Each patch and all of the text The correlation scores among the tokens are summed exponentially. S52: Based on the distribution of key text evidence and the attention weights of text tokens on image patches, obtain the importance score of each patch in the image inferred from the text perspective:

[0043] In the formula, The first image inferred from the textual perspective Importance score of each patch It is the first in the text Distribution of key textual evidence for each token; S53: Based on the distribution of key regions in the image and the attention weights of image patches on text tokens, obtain the importance score of each token in the text inferred from the image perspective:

[0044] In the formula, The first in the text is inferred from the image visually. The importance score of each token It is the first in the image Distribution of key regions in the image of each patch; S54: Perform bidirectional key evidence alignment based on the importance scores of each patch in the image inferred from the text perspective and the importance scores of each token in the text inferred from the image perspective:

[0045]

[0046]

[0047] In the formula, It is the text-to-image alignment loss. It is the image-to-text alignment loss. It is a loss of alignment of key evidence in both directions. It is the key region vector of the image. It is the key evidence vector of the text.

[0048] Preferably, step S6 specifically includes: S61: Construct a weighted text representation based on the token-level text feature representation and the distribution of key text evidence:

[0049] In the formula, It is a weighted representation of text. It is the first in the text Distribution of key textual evidence for each token It is the first in the text Token-level textual feature representation of each token It is the total number of tokens in the text; S62: Construct a weighted image representation based on the patch-level image feature representation and the distribution of key image regions:

[0050] In the formula, It is a weighted representation of the image. It is the first in the image Distribution of key regions in the image of each patch It is the first in the image Patch-level image feature representation of each patch It is the total number of patches in the image; S63: The weighted text representation and the weighted image representation are fused using a gated fusion method to obtain a global representation:

[0051]

[0052] In the formula, It is the gating factor. and It consists of a learnable weight matrix and learnable bias terms. For the Sigmoid function, Weighted representation of text Image weighted representation Concatenate along the feature dimension For global representation, ⊙ indicates element-wise multiplication; S64: Obtain the classification probability based on the global representation using the visual language model:

[0053] In the formula, For classification probability, and It consists of a learnable weight matrix and learnable bias terms. yes Activation function.

[0054] Preferably, step S7 specifically includes: S71: Constructing the classification loss :

[0055] In the formula, For the category label of the first One real category, For the predicted classification probability, The total number of actual categories in the category labels; S72: Constructing a Loss Monitoring System for Key Textual Evidence :

[0056] In the formula, It is the textual key evidence vector of the text. It is a pseudo-label of the key textual evidence in the text. It is a binary cross-entropy loss; S73: Constructing a supervision loss for key image regions :

[0057] In the formula, It is the key region vector of the image. These are pseudo-labels for key image regions of the image. It is a binary cross-entropy loss; S74: Constructing a global key evidence representation aligned with the loss :

[0058] In the formula, It is a weighted representation of text. It is a weighted representation of the image; S75: Constructing Sparsity Constraints and :

[0059]

[0060] In the formula, It is the first in the text Distribution of key textual evidence for each token It is the total number of tokens in the text. It is the first in the image Distribution of key regions in the image of each patch It is the total number of patches in the image; S76: Total Construction Loss :

[0061] In the formula, These are the weighting coefficients for each loss term; S77: Multi-objective joint optimization With the total loss mentioned above As a loss, the visual language model is subjected to multi-objective joint optimization to obtain a multimodal information classification model.

[0062] Preferably, the classification result in step S8 includes the classification category, key textual evidence, and key image regions.

[0063] Furthermore, the present invention also provides a multimodal interpretable classification system, characterized in that it includes: A multimodal sample acquisition module is used to acquire multimodal samples, each of which includes text, images, and a set of category labels; The pseudo-label generation module is used to generate pseudo-labels for key text evidence and pseudo-labels for key image regions from the multimodal samples using a visual language model. The feature representation acquisition module is used to encode the text and image of the multimodal samples respectively to obtain token-level text feature representation and patch-level image feature representation; The key distribution acquisition module is used to obtain the distribution of key text evidence and the distribution of key image regions based on the token-level text feature representation and the patch-level image feature representation, respectively. A two-way key evidence alignment module is used to perform two-way key evidence alignment based on the token-level text feature representation and patch-level image feature representation, as well as the text key evidence distribution and image key region distribution, so that the text and image are consistent at the key evidence level. The classification probability acquisition module is used to construct a text weighted representation based on the token-level text feature representation and the distribution of key text evidence, construct an image weighted representation based on the patch-level image feature representation and the distribution of key image regions, fuse the text weighted representation and the image weighted representation to obtain a global representation, and use a visual language model to obtain the classification probability based on the global representation; The model training module is used to perform multi-objective joint optimization of the visual language model based on classification loss, text key evidence supervision loss, image key region supervision loss, global key evidence representation alignment loss and sparsity constraints, so as to obtain a multimodal information classification model. The classification result output module is used to perform multimodal interpretable classification based on the multimodal information classification model to obtain and output the classification results.

[0064] Furthermore, the present invention also provides a multimodal interpretable classification device, characterized in that it includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal interpretable classification method as described above. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the multimodal interpretable classification method as described above.

[0065] Compared with existing technologies, the multimodal interpretable classification method, system, device, and storage medium of the present invention based on large model weak supervision and bidirectional key evidence alignment has one or more of the following beneficial technical effects: (1) The training method has been upgraded from coarse-grained training that relies solely on the overall category label to dual-modal key evidence collaborative learning based on weak supervision of a large model, which significantly improves the model's ability to extract disaster discrimination criteria and its training scalability.

[0066] Traditional disaster multimodal classification methods typically use only text, images, and category labels as training inputs, with the classification results of the entire sample serving as the primary supervision target. While this training approach can improve the model's category recognition ability to some extent, the lack of explicit supervision over key textual evidence and key image regions means that the model often only learns "which category a sample belongs to," but struggles to consistently learn "which textual words and which image regions truly support that category judgment." This can easily lead to problems such as relying on background bias, local noise, or superficial statistical patterns to complete the classification. Furthermore, even when traditional methods attempt to introduce manual annotation of key evidence, they often face practical limitations such as high text annotation costs, even greater difficulty in image region annotation, and the inability to support continuous construction of large-scale samples.

[0067] This invention constructs a large-model, weakly supervised key evidence generation mechanism for disaster multimodal samples. The large model generates text token-level key evidence pseudo-labels and image patch-level key region pseudo-labels, providing a fine-grained, low-cost, and scalable bimodal supervision foundation for disaster multimodal classification tasks. This mechanism enables the system to learn not only "classification labels" but also "classification criteria," achieving an upgrade from "supervising only the results" to "jointly supervising textual evidence, image evidence, and classification results."

[0068] By employing this mechanism, this invention significantly reduces the cost of manual annotation of traditional key evidence and improves the efficiency of model training data construction. Furthermore, it enables the model to more accurately identify disaster keywords and phrases in text and key disaster-affected areas, damaged objects, and rescue areas in images, thereby enhancing the model's ability to learn from real disaster discrimination criteria. Therefore, this invention upgrades disaster multimodal classification training from "coarse-grained learning oriented towards whole-sample labels" to "fine-grained collaborative learning oriented towards dual-modal key evidence," significantly improving the model's discrimination ability, training efficiency, and application expansion potential.

[0069] (2) The shift from text-driven one-way key evidence transfer to bidirectional alignment optimization of text and image key evidence significantly improves multimodal collaborative discrimination ability and classification stability.

[0070] While some existing interpretable disaster multimodal classification schemes have attempted to leverage key textual information to guide image region selection or improve image-text fusion capabilities through attention mechanisms, they generally still rely on one-way transfer and shallow fusion. Particularly in key evidence modeling, existing schemes often assume that the semantic representation of disasters in the text is easier to obtain and annotate, thus employing a one-way mechanism of "textual key evidence guiding key image regions." While this approach has some merit, if the text contains incomplete descriptions, ambiguous expressions, semantic gaps, or noise interference, key image regions can easily be misguided, affecting the final classification results. Furthermore, when images contain stronger and more intuitive disaster evidence that the text fails to adequately represent, traditional methods struggle to use image-level information to correct the textual focus.

[0071] This invention constructs a bidirectional key evidence projection and alignment mechanism from text to image and from image to text, transforming the one-way dependency between key text evidence and key image regions into a collaborative relationship of mutual verification, constraint, and complementarity. During training, the system not only infers the regions of interest in the image based on key text evidence but also uses key image regions to correct key words and phrases in the text that should be emphasized. This upgrades the system from a one-way transfer model where "text is the teacher and image is the student" to a bidirectional collaborative model where "text and image mutually check and optimize each other."

[0072] This mechanism can maintain a more stable correspondence between key textual and image evidence across various complex disaster samples. For example, in samples where the textual description is weak but the image disaster signal is strong, the system can improve the accuracy of the textual focus by leveraging strong image evidence. When the image background is complex or contains interfering information, the system can constrain the selection of image regions using explicit semantics in the text. Thus, this invention significantly enhances the complementary utilization of text and images, reduces the impact of single-modal bias on overall classification, and strengthens the model's classification stability and robustness in complex, modally asymmetric, and noisy sample scenarios.

[0073] (3) The optimization and upgrading of the single-layer output with only the final classification result as the core is to the collaborative optimization of key evidence supervision, evidence alignment and key evidence-driven decision-making, so as to realize the refinement and full-link of the disaster multimodal classification mechanism.

[0074] Existing disaster multimodal classification methods typically focus on optimizing the final category prediction result, emphasizing whether the model "classifies correctly," while paying less attention to intermediate processes such as "what information the model used to complete the classification," "whether this information comes from valid key evidence in the text and images," and "whether the evidence from the two modalities is consistent and can jointly support the classification result." Especially in disaster scenarios, even if the model outputs the correct category, it does not mean that it has truly focused on the correct text keywords and key image regions, thus making it difficult to guarantee that the system has stable evidence utilization capabilities and reliable classification interpretability.

[0075] This invention introduces key evidence pseudo-label supervision, local key evidence bidirectional alignment, global key evidence representation consistency constraints, and a classification decision mechanism based on key evidence weighted representation during the training process. By unifying the modeling of the relationship between text evidence extraction, image evidence extraction, cross-modal key evidence correspondence, and the final classification output, the system can achieve comprehensive coordination among "correct evidence, consistent evidence, and reliable decision" at the overall level.

[0076] Specifically, this invention not only constrains the disaster category output by the model to be consistent with the true category, but also further constrains the model to extract keywords and phrases highly related to the disaster category from the text and to extract disaster-affected areas, damaged objects, and rescue areas highly related to the disaster semantics from the image. It also requires strong consistency between the two modalities in the distribution of local key evidence and the semantic representation of global key evidence. Furthermore, the final classification is not based on a coarse-grained fusion of all original inputs, but rather on a multimodal representation after key evidence screening and weighting, thus tightly coupling the classification decision-making process with the key evidence extraction process.

[0077] With this mechanism, the present invention achieves an integrated optimization upgrade from "optimizing only the final category output" to "key evidence supervision - two-way alignment - key evidence-driven classification", which enables disaster multimodal classification training to have stronger refinement capabilities, interpretation reliability and process traceability.

[0078] (4) The system has been upgraded from locally interpretable enhancement to a key evidence-driven classification closed loop that is trainable, constrainable, and scalable, which significantly improves interpretability and system credibility in complex disaster scenarios.

[0079] Traditional interpretability improvement schemes in multimodal classification mostly generate saliency maps, heatmaps, or attention visualizations after inference, essentially resembling "result display" rather than "training constraints." These schemes often only indicate which regions or words the model seems to focus on, but cannot guarantee that these points of focus actually constitute the classification basis, nor can they guarantee the interpretability stability of the model across different scenarios and samples. Therefore, interpretability in existing technologies usually remains at the surface level, failing to meet the requirements of disaster assessment, emergency decision-making, and highly reliable auxiliary systems where "explanations must be credible and the basis must be traceable."

[0080] This invention constructs a complete closed loop from key evidence generation, key evidence learning, key evidence mutual verification to key evidence-driven output by unifying the key evidence-based classification decision-making process into a large-scale, weakly supervised key evidence pseudo-labeling, bidirectional key evidence alignment, and key evidence-based classification framework. This closed loop enables the system to not only output classification results during the inference stage, but also simultaneously output which words in the text are key evidence, which regions in the image are key evidence, and how these evidences collectively support the final disaster category determination.

[0081] This mechanism significantly improves the transparency, interpretability, and verifiability of the model output, enabling the system to provide more convincing classification criteria for emergency responders, analysts, and decision-making systems in complex disaster scenarios. Especially in high-risk, high-reliability applications, this invention effectively reduces problems such as "correct results but unclear evidence," "interpretation drift," and "inconsistencies between textual and graphical evidence," thereby improving the overall credibility and engineering feasibility of the system.

[0082] (5) The static coarse-grained fusion classification has been upgraded to a continuous robust classification mechanism for complex disaster scenarios, which significantly enhances the model’s generalization ability and multi-scenario adaptability.

[0083] Traditional disaster multimodal classification systems, after training, typically only exhibit good adaptability to limited data distributions, limited representation methods, and limited scene complexity. Once faced with new disaster representations, image data of different styles, samples with missing modal information, or samples with enhanced noise, the coarse-grained image-text fusion capability formed by the original training of the model often fails to maintain its optimal performance, easily leading to problems such as good judgment for some categories, significant dependency bias for some categories, and unstable interpretation in certain scenarios.

[0084] This invention constructs a large-model weakly supervised key evidence generation mechanism, a bimodal key evidence prediction mechanism, and a bidirectional key evidence alignment mechanism. This enables the system to continuously compare the differences and consistency between textual key evidence and key image regions during training, and to jointly optimize model parameters based on classification error and evidence alignment error. Specifically, the system can gradually enhance its ability to capture real disaster evidence during training, while reducing its dependence on surface patterns, background bias, and noise features, thereby forming a more stable disaster semantic discrimination capability.

[0085] By leveraging this mechanism, this invention overcomes the static limitation of traditional schemes that "only learn the fusion results, not the evidence structure," enabling the disaster multimodal classification system to form a collaborative closed loop of "pseudo-label generation—key evidence learning—bidirectional evidence alignment—evidence-driven classification." This closed loop allows the model to continuously enhance its adaptability to complex disaster scenarios, asymmetric image-text samples, and cross-scenario transfer tasks as the sample size increases, disaster types expand, and application requirements improve. This achieves a system upgrade from "simply improving classification accuracy" to "emphasizing a balance between improving classification accuracy, evidence consistency, and scenario robustness."

[0086] Through the above innovations, this invention achieves a holistic technological upgrade across four levels: supervision methods, modal collaboration mechanisms, optimization target design, and classification decision mechanisms. The training structure has shifted from coarse-grained training relying solely on whole-sample category labels to joint learning of dual-modal key evidence based on weak supervision of a large model. Modal relationships have evolved from traditional unidirectional transfer or shallow fusion to bidirectional alignment and mutual verification of key text and image evidence. The training objective has shifted from traditional single-classification accuracy optimization to a comprehensive balance of key evidence supervision, evidence consistency, and final classification decision. The classification logic has transformed from coarse-grained text-image fusion decision-making to key evidence-driven decision-making. With these improvements, this invention enables the disaster multimodal information classification model to maintain classification accuracy while possessing high evidence extraction capabilities, strong modal collaboration capabilities, good output interpretability, and strong adaptability to complex scenarios. This provides a new technical path for building a highly reliable, interpretable, and scalable intelligent multimodal disaster analysis system. Attached Figure Description

[0087] Figure 1 This is a flowchart of the multimodal interpretable classification method of the present invention.

[0088] Figure 2 This is a schematic diagram of the multimodal interpretable classification system of the present invention.

[0089] Figure 3 This is a structural block diagram of the multimodal interpretable classification device of the present invention. Detailed Implementation

[0090] Before detailing any embodiment of the invention, it should be understood that the invention, in its application, is not limited to the details of the construction and arrangement of the components set forth in the following description or illustrated in the following figures. The invention can have other embodiments and can be practiced or carried out in various ways. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered limiting. The use of “comprising” or “having” and variations thereof in this invention is intended to cover the items set forth below and their equivalents, as well as any additional items. Unless otherwise specified or limited, the terms “installation,” “connection,” “support,” and “linkage,” and variations thereof are used broadly and cover both direct and indirect installation, connection, support, and linking. Moreover, “connection” and “linkage” are not limited to physical or mechanical connections or links. Furthermore, firstly, in the disclosure of this invention, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting this invention. Secondly, the term "a" should be understood as "at least one" or "one or more," that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple. The term "a" should not be construed as a limitation on the quantity.

[0091] To address the problems existing in current multimodal information classification schemes, this invention proposes a multimodal interpretable classification method, system, device, and storage medium based on large-model weak supervision and bidirectional key evidence alignment. It constructs a text-based key evidence pseudo-label generation mechanism, an image key region pseudo-label generation mechanism, a text-image bidirectional key evidence alignment mechanism, and a classification decision mechanism based on weighted representation of key evidence for disaster multimodal samples. This enables the model to learn not only "correct classification" during training, but also "which content in the text is key evidence," "which regions in the image are key evidence," "key evidence in the two modalities should correspond and complement each other," and "the final classification result should be based on the consistency of key evidence." This achieves an upgrade from "result-oriented coarse-grained fusion classification" to "interpretable multimodal classification based on weak supervision and bidirectional alignment of key evidence."

[0092] To this end, the present invention introduces three key technological innovations in its system: (1) Construction of a key evidence generation mechanism for large models with weak supervision for disaster multimodal samples.

[0093] This invention addresses the joint classification task of disaster text and images by introducing a large-scale model with weak supervision. It generates pseudo-labels for key evidence at the text token level and key regions at the image patch level. For the same disaster sample, both the text description and the on-site image utilize the semantic understanding and visual-language association capabilities of the large-scale model to automatically determine which words, phrases, and image regions are more likely to support the current disaster category judgment, and establishes fine-grained importance supervision signals. By constructing a dual-modal key evidence pseudo-label system, it overcomes the limitations of traditional methods that "only have category labels and lack key evidence supervision" and "it is difficult to obtain key image regions at low cost."

[0094] (2) Text-image collaborative optimization mechanism based on bidirectional key evidence alignment.

[0095] This invention introduces a bidirectional key evidence projection and alignment mechanism from text to image and from image to text, jointly modeling the distribution of key evidence in text and the distribution of key regions in images. This mechanism no longer relies solely on text to guide images, but simultaneously utilizes high-value disaster evidence in images to conversely constrain the focus of text, enabling a more stable correspondence and synergistic relationship between the two modalities at the key evidence level. By constraining the consistency of local key evidence distribution and global key evidence semantic representation, it achieves an upgrade from "one-way transfer" to "two-way mutual verification."

[0096] (3) A key evidence-driven classification optimization mechanism aimed at synergistic improvement of accuracy, robustness and interpretability.

[0097] This invention no longer focuses solely on minimizing the final category prediction error. Instead, it incorporates textual key evidence supervision, image key region supervision, bidirectional key evidence alignment, and key evidence-based classification decisions into a unified optimization framework. By jointly modeling textual evidence, image evidence, evidence alignment relationships, and the final classification result, the system can improve the accuracy of disaster category identification while reducing the model's sensitivity to modal noise, surface bias, and locally irrelevant features. Furthermore, it significantly enhances the interpretability, stability, and cross-scenario generalization ability of the model output.

[0098] Through the above technical solution, this invention upgrades the disaster multimodal information classification mechanism from "optimization based solely on category labels and coarse-grained image-text fusion results" to "integrated optimization based on large-model weak supervision, bidirectional key evidence alignment, and key evidence-driven decision-making." This method can establish a unified key evidence supervision and collaborative constraint mechanism for text descriptions and image content within the same sample. This enables the model to maintain more accurate evidence extraction capabilities, more stable cross-modal correspondence capabilities, and more reliable interpretive output capabilities during disaster category judgment. This reduces the model's dependence on single-modal bias and superficial statistical patterns, improving its robustness, stability, and interpretability in complex disaster scenarios, diverse social media expressions, and highly reliable emergency decision-making tasks.

[0099] Before detailing the specific embodiments of the present invention, let's briefly introduce some technical terms used in the present invention so that those skilled in the art can better understand the present invention.

[0100] (1) Large-scale weak supervision: This refers to the method of automatically generating fine-grained supervision signals for the text and image content in disaster multimodal samples by using a large model with strong text understanding, visual understanding, or joint visual-language understanding capabilities. In this invention, large-scale weak supervision is mainly used to generate pseudo-labels for key evidence at the text token level and pseudo-labels for key regions at the image patch level. Its core is to provide trainable key evidence learning targets for subsequent disaster multimodal classification models with low manual annotation costs.

[0101] (2) Pseudo-labels: These are approximate supervised labels that are not directly obtained through precise manual annotation, but rather through large model inference, rule generation, semi-automatic annotation, or other weakly supervised methods. In this invention, pseudo-labels mainly include pseudo-labels for key text evidence and pseudo-labels for key image regions, used to represent the degree of support of each token in the text or each patch in the image for the current disaster category judgment. Pseudo-labels are usually represented by importance scores between 0 and 1, and in some implementations, they can be further discretized into binary labels based on a threshold.

[0102] (3) Key evidence: refers to the discriminative information that best supports the model's classification conclusion in the current disaster category determination task. For text modality, key evidence usually manifests as keywords, phrases, or local semantic fragments; for image modality, key evidence usually manifests as key regions, local targets, damaged areas, rescue areas, or visual local content with disaster semantics. The core characteristic of key evidence is that it is not a simple collection of all information in the sample, but rather a subset of information that has the most explanatory and discriminative value relative to the current classification target.

[0103] (4) Key textual evidence: refers to words, phrases, or semantic fragments in disaster text that can directly support the current category judgment. For example, in texts such as "bridge collapsed," "road flooded," and "rescue workers evacuating people," the contents of "bridge," "collapsed," "flooded," "rescue workers," and "evacuating people" may all constitute key textual evidence. In this invention, key textual evidence is usually represented in the form of token-level importance distribution, used to characterize the strength of support for the current disaster category at different locations in the text.

[0104] (5) Key Image Regions: These refer to local visual regions in a disaster image that can directly support the current category determination. For example, in a flood image, areas of floodwater accumulation, collapsed bridges, trapped people, and rescue boats may all constitute key image regions. In this invention, key image regions are typically represented by patch-level or local region-level importance distributions to characterize the strength of support for the current disaster category at different locations in the image.

[0105] (6) Key Evidence Pseudo-Label Generation: This refers to the process of automatically generating corresponding key evidence supervision signals by performing semantic analysis on text and image samples using a large model. For the text side, this process will output the importance score of each token in the text; for the image side, this process will output the importance score of each patch or local region in the image. This mechanism is used to provide fine-grained weak supervision targets for model training, enabling the system to learn "which information should be regarded as the key basis for disaster classification".

[0106] (7) Key Evidence Distribution Prediction: This refers to the process by which the model to be trained predicts the importance of each token or patch based on text encoding features or image encoding features and forms a distribution. This distribution reflects which text locations and image regions the model currently considers more likely to constitute key evidence. In this invention, key evidence distribution prediction differs from pseudo-label generation; the former is the learning output of the target model, while the latter is a weakly supervised objective provided during the training phase.

[0107] (8) Two-way key evidence alignment: This refers to establishing, during model training, both the projection relationship from textual key evidence to key image regions and the reverse projection relationship from key image regions to textual key evidence. Consistency constraints are used to ensure that the key evidence from both modalities mutually validates, constrains, and maintains semantic consistency. The core of this mechanism is that it no longer relies solely on text to guide the image, but instead enables text and image to form a two-way mutual verification relationship at the key evidence level, thereby enhancing multimodal collaborative discrimination capabilities.

[0108] (9) Text-to-Image Key Evidence Projection: This refers to the process of mapping the importance of key evidence in a text to the image space based on the distribution of key evidence in the text and the cross-modal similarity between text tokens and image patches. This process is used to infer "which areas in the image should be given priority based on the key information in the text", thereby establishing the guiding role of text in the selection of image regions.

[0109] (10) Image-to-text key evidence projection: This refers to the process of mapping the importance of key regions in an image back to the text space based on the distribution of key regions in each patch of the image and the cross-modal similarity relationship between image patches and text tokens. This process is used to infer "which words in the text should be given further attention based on the key information in the image", thereby establishing the reverse correction effect of the image on the focus of the text.

[0110] (11) Local key evidence alignment: This refers to the process of imposing consistency constraints on the distribution of key text evidence and key image regions at the token-patch or local region level. This alignment focuses on whether a reasonable, stable, and fine-grained correspondence has been established between specific words in the text and specific local regions in the image. The purpose of local key evidence alignment is to enable the model to learn at the micro level "which word corresponds to which region" and "which region reflects which text evidence".

[0111] (12) Global key evidence representation: refers to the overall textual evidence representation obtained by weighted aggregation of text features using the distribution of key textual evidence, and the overall image evidence representation obtained by weighted aggregation of image features using the distribution of key image regions. Global key evidence representation reflects the overall disaster semantic evidence extracted by the model from the text and image ends in the current sample, and is an important intermediate representation connecting local evidence extraction and final classification decision.

[0112] (13) Alignment of global key evidence representations: This refers to the mechanism of imposing similarity constraints on the global key evidence representations on the text side and the global key evidence representations on the image side during model training. This mechanism is used to ensure that the overall disaster semantic evidence extracted from the text and the overall disaster semantic evidence extracted from the image are consistent in the semantic space, thereby achieving consistency from local evidence to overall semantic consistency.

[0113] (14) Cross-modal similarity relationship: refers to the association between text token representation and image patch representation based on vector similarity, attention weight, optimal transmission relationship or other matching mechanisms. This relationship is used to characterize whether a word in the text corresponds semantically to a local region in the image, and is the basis for realizing bidirectional projection and alignment of key text-image evidence.

[0114] (15) Critical Evidence-Driven Classification: This refers to a classification method where the model does not directly classify based on all original text features and image features, but first extracts key text evidence and key image regions, and then fuses and determines the category based on the weighted representation of the key evidence. The core of this mechanism is to make the final classification result directly based on the extracted key evidence, thereby enhancing the transparency, interpretability and traceability of the classification process.

[0115] (16) Key Evidence Supervision Loss: This refers to the loss term used to constrain the distribution of key text evidence and key image regions predicted by the model to approximate the distribution of pseudo-labels generated by the large model. This loss is the direct supervision basis for key evidence learning, enabling the model to learn step by step from the training samples which words and regions should be considered important evidence.

[0116] (17) Bidirectional alignment loss: This refers to the loss term used to constrain the distribution of key text evidence to be consistent with the distribution of text importance obtained by back-projection of the image, and to constrain the distribution of key image regions to be consistent with the distribution of image importance obtained by text projection. The role of this loss is to make the two modalities form a more stable correspondence at the level of key evidence, and reduce the error propagation problem caused by one-way migration.

[0117] (18) Weighted representation of key evidence: This refers to the feature representation obtained by weighting and summing, weighted pooling, or other weighted aggregation of text token representation or image patch representation using the distribution of key evidence. This representation can highlight the information with the most discriminative value for the current disaster category and suppress information that is irrelevant to the classification or has a weak relationship with it. It is an important intermediate representation for realizing key evidence-driven classification in this invention.

[0118] (19) Disaster Multimodal Information Classification: This refers to the task of classifying samples that simultaneously contain disaster text and disaster images. The categories may include, but are not limited to, disaster-related categories such as infrastructure damage, trapped personnel, rescue operations, post-disaster environment, material needs, road interruption, and building damage. This invention particularly emphasizes that in the process of disaster multimodal classification, not only should the category results be output, but also the key textual evidence and key image regions supporting the results should be output simultaneously.

[0119] (20) Explainable classification output: This refers to the system's ability to provide classification results, including which words or phrases in the text constitute key evidence and which areas in the image constitute key regions, in addition to outputting disaster category results. This output method makes the classification results no longer just isolated labels, but includes specific, verifiable, and traceable evidence, thereby enhancing the system's credibility in disaster analysis and decision support scenarios.

[0120] (21) Training Feedback and Evidence Enhancement: This refers to the process by which the system continuously monitors the prediction error, bidirectional alignment error, and classification error of key evidence during training, and dynamically adjusts the pseudo-label quality control strategy, loss weight configuration, key evidence screening threshold, or alignment intensity setting based on training feedback. This mechanism enables the model to continuously improve its key evidence extraction ability, modal collaboration ability, and classification robustness as it trains, thereby forming a continuously optimized disaster multimodal classification training closed loop.

[0121] Figure 1 A flowchart of the multimodal interpretable classification method of the present invention is shown. Figure 1 As shown, the multimodal interpretable classification method of the present invention includes the following steps: S1: Multimodal sample acquisition.

[0122] In this invention, the first step is to collect multimodal samples from social media in disaster scenarios. Each multimodal sample includes text, images, and a set of category labels. The text sequence obtained after word segmentation is denoted as... The image sequence obtained after image segmentation is denoted as The set of category labels is denoted as .

[0123] Then the text sequence for:

[0124] In the formula, Represents the text sequence The Each text token (word element); For the text sequence The total number of text tokens in the dataset.

[0125] The image is segmented to obtain an image sequence. for:

[0126] In the formula, Represents the image sequence The One image patch (image block); The image sequence The total number of image patches in the dataset.

[0127] Category tag collection It can be represented as:

[0128] In the formula, Represents the first in the set of label categories One real category; This represents the total number of preset real categories.

[0129] S2: Pseudo-tag generation.

[0130] Using a visual language model, pseudo-labels for key textual evidence and key image regions are generated for each of the multimodal samples, specifically including: 1. Generation of pseudo-labels for key textual evidence.

[0131] The visual language model is input with the text and image of the multimodal sample, as well as the target category definition. Using prompt templates, the visual language model outputs a set of keywords or phrases in the text that support the target category determination; that is, a set of pseudo-labels for key textual evidence. :

[0132] In the formula, Indicates the first in the text Each token represents the confidence level of key textual evidence; the more crucial the token, the higher its confidence level. The larger; It is the total number of tokens in the text.

[0133] Therefore, large models can be used to generate text-level key evidence supervision signals at low cost, reducing the cost of manual annotation.

[0134] 2. Generation of pseudo-labels for key regions of an image.

[0135] The visual language model is input with the text and image of the multimodal sample, as well as the target category definition. Using a prompt template, the visual language model outputs a set of pseudo-labels for key image regions in the image that support target category determination. In this invention, semantic descriptions can be mapped to patch-level pseudo-labels using a grounding model or region detection module. The set of pseudo-labels for key image regions is represented as follows:

[0136] In the formula, Indicates the first in the image The confidence level of an image patch belonging to a key region of the image; the more key the patch, the higher the confidence level. The larger; It is the total number of patches in the image.

[0137] This allows for direct weak supervision of the image, preventing key image regions from relying solely on one-way text migration.

[0138] 3. Filtering for fake labels.

[0139] Since pseudo-labels generated by visual language models may contain noise, a confidence-based filtering process is performed on the pseudo-label set. Let the text pseudo-label retention threshold be... The threshold for retaining image pseudo-labels is Then we have:

[0140]

[0141] After filtering, the final textual key evidence pseudo-labels are obtained. and image key region pseudo-labels .

[0142] S3: Feature representation acquisition.

[0143] The text and images of the multimodal samples are encoded respectively to obtain token-level text feature representations and patch-level image feature representations.

[0144] Specifically, a text encoder and an image encoder are used to extract text and image features respectively. The text encoder can employ BERT, RoBERTa, or a domain-adapted model, and the text features are represented as follows:

[0145] In the formula, It is the first in the text Token-level textual feature representation of each token.

[0146] The image encoder can use ViT or CLIP-ViT, and the image features are represented as follows:

[0147] In the formula, It is the first in the image Patch-level image feature representation of each patch.

[0148] This step maps text and images to a unified dimension. The semantic space provides a basic representation for subsequent key evidence extraction and two-way alignment.

[0149] S4: Key Distribution Acquisition.

[0150] Based on the token-level text feature representation and the patch-level image feature representation, the distribution of key text evidence and the distribution of key image regions are obtained, respectively, which specifically include: 1. Extraction of key evidence from text.

[0151] First, each token-level text feature representation is scored to obtain the distribution of key text evidence:

[0152] In the formula, It is the first in the text Distribution of key textual evidence for each token; It is the first in the text Token-level textual feature representation of each token; For the Sigmoid function; and These are learnable parameters.

[0153] Then, based on each of the textual key evidence distributions, the textual key evidence vector of the text is obtained. :

[0154] In the formula, It is the total number of tokens in the text.

[0155] By extracting key evidence from the text, we can determine which words or phrases in the text best support the classification conclusion. 2. Extraction of key regions in the image.

[0156] First, each patch-level image feature representation is scored to obtain the distribution of key regions in the image:

[0157] In the formula, It is the first in the image Distribution of key regions in the image of each patch; It is the first in the image Patch-level image feature representation of each patch; For the Sigmoid function; and These are learnable parameters.

[0158] Then, based on the distribution of each image key region, the image key region vector of the image is obtained. :

[0159] In the formula, It is the total number of patches in the image.

[0160] By extracting key regions from an image, it is possible to determine which regions in the image best support the classification conclusion.

[0161] S5: Two-way alignment of key evidence. Based on the token-level text feature representation and patch-level image feature representation, as well as the distribution of key text evidence and the distribution of key image regions, bidirectional key evidence alignment is performed to constrain the key text evidence and key image regions, thereby ensuring consistency between the text and the image at the key evidence level. Specifically, this includes: 1. Construct a cross-modal similarity matrix.

[0162] Based on the token-level text feature representation and patch-level image feature representation, the attention weight of the text token to the image patch and the attention weight of the image patch to the text token are obtained:

[0163]

[0164]

[0165] In the formula, It is the first in the text The token is related to the first token in the image. Relevance scores between patches It is the first in the text Token-level textual feature representation of each token It is the first in the image Patch-level image feature representation of each patch It refers to the dimensions of the token-level text feature representation and the patch-level image feature representation. It is the total number of tokens in the text. It is the total number of patches in the image. It is the first in the text The token is added to the image. Attention weights for each patch It is to make the first part of the text Each token is associated with all of the images in the image. The correlation scores between patches are summed exponentially. It is the first in the image The nth patch is the first one in the text. Attention weight of each token It is to take the first one in the image Each patch and all of the text The correlation scores between the tokens are summed exponentially.

[0166] By constructing a cross-modal similarity matrix, we can establish semantic correspondences between text words and image regions, providing a foundation for bidirectional projection.

[0167] 2. Text-to-Image Key Evidence Projection: Based on the distribution of key evidence in the text, predict the regions of interest in the image.

[0168] Based on the distribution of key text evidence and the attention weights of text tokens on image patches, the importance score of each patch in the image inferred from the text perspective is obtained:

[0169] In the formula, The first image inferred from the textual perspective Importance score of each patch It is the first in the text Distribution of key textual evidence for each token.

[0170] but

[0171] By projecting key evidence from text to image, we can use the key information explicitly expressed in the text to guide the model to find the corresponding region in the image.

[0172] 3. Image-to-text key evidence projection: Based on the distribution of key regions in an image, predict the words in the text that should be of particular interest.

[0173] Based on the distribution of key regions in the image and the attention weights of image patches on text tokens, the importance scores of each token in the text inferred from the image perspective are obtained as follows:

[0174] In the formula, The first in the text is inferred from the image visually. The importance score of each token It is the first in the image The key regions of the image are distributed across each patch.

[0175] but

[0176] By projecting key evidence from images to text, strong evidence in images can be used to correct textual concerns in reverse, avoiding one-sided bias in the text.

[0177] 4. Bidirectional alignment loss.

[0178] Bidirectional key evidence alignment is performed based on the importance scores of each patch in the image inferred from the text perspective and the importance scores of each token in the text inferred from the image perspective:

[0179]

[0180]

[0181] In the formula, It is the text-to-image alignment loss. It is the image-to-text alignment loss. It is a loss of alignment of key evidence in both directions. It is the key region vector of the image. It is the key evidence vector of the text.

[0182] By using bidirectional alignment loss, text and images can be made consistent at the level of key evidence, thereby improving the reliability of bimodal interpretation.

[0183] S6: Obtaining classification probability.

[0184] A text-weighted representation is constructed based on the token-level text feature representation and the distribution of key text evidence. An image-weighted representation is constructed based on the patch-level image feature representation and the distribution of key image regions. The text-weighted representation and the image-weighted representation are then fused to obtain a global representation. A visual language model is used to obtain the classification probability based on the global representation. Specifically, this includes: 1. Construct a weighted representation of the text.

[0185] Based on the token-level text feature representation and the distribution of key text evidence, a weighted text representation is constructed:

[0186] In the formula, It is a weighted representation of text. It is the first in the text Distribution of key textual evidence for each token It is the first in the text Token-level textual feature representation of each token It is the total number of tokens in the text.

[0187] 2. Construct a weighted representation of the image.

[0188] Based on the patch-level image feature representation and the distribution of key image regions, a weighted image representation is constructed:

[0189] In the formula, It is a weighted representation of the image. It is the first in the image Distribution of key regions in the image of each patch It is the first in the image Patch-level image feature representation of each patch It is the total number of patches in the image.

[0190] 3. Integration.

[0191] The weighted text representation and the weighted image representation are fused to obtain a global representation. In this invention, a gated fusion method can be used:

[0192]

[0193] In the formula, It is the gating factor. and It consists of a learnable weight matrix and learnable bias terms. For the Sigmoid function, Weighted representation of text Image weighted representation Concatenate along the feature dimension For global representation, ⊙ indicates element-wise multiplication.

[0194] 4. Output the classification probability.

[0195] The classification probability is obtained based on the global representation using a visual language model:

[0196] In the formula, For classification probability, and It consists of a learnable weight matrix and learnable bias terms. yes Activation function.

[0197] This step enables the model to classify data based on key, selected evidence rather than all the original information, thus improving interpretability.

[0198] S7: Model training.

[0199] In order for the model to learn classification, key text evidence extraction, key image region extraction, and bidirectional consistency simultaneously, multiple loss terms need to be jointly optimized.

[0200] Specifically, a multi-objective joint optimization of the visual language model is performed based on classification loss, text key evidence supervision loss, image key region supervision loss, global key evidence representation alignment loss, and sparsity constraints to obtain a multimodal information classification model, which specifically includes: 1. Construct the classification loss. :

[0201] In the formula, For the category label of the first One real category, For the predicted classification probability, This represents the total number of actual categories in the category labels.

[0202] The classification loss described above ensures that the model can ultimately classify accurately.

[0203] 2. Constructing a system for monitoring loss of key textual evidence. :

[0204] In the formula, It is the textual key evidence vector of the text. It is a pseudo-label of the key textual evidence in the text. It is a binary cross-entropy loss.

[0205] By using textual key evidence supervised loss, the model can learn pseudo-labels of textual key evidence that are close to those generated by large models.

[0206] 3. Construct a supervised loss for key regions of the image. :

[0207] In the formula, It is the key region vector of the image. These are pseudo-labels for key image regions of the image. It is a binary cross-entropy loss.

[0208] By using image key region supervision loss, the model can learn pseudo-labels for image key regions that are close to those generated by large models.

[0209] 4. Construct a global key evidence representation to align the loss. : To ensure semantic similarity between textual and image-based key evidence representations, cosine alignment loss can be used as the global key evidence representation alignment loss. :

[0210] In the formula, It is a weighted representation of text. It is a weighted representation of the image.

[0211] By using the global key evidence representation alignment loss, the textual key evidence representation and the image key evidence representation of the same sample can be kept consistent in overall semantics.

[0212] 5. Construct sparsity constraints and : To prevent the model from treating all tokens and patches as key evidence, a sparsity constraint is introduced:

[0213]

[0214] In the formula, It is the first in the text Distribution of key textual evidence for each token It is the total number of tokens in the text. It is the first in the image Distribution of key regions in the image of each patch It is the total number of patches in the image.

[0215] Through sparsity constraints and This encourages models to select only a small amount of truly crucial evidence, thereby improving the compactness of the explanation.

[0216] 6. Construct the total loss :

[0217] In the formula, These are the weighting coefficients for each loss term.

[0218] 7. Multi-objective joint optimization.

[0219] With the total loss mentioned above As a loss, the visual language model is subjected to multi-objective joint optimization to obtain a multimodal information classification model.

[0220] Through multi-objective joint optimization, multimodal information classification models can simultaneously possess classification capabilities and the ability to extract interpretable key evidence.

[0221] S8: Output of classification results Multimodal interpretable classification is performed based on the aforementioned multimodal information classification model to obtain and output the classification results.

[0222] After the model training is completed, new disaster texts and images are input into the trained multimodal information classification model. The multimodal information classification model then sequentially completes encoding, key evidence extraction, bidirectional alignment, and classification prediction, and finally outputs the classification result.

[0223] In this invention, the final classification result includes: classification category, text key evidence highlighting result, and image key region heatmap or highlight box.

[0224] Therefore, in this invention, the output of the multimodal information classification model not only includes the "conclusion" but also "why this conclusion was reached".

[0225] Compared with existing multimodal classification methods that rely solely on whole-sample category labels, interpretable methods that rely solely on one-way text-guided images, methods that only perform shallow image-text fusion, and methods that only interpret results through attention visualization, this invention solves the problems of traditional methods that are difficult to simultaneously achieve low-cost key evidence acquisition, collaborative optimization of text and image evidence, and transparent output of classification criteria by means of mechanisms such as large-model weak supervision, bimodal key evidence prediction, bidirectional key evidence alignment, and key evidence-driven classification. It realizes systematic interpretation enhancement capabilities and robust classification capabilities for disaster multimodal classification tasks, and significantly improves the usability, reliability, and engineering deployment value of disaster analysis systems in complex real-world scenarios.

[0226] Figure 2 A schematic diagram of the multimodal interpretable classification system of the present invention is shown. Figure 2 As shown, the multimodal interpretable classification system of the present invention includes: 1. Multimodal sample acquisition module.

[0227] The multimodal sample acquisition module is used to acquire multimodal samples, each of which includes text, images, and a set of category labels.

[0228] 2. Pseudo-tag generation module.

[0229] The pseudo-label generation module is used to generate pseudo-labels for key text evidence and key image regions from the multimodal samples using a visual language model.

[0230] 3. Feature representation acquisition module.

[0231] The feature representation acquisition module is used to encode the text and image of the multimodal samples respectively to obtain token-level text feature representation and patch-level image feature representation.

[0232] 4. Key Distribution Acquisition Module.

[0233] The key distribution acquisition module is used to obtain the distribution of key text evidence and the distribution of key image regions based on the token-level text feature representation and the patch-level image feature representation, respectively.

[0234] 5. Two-way key evidence alignment module.

[0235] The bidirectional key evidence alignment module is used to perform bidirectional key evidence alignment based on the token-level text feature representation and patch-level image feature representation, as well as the text key evidence distribution and image key region distribution, so that the text and image achieve consistency at the key evidence level.

[0236] 6. Classification probability acquisition module.

[0237] The classification probability acquisition module is used to construct a text weighted representation based on the token-level text feature representation and the distribution of key text evidence, construct an image weighted representation based on the patch-level image feature representation and the distribution of key image regions, fuse the text weighted representation and the image weighted representation to obtain a global representation, and use a visual language model to obtain the classification probability based on the global representation.

[0238] 7. Model training module.

[0239] The model training module is used to perform multi-objective joint optimization of the visual language model based on classification loss, text key evidence supervision loss, image key region supervision loss, global key evidence representation alignment loss and sparsity constraints, so as to obtain a multimodal information classification model.

[0240] 8. Classification result output module.

[0241] The classification result output module is used to perform multimodal interpretable classification based on the multimodal information classification model to obtain and output the classification results.

[0242] Furthermore, this invention also provides a multimodal interpretable classification device. For example... Figure 3 As shown, the multimodal interpretable classification device of the present invention includes: a memory 11 for storing one or more programs; one or more processors 12; when the one or more programs are executed by the one or more processors 12, the one or more processors 12 implement the multimodal interpretable classification method of the present invention. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal interpretable classification method of the present invention.

[0243] The computer-readable storage medium includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined in this invention, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0244] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0245] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this invention can be implemented in hardware, software modules executed by a processor, or a combination of both. The software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0246] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention based on the concept of the present invention, without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A multimodal interpretable classification method, characterized in that, Includes the following steps: S1: Obtain multimodal samples, each of which includes a set of text, images, and category labels; S2: Use a visual language model to generate pseudo-labels for key text evidence and pseudo-labels for key image regions for the multimodal samples respectively; S3: Encode the text and image of the multimodal samples respectively to obtain token-level text feature representation and patch-level image feature representation; S4: Based on the token-level text feature representation and the patch-level image feature representation, the distribution of key text evidence and the distribution of key image regions are obtained respectively; S5: Based on the token-level text feature representation and patch-level image feature representation, as well as the distribution of key text evidence and the distribution of key image regions, perform bidirectional key evidence alignment, so that the text and image achieve consistency at the key evidence level; S6: Construct a text weighted representation based on the token-level text feature representation and the distribution of key text evidence; construct an image weighted representation based on the patch-level image feature representation and the distribution of key image regions; fuse the text weighted representation and the image weighted representation to obtain a global representation; and use a visual language model to obtain the classification probability based on the global representation. S7: Multi-objective joint optimization of the visual language model is performed based on classification loss, text key evidence supervision loss, image key region supervision loss, global key evidence representation alignment loss and sparsity constraints to obtain a multimodal information classification model. S8: Perform multimodal interpretable classification based on the multimodal information classification model to obtain and output the classification results.

2. The multimodal interpretable classification method according to claim 1, characterized in that, Step S2 specifically includes: S21: Input the text and image of the multimodal sample and the target category definition into the visual language model, and prompt the visual language model to output a set of pseudo-labels for key textual evidence that support the target category determination. : In the formula, Indicates the first in the text Each token represents the confidence level of key textual evidence. It is the total number of tokens in the text; S22: Input the text and image of the multimodal sample and the target category definition into the visual language model, and prompt the visual language model to output a set of pseudo-labels for key image regions in the image that support target category determination. : In the formula, Indicates the first in the image The confidence level that an image patch belongs to a key region of the image. It is the total number of patches in the image; S23: Based on the text pseudo-label retention threshold, analyze the set of pseudo-labels for the key text evidence. Confidence filtering is performed and the set of pseudo-labels for key regions of the image is retained based on the image pseudo-label retention threshold. Confidence filtering is performed to obtain the final pseudo-labels for key textual evidence. and image key region pseudo-labels .

3. The multimodal interpretable classification method according to claim 1, characterized in that, Step S4 specifically includes: S41: Score each of the token-level text feature representations to obtain the distribution of key text evidence: In the formula, It is the first in the text Distribution of key textual evidence for each token; It is the first in the text Token-level textual feature representation of each token; For the Sigmoid function; and These are learnable parameters; S42: Based on the distribution of each textual key evidence, obtain the textual key evidence vector of the text. : In the formula, It is the total number of tokens in the text; S43: Score each patch-level image feature representation to obtain the distribution of key image regions: In the formula, It is the first in the image Distribution of key regions in the image of each patch; It is the first in the image Patch-level image feature representation of each patch; For the Sigmoid function; and These are learnable parameters; S44: Based on the distribution of each of the image's key regions, obtain the image key region vector of the image. : In the formula, It is the total number of patches in the image.

4. The multimodal interpretable classification method according to claim 1, characterized in that, Step S5 specifically includes: S51: Based on the token-level text feature representation and patch-level image feature representation, obtain the attention weight of the text token to the image patch and the attention weight of the image patch to the text token: In the formula, It is the first in the text The token is related to the first token in the image. Relevance scores between patches It is the first in the text Token-level textual feature representation of each token It is the first in the image Patch-level image feature representation of each patch It refers to the dimensions of the token-level text feature representation and the patch-level image feature representation. It is the total number of tokens in the text. It is the total number of patches in the image. It is the first in the text The token is added to the image. Attention weights for each patch It is to make the first part of the text Each token is associated with all of the images in the image. The correlation scores between patches are summed exponentially. It is the first in the image The nth patch is the first one in the text. Pay attention to the weight of each token. It is to take the first one in the image Each patch and all of the text The correlation scores among the tokens are summed exponentially. S52: Based on the distribution of key text evidence and the attention weights of text tokens on image patches, obtain the importance score of each patch in the image inferred from the text perspective: In the formula, The first image inferred from the textual perspective Importance score of each patch It is the first in the text Distribution of key textual evidence for each token; S53: Based on the distribution of key regions in the image and the attention weights of image patches on text tokens, obtain the importance score of each token in the text inferred from the image perspective: In the formula, The first in the text is inferred from the image visually. The importance score of each token It is the first in the image Distribution of key regions in the image of each patch; S54: Perform bidirectional key evidence alignment based on the importance scores of each patch in the image inferred from the text perspective and the importance scores of each token in the text inferred from the image perspective: In the formula, It is the text-to-image alignment loss. It is the image-to-text alignment loss. It is a loss of alignment of key evidence in both directions. It is the key region vector of the image. It is the key evidence vector of the text.

5. The multimodal interpretable classification method according to claim 1, characterized in that, Step S6 specifically includes: S61: Construct a weighted text representation based on the token-level text feature representation and the distribution of key text evidence: In the formula, It is a weighted representation of text. It is the first in the text Distribution of key textual evidence for each token It is the first in the text Token-level textual feature representation of each token It is the total number of tokens in the text; S62: Construct a weighted image representation based on the patch-level image feature representation and the distribution of key image regions: In the formula, It is a weighted representation of the image. It is the first in the image Distribution of key regions in the image of each patch It is the first in the image Patch-level image feature representation of each patch It is the total number of patches in the image; S63: The weighted text representation and the weighted image representation are fused using a gated fusion method to obtain a global representation: In the formula, It is the gating factor. and It consists of a learnable weight matrix and learnable bias terms. For the Sigmoid function, Weighted representation of text Image weighted representation Concatenate along the feature dimension For global representation, ⊙ indicates element-wise multiplication; S64: Obtain the classification probability based on the global representation using the visual language model: In the formula, For classification probability, and It consists of a learnable weight matrix and learnable bias terms. yes Activation function.

6. The multimodal interpretable classification method according to claim 4, characterized in that, Step S7 specifically includes: S71: Constructing the classification loss : In the formula, For the category label of the first One real category, For the predicted classification probability, The total number of actual categories in the category labels; S72: Constructing a Loss Monitoring System for Key Textual Evidence : In the formula, It is the textual key evidence vector of the text. It is a pseudo-label of the key textual evidence in the text. It is a binary cross-entropy loss; S73: Constructing a supervision loss for key image regions : In the formula, It is the key region vector of the image. These are pseudo-labels for key image regions of the image. It is a binary cross-entropy loss; S74: Constructing a global key evidence representation aligned with the loss : In the formula, It is a weighted representation of text. It is a weighted representation of the image; S75: Constructing Sparsity Constraints and : In the formula, It is the first in the text Distribution of key textual evidence for each token It is the total number of tokens in the text. It is the first in the image Distribution of key regions in the image of each patch It is the total number of patches in the image; S76: Total Construction Loss : In the formula, These are the weighting coefficients for each loss term; S77: Multi-objective joint optimization With the total loss mentioned above As a loss, the visual language model is subjected to multi-objective joint optimization to obtain a multimodal information classification model.

7. The multimodal interpretable classification method according to claim 1, characterized in that, The classification results in step S8 include the classification category, key textual evidence, and key image regions.

8. A multimodal interpretable classification system, characterized in that, include: A multimodal sample acquisition module is used to acquire multimodal samples, each of which includes text, images, and a set of category labels; The pseudo-label generation module is used to generate pseudo-labels for key text evidence and pseudo-labels for key image regions from the multimodal samples using a visual language model. The feature representation acquisition module is used to encode the text and image of the multimodal samples respectively to obtain token-level text feature representation and patch-level image feature representation; The key distribution acquisition module is used to obtain the distribution of key text evidence and the distribution of key image regions based on the token-level text feature representation and the patch-level image feature representation, respectively. A two-way key evidence alignment module is used to perform two-way key evidence alignment based on the token-level text feature representation and patch-level image feature representation, as well as the text key evidence distribution and image key region distribution, so that the text and image are consistent at the key evidence level. The classification probability acquisition module is used to construct a text weighted representation based on the token-level text feature representation and the distribution of key text evidence, construct an image weighted representation based on the patch-level image feature representation and the distribution of key image regions, fuse the text weighted representation and the image weighted representation to obtain a global representation, and use a visual language model to obtain the classification probability based on the global representation; The model training module is used to perform multi-objective joint optimization of the visual language model based on classification loss, text key evidence supervision loss, image key region supervision loss, global key evidence representation alignment loss and sparsity constraints, so as to obtain a multimodal information classification model. The classification result output module is used to perform multimodal interpretable classification based on the multimodal information classification model to obtain and output the classification results.

9. A multimodal interpretable classification device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal interpretable classification method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the multimodal interpretable classification method as described in any one of claims 1-7.

Images