Incomplete multi-modal crisis event detection method based on memory pool and modal perception expert system

Abstract

The application discloses an incomplete multi-modal crisis event detection method based on a memory pool and a modal perception expert system, relates to the field of machine learning, and specifically comprises the following steps: constructing a prompt strongly related to an event instance based on a memory pool, and complementing a missing mode by using a large model; constructing a double-flow encoder by using a CNN, a ResNet and a CLIP encoder to realize text and visual feature extraction; capturing fine-grained event information and clues; identifying the distribution difference between a complemented sample and a complete sample based on a modal perception expert system to improve the quality of a fused representation; and inputting a mixed feature into a classification head to complete a crisis event detection task. The application can effectively deal with a multi-modal data missing scene and improve the accuracy and timeliness of crisis event detection by constructing a prompt based on a memory pool, introducing a large model to realize detailed complementation of a missing mode, and accurately identifying the distribution difference between a complemented sample and a complete sample based on a modal perception expert system.

Description

Technical Field

[0001] This invention relates to the field of machine learning, and more specifically to a method for detecting incomplete multimodal crisis events based on memory pools and modality-aware expert systems. Background Technology

[0002] In multimodal crisis event detection, comprehensive analysis and identification typically rely on data from different modalities, including images and text. These modalities provide rich scene information; for example, identifying a fire scene through images or analyzing emergency reports through text analysis. However, real-world multimodal data is often incomplete, especially during actual crisis events, where data loss, transmission delays, or platform limitations often mean that only data from a single modality is available.

[0003] For example, in some sudden crisis events, image data may be lost or unavailable due to transmission problems, leaving only text data to provide a preliminary description of the event. Conversely, in some cases, image data may be available, but corresponding textual descriptions are lacking. This modal incompleteness leads to cross-modal information loss, affecting the accuracy and timeliness of event detection.

[0004] To address this challenge, recovering and completing missing modalities when they exist has become a key technical challenge for improving the performance of multimodal crisis event detection systems. Existing technologies often rely on simple interpolation techniques or generate missing modalities based on existing data. However, these methods often fail to accurately capture the specific context or features of crisis events, resulting in insufficiently precise recovered modal information that cannot meet the requirements for high-precision detection.

[0005] Therefore, how to generate high-quality completed data when a modality is missing through an intelligent modality completion mechanism, and effectively combine the completed modality with the existing modality, is an important direction for improving multimodal crisis event detection technology. Summary of the Invention

[0006] To achieve accurate and timely crisis event detection in scenarios with missing modalities, this invention proposes an incomplete multimodal crisis event detection method based on a memory pool and a modality-aware expert system, comprising the following steps:

[0007] An incomplete multimodal crisis event detection method based on memory pools and modality-aware expert systems includes the following steps:

[0008] Missing Modality Completion: A cross-modal memory pool is constructed based on text-image pairs in the training set. Text and images are extracted using encoders and stored as vectors in the memory pool. For samples lacking the text modality, the corresponding image features are used as the query vector, and a cross-modal retrieval based on cosine similarity is performed in the text feature set of the memory pool. For samples lacking the image modality, the text features are used as the query vector, and a cross-modal retrieval based on cosine similarity is performed in the image feature set of the memory pool. After retrieval, P pairs of samples with the highest cosine similarity are selected for memory, and keywords are extracted from the text data of these samples. The Q keywords with the highest word frequency are statistically analyzed and selected. Subsequently, for samples lacking the text modality, the original image features, the retrieved P pairs of text data, and the extracted keywords are combined to construct a prompt message, which is then input into the image-to-text model to generate the completed text. For samples lacking the image modality, the original text features, the retrieved P pairs of text data, and the keywords are combined to construct a prompt message, which is then input into the image-to-text model to achieve image completion. P=3, Q=3.

[0009] Text and visual feature extraction: Text features are extracted using TEXTCNN encoder and CLIP text encoder, and image features are extracted using ResNet and CLIP visual encoder. The features are then concatenated by dimension, and image-text contrast loss is used to reduce the feature difference between the completed features and the real features.

[0010] Deep Feature Interaction: Utilizing a transformer-based multi-head attention mechanism to achieve deep interaction between text and image features, extracting fine-grained feature information, and uncovering subtle event clues in images and text;

[0011] Gated expert modality perception: Different experts are used to process different combinations of modalities to identify the differences in modal features between samples;

[0012] Crisis event classification: Based on the classification head, mixed features are processed to identify the event category to which the current sample belongs.

[0013] As a further aspect of the present invention, in the missing modality completion step, if the missing modality is text, the missing description will be synthesized through the image-to-text big data model; if the missing modality is an image, the text-to-image big data model will generate a visually consistent image through back diffusion, guided by prompts.

[0014] As a further aspect of the present invention, in the text and visual feature extraction step, for text features, a TEXTCNN encoder and a pre-trained CLIP text encoder are used to extract modality-specific features and modality-consistent features:

[0015]

[0016] For image features, the ResNet architecture and a pre-trained CLIP image encoder are used to extract modality-specific and modality-consistent features:

[0017]

[0018] in, and These represent modality-specific and modality-consistent encoders of text, respectively. and These represent modality-specific and modality-consistent encoders of an image, respectively. and This indicates modality-specific and modality-consistent features of text. and Represents modality-specific and modality-consistent features of an image;

[0019] Concatenate modality-specific and modality-consistent features as output features for text and image modalities:

[0020]

[0021] in, or , This indicates a splicing operation.

[0022] As a further aspect of the present invention, the image-text contrast loss method involved is as follows:

[0023]

[0024] Where s is a learnable temperature parameter used to control the smoothness of the contrast learning; Represents the similarity moments from text features to image features; A similarity matrix representing image features to text features; Represents the transpose of the image matrix; This indicates that the temperature parameter has been exponentialized.

[0025] The contrastive loss for each sample is as follows:

[0026]

[0027] Where B is the size of each batch of training samples; For text-to-image contrast loss; For image-to-text contrast loss; Indicates the index of the current sample; Indicates the index of the candidate matching sample.

[0028] The final comparison loss is:

[0029]

[0030] in, To compare the hyperparameters of the loss weights.

[0031] As a further aspect of the present invention, in the deep feature interaction step, a hybrid serial-parallel encoder is used to perform fine-grained feature extraction and capture subtle event cues.

[0032] A transformer-based multi-head self-attention mechanism is employed to process text and image features separately, enhancing the representation of each modality. Self-attention is computed for each attention head, independently enhancing text and image features.

[0033]

[0034]

[0035]

[0036] in, , and They are the first The query size, key and value embedding, For learnable parameter matrix, Indicates the input feature dimension. For the output dimension, then compute the self-attention for each attention head:

[0037]

[0038]

[0039]

[0040] in, Indicates the first The output matrix of each attention head, This represents the concatenation matrix of the attention outputs from m heads. To represent the final output matrix of multi-head attention, , Indicates the number of heads. This indicates a splicing operation. This indicates the output linear transformation matrix;

[0041] Subsequently, an FFN network consisting of two fully connected layers and a ReLU activation function was used to obtain the modes. Enhanced features:

[0042]

[0043] in, This indicates a bullish focus operation;

[0044] The enhanced text and image features are fused through a cross-modal fusion module to generate hybrid features:

[0045]

[0046] .

[0047] in, Indicates image enhancement features, Enhance text features, and It is a hybrid feature of images and text.

[0048] As a further aspect of the present invention, the step of employing different experts to process different combinations of modes includes the following steps:

[0049] Employing a lightweight routing network Based on semantic content and detected modal configurations, the network predicts adaptive route scores; when completing text modalities, the crisis-aware routing network first calculates the score for each route:

[0050]

[0051] in, These represent three specific modality experts. , , The score, General experts The scores are used to select the top k experts from the general expert set based on their routing scores, denoted as [the top k experts]. Combined with specific modality experts Routing score The normalized route weights are calculated using the softmax function:

[0052]

[0053] in, This represents the weights assigned to a specific modality expert and a selected general expert;

[0054] Each expert processes the input features separately:

[0055]

[0056] in, For the output of a specific modality expert, For the first The output of a general expert.

[0057] Final output features The following formula is used to derive:

[0058] .

[0059] As a further aspect of the present invention, crisis events are classified based on hybrid characteristics:

[0060]

[0061] in, It is a fully connected layer, except for the last one. In addition to the layers, each layer has an activation function;

[0062] The standard cross-entropy loss function is used to optimize the classification performance of crisis event detection:

[0063]

[0064] Where N is the total number of samples, and C is the number of categories. It's a real label. It is the predicted probability of category c;

[0065] The overall loss function is as follows: .

[0066] Compared with existing technologies, the advantages of this invention are as follows: This invention demonstrates higher accuracy and robustness in addressing the modality incompleteness problem in multimodal crisis event detection. First, by constructing event instance-related prompts based on a memory pool and combining this with a large model for modality completion, the system can still recover missing information and ensure the accuracy of detection results even when a certain modality is missing. Traditional methods often rely on simple interpolation or rule-based imputation, which struggles to guarantee the contextual consistency and event perception capabilities of the completed information. Second, by utilizing CNN, ResNet, and CLIP encoders to construct a dual-stream encoder and deep feature interaction, subtle event clues in images and text can be meticulously extracted, resulting in richer feature information. Finally, the modality-aware expert system can flexibly schedule different experts for processing based on the routing scores of modality-specific and general experts, thereby achieving more accurate modality fusion and information completion. This dynamic routing-based design allows the model to interact more deeply between different modalities while avoiding the limitations of a fixed expert structure, greatly improving the system's adaptability and accuracy. Furthermore, processing mixed features based on the classification head can effectively fuse information from different modalities, thereby improving the recognition accuracy of complex event categories. In summary, this invention not only solves the challenges posed by incomplete modalities but also enhances the model's real-time response capability, accuracy, and flexibility, making it of significant application value in crisis event detection. Attached Figure Description

[0067] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0068] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] The incomplete multimodal crisis event detection method proposed in this invention, based on a memory pool and a modality-aware expert system, achieves accurate and timely multimodal crisis event detection while accurately completing missing modality information through the following steps. This method includes multiple steps: missing modality completion, text and image feature extraction, deep feature interaction, gated expert modality perception, and crisis event classification. Each step can be adjusted and optimized according to task requirements. The implementation method for each step is as follows.

[0070] (1) Step 1: Missing mode completion

[0071] This step is the starting point for model training, ensuring effective completion of missing modality information. This step includes three parts: building a memory pool, extracting similar text and keywords, and generating and distributing missing modality data.

[0072] A. Constructing a memory pool:

[0073] This invention constructs a high-quality text visual memory pool from a complete training set, wherein text and visual features are encoded using a CLIP encoder.

[0074] B. Extracting similar text and keywords:

[0075] Based on cosine similarity, the current modality information is retrieved from the memory pool to obtain the three most similar texts. KeyBERT is then used to extract keywords from the extracted text, and the three words with the highest frequency are selected.

[0076] C. Missing mode generation:

[0077] If the missing modality is text, the missing description will be synthesized using the InstructBLIP (Graph-to-Text Big Data) model. If the missing modality is an image, the Text-to-Graph Big Data model (such as Stable Diffusion) generates a visually consistent image through backdiffusion, guided by prompts.

[0078] (2) Step Two: Text and Image Feature Extraction

[0079] This step fully utilizes the visual-language relationship to extract comprehensive feature information. The implementation method is as follows:

[0080] A. Text Feature Extraction:

[0081] For text features, the TEXCNN architecture and the pre-trained CLIP text encoder are used to extract modality-specific and modality-consistent features:

[0082]

[0083] B. Image Feature Extraction:

[0084] For image features, the ResNet architecture and a pre-trained CLIP image encoder are used to extract modality-specific and modality-consistent features:

[0085]

[0086] in, and These represent modality-specific and modality-consistent encoders of text, respectively. and These represent modality-specific and modality-consistent encoders of an image, respectively. and This indicates modality-specific and modality-consistent features of text. and Represents modality-specific and modality-consistent features of an image;

[0087] Concatenate modality-specific and modality-consistent features as output features for text and image modalities:

[0088]

[0089] in, or , Indicates a splicing operation;

[0090] C. Comparison of losses:

[0091] To further narrow the gap between the modality-specific feature space and the modality-invariant feature space, an image-text contrast loss was employed.

[0092]

[0093] Where s is a learnable temperature parameter used to control the smoothness of the contrast learning; Represents the similarity moments from text features to image features; A similarity matrix representing image features to text features; Represents the transpose of the image matrix; This indicates that the temperature parameter has been exponentialized.

[0094] The contrastive loss for each sample is as follows:

[0095]

[0096] Where B is the size of each batch of training samples; For text-to-image contrast loss; For image-to-text contrast loss; Indicates the index of the current sample; Indicates the index of the candidate matching sample.

[0097] The final comparison loss is:

[0098]

[0099] in, To compare the hyperparameters of the loss weights.

[0100] (3) Step 3: Deep feature interaction

[0101] This method enables deep interaction between text and image features, allowing for detailed mining of subtle event clues within images and text, thereby extracting richer feature information. The implementation method is as follows:

[0102] A transformer-based multi-head self-attention mechanism is employed to process text and image features separately, enhancing the representation of each modality. Self-attention is computed for each attention head, independently enhancing text and image features.

[0103]

[0104]

[0105]

[0106] in, , and They are the first The query size, key and value embedding, For learnable parameter matrix, Indicates the input feature dimension. The output dimension is used, and then the self-attention of each attention head is calculated.

[0107]

[0108]

[0109]

[0110] in, Indicates the first The output matrix of each attention head, This represents the concatenation matrix of the attention outputs from m heads. To represent the final output matrix of multi-head attention, , Indicates the number of heads. This indicates a splicing operation. This indicates the output linear transformation matrix;

[0111] Subsequently, by employing an FFN network consisting of two fully connected layers and a ReLU activation function, the modalities can be obtained. Enhanced features,

[0112]

[0113] in, This indicates a bullish focus operation;

[0114] The enhanced text and image features are fused through a cross-modal fusion module to generate hybrid features:

[0115]

[0116] .

[0117] in, Indicates image enhancement features, Enhance text features, and It is a hybrid feature of images and text.

[0118] (4) Step 4: Gated expert modal perception

[0119] By assigning specialized expert models to different modal combinations, the feature differences between each modality can be accurately identified, thereby avoiding conflicts or loss during information fusion. The implementation method is as follows:

[0120] A lightweight routing network is employed to predict adaptive route scores based on semantic content and detected modal configurations. Taking text completion modality as an example, the crisis-aware routing network first calculates the score for each route.

[0121]

[0122] in, These represent three specific modality experts. , , The score, General experts Score

[0123] From the general set of experts, we select the top k experts based on their routing scores, denoted as [k experts]. Combining specific modality experts Routing score We calculate the normalized route weights using the softmax function:

[0124]

[0125] in, This represents the weights assigned to specific modality experts and selected general experts.

[0126] Each expert processes the input features separately:

[0127]

[0128] in, For the output of a specific modality expert, For the first The output of a general expert.

[0129] Final output features The following can be derived:

[0130]

[0131] This dynamic routing allows the network to emphasize evidence of a specific modality while still utilizing extensively trained experts.

[0132] (5) Step Five: Crisis Event Classification

[0133] By processing mixed features using a classification head, information from different modalities can be fused, and the event category of the current sample can be identified, thereby improving the accuracy of crisis event detection. The implementation method is as follows:

[0134]

[0135] in, It is a fully connected layer, except for the last one. In addition to the layers, each layer has an activation function.

[0136] To optimize the classification performance of crisis event detection, we adopt the standard cross-entropy loss function:

[0137]

[0138] Where N is the total number of samples, and C is the number of categories. It's a real label. It is the predicted probability of category c.

[0139] The overall loss function is as follows:

[0140] .

[0141] This invention employs a joint optimization of the cross-entropy loss function and the text-image contrast loss function in the multimodal crisis event classification process, simultaneously improving the model's classification and discrimination capabilities as well as its cross-modal semantic alignment. Through this joint optimization strategy, the model significantly improves its recognition accuracy across different crisis types and exhibits higher robustness and generalization ability at the classification level, thereby effectively enhancing the overall performance of multimodal crisis event detection.

[0142] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0143] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for detecting incomplete multimodal crisis events based on a memory pool and a modality-aware expert system, characterized in that, Includes the following steps: Missing Modality Completion: A cross-modal memory pool is constructed based on text-image pairs in the training set. Text and images are extracted using encoders and stored as vectors in the memory pool. For samples lacking the text modality, the corresponding image features are used as the query vector, and a cross-modal retrieval based on cosine similarity is performed in the text feature set of the memory pool. For samples lacking the image modality, the text features are used as the query vector, and a cross-modal retrieval based on cosine similarity is performed in the image feature set of the memory pool. After retrieval, P pairs of samples with the highest cosine similarity are selected for memory, and keywords are extracted from the text data of these samples. The Q keywords with the highest word frequency are statistically analyzed and selected. Subsequently, for samples lacking the text modality, the original image features, the retrieved P pairs of text data, and the extracted keywords are combined to construct a prompt message, which is then input into the image-to-text model to generate the completed text. For samples lacking the image modality, the original text features, the retrieved P pairs of text data, and the keywords are combined to construct a prompt message, which is then input into the image-to-text model to achieve image completion. Text and visual feature extraction: Text features are extracted using TEXTCNN encoder and CLIP text encoder, and image features are extracted using ResNet and CLIP visual encoder. The features are then concatenated by dimension, and image-text contrast loss is used to reduce the feature difference between the completed features and the real features. Deep Feature Interaction: Utilizing a transformer-based multi-head attention mechanism to achieve deep interaction between text and image features, extracting fine-grained feature information, and uncovering subtle event clues in images and text; Gated expert modality perception: Different experts are used to process different combinations of modalities to identify the differences in modal features between samples; Crisis event classification: Based on the classification head, mixed features are processed to identify the event category to which the current sample belongs.

2. The incomplete multimodal crisis event detection method based on a memory pool and modality-aware expert system according to claim 1, characterized in that, In the missing modality completion step, if the missing modality is text, the missing description will be synthesized by the image-to-text big data model. If the missing modality is an image, the text-to-image big data model will generate a visually consistent image through back diffusion, guided by prompts.

3. The incomplete multimodal crisis event detection method based on a memory pool and modality-aware expert system according to claim 1, characterized in that, In the text and visual feature extraction steps, for text features, the TEXTCNN encoder and the pre-trained CLIP text encoder are used to extract modality-specific features and modality-consistent features: For image features, the ResNet architecture and a pre-trained CLIP image encoder are used to extract modality-specific and modality-consistent features: in, and These represent modality-specific and modality-consistent encoders of text, respectively. and These represent modality-specific and modality-consistent encoders of an image, respectively. and This indicates modality-specific and modality-consistent features of text. and Represents modality-specific and modality-consistent features of an image; Concatenate modality-specific and modality-consistent features as output features for text and image modalities: in, or , This indicates a splicing operation.

4. The incomplete multimodal crisis event detection method based on a memory pool and modality-aware expert system according to claim 3, characterized in that, The image-text contrast loss involved is as follows: Where s is a learnable temperature parameter used to control the smoothness of the contrast learning; Represents the similarity moments from text features to image features; A similarity matrix representing image features to text features; Represents the transpose of the image matrix; This indicates that the temperature parameter has been exponentialized. The contrastive loss for each sample is as follows: Where B is the size of each batch of training samples; For text-to-image contrast loss; For image-to-text contrast loss; Indicates the index of the current sample; Indicates the index of the candidate matching sample; The final comparison loss is: in, To compare the hyperparameters of the loss weights.

5. The incomplete multimodal crisis event detection method based on a memory pool and modality-aware expert system according to claim 1, characterized in that, In the deep feature interaction step, a hybrid serial-parallel encoder is used for fine-grained feature extraction and to capture subtle event cues. A transformer-based multi-head self-attention mechanism is employed to process text and image features separately, enhancing the representation of each modality. Self-attention is computed for each attention head, independently enhancing text and image features. in, , and They are the first The query size, key and value embedding, For learnable parameter matrix, Indicates the input feature dimension. The output dimension is used, and then the self-attention of each attention head is calculated. in, Indicates the first The output matrix of each attention head, This represents the concatenation matrix of the attention outputs from m heads. To represent the final output matrix of multi-head attention, , Indicates the number of heads. This indicates a splicing operation. This indicates the output linear transformation matrix; Subsequently, an FFN network consisting of two fully connected layers and a ReLU activation function was used to obtain the modes. Enhanced features: in, This indicates a bullish attention-based strategy; The enhanced text and image features are fused through a cross-modal fusion module to generate hybrid features: in, Indicates image enhancement features, Enhance text features, and It is a hybrid feature of images and text.

6. The incomplete multimodal crisis event detection method based on a memory pool and modality-aware expert system according to claim 1, characterized in that, The method of using different experts to handle different combinations of modalities includes the following steps: Use a lightweight routing network Predict adaptive route scores based on semantic content and detected modal configurations; complete text modalities. At that time, the crisis-aware routing network first calculates the score for each route. : in, These represent three specific modality experts. , , The score, General experts The scores are used to select the top k experts from the general expert set based on their routing scores, denoted as [the top k experts]. Combined with specific modality experts Routing score The normalized route weights are calculated using the softmax function: in, This represents the weights assigned to specific modal experts and selected general experts; Each expert processes the input features separately: in, For the output of a specific modality expert, For the first The output of a general expert; Final output features The following formula is used to derive: 。 7. The incomplete multimodal crisis event detection method based on a memory pool and modality-aware expert system according to claim 5, characterized in that, Classification of crisis events based on mixed characteristics: in, It is a fully connected layer, except for the last one. In addition to the layers, each layer has an activation function; The standard cross-entropy loss function is used to optimize the classification performance of crisis event detection: Where N is the total number of samples, and C is the number of categories. It's a real label. It is the predicted probability of category c; The overall loss function is as follows: 。

Images