A multimodal social event detection method with structural information optimization

Abstract

This invention discloses a multimodal social event detection method using structural information optimization, comprising the following steps: S10, acquiring and preprocessing multimodal social media data; S20, processing the multimodal social media data using an adaptive incremental learning model guided by structural entropy, outputting detected social events and their features; and S30, inputting the social event information and features output by the adaptive incremental learning model guided by structural entropy into an unsupervised graph clustering model guided by deep structural entropy to obtain clustering results. This invention overcomes the limitations of the prior art and improves the accuracy, robustness, and efficiency of event detection.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a multimodal social event detection method with optimized structural information. Background Art

[0002] The complexity of social media data, primarily due to its multimodal nature, massive data volume, and diverse content, presents significant limitations for traditional unimodal social event detection methods. Many traditional methods rely on static models, are unable to adapt to rapidly changing environments, and often require high manual annotation costs. Furthermore, existing graph clustering techniques perform poorly with sparse and noisy data, resulting in unstable clustering results. Social event detection also faces challenges related to dynamic changes and real-time requirements, requiring models to rapidly respond to new events and effectively process diverse forms of information.

[0003] In the social media environment, the dynamics and complexity of events make effective detection and analysis a challenging task. Existing methods for dealing with these problems are often limited by label dependence, data noise, and insufficient model interpretability. Summary of the Invention

[0004] In order to solve the above problems, the present invention proposes a multimodal social event detection method with structural information optimization to overcome the above limitations and improve the accuracy, robustness and efficiency of event detection.

[0005] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a multimodal social event detection method with optimized structural information, comprising the following steps:

[0006] S10, acquiring multimodal social media data and preprocessing the data;

[0007] S20, after processing by an adaptive incremental learning model guided by the multimodal social media data structure entropy, outputs the detected social events and their features, including the following steps:

[0008] S201, initial model pre-training: The multimodal event encoder introduced into the adaptive incremental learning model guided by the structural entropy of the pre-processed multimodal social media data is used to construct positive samples through contrastive learning, train the loss, and obtain the feature representation of the message;

[0009] S202, perform multimodal social event detection in the open world: After obtaining the feature representation of the message, the message is modeled as a graph G M , the graph contains two types of edges: edges based on feature similarity E s and edges E based on shared attributes a ; A hierarchical clustering algorithm based on structural entropy is introduced. The input of the algorithm is the constructed message graph GM ; First, perform hierarchical clustering and generate a hierarchical clustering tree T h ; Then, for T h For each layer, the corresponding clustering results need to be calculated, and the two-dimensional SE under each clustering result is calculated; the clustering result with the smallest two-dimensional SE is selected as the final output of the algorithm;

[0010] S203, Adaptive Incremental Learning: First, an adaptive noise filter is used to process the dynamic updates and noise of social media data. Then, incremental learning is performed to retain event-related information and output the detected social events and their characteristics.

[0011] S30, inputting the social event information and its features output by the adaptive incremental learning guided by structural entropy into the unsupervised graph clustering model guided by deep structural entropy to obtain a clustering result; comprising the steps of:

[0012] S301, structure quantization: first perform probability clustering assignment on the input data;

[0013] S302, structure learning layer: first receive node features X and adjacency matrix A g As input; then the original graph structure is enhanced by the available features X, and the graph is dynamically optimized and updated during the training process;

[0014] S303, the cluster assignment layer, uses the initial embedding and adjacency matrix to learn the soft assignment and embedding of nodes, and updates the graph structure and cluster embedding after aggregation, and finally outputs the clustering results, showing the grouping of related events and their structural information.

[0015] Furthermore, multimodal social media data includes: collecting multimodal data from multiple social media platforms, obtaining the latest news reports and user discussions through news aggregation platforms, and obtaining academic datasets and open source datasets;

[0016] Data preprocessing includes data cleaning, data conversion, feature extraction and data integration in sequence.

[0017] Furthermore, a contrastive learning strategy without negative samples is adopted. In the adaptive incremental learning model guided by structural entropy, a multimodal large language model is used to perform visual question answering on the original multimodal message, thereby reconstructing the message and generating positive samples using the multimodal large language model. These positive samples are constructed from three aspects: event type, event topic and image title. The event type serves as a coarse-grained representation of the message, helping to understand the event category described in the message; the event topic provides a finer-grained understanding; and the image title enhances the comprehensiveness of the message representation by supplementing the image information.

[0018] Furthermore, the multimodal event encoder comprises the steps of:

[0019] First, text embedding is obtained through a pre-trained language model; for image features, a frozen pre-trained visual transformer is used for feature extraction;

[0020] Subsequently, a bidirectional cross-attention mechanism is used to fuse cross-modal features to generate image-to-text and text-to-image attention scores. The attention scores are then compositely mapped with the features to generate new features for text and image.

[0021] Finally, the new features of text and image are fused through learnable weighted fusion to obtain event embedding;

[0022] Next, a predictor is introduced to maximize the output of the multimodal event encoder on the original data and enhanced data through contrastive learning, which is expressed as this loss function:

[0023]

[0024] Among them, F or is the multimodal event encoder output on the original data side after being processed by the predictor, F en It is the output of the enhanced data side.

[0025] Furthermore, for E s , select another node that is most similar to the pre-trained language model output of a given node to connect, and set the weight of the edge to the cosine similarity between the two nodes; for E a , then connect nodes with shared attributes.

[0026] Furthermore, the adaptive noise filter can handle the dynamic updates and noise of social media data. N For each cluster in N, we first calculate the entropy of each node in the cluster and add it to the set N E Then, according to N E The average value of the threshold a is calculated, where is composed of all the numbers greater than N E Finally, nodes whose entropy exceeds this threshold are considered as noise and are removed.

[0027] Furthermore, during the incremental learning process, a dedicated distillation learning module is used to consolidate prior knowledge, freezing the parameters of the multimodal event encoder and predictor from the previous model, which serves as a teacher model to transfer prior knowledge. The remaining two shared-weight multimodal event encoders have the same settings as the previous stage. During the incremental learning training process, 50% of the clean data is used for knowledge distillation and knowledge transfer with the previous model, and the remaining 50% is input into the shared-weight multimodal event encoder to learn new knowledge.

[0028] In terms of incremental learning loss, the adaptive incremental learning stage includes two losses: one is the distillation loss L2, and the other is the contrast loss L3; the distillation loss and contrast loss are unified, and the specific formula is:

[0029]

[0030] Among them, F pre represents the output of the previous model, F or is the multimodal event encoder output on the original data side after being processed by the predictor, F en is the output of the enhanced data side;

[0031] Finally, the total loss L In is the sum of the above two losses:

[0032] L In =β·L2+(1-β)·L3;

[0033] Among them, β is a hyperparameter that controls the weights of the two losses and satisfies O<β<1.

[0034] Furthermore, when assigning probabilistic clusters, an assignment matrix is ​​used;

[0035] Allocation matrix:

[0036]

[0037] C k =S h ·S h-1 ...S k+1 ;

[0038] Among them, H sa (G) represents the total structural entropy of the coding tree; H sa (G;k) represents the structural entropy of the kth layer, with N k vertices, and the total height of the coding tree is h; S k represents the distribution matrix between the k-th layer vertices and the k-1-th layer vertices; C k It is the direct assignment matrix between nodes and vertices in the k-th layer, indicating the probability that each node belongs to the k-th layer cluster;

[0039] The calculation formulas for the cut edge and volume are redefined as follows:

[0040] vol k [i]=D(C k ) i ;

[0041]

[0042] Among them, the volume vol of vertex i in the kth layer isk [i] is the sum of the assignment probabilities of all node degrees; D is the degree vector of all nodes, which is calculated by the edge weight matrix W, and the formula is:

[0043]

[0044] in, Is a vector of length N with all 1s, weight matrix W; d matrix C k The subscript i of represents the i-th column vector, which represents the direct clustering probability of N nodes in the graph for the k-th layer clustering; Represents the tangent value of the i-th vertex in the k-th layer, calculated as the volume vol of vertex i k [i] and its internal volume The difference between the internal volume represents the sum of the weighted probabilities of all edges, where probability refers to the possibility that two nodes connected by an edge belong to the same cluster i in the kth layer;

[0045] The structural entropy is calculated as follows:

[0046]

[0047] Among them, N k is the number of vertices in the kth layer of the coding tree, C O is the original clustering probability, D· In the soft assignment method, the original volume associated with a single parent vertex is replaced by the probabilistic sum of the volumes of all parent vertices.

[0048] Furthermore, the structure learning layer includes:

[0049] Feature Mapping: Use a multi-layer perceptron to map node features into a dense space of lower dimensions:

[0050] A f =KNN(MPL(X;Θ f );K);

[0051] Among them, MPL is a multi-layer perceptron, Θ f is the multilayer perceptron parameter, and KNN is the K nearest neighbor operation, K represents a K value in the K nearest neighbor algorithm;

[0052] Neighbor selection: Select the first K neighbors for each node through the K nearest neighbor algorithm and build an adjacency matrix A f ;

[0053] Adjust the adjacency matrix: Adjust the adjacency matrix of the attribute graph:

[0054]

[0055] Fuse adjacency matrix: Combine the adjacency matrix of the original graph with the adjacency matrix of the attributed graph to obtain an enhanced graph:

[0056] W′=A g +β f A f ;

[0057] Among them, β f is a hyperparameter used to control the weight of the attribute graph in the fusion, and W′ is the new adjacency matrix, A g is the adjacency matrix of the original graph.

[0058] Furthermore, the cluster assignment layer includes an embedding learner, a soft assignment learner, and an aggregator;

[0059] Embedding Learner: The embedding learner is based on a graph neural network architecture. The embedding learner performs a linear transformation on the initial embedding to map it into the embedding space. It then aggregates the average embeddings of connected nodes to generate new node embeddings and applies an activation function.

[0060] Soft Assignment Learner: Soft Assignment Learner extends the attention mechanism in the graph neural network architecture;

[0061] The soft assignment learner linearly transforms the initial embedding into the cluster space; it computes the attention matrix for each edge as the aggregation weight and performs non-averaged embedding aggregation to obtain the cluster embedding; the attention calculation involves linearly transforming the connection embeddings of the nodes at both ends of the edge into the weight space, followed by activation and normalization.

[0062] Aggregator: The goal of the aggregator is to update the cluster embeddings and adjacency matrix. The cluster embeddings are calculated using the probabilistic sum of node embeddings. The new adjacency matrix is ​​formed by combining the property graph adjacency matrix and the structure graph adjacency matrix. The cluster assignment layer dynamically updates the node embeddings and graph structure. Finally, the clustering results are output, showing the grouping of related events and their structural information.

[0063] The beneficial effects of adopting this technical solution are:

[0064] This paper proposes a multimodal social event detection technique based on structural information optimization. This technique aims to improve the accuracy and efficiency of social event detection by effectively integrating multiple information modalities. This technique can respond to the dynamic changes in the social media environment in real time, overcome the limitations of traditional single-modal methods, and reduce reliance on manual annotation. Furthermore, by optimizing the graph clustering process using structural entropy, it helps improve the stability and interpretability of clustering when dealing with sparse and noisy data, thereby providing stronger support for social event analysis.

[0065] By introducing a method for structural information optimization, this paper provides a deeper understanding of the composition of complex events in social media. Social events are typically composed of multiple factors, such as time, location, participants, and content, which cannot be fully captured by single-modal data. By integrating multimodal data such as text, images, and video, combined with structural information optimization, key event features can be more accurately identified and extracted, thereby improving the accuracy and reliability of event detection.

[0066] This invention prioritizes model interpretability. Traditional deep learning models are often viewed as "black boxes," making their internal decision-making processes difficult to understand. This is particularly important for social event detection, as researchers and decision-makers need to clearly understand how the model reaches its conclusions. By applying structural information theory, this invention provides model transparency, making detection results easier to analyze and understand. This interpretability not only enhances the model's credibility but also provides important support for subsequent decision-making.

[0067] To address label dependency, this invention enables unsupervised and incremental learning. Existing social event detection methods typically require large amounts of labeled data, limiting their flexibility and scope of application. By leveraging multimodal contrastive learning and adaptive incremental learning, this invention can rapidly identify emerging social events without explicit labels. This capability enables the model to adapt to the ever-changing social media landscape and rapidly respond to new events and trends.

[0068] This invention can reduce the impact of data noise on model performance. Social media data is often accompanied by a large amount of noise and irrelevant information, which can interfere with the model's learning process. By introducing a noise filtering mechanism guided by structural entropy, this invention can effectively identify and filter out low-quality data, thereby improving the model's robustness when processing social media streams. This noise filtering capability ensures that the model remains efficient over the long term, avoiding performance degradation caused by degraded data quality.

[0069] This paper designs an efficient real-time application framework to adapt to the rapid changes in social media data. Social events are often time-sensitive, making real-time detection and analysis particularly important. Through rational algorithm design and system architecture, this paper ensures rapid processing of large-scale social media data and enables real-time event detection and analysis. This feature gives this paper broad application potential in a variety of fields, including social governance, public opinion analysis, and information dissemination.

[0070] By combining the concepts of adaptive incremental learning and structural entropy, this paper provides a novel and effective social event detection technology, aiming to improve the effectiveness and interpretability of multimodal social event detection and meet the diverse needs in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A schematic diagram of a framework of a multimodal social event detection method based on structural information optimization according to the present invention;

[0072] Figure 2 Schematic diagram of the adaptive incremental learning model framework guided by structural entropy in an embodiment of the present invention;

[0073] Figure 3 Schematic diagram of a framework of a multimodal event encoder according to an embodiment of the present invention;

[0074] Figure 4 Schematic diagram of the framework of the unsupervised graph clustering model guided by deep structural entropy in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings.

[0076] In this embodiment, see Figures 1-4 As shown, the present invention proposes a multimodal social event detection method with structural information optimization, comprising the steps of:

[0077] S10, acquiring multimodal social media data and preprocessing the data;

[0078] S20, after processing by an adaptive incremental learning model guided by the multimodal social media data structure entropy, outputs the detected social events and their features, including the following steps:

[0079] S201, Initial Model Pre-training: The multimodal event encoder introduced into the adaptive incremental learning model guided by the structural entropy of the pre-processed multimodal social media data is used to construct positive samples through contrastive learning and train the loss to improve the efficiency of social event detection and obtain the feature representation of the message;

[0080] S202, perform multimodal social event detection in the open world: After obtaining the feature representation of the message, the message is modeled as a graph G M , the graph contains two types of edges: edges based on feature similarity E s and edges E based on shared attributes a In order to cluster messages without determining the total number of events in advance, a hierarchical clustering algorithm based on structural entropy is introduced. The input of the algorithm is the constructed message graph G M ; First, perform hierarchical clustering and generate a hierarchical clustering tree T h ; Then, for T hFor each layer, the corresponding clustering results need to be calculated, and the two-dimensional SE under each clustering result is calculated; the clustering result with the smallest two-dimensional SE is selected as the final output of the algorithm;

[0081] S203, Adaptive Incremental Learning: First, an adaptive noise filter is used to process the dynamic updates and noise of social media data. Then, incremental learning is performed to retain event-related information and output the detected social events and their characteristics.

[0082] S30, inputting the social event information and its features output by the adaptive incremental learning guided by structural entropy into the unsupervised graph clustering model guided by deep structural entropy to obtain a clustering result; comprising the steps of:

[0083] S301, structure quantization: first perform probability clustering assignment on the input data;

[0084] S302, structure learning layer: first receive node features X and adjacency matrix A g As input; then the original graph structure is enhanced by the available features X, and the graph is dynamically optimized and updated during the training process;

[0085] S303, the cluster assignment layer, uses the initial embedding and adjacency matrix to learn the soft assignment and embedding of nodes, while updating the graph structure and cluster embedding after aggregation, and finally outputs the clustering results, showing the grouping of related events and their structural information.

[0086] As an optimization solution to the above embodiment, multimodal social media data includes: collecting multimodal data from multiple social media platforms, obtaining the latest news reports and user discussions through news aggregation platforms, and obtaining academic datasets and open source datasets;

[0087] Social media platforms are important data sources, and we automatically collect multimodal data from multiple social media platforms. We can not only extract text information, such as user status updates, comments, and forwarded content, but also capture various forms of data related to these text messages, such as images, videos, and audio.

[0088] News websites and blogs are also key data sources. News aggregation platforms provide access to the latest news reports and user discussions. This content enhances multi-perspective understanding of events, providing a broader perspective and deeper analysis of social media data.

[0089] Academic and open-source datasets support model benchmarking. Using existing public datasets ensures the effectiveness of the method, while labeled data obtained from platforms such as Kaggle facilitates model training and validation.

[0090] Other data sources such as open APIs and third-party data providers also play an important role. Obtaining domain-specific data (such as weather and traffic) through these channels provides more comprehensive context for event detection systems.

[0091] Data preprocessing includes data cleaning, data conversion, feature extraction and data integration in sequence.

[0092] Data cleaning: This step involves removing duplicate data and irrelevant information to ensure that each piece of data is unique and relevant. For social media data, common noise includes advertisements, spam, and meaningless content, which require effective filtering rules to remove. Furthermore, missing values ​​must be addressed, either through interpolation or deletion, to ensure data integrity.

[0093] Data conversion: During this phase, raw data is converted into a format suitable for model processing. For example, text data can be processed through tokenization, stop word removal, and stemming to extract meaningful keywords. Furthermore, word embedding techniques such as Word2Vec or GloVe can be used to convert text into vector representations for subsequent analysis. For image and video data, image scaling, normalization, and feature extraction may be required to ensure a uniform input format.

[0094] Feature extraction and selection: This process aims to extract meaningful features from multimodal data and construct feature vectors. For example, sentiment features can be extracted from text, visual features from images, and dynamic features from videos. Feature selection techniques (such as principal component analysis or LASSO regression) can further reduce feature dimensionality and retain the most helpful features for the model, thereby improving computational efficiency and model performance.

[0095] Data integration: Data from different modalities needs to be integrated into a unified structure for easier processing. This can be achieved by constructing a graph structure, where nodes represent events, users, or keywords, and edges represent the relationships between them. This integrated data provides a comprehensive perspective for social event detection, ensuring that the invention can fully utilize all types of information.

[0096] As an optimization scheme for the above embodiment, in order to make the adaptive incremental learning model guided by structural entropy not restricted by the supervised labels in the training data, a contrastive learning strategy without negative samples is adopted. In the adaptive incremental learning model guided by structural entropy, a multimodal large language model is used to perform visual question answering on the original multimodal message, so as to reconstruct the message and generate positive samples using the multimodal large language model; these positive samples are constructed from three aspects: event type, event theme and image title, so as to more comprehensively represent the message; event type serves as a coarse-grained representation of the message, helping to understand the event category described in the message, such as sports, natural disasters or terrorist incidents; event theme provides a finer-grained understanding, such as the impact of hurricanes on coastal residents or the post-disaster reconstruction after an earthquake; image title enhances the comprehensiveness of message representation by supplementing image information.

[0097] As an optimization solution of the above embodiment, Figure 3 As shown, the multimodal event encoder comprises the steps of:

[0098] First, obtain text embeddings through a pre-trained language model;

[0099] The specific formula is:

[0100] E t =PLM(text);

[0101] Among them, E t is a text embedding of dimension 384, and the text can be T or or T en ;

[0102] For image features, a frozen pre-trained visual transformer is used for feature extraction;

[0103] The specific formula is:

[0104] E i =ViT(I or );

[0105] Among them, E i is an image feature of dimension 384;

[0106] Subsequently, a bidirectional cross-attention mechanism is used to fuse cross-modal features and generate image-to-text and text-to-image attention scores;

[0107] The specific formula is:

[0108]

[0109] Among them, Q T and Q I are the query matrices of the target and input, respectively, K Iand K T are the key matrices of input and target, V I and V T are the value matrices of input and target respectively, d k is the dimension of the bond matrix.

[0110] Combine the attention scores with the features to generate new features for text and images;

[0111] The specific formula is:

[0112]

[0113] Finally, the new features of text and image are fused through learnable weighted fusion to obtain event embedding;

[0114] The formula is:

[0115] F message =α·F text +(1-α)·F image ;

[0116] In this process, α is a learnable parameter and satisfies O<α<1.

[0117] Next, a predictor is introduced to maximize the output of the multimodal event encoder on the original data and enhanced data through contrastive learning, which is expressed as this loss function:

[0118]

[0119] Among them, F or is the multimodal event encoder output on the original data side after being processed by the predictor, F en It is the output of the enhanced data side.

[0120] As an optimization solution of the above embodiment, in the multimodal social event detection in the open world, for E s , select another node that is most similar to the pre-trained language model output of a given node to connect, and set the weight of the edge to the cosine similarity between the two nodes; for E a , then nodes with shared properties are connected. For example, posts from the same user, posts with the same hashtag (#), and posts mentioning the same user (@).

[0121] As an optimization solution of the above embodiment, the adaptive noise filter processes the dynamic updates and noise of social media data. N For each cluster in N, we first calculate the entropy of each node in the cluster and add it to the set N E Then, according to N EThe average value of the threshold a is calculated, where is composed of all the numbers greater than N E Finally, nodes whose entropy exceeds this threshold are considered as noise and are removed.

[0122] During incremental learning, a dedicated distillation learning module is used to consolidate prior knowledge and freeze the parameters of the multimodal event encoder and predictor from the previous model, serving as a teacher model to transfer prior knowledge. The remaining two weight-sharing multimodal event encoders use the same settings as the previous stage. During incremental learning training, 50% of the clean data is used for knowledge distillation and knowledge transfer with the previous model, while the remaining 50% is input into the weight-sharing multimodal event encoder to learn new knowledge.

[0123] In terms of incremental learning loss, the adaptive incremental learning stage includes two losses: one is the distillation loss L2, and the other is the contrast loss L3; the distillation loss and contrast loss are unified, and the specific formula is:

[0124]

[0125] Among them, F pre represents the output of the previous model, F or is the multimodal event encoder output on the original data side after being processed by the predictor, F en is the output of the enhanced data side;

[0126] Finally, the total loss L In is the sum of the above two losses:

[0127] L In =β·L2+(1-β)·L3;

[0128] Among them, β is a hyperparameter that controls the weights of the two losses and satisfies O<β<1.

[0129] As an optimization solution of the above embodiment, Figure 4 As shown, when assigning probability clusters, the assignment matrix is ​​used;

[0130] Allocation matrix:

[0131]

[0132] C k =S h ·S h-1 ...S k+1 ;

[0133] Among them, H sa (G) represents the total structural entropy of the coding tree; H sa(G;k) represents the structural entropy of the kth layer, with N k vertices, and the total height of the coding tree is h; S k represents the distribution matrix between the k-th layer vertices and the k-1-th layer vertices; C k It is the direct assignment matrix between nodes and vertices in the k-th layer, indicating the probability that each node belongs to the k-th layer cluster;

[0134] The calculation formulas for the cut edge and volume are redefined as follows:

[0135] vol k [i]=D(C k ) i ;

[0136]

[0137] Among them, the volume vol of vertex i in the kth layer is k [i] is the sum of the assignment probabilities of all node degrees; D is the degree vector of all nodes, which is calculated by the edge weight matrix W, and the formula is:

[0138]

[0139] in, Is a vector of length N with all 1s, weight matrix W; d matrix C k The subscript i of represents the i-th column vector, which represents the direct clustering probability of N nodes in the graph for the k-th layer clustering; Represents the tangent value of the i-th vertex in the k-th layer, calculated as the volume vol of vertex i k [i] and its internal volume The difference between the internal volume represents the sum of the weighted probabilities of all edges, where probability refers to the possibility that two nodes connected by an edge belong to the same cluster i in the kth layer;

[0140] The structural entropy is calculated as follows:

[0141]

[0142] Among them, N k is the number of vertices in the kth layer of the coding tree, C O is the original clustering probability, D· In the soft assignment method, the original volume associated with a single parent vertex is replaced by the probabilistic sum of the volumes of all parent vertices.

[0143] As an optimization solution of the above embodiment, Figure 4 As shown, the structure learning layer includes:

[0144] Feature Mapping: Use a multi-layer perceptron to map node features into a dense space of lower dimensions:

[0145] A f =KNN(MPL(X;Θ f );K);

[0146] Among them, MPL is a multi-layer perceptron, Θ f is the multilayer perceptron parameter, and KNN is the K nearest neighbor operation, K represents a K value in the K nearest neighbor algorithm;

[0147] Neighbor selection: Select the first K neighbors for each node through the K nearest neighbor algorithm and build an adjacency matrix A f ;

[0148] Adjust the adjacency matrix: To ensure the symmetry of the adjacency matrix, adjust the adjacency matrix of the attribute graph:

[0149]

[0150] Fuse adjacency matrix: Combine the adjacency matrix of the original graph with the adjacency matrix of the attributed graph to obtain an enhanced graph:

[0151] W′=A g +β f A f ;

[0152] Among them, β f is a hyperparameter used to control the weight of the attribute graph in the fusion, and W′ is the new adjacency matrix, A g is the adjacency matrix of the original graph.

[0153] As an optimization solution of the above embodiment, Figure 4 As shown, the clustering assignment layer includes an embedding learner, a soft assignment learner, and an aggregator;

[0154] Embedding Learner: The embedding learner is based on a graph neural network architecture. The embedding learner performs a linear transformation on the initial embedding to map it into the embedding space. It then aggregates the average embeddings of connected nodes to generate new node embeddings and applies an activation function.

[0155] Embedding learner mathematical representation:

[0156] H=GNN emb (E, W; Θ1)=ReLU(mean(WEΘ1));

[0157] The learnable parameters of the linear transformation are Representation, embedding learner GNN emb , initial embedding E.

[0158] Soft Assignment Learner: Soft Assignment Learner extends the attention mechanism in the graph neural network architecture;

[0159] The soft assignment learner linearly transforms the initial embedding into the cluster space; it computes the attention matrix for each edge as the aggregation weight and performs non-averaged embedding aggregation to obtain the cluster embedding; the attention calculation involves linearly transforming the connection embeddings of the nodes at both ends of the edge into the weight space, followed by activation and normalization.

[0160] The mathematical representation of the soft assignment learner is as follows:

[0161]

[0162] GNN ass is a soft assignment learner, Γ is the attention matrix, and the learnable parameters of the linear transformation in the attention mechanism are Indicates that e i and e j Represents node v i and node v j The embedding of ; the learnable parameters of the linear transformation in the attention mechanism are used express.

[0163] Aggregator: The goal of the aggregator is to update the cluster embeddings and adjacency matrix. The cluster embeddings are calculated using the probabilistic sum of node embeddings. The new adjacency matrix is ​​formed by combining the property graph adjacency matrix and the structure graph adjacency matrix. The cluster assignment layer dynamically updates the node embeddings and graph structure. Finally, the clustering results are output, showing the grouping of related events and their structural information.

[0164] The cluster embedding is calculated using the probability sum of node embeddings, expressed as:

[0165]

[0166] The new adjacency matrix is ​​formed by combining the attribute graph adjacency matrix and the structure graph adjacency matrix. The formula is:

[0167]

[0168] in The multilayer perceptron representing the clustering learns parameters, and the new weighted adjacency matrix is:

[0169] W′=A′ g +β f A′ f .

[0170] This patent aims to invent an innovative multimodal social event detection technology that improves the accuracy, interpretability and adaptability of event detection by combining the advantages of structural information optimization and deep learning.

[0171] The present invention proposes the design of a multimodal event encoder: The design of a multimodal event encoder is an important component of the present invention, which aims to effectively fuse data from different sources (such as text, images, videos, etc.). In specific implementation, the architecture of the encoder includes an input layer, a feature extraction layer, and an output layer. Through a comparative learning framework, the encoder can integrate data of different modalities into a unified representation. In addition, the loss function and optimization strategy selected during the model training process will also be elaborated in detail to demonstrate its effectiveness and uniqueness in multimodal data fusion.

[0172] This paper proposes a structural information-guided clustering algorithm: The clustering algorithm is one of the core innovations of this invention, particularly in its use of structural entropy to optimize the clustering process. The algorithm's operating principle is based on structural information theory. The specific implementation includes how to calculate and optimize structural entropy during the clustering process and how to use these calculations to improve clustering effectiveness. Evaluation metrics for clustering results are also detailed to ensure the algorithm's adaptability and versatility when working with diverse datasets, thereby enhancing its effectiveness in practical applications.

[0173] This paper proposes an adaptive incremental learning mechanism. Its innovative nature lies in its ability to effectively implement incremental learning under unsupervised conditions. The implementation process of this mechanism is described in detail, including the specific steps and strategies for introducing new knowledge, retaining old knowledge, and updating the model. This design not only reduces manual intervention but also ensures the model's long-term effectiveness and stability, further enhancing the application value of this technology.

[0174] This paper proposes a technique for dynamically filtering noise samples, using a structural entropy-guided approach to improve the robustness of the model. Specifically, the implementation details of this technique will include the steps for identifying and removing noise samples, as well as the implementation of the filtering algorithm.

[0175] This paper proposes a method for calculating deep structural entropy: This method is an important component of the present invention, ensuring the uniqueness and technological advancement of the calculation process. The specific implementation includes converting structural entropy into a continuously differentiable form, constructing a probability matrix, and applying these calculations in the clustering process.

[0176] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal social event detection method with structural information optimization, characterized in that: Including steps: S10, obtaining multimodal social media data and preprocessing the data; S20, after processing by an adaptive incremental learning model guided by the multimodal social media data structure entropy, outputs the detected social events and their features, including the following steps: S201, initial model pre-training: The multimodal event encoder introduced into the adaptive incremental learning model guided by the structural entropy of the pre-processed multimodal social media data is used to construct positive samples through contrastive learning, train the loss, and obtain the feature representation of the message; S202, perform multimodal social event detection in the open world: After obtaining the feature representation of the message, the message is modeled as a graph G M , the graph contains two types of edges: edges based on feature similarity E s and edges E based on shared attributes a ; A hierarchical clustering algorithm based on structural entropy is introduced. The input of the algorithm is the constructed message graph G M ; First, perform hierarchical clustering and generate a hierarchical clustering tree T h ; Then, for T h For each layer, the corresponding clustering results need to be calculated, and the two-dimensional SE under each clustering result is calculated; the clustering result with the smallest two-dimensional SE is selected as the final output of the algorithm; S203, Adaptive Incremental Learning: First, an adaptive noise filter is used to process the dynamic updates and noise of social media data. Then, incremental learning is performed to retain event-related information and output the detected social events and their characteristics. S30, inputting the social event information and its features output by the adaptive incremental learning guided by structural entropy into the unsupervised graph clustering model guided by deep structural entropy to obtain a clustering result; Including steps: S301, structure quantization: first perform probability clustering assignment on the input data; S302, structure learning layer: first receive node features X and adjacency matrix A g As input; then the original graph structure is enhanced by the available features X, and the graph is dynamically optimized and updated during the training process; S303, the cluster assignment layer, uses the initial embedding and adjacency matrix to learn the soft assignment and embedding of nodes, and updates the graph structure and cluster embedding after aggregation, and finally outputs the clustering results, showing the grouping of related events and their structural information.

2. The multimodal social event detection method based on structural information optimization according to claim 1, characterized in that: Multimodal social media data includes: collecting multimodal data from multiple social media platforms, obtaining the latest news reports and user discussions through news aggregation platforms, and obtaining academic datasets and open source datasets; Data preprocessing includes data cleaning, data conversion, feature extraction and data integration in sequence.

3. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: Using a contrastive learning strategy without negative samples, the system uses a multimodal large language model in an adaptive incremental learning model guided by structural entropy to perform visual question answering on the original multimodal message. This model then reconstructs the message and generates positive samples. These positive samples are constructed based on three aspects: event type, event topic, and image title. The event type serves as a coarse-grained representation of the message, helping to understand the event category described by the message; the event subject provides a finer-grained understanding; and the image title enhances the comprehensiveness of the message representation by supplementing the image information.

4. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: The multimodal event encoder comprises the steps of: First, text embedding is obtained through a pre-trained language model; for image features, a frozen pre-trained visual transformer is used for feature extraction; Subsequently, a bidirectional cross-attention mechanism is used to fuse cross-modal features to generate image-to-text and text-to-image attention scores. The attention scores are then compositely mapped with the features to generate new features for text and image. Finally, the new features of text and image are fused through learnable weighted fusion to obtain event embedding; Next, a predictor is introduced to maximize the output of the multimodal event encoder on the original data and enhanced data through contrastive learning, which is expressed as this loss function: Among them, F or is the multimodal event encoder output on the original data side after being processed by the predictor, F en It is the output of the enhanced data side.

5. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: For E s , select another node that is most similar to the pre-trained language model output of a given node to connect, and set the weight of the edge to the cosine similarity between the two nodes; for E a , then connect nodes with shared attributes.

6. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: Adaptive noise filter, when dealing with dynamic updates and noise in social media data, for graph G N For each cluster in N, we first calculate the entropy of each node in the cluster and add it to the set N E Then, according to N E The average value of the threshold a is calculated, where is composed of all the numbers greater than N E Finally, nodes whose entropy exceeds this threshold are considered as noise and are removed.

7. The multimodal social event detection method with structural information optimization according to claim 6, characterized in that: During incremental learning, a distillation learning module is used to consolidate prior knowledge and freeze the parameters of the multimodal event encoder and predictor from the previous model, serving as a teacher model to transfer prior knowledge. The remaining two weight-sharing multimodal event encoders use the same settings as the previous stage. During incremental learning training, 50% of the clean data is used for knowledge distillation and knowledge transfer with the previous model, while the remaining 50% is input into the weight-sharing multimodal event encoder to learn new knowledge. In terms of incremental learning loss, the adaptive incremental learning stage includes two losses: one is the distillation loss L2, and the other is the contrast loss L3; the distillation loss and contrast loss are unified, and the specific formula is: Among them, F pre represents the output of the previous model, F or is the multimodal event encoder output on the original data side after being processed by the predictor, F en is the output of the enhanced data side; Finally, the total loss L In is the sum of the above two losses: L In =β·L2+(1-β)·L3; Among them, β is a hyperparameter that controls the weights of the two losses and satisfies 0<β<1.

8. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: When assigning probabilistic clusters, an assignment matrix is ​​used; Allocation matrix: C k =S h ·S h-1 …S k+1 ; Among them, H sa (G) represents the total structural entropy of the coding tree; H as (G;k) represents the structural entropy of the kth layer, with N k vertices, and the total height of the coding tree is h; S k represents the distribution matrix between the k-th layer vertices and the k-1-th layer vertices; C k It is the direct assignment matrix between nodes and vertices in the k-th layer, indicating the probability that each node belongs to the k-th layer cluster; The calculation formulas for the cut edge and volume are redefined as follows: vol k [i]=D(C k ) i ; Among them, the volume vol of vertex i in the kth layer is k [i] is the sum of the assignment probabilities of all node degrees; D is the degree vector of all nodes, which is calculated through the edge weight matrix E, and the formula is: in, Is a vector of length N with all 1s, weight matrix W; d matrix C k The subscript i of represents the i-th column vector, which represents the direct clustering probability of N nodes in the graph for the k-th layer clustering; Represents the tangent value of the i-th vertex in the k-th layer, calculated as the volume vol of vertex i k [i] and its internal volume The difference between the internal volume represents the sum of the weighted probabilities of all edges, where probability refers to the possibility that two nodes connected by an edge belong to the same cluster i in the kth layer; The structural entropy is calculated as follows: Among them, N k is the number of vertices in the kth layer of the coding tree, C 0 is the original clustering probability, In the soft assignment method, the original volume associated with a single parent vertex is replaced by the probabilistic sum of the volumes of all parent vertices.

9. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: The structure learning layer includes: Feature Mapping: Use a multi-layer perceptron to map node features into a dense space of lower dimensions: A f =KNN(MPL(X;Θ f );K); Among them, MPL is a multi-layer perceptron, Θ f is the multilayer perceptron parameter, and KNN is the K nearest neighbor operation, K represents a K value in the K nearest neighbor algorithm; Neighbor selection: Select the first K neighbors for each node through the K nearest neighbor algorithm and construct the adjacency matrix A f ; Adjust the adjacency matrix: Adjust the adjacency matrix of the attribute graph: Fuse adjacency matrix: Combine the adjacency matrix of the original graph with the adjacency matrix of the attributed graph to obtain an enhanced graph: W′=A g +b f A f ; Among them, β f is a hyperparameter used to control the weight of the attribute graph in the fusion, and W′ is the new adjacency matrix, A g is the adjacency matrix of the original graph.

10. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: Cluster assignment layer, including embedding learner, soft assignment learner and aggregator; Embedding Learner: The embedding learner is based on the graph neural network architecture; The embedding learner performs a linear transformation on the initial embedding, mapping it to the embedding space; The average embeddings of connected nodes are then aggregated to generate new node embeddings, and an activation function is applied; Soft Assignment Learner: Soft Assignment Learner extends the attention mechanism in the graph neural network architecture; The soft assignment learner linearly transforms the initial embedding into the cluster space; It computes the attention matrix of each edge as the aggregation weight and performs non-averaged embedding aggregation to obtain clustered embeddings; The calculation of attention involves linearly transforming the connection embedding of the nodes at both ends of the edge into the weight space, followed by activation and normalization; Aggregator: The goal of the aggregator is to update the cluster embeddings and adjacency matrix. The cluster embeddings are calculated using the probabilistic sum of node embeddings. The new adjacency matrix is ​​formed by combining the property graph adjacency matrix and the structure graph adjacency matrix. The clustering assignment layer dynamically updates node embeddings and graph structures; The final clustering result is output, showing the grouping of related events and their structural information.

Images