Structural information optimized multi-modal social event detection method

Abstract

The invention discloses a structural information optimized multi-modal social event detection method, which comprises the following steps: S10, acquiring multi-modal social media data, and preprocessing the data; s20, carrying out processing through a self-adaptive incremental learning model guided by a multi-modal social media data structure entropy, and outputting a detected social event and features thereof; and S30, inputting the social event information output by the adaptive incremental learning guided by the structure entropy and the features of the social event information into the unsupervised graph clustering model guided by the deep structure entropy to obtain a clustering result. According to the method, the limitation in the prior art is overcome, and the event detection accuracy, robustness and efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a multimodal social event detection method with optimized structural information. Background Art

[0002] The complexity of social media data is mainly reflected in its multimodal characteristics, huge data volume and content diversity, which makes traditional unimodal social event detection methods face significant limitations. Many traditional methods rely on static models, cannot adapt to rapidly changing environments, and usually require high manual annotation costs. At the same time, existing graph clustering techniques perform poorly when dealing with sparse and noisy data, resulting in unstable clustering effects. The challenges of social event detection also include dynamic changes and real-time requirements, which require the model to respond quickly to new events and effectively process different forms of information.

[0003] In the social media environment, the dynamics and complexity of events make effective detection and analysis a challenging task. Existing methods are often limited by label dependence, data noise, and lack of model interpretability when dealing with these problems. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a multimodal social event detection method with structural information optimization to overcome the above limitations and improve the accuracy, robustness and efficiency of event detection.

[0005] To achieve the above object, the technical solution adopted by the present invention is: a multimodal social event detection method with optimized structural information, comprising the steps of:

[0006] S10, obtaining multimodal social media data and preprocessing the data;

[0007] S20, after being processed by an adaptive incremental learning model guided by the multimodal social media data structure entropy, outputs the detected social events and their features, including the steps of:

[0008] S201, initial model pre-training: the multimodal event encoder introduced in the adaptive incremental learning model guided by the structure entropy of the pre-processed multimodal social media data is used to construct positive samples through the contrastive learning method, train the loss, and obtain the feature representation of the message;

[0009] S202, perform multimodal social event detection in the open world: After obtaining the feature representation of the message, the message is modeled as a graph G M , the graph contains two types of edges: edges based on feature similarity E s and edges E based on shared attributes a ; A hierarchical clustering algorithm based on structural entropy is introduced. The input of the algorithm is the constructed message graph GM ; First, perform hierarchical clustering and generate a hierarchical clustering tree T h ; Then, for T h For each layer, the corresponding clustering results need to be calculated, and the two-dimensional SE under each clustering result needs to be calculated; the clustering result with the smallest two-dimensional SE is selected as the final output of the algorithm;

[0010] S203, adaptive incremental learning: first, an adaptive noise filter is used to process the dynamic updates and noise of social media data; then, incremental learning is performed to retain information related to the event and output the detected social events and their characteristics;

[0011] S30, inputting the social event information and its features output by the adaptive incremental learning guided by the structural entropy into the unsupervised graph clustering model guided by the deep structural entropy to obtain a clustering result; comprising the steps of:

[0012] S301, structural quantization: first perform probability clustering assignment on the input data;

[0013] S302, structure learning layer: first receive node features X and adjacency matrix A g As input; the original graph structure is then enhanced by the available features X, and the graph is dynamically optimized and updated during the training process;

[0014] S303, the clustering assignment layer, uses the initial embedding and adjacency matrix to learn the soft assignment and embedding of nodes, and updates the graph structure and cluster embedding after aggregation, and finally outputs the clustering results, showing the grouping of related events and their structural information.

[0015] Furthermore, multimodal social media data includes: collecting multimodal data from multiple social media platforms, obtaining the latest news reports and user discussions through news aggregation platforms, and obtaining academic datasets and open source datasets;

[0016] Data preprocessing includes data cleaning, data conversion, feature extraction and data integration in sequence.

[0017] Furthermore, a contrastive learning strategy without negative samples is adopted, and a multimodal large language model is used in the adaptive incremental learning model guided by structural entropy to conduct visual question answering on the original multimodal messages, so as to reconstruct the messages and generate positive samples using the multimodal large language model; these positive samples are constructed from three aspects: event type, event topic and image title; the event type serves as a coarse-grained representation of the message, which helps to understand the event category described in the message; the event topic provides a finer-grained understanding; and the image title enhances the comprehensiveness of the message representation by supplementing the image information.

[0018] Furthermore, the multimodal event encoder comprises the steps of:

[0019] First, text embedding is obtained through a pre-trained language model. For image features, a frozen pre-trained visual transformer is used for feature extraction.

[0020] Then, a bidirectional cross-attention mechanism is used to fuse cross-modal features and generate image-to-text and text-to-image attention scores. The attention scores are compositely mapped with the features to generate new features for text and image.

[0021] Finally, the new features of text and image are fused through learnable weighted fusion to obtain event embedding;

[0022] Next, a predictor is introduced to maximize the output of the multimodal event encoder on the original data and enhanced data through contrastive learning, which is expressed as this loss function:

[0023]

[0024] Among them, F or is the multimodal event encoder output on the original data side after being processed by the predictor, and F en It is the output of the enhanced data side.

[0025] Furthermore, for E s , select another node that is most similar to the pre-trained language model output of the given node for connection, and set the weight of the edge to the cosine similarity between the two nodes; for E a , then nodes with shared attributes are connected.

[0026] Furthermore, the adaptive noise filter can handle the dynamic updates and noise of social media data for graph G. N For each cluster in N, we first calculate the entropy of each node in the cluster and add it to the set N. E Then, according to N E The average value of the threshold a is calculated, where is composed of all the E The set consists of elements with the average value; finally, nodes whose entropy exceeds this threshold are considered noise and are removed.

[0027] Furthermore, during the incremental learning process, a dedicated distillation learning module is used to consolidate previous knowledge and freeze the parameters of the multimodal event encoder and predictor from the previous model as a teacher model to transfer previous knowledge; the remaining two shared-weight multimodal event encoders are the same as the settings of the previous stage; during the incremental learning training process, 50% of the clean data is used for knowledge distillation and knowledge transfer with the previous model, and the remaining 50% is input into the shared-weight multimodal event encoder to learn new knowledge;

[0028] In terms of the loss of incremental learning, the adaptive incremental learning stage includes two losses: one is the distillation loss L2, and the other is the contrast loss L3; the distillation loss and the contrast loss are unified, and the specific formula is:

[0029]

[0030] Among them, F pre represents the output of the previous model, F or is the multimodal event encoder output on the original data side after being processed by the predictor, and F en is the output of the enhanced data side;

[0031] Finally, the total loss L In is the sum of the above two losses:

[0032] L In =β·L2+(1-β)·L3;

[0033] Among them, β is a hyperparameter that controls the weights of the two losses and satisfies O<β<1.

[0034] Furthermore, when assigning probabilistic clusters, an assignment matrix is ​​used;

[0035] Allocation matrix:

[0036]

[0037] C k =S h ·S h-1 ...S k+1 ;

[0038] Among them, H sa (G) represents the total structural entropy of the coding tree; H sa (G; k) represents the structural entropy of the kth layer, with N k vertices, and the total height of the coding tree is h; S k represents the allocation matrix between the k-th layer vertices and the k-1-th layer vertices; C k It is the direct assignment matrix between nodes and vertices at the kth level, indicating the probability that each node belongs to the kth level cluster;

[0039] Redefine the calculation formula of the cut edge and volume as follows:

[0040] vol k [i]=D(C k ) i ;

[0041]

[0042] Among them, the volume vol of vertex i in the kth layer isk [i] is the sum of the assignment probabilities of all node degrees; D is the degree vector of all nodes, calculated by the edge weight matrix W, the formula is:

[0043]

[0044] in, is a vector of length N with all 1s, weight matrix W; d matrix C k The subscript i of represents the i-th column vector, which indicates the direct clustering probability of the N nodes in the graph for the k-th layer clustering; Represents the tangent value of the i-th vertex in the k-th layer, calculated as the volume vol of vertex i k [i] and its internal volume The difference between the internal volume represents the sum of the weighted probabilities of all edges, where probability refers to the possibility that two nodes connected by an edge belong to the same cluster i in the kth layer;

[0045] The structural entropy is calculated using the formula:

[0046]

[0047] Among them, N k is the number of vertices in the kth layer of the coding tree, C O is the original clustering probability, D In the soft assignment method, the original volume associated with a single parent vertex is replaced by the probabilistic sum of the volumes of all parent vertices.

[0048] Furthermore, the structure learning layer includes:

[0049] Feature Mapping: Use a multi-layer perceptron to map node features to a dense space of lower dimension:

[0050] A f =KNN(MPL(X;θ f );K);

[0051] Among them, MPL is a multi-layer perceptron, Θ f is the multilayer perceptron parameter, and KNN is the K nearest neighbor operation, K represents a K value in the K nearest neighbor algorithm;

[0052] Neighbor selection: Select the first K neighbors for each node through the K nearest neighbor algorithm and construct an adjacency matrix A f ;

[0053] Adjust the adjacency matrix: Adjust the adjacency matrix of the attribute graph:

[0054]

[0055] Fuse adjacency matrix: Combine the adjacency matrix of the original graph with the adjacency matrix of the attributed graph to obtain an enhanced graph:

[0056] W′=A g +β f A f ;

[0057] Among them, β f is a hyperparameter used to control the weight of the attribute graph in the fusion, and W′ is the new adjacency matrix, A g is the adjacency matrix of the original graph.

[0058] Further, a clustering assignment layer, including an embedding learner, a soft assignment learner and an aggregator;

[0059] Embedding learner: The embedding learner is based on a graph neural network architecture; the embedding learner performs a linear transformation on the initial embedding to map it to the embedding space; it then aggregates the average embeddings of connected nodes to generate new node embeddings and applies an activation function;

[0060] Soft Assignment Learner: Soft Assignment Learner extends the attention mechanism in the graph neural network architecture;

[0061] The soft assignment learner linearly transforms the initial embedding to the cluster space; it calculates the attention matrix for each edge as the aggregation weight and performs non-average embedding aggregation to obtain the cluster embedding; the calculation of attention involves linearly transforming the connection embeddings of the nodes at both ends of the edge to the weight space, followed by activation and normalization.

[0062] Aggregator: The goal of the aggregator is to update the cluster embedding and adjacency matrix; the cluster embedding is calculated using the probability sum of the node embedding; the new adjacency matrix is ​​a combination of the property graph adjacency matrix and the structure graph adjacency matrix; the cluster assignment layer completes the dynamic update of the node embedding and graph structure; and finally the clustering result is output, showing the grouping of related events and their structural information.

[0063] The beneficial effects of adopting this technical solution are:

[0064] This paper proposes a multimodal social event detection technology with structural information optimization, which aims to improve the accuracy and efficiency of social event detection by effectively integrating multiple information modalities. This technology can respond to the dynamic changes of the social media environment in real time, overcome the limitations of traditional single-modal methods, and reduce the reliance on manual annotation. In addition, the use of structural entropy to optimize the graph clustering process helps to improve the stability and interpretability of clustering when dealing with sparse and noisy data, thereby providing stronger support for social event analysis.

[0065] By introducing the method of structural information optimization, the present invention can more deeply understand the composition of complex events in social media. Social events are usually composed of multiple factors, such as time, location, participants and content, etc. Single-modal data cannot fully capture this information. By integrating multimodal data such as text, images, and videos, combined with structural information optimization, the key features of events can be more accurately identified and extracted, thereby improving the accuracy and credibility of event detection.

[0066] The present invention attaches importance to the interpretability of the model. Traditional deep learning models are often regarded as "black boxes" and their internal decision-making processes are difficult to understand. This is especially important for social event detection, because researchers and decision makers need to clearly understand how the model draws conclusions. By applying structural information theory, the present invention can provide model transparency, making the detection results easier to analyze and understand. This interpretability not only improves the credibility of the model, but also provides important support for subsequent decision-making.

[0067] In order to solve the label dependency problem, the present invention realizes the capability of unsupervised and incremental learning. Existing social event detection methods usually require a large amount of labeled data, which limits the flexibility and scope of their application. Through multimodal contrastive learning and adaptive incremental learning methods, the present invention can quickly identify emerging social events without explicit labels. This capability enables the model to adapt to the ever-changing social media environment and quickly respond to new events and trends.

[0068] The present invention can reduce the impact of data noise on model performance. Social media data is often accompanied by a large amount of noise and irrelevant information, which interferes with the learning process of the model. By introducing a noise filtering mechanism guided by structural entropy, the present invention can effectively identify and filter out low-quality data, thereby improving the robustness of the model when processing social media streams. This noise filtering capability ensures that the model remains efficient in long-term use and avoids performance degradation due to deterioration in data quality.

[0069] The present invention designs an efficient real-time application framework to adapt to the rapid changes in social media data. Social events are often time-sensitive, so real-time detection and analysis become particularly important. The present invention ensures that large-scale social media data can be quickly processed and real-time event detection and analysis can be achieved through reasonable algorithm design and system architecture. This feature gives the present invention broad application potential in multiple fields such as social governance, public opinion analysis, and information dissemination.

[0070] By combining the concepts of adaptive incremental learning and structural entropy, the present invention provides a novel and effective social event detection technology, aiming to improve the effect and interpretability of multimodal social event detection and meet the diverse needs in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A schematic diagram of a framework of a multimodal social event detection method for optimizing structural information of the present invention;

[0072] Figure 2 Schematic diagram of the adaptive incremental learning model framework guided by structural entropy in an embodiment of the present invention;

[0073] Figure 3 Schematic diagram of a framework of a multimodal event encoder in an embodiment of the present invention;

[0074] Figure 4 Schematic diagram of the framework of an unsupervised graph clustering model guided by deep structural entropy in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings.

[0076] In this embodiment, see Figure 1-Figure 4 As shown, the present invention proposes a multimodal social event detection method with structural information optimization, comprising the steps of:

[0077] S10, obtaining multimodal social media data and preprocessing the data;

[0078] S20, after being processed by an adaptive incremental learning model guided by the multimodal social media data structure entropy, outputs the detected social events and their features, including the steps of:

[0079] S201, initial model pre-training: the multimodal event encoder introduced in the adaptive incremental learning model guided by the structure entropy of the multimodal social media data after data pre-processing is used to construct positive samples through a contrastive learning method and train the loss to improve the efficiency of social event detection and obtain the feature representation of the message;

[0080] S202, perform multimodal social event detection in the open world: After obtaining the feature representation of the message, the message is modeled as a graph G M , the graph contains two types of edges: edges based on feature similarity E s and edges E based on shared attributes a In order to cluster messages without determining the total number of events in advance, a hierarchical clustering algorithm based on structural entropy is introduced. The input of the algorithm is the constructed message graph G M ; First, perform hierarchical clustering and generate a hierarchical clustering tree T h ; Then, for T hFor each layer, the corresponding clustering results need to be calculated, and the two-dimensional SE under each clustering result needs to be calculated; the clustering result with the smallest two-dimensional SE is selected as the final output of the algorithm;

[0081] S203, adaptive incremental learning: first, an adaptive noise filter is used to process the dynamic updates and noise of social media data; then, incremental learning is performed to retain information related to the event and output the detected social events and their characteristics;

[0082] S30, inputting the social event information and its features output by the adaptive incremental learning guided by the structural entropy into the unsupervised graph clustering model guided by the deep structural entropy to obtain a clustering result; comprising the steps of:

[0083] S301, structural quantization: first perform probability clustering assignment on the input data;

[0084] S302, structure learning layer: first receive node features X and adjacency matrix A g As input; the original graph structure is then enhanced by the available features X, and the graph is dynamically optimized and updated during the training process;

[0085] S303, cluster assignment layer, uses the initial embedding and adjacency matrix to learn the soft assignment and embedding of nodes, and updates the graph structure and cluster embedding after aggregation, and finally outputs the clustering results, showing the grouping of related events and their structural information.

[0086] As an optimization solution of the above embodiment, the multimodal social media data includes: collecting multimodal data from multiple social media platforms, obtaining the latest news reports and user discussions through a news aggregation platform, and obtaining academic data sets and open source data sets;

[0087] Social media platforms are important data sources, and multimodal data can be automatically collected from multiple social media platforms. Not only can text information, such as user-posted status, comments, and forwarded content, but also multiple forms of data such as images, videos, and audio related to these text information can be captured.

[0088] News websites and blogs are also key data sources. Through news aggregation platforms, the latest news reports and user discussions can be obtained. These contents enhance the multi-angle understanding of events and provide a broader perspective and in-depth analysis of social media data.

[0089] Academic and open source datasets provide support for benchmarking models. Using existing public datasets can ensure the effectiveness of the method, while annotated data obtained from platforms such as Kaggle can help with model training and verification.

[0090] Other data sources such as open APIs and third-party data providers also play an important role. Obtaining data in specific fields (such as weather and traffic) through these channels provides more comprehensive context for event detection systems.

[0091] Data preprocessing includes data cleaning, data conversion, feature extraction and data integration in sequence.

[0092] Data cleaning: This step involves removing duplicate data and irrelevant information to ensure the uniqueness and relevance of each piece of data. For social media data, common noise includes advertisements, spam, and meaningless content, which need to be removed by designing effective filtering rules. In addition, missing values ​​need to be processed and missing data needs to be interpolated or deleted to ensure data integrity.

[0093] Data conversion: In this phase, the raw data will be converted into a format suitable for model processing. For example, text data can be processed by tokenization, stop word removal, and stemming to extract meaningful keywords. In addition, word embedding techniques (such as Word2Vec or GloVe) can be used to convert text into vector representations for subsequent analysis. For image and video data, image scaling, normalization, and feature extraction may be required to ensure a uniform input format.

[0094] Feature extraction and selection: This process aims to extract meaningful features from multimodal data and construct feature vectors. For example, emotional features can be extracted from text, visual features can be extracted from images, and dynamic features can be extracted from videos. Feature selection techniques (such as principal component analysis or LASSO regression) can further reduce feature dimensions and retain the features that are most helpful to the model, thereby improving computational efficiency and model performance.

[0095] Data integration: Data from different modalities need to be integrated into a unified structure for subsequent processing. This can be achieved by building a graph structure, where nodes represent events, users or keywords, and edges represent the relationship between them. The integrated data will provide a comprehensive perspective for social event detection, ensuring that the invention can make full use of various types of information.

[0096] As an optimization scheme of the above embodiment, in order to make the adaptive incremental learning model guided by structural entropy not restricted by the supervision labels in the training data, a contrastive learning strategy without negative samples is adopted. In the adaptive incremental learning model guided by structural entropy, a multimodal large language model is used to perform visual question and answer on the original multimodal message, so as to reconstruct the message and generate positive samples using the multimodal large language model; these positive samples are constructed from three aspects: event type, event theme and image title, so as to more comprehensively represent the message; the event type, as a coarse-grained representation of the message, helps to understand the event category described in the message, such as sports, natural disasters or terrorist incidents; the event theme provides a finer-grained understanding, such as the impact of hurricanes on coastal residents or the post-disaster reconstruction after an earthquake; the image title enhances the comprehensiveness of the message representation by supplementing the image information.

[0097] As an optimization solution of the above embodiment, Figure 3 As shown, the multimodal event encoder comprises the steps of:

[0098] First, obtain text embedding through a pre-trained language model;

[0099] The specific formula is:

[0100] E t =PLM(text);

[0101] Among them, E t is a text embedding of dimension 384, and the text can be T or or T en ;

[0102] For image features, a frozen pre-trained visual transformer is used for feature extraction;

[0103] The specific formula is:

[0104] E i =ViT(I or );

[0105] Among them, E i is an image feature of dimension 384;

[0106] Subsequently, a bidirectional cross-attention mechanism is used to fuse cross-modal features and generate image-to-text and text-to-image attention scores;

[0107] The specific formula is:

[0108]

[0109] Among them, Q T and Q I are the query matrices of the target and input, respectively, K Iand K T are the key matrices of input and target, V I and V T are the value matrices of input and target, respectively, and d k is the dimension of the key matrix.

[0110] The attention score is compositely mapped to the feature to generate new features for text and images;

[0111] The specific formula is:

[0112]

[0113] Finally, the new features of text and image are fused through learnable weighted fusion to obtain event embedding;

[0114] The formula is:

[0115] F message =α·F text +(1-α)·F image ;

[0116] In this process, α is a learnable parameter and satisfies O<α<1.

[0117] Next, a predictor is introduced to maximize the output of the multimodal event encoder on the original data and enhanced data through contrastive learning, which is expressed as this loss function:

[0118]

[0119] Among them, F or is the multimodal event encoder output on the original data side after being processed by the predictor, and F en It is the output of the enhanced data side.

[0120] As an optimization solution of the above embodiment, in the multimodal social event detection in the open world, for E s , select another node that is most similar to the pre-trained language model output of the given node for connection, and set the weight of the edge to the cosine similarity between the two nodes; for E a , then nodes with shared properties are connected. For example, posts from the same user, posts with the same hashtag (#), and posts that mention the same user (@).

[0121] As an optimization solution of the above embodiment, the adaptive noise filter processes the dynamic update and noise of social media data. N For each cluster in N, we first calculate the entropy of each node in the cluster and add it to the set N. E Then, according to N EThe average value of the threshold a is calculated, where is composed of all the E The set consists of elements with the average value; finally, nodes whose entropy exceeds this threshold are considered noise and are removed.

[0122] During the incremental learning process, a dedicated distillation learning module is used to consolidate previous knowledge and freeze the parameters of the multimodal event encoder and predictor from the previous model as a teacher model to transfer previous knowledge. The remaining two multimodal event encoders with shared weights are set the same as in the previous stage. During the incremental learning training process, 50% of the clean data is used for knowledge distillation and knowledge transfer with the previous model, and the remaining 50% is input into the multimodal event encoder with shared weights to learn new knowledge.

[0123] In terms of the loss of incremental learning, the adaptive incremental learning stage includes two losses: one is the distillation loss L2, and the other is the contrast loss L3; the distillation loss and the contrast loss are unified, and the specific formula is:

[0124]

[0125] Among them, F pre represents the output of the previous model, F or is the multimodal event encoder output on the original data side after being processed by the predictor, and F en is the output of the enhanced data side;

[0126] Finally, the total loss L In is the sum of the above two losses:

[0127] L In =β·L2+(1-β)·L3;

[0128] Among them, β is a hyperparameter that controls the weights of the two losses and satisfies O<β<1.

[0129] As an optimization solution of the above embodiment, Figure 4 As shown, when probability clustering is assigned, the assignment matrix is ​​used;

[0130] Allocation matrix:

[0131]

[0132] C k =S h ·S h-1 ...S k+1 ;

[0133] Among them, H sa (G) represents the total structural entropy of the coding tree; H sa(G; k) represents the structural entropy of the kth layer, with N k vertices, and the total height of the coding tree is h; S k represents the allocation matrix between the k-th layer vertices and the k-1-th layer vertices; C k It is the direct assignment matrix between nodes and vertices at the kth level, indicating the probability that each node belongs to the kth level cluster;

[0134] Redefine the calculation formula of the cut edge and volume as follows:

[0135] vol k [i]=D(C k ) i ;

[0136]

[0137] Among them, the volume vol of vertex i in the kth layer is k [i] is the sum of the assignment probabilities of all node degrees; D is the degree vector of all nodes, calculated by the edge weight matrix W, the formula is:

[0138]

[0139] in, is a vector of length N with all 1s, weight matrix W; d matrix C k The subscript i of represents the i-th column vector, which indicates the direct clustering probability of the N nodes in the graph for the k-th layer clustering; Represents the tangent value of the i-th vertex in the k-th layer, calculated as the volume vol of vertex i k [i] and its internal volume The difference between the internal volume represents the sum of the weighted probabilities of all edges, where probability refers to the possibility that two nodes connected by an edge belong to the same cluster i in the kth layer;

[0140] The structural entropy is calculated using the formula:

[0141]

[0142] Among them, N k is the number of vertices in the kth layer of the coding tree, C O is the original clustering probability, D In the soft assignment method, the original volume associated with a single parent vertex is replaced by the probabilistic sum of the volumes of all parent vertices.

[0143] As an optimization solution of the above embodiment, Figure 4 As shown, the structure learning layer includes:

[0144] Feature Mapping: Use a multi-layer perceptron to map node features to a dense space of lower dimension:

[0145] A f =KNN(MPL(X;θ f );K);

[0146] Among them, MPL is a multi-layer perceptron, Θ f is the multilayer perceptron parameter, and KNN is the K nearest neighbor operation, K represents a K value in the K nearest neighbor algorithm;

[0147] Neighbor selection: Select the first K neighbors for each node through the K nearest neighbor algorithm and construct an adjacency matrix A f ;

[0148] Adjust the adjacency matrix: To ensure the symmetry of the adjacency matrix, adjust the adjacency matrix of the attribute graph:

[0149]

[0150] Fuse adjacency matrix: Combine the adjacency matrix of the original graph with the adjacency matrix of the attributed graph to obtain an enhanced graph:

[0151] W′=A g +β f A f ;

[0152] Among them, β f is a hyperparameter used to control the weight of the attribute graph in the fusion, and W′ is the new adjacency matrix, A g is the adjacency matrix of the original graph.

[0153] As an optimization solution of the above embodiment, Figure 4 As shown, the clustering assignment layer includes an embedding learner, a soft assignment learner, and an aggregator;

[0154] Embedding learner: The embedding learner is based on a graph neural network architecture; the embedding learner performs a linear transformation on the initial embedding to map it to the embedding space; it then aggregates the average embeddings of connected nodes to generate new node embeddings and applies an activation function;

[0155] Embedding learner mathematical representation:

[0156] H=GNN emb (E, W; Θ1)=ReLU(mean(WEΘ1));

[0157] The learnable parameters of the linear transformation are Represents, embedding learner GNN emb , initial embedding E.

[0158] Soft Assignment Learner: Soft Assignment Learner extends the attention mechanism in the graph neural network architecture;

[0159] The soft assignment learner linearly transforms the initial embedding to the cluster space; it calculates the attention matrix for each edge as the aggregation weight and performs non-average embedding aggregation to obtain the cluster embedding; the calculation of attention involves linearly transforming the connection embeddings of the nodes at both ends of the edge to the weight space, followed by activation and normalization.

[0160] The mathematical representation of the soft assignment learner is as follows:

[0161]

[0162] GNN ass is the soft assignment learner, Γ is the attention matrix, and the learnable parameters of the linear transformation in the attention mechanism are Indicates that e i and e j Respectively represent the node v i and node v j The learnable parameters of the linear transformation in the attention mechanism are used express.

[0163] Aggregator: The goal of the aggregator is to update the cluster embedding and adjacency matrix; the cluster embedding is calculated using the probability sum of the node embedding; the new adjacency matrix is ​​a combination of the property graph adjacency matrix and the structure graph adjacency matrix; the cluster assignment layer completes the dynamic update of the node embedding and graph structure; and finally the clustering result is output, showing the grouping of related events and their structural information.

[0164] The cluster embedding is calculated using the probability sum of node embeddings, expressed as:

[0165]

[0166] The new adjacency matrix is ​​formed by combining the attribute graph adjacency matrix and the structure graph adjacency matrix. The formula is:

[0167]

[0168] in The multilayer perceptron representing the clustering can learn parameters, and the new weighted adjacency matrix is:

[0169] W′=A′ g +β f A′ f .

[0170] This patent aims to invent an innovative multimodal social event detection technology that improves the accuracy, interpretability and adaptability of event detection by combining the advantages of structural information optimization and deep learning.

[0171] The present invention proposes the design of a multimodal event encoder: The design of a multimodal event encoder is an important component of the present invention, which aims to effectively fuse data from different sources (such as text, images, videos, etc.). In specific implementation, the architecture of the encoder includes an input layer, a feature extraction layer, and an output layer. Through the contrast learning framework, the encoder can integrate data of different modalities into a unified representation. In addition, the loss function and optimization strategy selected in the model training process will also be elaborated in detail to demonstrate its effectiveness and uniqueness in multimodal data fusion.

[0172] The present invention proposes a clustering algorithm guided by structural information: The clustering algorithm is one of the core innovations of the present invention, especially in terms of optimizing the clustering process using structural entropy. The working principle of the algorithm is based on structural information theory, and the specific implementation includes how to calculate and optimize structural entropy in the clustering process, and use these calculation results to improve the clustering effect. The evaluation indicators of the clustering results will also be described in detail to ensure the adaptability and versatility of the algorithm when processing different data sets, thereby enhancing its effect in practical applications.

[0173] This invention proposes an adaptive incremental learning mechanism: The innovativeness of the adaptive incremental learning mechanism is that it can effectively realize incremental learning under unsupervised conditions. The implementation process of the mechanism will be described in detail, including the introduction of new knowledge, the retention of old knowledge, and the specific steps and strategies for model updating. The design of this mechanism can not only reduce manual intervention, but also ensure the effectiveness and stability of the model in long-term use, further enhancing the application value of the technology.

[0174] The present invention proposes a technique for dynamically filtering noise samples, using a structural entropy-guided method to improve the robustness of the model. Specifically, the implementation details of the technique will include steps on how to identify and remove noise samples, as well as the implementation of the filtering algorithm.

[0175] The present invention proposes a method for calculating deep structural entropy: The method for calculating deep structural entropy is an important part of the present invention, the purpose of which is to ensure the uniqueness and technological advancement of the calculation process. The specific implementation includes converting structural entropy into a continuously differentiable form, constructing a probability matrix, and how to apply these calculations in the clustering process.

[0176] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multimodal social event detection method with structural information optimization, characterized in that: Includes steps: S10, obtaining multimodal social media data and preprocessing the data; S20, after being processed by an adaptive incremental learning model guided by the multimodal social media data structure entropy, outputs the detected social events and their features, including the steps of: S201, initial model pre-training: the multimodal event encoder introduced in the adaptive incremental learning model guided by the structure entropy of the pre-processed multimodal social media data is used to construct positive samples through a contrastive learning method, train the loss, and obtain the feature representation of the message; S202, perform multimodal social event detection in the open world: After obtaining the feature representation of the message, the message is modeled as a graph G M , the graph contains two types of edges: edges based on feature similarity E s and the edge E based on the shared attribute a ; A hierarchical clustering algorithm based on structural entropy is introduced. The input of the algorithm is the constructed message graph G M ; First, perform hierarchical clustering and generate a hierarchical clustering tree T h ; Then, for T h For each layer, the corresponding clustering results need to be calculated, and the two-dimensional SE under each clustering result needs to be calculated; the clustering result with the smallest two-dimensional SE is selected as the final output of the algorithm; S203, adaptive incremental learning: first, an adaptive noise filter is used to process the dynamic updates and noise of social media data; then, incremental learning is performed to retain information related to the event and output the detected social events and their characteristics; S30, inputting the social event information and its features output by the adaptive incremental learning guided by the structural entropy into the unsupervised graph clustering model guided by the deep structural entropy to obtain a clustering result; Includes steps: S301, structural quantification: first perform probability clustering assignment on the input data; S302, structure learning layer: first receive node features X and adjacency matrix A g As input; the original graph structure is then enhanced by the available features X, and the graph is dynamically optimized and updated during the training process; S303, the clustering assignment layer, uses the initial embedding and adjacency matrix to learn the soft assignment and embedding of nodes, and updates the graph structure and cluster embedding after aggregation, and finally outputs the clustering results, showing the grouping of related events and their structural information.

2. According to the structure information optimized multimodal social event detection method of claim 1, it is characterized in that: Multimodal social media data includes: collecting multimodal data from multiple social media platforms, obtaining the latest news reports and user discussions through news aggregation platforms, and obtaining academic data sets and open source data sets; Data preprocessing includes data cleaning, data conversion, feature extraction and data integration in sequence.

3. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: A contrastive learning strategy without negative samples is adopted. In the adaptive incremental learning model guided by structural entropy, a multimodal large language model is used to conduct visual question-answering on the original multimodal message, so as to reconstruct the message and generate positive samples using the multimodal large language model. These positive samples are constructed from three aspects: event type, event theme and image title. The event type, as a coarse-grained representation of the message, helps to understand the event category described by the message; the event subject provides a more fine-grained understanding; and the image title enhances the comprehensiveness of the message representation by supplementing the image information.

4. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: The multimodal event encoder comprises the steps of: First, text embedding is obtained through a pre-trained language model. For image features, a frozen pre-trained visual transformer is used for feature extraction. Then, a bidirectional cross-attention mechanism is used to fuse cross-modal features and generate image-to-text and text-to-image attention scores. The attention scores are compositely mapped with the features to generate new features for text and image. Finally, the new features of text and image are fused through learnable weighted fusion to obtain event embedding; Next, a predictor is introduced to maximize the output of the multimodal event encoder on the original data and enhanced data through contrastive learning, which is expressed as this loss function: Among them, F or is the multimodal event encoder output on the original data side after being processed by the predictor, and F en It is the output of the enhanced data side.

5. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: For E s , select another node that is most similar to the pre-trained language model output of the given node for connection, and set the weight of the edge to the cosine similarity between the two nodes; for E a , then connect the nodes with shared attributes.

6. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: Adaptive noise filter, when dealing with dynamic updates and noise in social media data, for graph G N For each cluster in N, we first calculate the entropy of each node in the cluster and add it to the set N. E Then, according to N E The average value of the threshold a is calculated, where is composed of all the E The set consists of elements with the average value; finally, nodes whose entropy exceeds this threshold are considered noise and are removed.

7. The multimodal social event detection method with structural information optimization according to claim 6, characterized in that: During the incremental learning process, a distillation learning module is used to consolidate previous knowledge and freeze the parameters of the multimodal event encoder and predictor from the previous model as a teacher model to transfer previous knowledge. The remaining two shared-weight multimodal event encoders are set the same as in the previous stage. During the incremental learning training process, 50% of the clean data is used for knowledge distillation and knowledge transfer with the previous model, and the remaining 50% is input into the shared-weight multimodal event encoder to learn new knowledge. In terms of the loss of incremental learning, the adaptive incremental learning stage includes two losses: one is the distillation loss L2, and the other is the contrast loss L3; the distillation loss and the contrast loss are unified, and the specific formula is: Among them, F pre represents the output of the previous model, F or is the multimodal event encoder output on the original data side after being processed by the predictor, and F en is the output of the enhanced data side; Finally, the total loss L In is the sum of the above two losses: L In =β·L2+(1-β)·L3; Among them, β is a hyperparameter that controls the weights of the two losses and satisfies 0<β<1.

8. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: When assigning probabilistic clusters, an assignment matrix is ​​used; Allocation matrix: C k =S h ·S h-1 …S k+1 ; Among them, H sa (G) represents the total structural entropy of the coding tree; H as (G; k) represents the structural entropy of the kth layer, with N k vertices, and the total height of the coding tree is h; S k represents the allocation matrix between the k-th layer vertices and the k-1-th layer vertices; C k It is the direct assignment matrix between nodes and vertices at the kth level, indicating the probability that each node belongs to the kth level cluster; Redefine the calculation formula of the cut edge and volume as follows: vol k [i]=D(C k ) i ; Among them, the volume vol of vertex i in the kth layer is k [i] is the sum of the probability of assigning all node degrees; D is the degree vector of all nodes, calculated by the edge weight matrix E, the formula is: in, is a vector of length N with all 1s, weight matrix W; d matrix C k The subscript i of represents the i-th column vector, which indicates the direct clustering probability of the N nodes in the graph for the k-th layer clustering; Represents the tangent value of the i-th vertex in the k-th layer, calculated as the volume vol of vertex i k [i] and its internal volume The difference between the internal volume represents the sum of the weighted probabilities of all edges, where probability refers to the possibility that two nodes connected by an edge belong to the same cluster i in the kth layer; The structural entropy is calculated using the formula: Among them, N k is the number of vertices in the kth layer of the coding tree, C 0 is the original clustering probability, In the soft assignment method, the original volume associated with a single parent vertex is replaced by the probabilistic sum of the volumes of all parent vertices.

9. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: The structure learning layer includes: Feature Mapping: Use a multi-layer perceptron to map node features to a dense space of lower dimension: A f =KNN(MPL(X;Θ f );K); Among them, MPL is a multi-layer perceptron, Θ f is the multilayer perceptron parameter, and KNN is the K nearest neighbor operation, K represents a K value in the K nearest neighbor algorithm; Neighbor selection: Select the first K neighbors for each node through the K nearest neighbor algorithm and construct the adjacency matrix A f ; Adjust the adjacency matrix: Adjust the adjacency matrix of the attribute graph: Fuse adjacency matrix: Combine the adjacency matrix of the original graph with the adjacency matrix of the attributed graph to obtain an enhanced graph: W′=A g +b f A f ; Among them, β f is a hyperparameter used to control the weight of the attribute graph in the fusion, and W′ is the new adjacency matrix, A g is the adjacency matrix of the original graph.

10. The multimodal social event detection method with structural information optimization according to claim 1, characterized in that: The clustering assignment layer includes an embedding learner, a soft assignment learner, and an aggregator; Embedding Learner: The embedding learner is based on the graph neural network architecture; The embedding learner performs a linear transformation on the initial embedding, mapping it to the embedding space; The average embeddings of connected nodes are then aggregated to generate new node embeddings, and an activation function is applied; Soft Assignment Learner: Soft Assignment Learner extends the attention mechanism in the graph neural network architecture; The soft assignment learner linearly transforms the initial embedding into the cluster space; It computes the attention matrix of each edge as the aggregation weight and performs non-average embedding aggregation to obtain clustered embeddings; The calculation of attention involves linearly transforming the connection embedding of the nodes at both ends of the edge into the weight space, followed by activation and normalization; Aggregator: The goal of the aggregator is to update the cluster embedding and adjacency matrix; the cluster embedding is calculated using the probability sum of node embeddings; the new adjacency matrix is ​​a combination of the attribute graph adjacency matrix and the structure graph adjacency matrix; The clustering assignment layer completes the dynamic update of node embedding and graph structure; The final clustering result is output, showing the grouping of related events and their structural information.

Images