Intelligent event analysis method and system based on AI

Abstract

The invention relates to an intelligent event analysis method and system based on AI, and the method comprises the following steps: obtaining multi-modal original data related to an event, and extracting the feature representation of each modal data; the method comprises the following steps of: extracting features of an event, constructing a hierarchical multi-modal feature graph containing intra-modal sub-graphs and a cross-modal association graph on the basis of the extracted features, sequentially performing intra-modal GAT aggregation and modal perception cross-modal attention calculation by utilizing a graph attention network to obtain global features after cross-modal aggregation, and finally outputting event type probability distribution through a classification layer and identifying event types; and based on the identified event type and the multi-modal feature graph, constructing a layered multi-modal event knowledge graph, analyzing a logical relationship and an evolution path between the events by using time sequence modeling and a causal reasoning model, and outputting an analysis result. Compared with the prior art, the method has the advantages that the multi-modal event type identification precision and accuracy can be improved, the analysis efficiency is improved, and the like.

Description

Technical Field

[0001] This invention relates to the field of big data analytics, and in particular to an AI-based intelligent event analysis method and system. Background Technology

[0002] Currently, several event analysis systems based on rules or traditional machine learning methods exist both domestically and internationally. These systems detect and classify events from text, images, or video data through keyword matching, pattern recognition, or statistical analysis. Some systems employ deep learning models (such as CNN, RNN, and Transformer) for preliminary feature extraction and event recognition.

[0003] Chinese patent CN119557603A discloses a method, system, and apparatus for event analysis based on generative tasks and multimodal approaches. This method acquires multimodal data (text, images, and audio), utilizes a large generative model to generate structured event information, constructs an event graph, and uses a cross-modal attention mechanism and graph neural network for feature fusion and optimization to generate sentiment feature vectors and event prediction graphs for quickly and accurately grasping the development trend of events. However, the cross-modal attention mechanism of this method uses a single-level interaction logic, making it difficult to coordinate the inherent differences between different modal features, and modality-specific information is easily ignored during the interaction process. Furthermore, the event graph construction relies on structured information generated by a large model, resulting in a break in the information connection with the previous feature processing stages, making it difficult to guarantee information consistency. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an AI-based intelligent event analysis method and system.

[0005] The objective of this invention can be achieved through the following technical solutions: According to a first aspect of the present invention, an AI-based intelligent event analysis method is provided, the method comprising the following steps: Acquire event-related multimodal raw data and extract feature representations for each modality; Based on the extracted features, a hierarchical multimodal feature map containing intramodal subgraphs and cross-modal association graphs is constructed. A graph attention network is used to perform intramodal GAT aggregation and modality-aware cross-modal attention calculation in sequence to obtain the global features after cross-modal aggregation. Finally, the event type probability distribution is output through the classification layer to identify the event type. Based on the identified event types and multimodal feature maps, a hierarchical multimodal event knowledge graph is constructed, and the logical relationships and evolution paths between events are analyzed using temporal modeling and causal reasoning models, and the analysis results are output.

[0006] The modal subgraphs are determined based on the number of modalities in the acquired multimodal raw data. Each modality corresponds to one modal subgraph. For the text modality, the nodes of the corresponding modal subgraph are defined as semantic units of event-related text, and their features are semantic vectors extracted by a pre-trained language model. For the image modality, the nodes of the corresponding modal subgraph are defined as key visual objects in the image, and their features are visual vectors extracted by a pre-trained visual model. For the audio modality, the nodes of the corresponding modal subgraph are defined as acoustic events, and their features are acoustic vectors extracted by MFCC. For the video modality, the nodes of the corresponding modal subgraph are defined as video keyframes, and their features are temporal visual vectors extracted by a pre-trained visual feature extraction model. Within the same modal subgraph, the edge weights between nodes are calculated based on feature similarity.

[0007] The nodes of the cross-modal association graph are defined as core nodes connecting different modal subgraphs, namely, alignment units corresponding to semantic units, key visual objects, acoustic events, and video keyframes. The features are the average concatenation of the node features of the corresponding intra-modal subgraphs. The edge weights between nodes of different intra-modal subgraphs and nodes of the cross-modal association graph are calculated based on the cross-modal feature alignment score.

[0008] The intra-modal GAT aggregation specifically involves: independently performing GAT calculations on each intra-modal subgraph, aggregating local correlation features within the modality, and preserving the inherent characteristics of each modality, including the following steps: Intramodal attention coefficient calculation: For modal m Modal inner subgraph, compute nodes i For neighboring nodes j Attention coefficient: , in, For modality m The characteristic projection matrix, For modality m Next node i The feature is represented by ||, which is the feature concatenation operation. For modality m Next node i The set of neighboring nodes; Intramodal aggregated feature output: Calculates the feature for each modality based on the intramodal attention coefficient. m Modal inner subgraph nodes i aggregation features : , Where σ is the activation function; The global aggregated features of each modality are obtained by combining the aggregated features of all nodes within the modality: , in, For modality m The number of nodes, For modality m Global aggregation features.

[0009] The modality-aware cross-modal attention computation specifically includes the following steps: Modal importance weight calculation: Based on prior knowledge of the current event scenario and real-time feature distribution, the importance weight of each modality is dynamically calculated. , in, For modality m Importance weights M For modal sets, For modality m The significance score, It is a single-layer perceptron. For modality m Importance weights obtained through intramodal GAT aggregation; Cross-modal attention coefficient calculation: Based on the cross-modal association graph, the cross-modal attention coefficient is calculated by fusing modal importance weights. For mode m Nodes in the modal inner subgraph i With mode k Nodes in the modal inner subgraph j The cross-modal attention coefficients between the two modes are first calculated by the modality. m Nodes in the modal inner subgraph i Nodes in the cross-modal association graph c First attention coefficient between : , in, For modality m Nodes in the modal inner subgraph i The aggregation characteristics, For nodes c Features , Represents the cross-modal projection matrix. For modality m nodes i Nodes in the cross-modal association graph c Edge weights between them Representation and Modality m nodes i The core node set of the associated cross-modal association graph, where || represents the feature concatenation operation; Subsequently, the nodes in the cross-modal association graph are calculated.c With mode k Nodes in the modal inner subgraph j The second attention coefficient between : , in, For modality k Importance weights For modality k Nodes in the modal inner subgraph j The aggregation characteristics, For modality k The cross-modal projection matrix, For nodes in the cross-modal association graph c Associated modes k The set of nodes, For nodes in the cross-modal association graph c With mode k Nodes in the modal inner subgraph j Edge weights between them; Calculate the mode based on the first attention coefficient and the second attention coefficient. m The next node i and modality k The next node j Cross-modal attention coefficients ; , Cross-modal aggregated feature output: Global features are obtained by aggregating cross-modal attention coefficients. for: , in, This represents the activation function. , indicating the relationship between nodes in the cross-modal association graph i Indirectly related modes k Node set.

[0010] The event types are a predefined closed set, including lifecycle events, interaction events, transaction events, business activities, judicial events, military activities, and socio-political events.

[0011] The loss function of the graph attention network is defined as follows: , in, For cross-entropy loss, To predict the event type output by the graph attention network, For the event type truth value, The regularization coefficient is . , Modal m , k The corresponding set of nodes, M For modal sets, For modality m Nodes in the modal inner subgraph i The aggregation characteristics, For modality k Nodes in the modal inner subgraph j Aggregation characteristics.

[0012] The hierarchical multimodal event knowledge graph adopts a three-layer architecture, which is aligned with the hierarchical multimodal feature graph. The event core layer corresponds to the event type identification result, the modal entity layer corresponds to the intramodal subgraph, which stores the entities of a single modality and the intramodal associations, and the cross-modal association layer corresponds to the cross-modal association graph, which connects entities of different modalities and stores the cross-modal alignment relationship.

[0013] The construction of the hierarchical multimodal event knowledge graph includes the following steps: Define the entity types, relation types, and attributes of a hierarchical multimodal event knowledge graph. For the event core layer, the entity type is event entity; for the modal entity layer, the entity type is modal entity, including text entities, image entities, audio entities, and video entities; for the cross-modal association layer, the entity type is cross-modal association entity. The relation types include intra-modal relations, cross-modal relations, and event-entity association relations. Based on the intermediate processing and output of hierarchical multimodal feature maps and graph attention networks, modal entities and cross-modal associated entities are extracted from the hierarchical multimodal feature maps, and relation extraction is performed in combination with semantic rules. Attribute information is obtained through feature sequence mapping.

[0014] According to a second aspect of the present invention, an AI-based event intelligent analysis system is provided for implementing the method, the system comprising: Data acquisition and feature extraction module: used to acquire event-related multimodal raw data and extract feature representations for each modality of data; Event type identification module: It is used to construct a hierarchical multimodal feature map containing intramodal subgraphs and cross-modal association graphs based on the extracted features. It uses a graph attention network to perform intramodal GAT aggregation and modality-aware cross-modal attention calculation in sequence to obtain the global features after cross-modal aggregation. Finally, it outputs the event type probability distribution through the classification layer to identify the event type. Event Analysis Module: Based on the identified event types and multimodal feature maps, it constructs a hierarchical multimodal event knowledge graph, and uses time-series modeling and causal reasoning models to analyze the logical relationships and evolution paths between events, and outputs the analysis results.

[0015] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0016] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) The cross-modal attention architecture of the present invention adopts a hierarchical interaction logic and constructs a hierarchical multimodal feature map containing intramodal subgraphs and cross-modal association graphs. It can coordinate the inherent differences of different modal features and retain the exclusive information of each modality. It avoids the dilution of effective information during the interaction process, improves the accuracy of cross-modal feature fusion, and can still ensure a certain recognition accuracy in scenarios where some modalities are missing. (2) The hierarchical multimodal event knowledge graph constructed in this invention directly reuses the quantitative features of the previous feature processing stage, realizes the information connection between the feature layer and the knowledge layer, effectively ensures the consistency and continuity of information transmission, and reduces the error caused by information gaps. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0021] This embodiment provides an AI-based intelligent event analysis method, such as... Figure 1 As shown, the method includes the following steps: S1: Obtain the event-related multimodal raw data and extract the feature representation of each modality.

[0022] This step involves collecting data in real time from various data sources (such as social media, news websites, surveillance videos, sensors, etc.) and performing preprocessing operations such as cleaning, noise reduction, and standardization.

[0023] Pre-trained deep learning models (such as BERT for text, ResNet for images, and 3D-CNN for videos) are used to extract feature representations for each modality of data.

[0024] S2. Based on the extracted features, a hierarchical multimodal feature map containing intramodal subgraphs and cross-modal association graphs is constructed. The graph attention network is used to perform intramodal GAT aggregation and modality-aware cross-modal attention calculation in sequence to obtain the global features after cross-modal aggregation. Finally, the event type probability distribution is output through the classification layer to identify the event type.

[0025] 1. Definition of event types The classification system for event types can be customized according to specific application areas (such as news analysis, financial risk control, and medical diagnosis). A common and widely referenced standard is the ACE (Automatic Content Extraction) event standard. Table 1 shows some typical event types.

[0026] Table 1 In specific projects, event types are usually a predefined closed set. The task of this step is to determine whether the input multimodal data belongs to one or more of these types (multi-label classification).

[0027] 2. Construction of hierarchical multimodal feature maps Based on the extracted modal features (text semantic features, image visual features, audio acoustic features, and video frame temporal features), a hierarchical multimodal feature map is constructed, including intra-modal subgraphs and cross-modal association graphs, as follows: 1) Definition of node V 11) The modal subgraph is determined based on the number of modes in the acquired multimodal raw data, with each mode corresponding to a modal subgraph.

[0028] Specifically, for the text modality, the nodes of the corresponding subgraph within the modality are defined as semantic units (entities, phrases, sentences) of event-related text, and the features are semantic vectors extracted by a pre-trained language model (such as BERT); for the image modality, the nodes of the corresponding subgraph within the modality are defined as key visual objects in the image (such as people, scenes, and items identified by the YOLO object detection model), and the features are visual vectors extracted by a pre-trained visual model (such as ResNet50); for the audio modality, the nodes of the corresponding subgraph within the modality are defined as acoustic events (such as speech segments, ambient sounds, and explosion sounds), and the features are acoustic vectors extracted by MFCC; for the video modality, the nodes of the corresponding subgraph within the modality are defined as video keyframes, and the features are temporal visual vectors extracted by a pre-trained visual feature extraction model (such as 3D-CNN).

[0029] 12) The nodes of the cross-modal association graph are defined as the core nodes that connect different modal subgraphs, namely the alignment units corresponding to semantic units, key visual objects, acoustic events and video keyframes (such as the "fire" entity in text → the "flame" object in image → the "explosion sound" in audio). The features are the average concatenation of the node features of the corresponding intramodal subgraphs.

[0030] 2) Definition of edge E Within a subgraph of the same modality, edge weights between nodes are calculated based on feature similarity: , in, For modality m Next node i Features For modality m Next node i The neighboring nodes.

[0031] The edge weights between nodes in subgraphs within different modalities and nodes in cross-modal association graphs are calculated based on cross-modal feature alignment scores. These cross-modal feature alignment scores are obtained by calculating cosine similarity using the CLIP model. , in, Cross-modal association nodes c Its characteristics.

[0032] 3. Intramodal GAT aggregation Perform GAT calculation independently on each modal subgraph, aggregate local correlation features within the modality, and preserve the inherent characteristics of each modality, including the following steps: 31. Intramodal attention coefficient calculation: For modal... m Modal inner subgraph, compute nodes i For neighboring nodes j Attention coefficient: , in, For modality m The characteristic projection matrix, For modality m Next node i The feature is represented by ||, which is the feature concatenation operation. For modality m Next node i The set of neighboring nodes.

[0033] 32. Intramodal aggregated feature output: Based on the intramodal attention coefficient, calculate the aggregated feature output for each modality. m Modal inner subgraph nodes i aggregation features : , Where σ is the activation function.

[0034] 33. By combining the aggregated features of all nodes within a mode, the global aggregated features of each mode are obtained: , in, For modality m The number of nodes, For modality m Global aggregation features.

[0035] 4. Modality-aware cross-modal attention computation Specifically, the following steps are included: 41. Modal Importance Weight Calculation: Based on prior knowledge of the current event scenario and real-time feature distribution, dynamically calculate the importance weight of each modality. , in, For modality m Importance weights M For modal sets, For modality m The significance score, It is a single-layer perceptron. For modality m Importance weights obtained through intramodal GAT aggregation.

[0036] 42. Cross-modal attention coefficient calculation: Based on the cross-modal association graph, the cross-modal attention coefficient is calculated by fusing modal importance weights. For mode m Nodes in the modal inner subgraph i With mode k Nodes in the modal inner subgraph j The cross-modal attention coefficients between the two modes are first calculated by the modality. m Nodes in the modal inner subgraph i Nodes in the cross-modal association graph c First attention coefficient between : , in, For modality m Nodes in the modal inner subgraph i The aggregation characteristics, For nodes c Features , Represents the cross-modal projection matrix. For modality m nodes i Nodes in the cross-modal association graph c Edge weights between them Representation and Modality m nodes i The core node set of the associated cross-modal association graph, where || represents the feature concatenation operation; Subsequently, the nodes in the cross-modal association graph are calculated. c With mode k Nodes in the modal inner subgraph j The second attention coefficient between : , in, For modality k Importance weights For modality k Nodes in the modal inner subgraph j The aggregation characteristics, For modality k The cross-modal projection matrix, For nodes in the cross-modal association graph c Associated modes k The set of nodes, For nodes in the cross-modal association graph c With mode k Nodes in the modal inner subgraph j Edge weights between them; Calculate the mode based on the first attention coefficient and the second attention coefficient. m The next node i and modality k The next node j Cross-modal attention coefficients ; , 43. Cross-modal aggregated feature output: Obtain the global features after cross-modal aggregation based on cross-modal attention coefficient aggregation. for: , in, This represents the activation function. , indicating the relationship between nodes in the cross-modal association graph i Indirectly related modes k Node set.

[0037] 5. Event type identification Cross-modal aggregation features Input classification layer, output event type probability distribution: , in, , It is a fully connected layer.

[0038] 6. Loss Function Design In this embodiment, to enhance the consistency of features between modalities, a modality consistency regularization term is introduced based on the cross-entropy loss. The loss function of the graph attention network is defined as follows: , in, For cross-entropy loss, To predict the event type output by the graph attention network, For the event type truth value, The regularization coefficient is . , Modal m , k The corresponding set of nodes, M For modal sets, For modality m Nodes in the modal inner subgraph i The aggregation characteristics, For modality k Nodes in the modal inner subgraph j Aggregation characteristics.

[0039] S3 constructs a hierarchical multimodal event knowledge graph based on the identified event types and multimodal feature maps, and uses temporal modeling and causal reasoning models to analyze the logical relationships and evolution paths between events, and outputs the analysis results.

[0040] In this embodiment, the hierarchical multimodal event knowledge graph adopts a three-layer architecture, which is aligned with the hierarchical multimodal feature graph. The event core layer corresponds to the event type identification result, the modal entity layer corresponds to the intramodal subgraph, stores the entities of a single modality and the intramodal associations, and the cross-modal association layer corresponds to the cross-modal association graph, connects entities of different modalities, and stores the cross-modal alignment relationship.

[0041] S31, Construct a hierarchical multimodal event knowledge graph.

[0042] A1) Define the entity types, relation types, and attributes of a hierarchical multimodal event knowledge graph.

[0043] For the event core layer, the entity type is event entity; for the modal entity layer, the entity type is modal entity, including text entity, image entity, audio entity and video entity; for the cross-modal association layer, the entity type is cross-modal association entity, as shown in Table 2; the relationship types include intra-modal relationship, cross-modal relationship and event-entity association relationship, as shown in Table 3; the attribute definitions are shown in Table 4.

[0044] Table 2 Table 3 Table 4 A2) Based on the intermediate processing and output of hierarchical multimodal feature maps and graph attention networks, modal entities and cross-modal associated entities are extracted from the hierarchical multimodal feature maps, and relation extraction is performed in combination with semantic rules. Attribute information is obtained through feature sequence mapping.

[0045] (1) Entity extraction Event entity extraction: Extract event types from the classification layer output of the graph attention network in step S2, refine event entity attributes (event ID, event category, subcategory, occurrence time / location (if any), identification confidence) in combination with business rules; normalize event types using an event ontology library (such as an emergency event ontology).

[0046] Modal entity extraction: For text modality, entities (people, places, items, event keywords) in text subgraph nodes are extracted based on the BERT-NER model, and entities with feature similarity ≥0.9 are merged, retaining the core name; for image modality, visual objects in image subgraph nodes are extracted based on the YOLO+ visual attribute recognition model, and objects in the same visual region and of the same category are merged, retaining the mean of the feature vector; for audio modality, acoustic events in audio subgraph nodes are extracted based on acoustic event detection models (such as YAMNet), and acoustic events of the same type with temporal overlap ≥80% are merged; for video modality, entities in keyframes are extracted based on 3D-CNN+ temporal clustering, temporal information is associated, and entities of the same type in adjacent keyframes are merged, retaining the temporal interval.

[0047] Cross-modal associated entity extraction: From the nodes of the cross-modal association graph, select alignment units with a cross-modal attention coefficient ≥ 0.7 as core anchor points; aggregate the multimodal entities associated with the anchor points into a single cross-modal associated entity, with the naming rule being "Modal 1 entity - Modal 2 entity - ... (event type)"; perform uniqueness verification to ensure that cross-modal associated entities under the same event are not duplicated.

[0048] (2) Relation extraction Modal relation extraction: for modal m The subgraph is selected by choosing node pairs with modal inner edge weights ≥ 0.7. i , j After filtering, relation type matching is performed. For text modality, the weights of node pairs are associated with semantic relations based on dependency parsing. For image modality, the node pairs are associated with visual relations based on visual scene understanding. For audio modality, the node pairs are associated with acoustic temporal relations based on temporal analysis. Finally, the relation confidence is assigned by normalizing the edge weights to (0-1) as the relation confidence.

[0049] Cross-modal relation extraction uses cross-modal associated entities as intermediate nodes to associate entities of different modalities; modal entity pairs with cross-modal attention coefficients ≥ 0.6 are selected and uniformly labeled as feature alignment relations, with relation confidence based on the cross-modal attention coefficient value; for text-image cross-modal pairs, semantic similarity sub-relationships are supplemented by CLIP alignment scores.

[0050] Event-entity relationship extraction: Select modalities with a modality importance weight ≥ 0.5 as core modalities; within the core modalities, filter entities whose aggregated features and event type semantic similarity are ≥ 0.8; label event entities with core entities as "triggering" relationships and with other entities as "containing" relationships; finally, assign confidence scores, where the confidence score for triggering relationships = modality importance weight × semantic similarity, and the confidence score for containing relationships = modality importance weight.

[0051] (3) Attribute filling Vector feature processing: For high-dimensional vectors such as intramodal aggregated features and cross-modal attention coefficients, PCA is used to reduce the dimensionality to 64 dimensions before serialization and storage (or the original vectors are stored directly to adapt to subsequent inference needs). Numerical feature processing: Scalar features such as event confidence, modal importance weight, and edge weight are directly stored as floating-point numbers; Text feature processing: Standardize and store text information such as entity name, event type, and relationship type (e.g., unify to UTF-8 encoding); Temporal feature processing: Supplementing video and audio entities with temporal attributes (such as "start timestamp" and "end timestamp").

[0052] S32, Temporal Modeling and Representation Learning The original event graph is discrete and symbolic. In order for the model to perform inference, it needs to be transformed into a continuous vector representation and incorporate key time information.

[0053] Specifically, the following steps are included: Graph representation learning: Using graph neural networks (GNNs), such as graph convolutional networks (GCNs) and graph attention networks (GATs), to learn vector representations (embeddings) of events and entities.

[0054] The core idea of ​​GNN is that the representation of a node is determined by its own features and the features of its neighboring nodes. Through multi-layer propagation, the representation of each node can capture information about its local graph structure.

[0055] The output of this step is: each event / entity has a low-dimensional, dense vector representation that encodes its semantic information, structural information in the graph, and temporal information.

[0056] S33, Causal Reasoning and Evolutionary Analysis (Transformer-based Reasoning Networks) Specifically, the following steps are included: S331, Serialization: Extracting event sequences from the event knowledge graph based on timestamps. For example, for the entity "Company A", extract all its related events and sort them by time: [profit growth, new product launch, stock price increase].

[0057] S332, Construct the inference model: Take the vector representation of each event in the event sequence (from step S32) as input, which can be an encoder-decoder (such as BART, T5) or an encoder-only (such as BERT) Transformer architecture. More commonly, an encoder-only model is used for inference tasks.

[0058] S333, Pre-training and Fine-tuning: On large-scale event sequence data, using a BERT-like masked language model task, the model learns the co-occurrence and dependency patterns between events. For example, masking "stock price increase" and having the model predict it based on the context "profit growth" and "new product launch". On domain-specific labeled data, the model is trained to perform specific inference tasks. Common tasks include: Next event prediction: Given the first k events, predict the most likely next event.

[0059] Causality identification: Determining whether a causal relationship exists between two events. This is a binary classification task.

[0060] Evolutionary path completion: Given an incomplete chain of events, reason and complete the missing key events in the middle.

[0061] Analyzing the evolutionary path: By analyzing the Transformer's self-attention weight matrix, we can interpretably see which events in the historical sequence the model prioritizes when making predictions. For example, when predicting "stock price increase," if the model gives a very high attention weight to the event "new product launch," we can consider "new product launch" as a key factor leading to the "stock price increase." By tracing the high-weighted attention connections, we can outline a key causal evolutionary path.

[0062] This embodiment also provides an AI-based intelligent event analysis system for implementing the method, the system comprising: Data acquisition and feature extraction module: used to acquire event-related multimodal raw data and extract feature representations for each modality of data; Event type identification module: It is used to construct a hierarchical multimodal feature map containing intramodal subgraphs and cross-modal association graphs based on the extracted features. It uses a graph attention network to perform intramodal GAT aggregation and modality-aware cross-modal attention calculation in sequence to obtain the global features after cross-modal aggregation. Finally, it outputs the event type probability distribution through the classification layer to identify the event type. Event Analysis Module: Based on the identified event types and multimodal feature maps, it constructs a hierarchical multimodal event knowledge graph, and uses time-series modeling and causal reasoning models to analyze the logical relationships and evolution paths between events, and outputs the analysis results.

[0063] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0064] In one embodiment, the electronic device includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). The RAM may also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0065] Multiple components in an electronic device are connected to an I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0066] The computing unit can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing units include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit performs the various methods and processes described above, such as methods S1-S3. For example, in some embodiments, methods S1-S3 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the computing unit, one or more steps of methods S1-S3 described above may be performed. Alternatively, in other embodiments, the computing unit may be configured to perform methods S1-S3 by any other suitable means (e.g., by means of firmware).

[0067] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0068] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0069] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0070] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An AI-based intelligent event analysis method, characterized in that, The method includes the following steps: Acquire event-related multimodal raw data and extract feature representations for each modality; Based on the extracted features, a hierarchical multimodal feature map containing intramodal subgraphs and cross-modal association graphs is constructed. A graph attention network is used to perform intramodal GAT aggregation and modality-aware cross-modal attention calculation in sequence to obtain the global features after cross-modal aggregation. Finally, the event type probability distribution is output through the classification layer to identify the event type. Based on the identified event types and multimodal feature maps, a hierarchical multimodal event knowledge graph is constructed, and the logical relationships and evolution paths between events are analyzed using temporal modeling and causal reasoning models, and the analysis results are output.

2. The AI-based intelligent event analysis method according to claim 1, characterized in that, The modal subgraphs are determined based on the number of modalities in the acquired multimodal raw data. Each modality corresponds to one modal subgraph. For the text modality, the nodes of the corresponding modal subgraph are defined as semantic units of event-related text, and their features are semantic vectors extracted by a pre-trained language model. For the image modality, the nodes of the corresponding modal subgraph are defined as key visual objects in the image, and their features are visual vectors extracted by a pre-trained visual model. For the audio modality, the nodes of the corresponding modal subgraph are defined as acoustic events, and their features are acoustic vectors extracted by MFCC. For the video modality, the nodes of the corresponding modal subgraph are defined as video keyframes, and their features are temporal visual vectors extracted by a pre-trained visual feature extraction model. Within the same modal subgraph, the edge weights between nodes are calculated based on feature similarity.

3. The AI-based intelligent event analysis method according to claim 1, characterized in that, The nodes of the cross-modal association graph are defined as core nodes connecting different modal subgraphs, namely, alignment units corresponding to semantic units, key visual objects, acoustic events, and video keyframes. The features are the average concatenation of the node features of the corresponding intra-modal subgraphs. The edge weights between nodes of different intra-modal subgraphs and nodes of the cross-modal association graph are calculated based on the cross-modal feature alignment score.

4. The AI-based intelligent event analysis method according to claim 1, characterized in that, The intra-modal GAT aggregation specifically involves: independently performing GAT calculations on each intra-modal subgraph, aggregating local correlation features within the modality, and preserving the inherent characteristics of each modality, including the following steps: Intramodal attention coefficient calculation: For modal m Modal inner subgraph, compute nodes i For neighboring nodes j Attention coefficient: ， in, For modality m The characteristic projection matrix, For modality m Next node i The feature is represented by ||, which is the feature concatenation operation. For modality m Next node i The set of neighboring nodes; Intramodal aggregated feature output: Calculates the feature for each modality based on the intramodal attention coefficient. m Modal inner subgraph nodes i aggregation features : ， Where σ is the activation function; The global aggregated features of each modality are obtained by combining the aggregated features of all nodes within the modality: ， in, For modality m The number of nodes, For modality m Global aggregation features.

5. The AI-based intelligent event analysis method according to claim 1, characterized in that, The modality-aware cross-modal attention computation specifically includes the following steps: Modal importance weight calculation: Based on prior knowledge of the current event scenario and real-time feature distribution, the importance weight of each modality is dynamically calculated. ， in, For modality m Importance weights M For modal sets, For modality m The significance score, It is a single-layer perceptron. For modality m Importance weights obtained through intramodal GAT aggregation; Cross-modal attention coefficient calculation: Based on the cross-modal association graph, the cross-modal attention coefficient is calculated by fusing modal importance weights. For mode m Nodes in the modal inner subgraph i With mode k Nodes in the modal inner subgraph j The cross-modal attention coefficients between the two modes are first calculated by the modality. m Nodes in the modal inner subgraph i Nodes in the cross-modal association graph c First attention coefficient between : ， in, For modality m Nodes in the modal inner subgraph i aggregation characteristics, For nodes c Features , Represents the cross-modal projection matrix. For modality m nodes i Nodes in the cross-modal association graph c Edge weights between them Representation and Modality m nodes i The core node set of the associated cross-modal association graph, where || represents the feature concatenation operation; Subsequently, the nodes in the cross-modal association graph are calculated. c With mode k Nodes in the modal inner subgraph j The second attention coefficient between : ， in, For modality k Importance weights For modality k Nodes in the modal inner subgraph j aggregation characteristics, For modality k The cross-modal projection matrix, For nodes in the cross-modal association graph c Associated modes k The set of nodes, For nodes in the cross-modal association graph c With mode k Nodes in the modal inner subgraph j Edge weights between them; Calculate the mode based on the first attention coefficient and the second attention coefficient. m The next node i and modality k The next node j Cross-modal attention coefficients ; ， Cross-modal aggregated feature output: Global features are obtained by aggregating cross-modal attention coefficients. for: ， in, This represents the activation function. , indicating the relationship between nodes in the cross-modal association graph i Indirectly related modes k Node set.

6. The AI-based intelligent event analysis method according to claim 1, characterized in that, The event types are a predefined closed set, including lifecycle events, interaction events, transaction events, business activities, judicial events, military activities, and socio-political events.

7. The AI-based intelligent event analysis method according to claim 1, characterized in that, The loss function of the graph attention network is defined as follows: ， in, For cross-entropy loss, To predict the event type output by the graph attention network, For the event type truth value, The regularization coefficient is . , Modal m , k The corresponding set of nodes, M For modal sets, For modality m Nodes in the modal inner subgraph i aggregation characteristics, For modality k Nodes in the modal inner subgraph j Aggregation characteristics.

8. The AI-based intelligent event analysis method according to claim 1, characterized in that, The hierarchical multimodal event knowledge graph adopts a three-layer architecture, which is aligned with the hierarchical multimodal feature graph. The event core layer corresponds to the event type identification result, the modal entity layer corresponds to the intramodal subgraph, which stores the entities of a single modality and the intramodal associations, and the cross-modal association layer corresponds to the cross-modal association graph, which connects entities of different modalities and stores the cross-modal alignment relationship.

9. The AI-based intelligent event analysis method according to claim 8, characterized in that, The construction of the hierarchical multimodal event knowledge graph includes the following steps: Define the entity types, relation types, and attributes of a hierarchical multimodal event knowledge graph. For the event core layer, the entity type is event entity; for the modal entity layer, the entity type is modal entity, including text entities, image entities, audio entities, and video entities; for the cross-modal association layer, the entity type is cross-modal association entity. The relation types include intra-modal relations, cross-modal relations, and event-entity association relations. Based on the intermediate processing and output of hierarchical multimodal feature maps and graph attention networks, modal entities and cross-modal associated entities are extracted from the hierarchical multimodal feature maps, and relation extraction is performed in combination with semantic rules. Attribute information is obtained through feature sequence mapping.

10. An AI-based intelligent event analysis system, characterized in that, For implementing the method as described in any one of claims 1-9, the system comprises: Data acquisition and feature extraction module: used to acquire event-related multimodal raw data and extract feature representations for each modality of data; Event type identification module: It is used to construct a hierarchical multimodal feature map containing intramodal subgraphs and cross-modal association graphs based on the extracted features. It uses a graph attention network to perform intramodal GAT aggregation and modality-aware cross-modal attention calculation in sequence to obtain the global features after cross-modal aggregation. Finally, it outputs the event type probability distribution through the classification layer to identify the event type. Event Analysis Module: Based on the identified event types and multimodal feature maps, it constructs a hierarchical multimodal event knowledge graph, and uses time-series modeling and causal reasoning models to analyze the logical relationships and evolution paths between events, and outputs the analysis results.

Images