Real-time multi-modal data fusion method and device based on dynamic graph neural network

Abstract

The invention belongs to the field of artificial intelligence, and relates to a real-time multi-modal data fusion method and device based on a dynamic graph neural network, and the method comprises the steps: carrying out the time-space alignment and preprocessing of multi-modal data, and obtaining a multi-modal data instance; based on the multi-modal data instance, constructing a multi-modal graph structure to obtain a dynamic heterogeneous graph; inputting the dynamic heterogeneous graph into a dynamic graph neural network, aggregating neighbor information, and learning advanced and context-aware feature representations of nodes to obtain a global graph representation; designing an attention fusion layer focused on information interaction between modals, performing selective and weighted integration on node features from different modals, and generating fusion feature representation of each entity; real-time incremental updating and optimization are carried out; and outputting a fusion result according to the fusion feature representation and / or the global graph representation. The data relevance is obviously improved; the feature expression capability is enhanced; intelligent integration of information between modes is realized; and the low delay and the expandability of the system are ensured.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a real-time multimodal data fusion method and apparatus based on dynamic graph neural networks. Background Technology

[0002] With the rapid development of fields such as the Internet of Things, autonomous driving, and intelligent monitoring, the real-time fusion of multimodal data (such as images, videos, point clouds, sound, text, and inertial measurement unit data) has become a core challenge for perception and decision-making systems. Traditional data fusion methods often fail to effectively model the complex spatiotemporal relationships between different modalities and are difficult to adapt to dynamic changes in data. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a real-time multimodal data fusion method based on a dynamic graph neural network, employing the following technical solution, including the following steps:

[0004] The multimodal data is spatiotemporally aligned and preprocessed to obtain multimodal data instances;

[0005] Based on the aforementioned multimodal data instance, a multimodal graph structure is constructed to obtain a dynamic heterogeneous graph;

[0006] The dynamic heterogeneous graph is input into a dynamic graph neural network. Through a multi-layer message passing mechanism, neighbor information is aggregated, and high-level, context-aware feature representations of nodes are learned to obtain a global graph representation.

[0007] The design focuses on an attention fusion layer for intermodal information interaction, which selectively and weightedly integrates node features from different modalities to generate a fused feature representation for each entity.

[0008] Perform real-time incremental updates and optimizations;

[0009] The fusion result is output based on the fusion feature representation and / or global graph representation.

[0010] Preferably, the step of performing spatiotemporal alignment and preprocessing on the multimodal data to obtain multimodal data instances specifically includes:

[0011] The multimodal data is subjected to time synchronization and interpolation processing;

[0012] A unified spatial coordinate transformation and projection are performed on the multimodal data after time synchronization and interpolation processing;

[0013] The multimodal data, after undergoing unified spatial coordinate transformation and projection, is subjected to modality-specific preprocessing and primary feature extraction to obtain multimodal data instances.

[0014] Preferably, the step of constructing a multimodal graph structure based on the multimodal data instance to obtain a dynamic heterogeneous graph specifically includes:

[0015] Based on the multimodal data instance, the type and granularity of nodes in the multimodal graph structure are determined, and an initial feature vector is assigned to each node;

[0016] Establish various connection relationships between nodes, and the edge type represents spatial proximity, temporal association, modal complementarity, or semantic similarity;

[0017] Based on the current task and node importance, the graph is sparsified, preset connections are retained based on edge weights, and dynamic update rules for the graph structure are defined to obtain a dynamic heterogeneous graph.

[0018] Preferably, the step of inputting the dynamic heterogeneous graph into a dynamic graph neural network, aggregating neighbor information through a multi-layer message passing mechanism, learning high-level, context-aware feature representations of nodes, and obtaining a global graph representation specifically includes:

[0019] Attention message passing is performed on the dynamic heterogeneous graph. On the dynamic heterogeneous graph, the importance of different node types and edge types to message aggregation is distinguished to achieve information fusion.

[0020] The dynamic heterogeneous graph is subjected to temporal graph convolution and memory enhancement.

[0021] In multi-layer dynamic graph neural networks, pooling of graph structures is implemented to capture structural information at different scales, and a global graph representation is generated in the final layer for graph-level tasks.

[0022] Preferably, the design focuses on an attention fusion layer for intermodal information interaction, and the steps of selectively and weightedly integrating node features from different modalities to generate a fused feature representation for each entity specifically include:

[0023] The importance of features from each modality source to the target node during fusion is calculated, i.e., intra-modal attention, and the relative importance between different modalities is calculated, i.e., inter-modal attention.

[0024] Gating mechanisms are used to control how information from different modalities flows into and updates the fused representation of the target node;

[0025] A loss function is introduced during the training phase.

[0026] Preferably, the step of performing real-time incremental updates and optimization specifically includes:

[0027] When a new data frame arrives, the graph nodes and edges of the multimodal graph structure are updated;

[0028] Perform lightweight dynamic graph neural network inference and caching;

[0029] Conduct online learning and parameter adaptation.

[0030] Preferably, the step of outputting the fusion result based on the fusion feature representation and / or global graph representation specifically includes:

[0031] Based on application requirements, design a fully connected network or a multilayer perceptron to map the fused features to the output space required by the task.

[0032] Perform multi-task joint learning and output fusion;

[0033] The original output of the model is post-processed to generate the final result, which is then presented to the user or sent to the control system.

[0034] To address the aforementioned technical problems, this invention also provides a real-time multimodal data fusion device based on a dynamic graph neural network, employing the following technical solution, including:

[0035] The preprocessing module is used to perform spatiotemporal alignment and preprocessing on the multimodal data to obtain multimodal data instances;

[0036] The construction module is used to construct a multimodal graph structure based on the multimodal data instance to obtain a dynamic heterogeneous graph;

[0037] The learning module is used to input the dynamic heterogeneous graph into the dynamic graph neural network, aggregate neighbor information through a multi-layer message passing mechanism, learn the high-level, context-aware feature representation of nodes, and obtain a global graph representation.

[0038] The generation module is used to design an attention fusion layer that focuses on intermodal information interaction. It selectively and weightedly integrates node features from different modalities to generate a fusion feature representation for each entity.

[0039] The update module is used for real-time incremental updates and optimizations;

[0040] The output module is used to output the fusion result based on the fusion feature representation and / or global graph representation.

[0041] To address the aforementioned technical problems, the present invention also provides a computer device that employs the technical solution described below, comprising a memory and a processor. The memory stores computer-readable instructions, and the processor, when executing the computer-readable instructions, implements the steps of the aforementioned real-time multimodal data fusion method based on dynamic graph neural networks.

[0042] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium, which employs the technical solution described below. The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the aforementioned real-time multimodal data fusion method based on a dynamic graph neural network.

[0043] Compared with the prior art, the present invention has the following main advantages:

[0044] (1) The spatiotemporal alignment and dynamic graph modeling capabilities significantly improve data correlation. Through spatiotemporal alignment preprocessing of multimodal data, the differences in temporal resolution and spatial scale of heterogeneous data are effectively solved; the construction of dynamic heterogeneous graph structures can capture the dynamic correlation between data in real time, which is especially suitable for dynamic scenarios such as traffic and industrial monitoring, and enhances the adaptability of the model to complex environmental changes;

[0045] (2) Multi-layer message passing mechanism and context-aware feature learning enhance feature representation capabilities. Dynamic graph neural networks can mine high-order relationships between nodes by aggregating multi-layer neighbor information, generating context-aware feature representations containing global semantics. This structure not only preserves the local features of single-modal data, but also achieves feature complementarity through cross-modal interaction, significantly improving feature discriminability.

[0046] (3) The introduction of the attention fusion layer enables intelligent integration of information between modalities. By dynamically allocating the weights of different modalities, this layer can adaptively focus on key information and suppress noise interference. Especially when some modal data is missing or degraded, it can still maintain stable fusion performance. This selective integration mechanism significantly improves the robustness of the fusion results.

[0047] (4) The real-time incremental update mechanism ensures the system's low latency and scalability. Through local updates and parameter optimization of the dynamic graph structure, the model can adapt to changes in data distribution without full retraining, significantly reducing computational overhead while maintaining high accuracy, and providing efficient technical support for real-time decision-making scenarios (such as autonomous driving and intelligent security). Attached Figure Description

[0048] To more clearly illustrate the solutions in this invention, the accompanying drawings used in the description of the embodiments of this invention will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a flowchart of an embodiment of the real-time multimodal data fusion method based on dynamic graph neural networks of the present invention;

[0050] Figure 2 This is a schematic diagram of the structure of an embodiment of the real-time multimodal data fusion device based on dynamic graph neural network of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of an embodiment of the computer device of the present invention. Detailed Implementation

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the specification is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings are used to distinguish different objects and not to describe a particular order.

[0053] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0054] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0055] It should be noted that the real-time multimodal data fusion method based on dynamic graph neural networks provided in the embodiments of the present invention is generally executed by a server / terminal device, and correspondingly, the real-time multimodal data fusion device based on dynamic graph neural networks is generally set in the server / terminal device.

[0056] It should be understood that the number of terminal devices, networks, and servers is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be used.

[0057] Example 1

[0058] Please refer to Figure 1 The diagram illustrates a flowchart of an embodiment of the real-time multimodal data fusion method based on dynamic graph neural networks of the present invention. The real-time multimodal data fusion method based on dynamic graph neural networks includes the following steps:

[0059] Step S1: Perform spatiotemporal alignment and preprocessing on the multimodal data to obtain a multimodal data instance.

[0060] In this embodiment, the electronic device (e.g., a server / terminal device) on which the real-time multimodal data fusion method based on dynamic graph neural networks runs can receive real-time multimodal data fusion requests based on dynamic graph neural networks via wired or wireless connections. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future-developed wireless connection methods.

[0061] In this embodiment, step S1, which involves spatiotemporal alignment and preprocessing of the multimodal data to obtain a multimodal data instance, may specifically include the following steps:

[0062] S11, Time synchronization and interpolation processing are performed on the multimodal data.

[0063] A hybrid synchronization strategy based on hardware clock signals and software filtering is adopted.

[0064] For sensors that support PPS (pulses per second) or IEEE 1588 (PTP), use a synchronization signal to align the acquisition clock.

[0065] For situations where hardware synchronization is impossible or data packets are lost, a method based on Dynamic Time Warping (DTW) and cubic spline interpolation is employed. Assume there is a high-frequency reference sequence T from sensor A. ref =[t1,t2,...,t m ] and the low-frequency or irregular sequence T of sensor B b =[τ1,τ2,...,τ n ] and its data D b First, use DTW to find T. ref and T b The optimal path between, and then, guided by the path, for D b In T ref Each time point t k Spline interpolation is performed on the data to obtain the aligned data.

[0066] Spline interpolation formula (taking piecewise cubic splines as an example):

[0067] S j (τ)=a j +b j (τ-τ j )+c j (τ-τ j) 2 +d j (τ-τ j ) 3 ,τ∈[τ j ,τ j+1 ], where τ: the time point to be interpolated (belonging to T) ref ). τ j ,τ j+1 : Adjacent time points in the raw data of sensor B. S j (τ): in the interval [τ j ,τ j+1 Interpolation function on ]. a j ,b j ,c j ,d j : From data point D b (τ j D b (τ j+1 The coefficients are determined by the boundary continuity condition (continuity of first and second derivatives).

[0068] This formula constructs a smooth curve that makes the function value and lower-order derivative continuous at known data points, thus allowing for a smooth estimation of the data value of sensor B at an unknown time point τ.

[0069] The purpose of step S11 is to resolve the timing misalignment problem caused by different sensor acquisition frequencies and trigger times.

[0070] S12, perform unified spatial coordinate transformation and projection on the multimodal data after time synchronization and interpolation processing.

[0071] The multi-sensor system is pre-calibrated offline to obtain the extrinsic parameters (rotation matrix R) of each sensor relative to a common reference (such as the vehicle center or robot base). i Translation vector t i ) and internal parameters (such as camera focal length f) x ,f y (distortion coefficient).

[0072] For a point P in the sensor i coordinate system i =[x i ,y i ,z i ] T Its coordinates P in the world coordinate system w Calculate P using the following formula: w =R i ·P i +t i , where P i : 3D coordinates in the local coordinate system of sensor i. Ri : A 3x3 rotation matrix describing the rotation from the sensor i-coordinate system to the world coordinate system. t i : 3x1 translation vector, describing the position of the origin of the sensor i coordinate system in the world coordinate system. P w : Transformed world coordinates.

[0073] Rigid body transformation unifies the observations of all sensors into the same three-dimensional spatial framework, which is the basis for subsequent spatial correlation (such as determining whether image pixels and LiDAR points correspond to the same object).

[0074] Projection (for image data): For a 3D point P w Projected onto the pixel plane of camera j:

[0075] Where (u,v): the projected pixel coordinates. K j : The 3x3 intrinsic parameter matrix of camera j. Rotation matrix R j The transpose of t. j : The translation vector of camera j. ∝ indicates equality in homogeneous coordinates (difference by a scale factor).

[0076] Establish geometric correspondences between the 3D world and 2D images for cross-modal (visual-geometric) feature association.

[0077] The purpose of step S12 is to convert the data collected by different sensors (such as cameras, lidar, and radar) into the same world coordinate system or reference coordinate system, so that the spatial positions are comparable.

[0078] S13. After the multimodal data has undergone unified spatial coordinate transformation and projection, modality-specific preprocessing and primary feature extraction are performed to obtain multimodal data instances.

[0079] For image / video data, use lightweight CNNs (such as shallow layers of MobileNetV3) or traditional filters (such as Gaussian filtering for noise reduction, histogram equalization for enhancement) to extract local feature maps or key point descriptors (such as a simplified version of SIFT or learned sparse features).

[0080] For point cloud data, voxel mesh downsampling or farthest point sampling (FPS) is performed to reduce the number of points; local geometric features such as normal vectors and curvature are calculated.

[0081] For text / speech data, perform word segmentation and stop word removal; extract bag-of-words (BoW) vectors or MFCC features.

[0082] For IMU / sensor time-series data, Butterworth low-pass filters are applied for noise reduction; statistical characteristics (mean, variance, FFT spectral energy) within the sliding window are calculated.

[0083] The purpose of step S13 is to denoise and enhance the data for each modality, and extract lightweight primary features to reduce the dimensionality of the original data and improve the efficiency of subsequent processing.

[0084] The purpose of step S1 is to align the data streams from different sensors in time and space, and to perform necessary cleaning, standardization, and preliminary feature extraction, as raw multimodal data often suffers from issues such as asynchronous timestamps, inconsistent coordinate systems, and varying data formats and scales. This step lays the foundation for constructing a unified graph structure. The raw multimodal data stream is received, and its output is a set of standardized features after cleaning and alignment, which is directly used in step S2 to construct graph nodes and edges.

[0085] Step S2: Based on the multimodal data instance, construct a multimodal graph structure to obtain a dynamic heterogeneous graph.

[0086] In this embodiment, step S2, constructing a multimodal graph structure based on the multimodal data instance to obtain a dynamic heterogeneous graph, may specifically include the following steps:

[0087] S21, based on the multimodal data instance, determine the type and granularity of nodes in the multimodal graph structure, and assign an initial feature vector to each node.

[0088] Multiple node types are defined based on modality and semantics, such as: Object_Node (detected object), Region_Node (image superpixel or feature map region), PointCluster_Node (point cloud segmentation block), Entity_Node (named entity extracted from text), and Event_Node (event summary within a specific time window).

[0089] Use lightweight real-time detection or segmentation algorithms (such as YOLO-Lite, a simplified version of PointNet++) to quickly extract candidate node instances from various modal datasets.

[0090] For each node, its modality-specific primary features from step S1 are concatenated with its spatial location (coordinates), timestamp, and modality type encoding to form a unified initial node feature vector.

[0091] Step S21 is used to define and initialize multi-granularity nodes.

[0092] S22 establishes various connection relationships between nodes, with edge types representing spatial proximity, temporal association, modal complementarity, or semantic similarity.

[0093] Spatial Proximity Edge: For nodes in 3D space (such as Object_Node, PointCluster_Node), calculate their Euclidean distance d. ij =‖p i -p j ‖2. If d ij <θ space If so, a "spatial proximity" edge is established. The edge weight... It can be set to Where, p i ,p j θ: Spatial coordinates of nodes i and j. space : Spatial distance threshold. σ: Parameter controlling the rate of weight decay.

[0094] Temporally related edges: For instances of the same node (tracking target) at different time steps, "temporally related" edges are established based on the matching degree between the predicted position and the observed position by the motion model (such as Kalman filtering). The matching degree can be measured by Mahalanobis distance and converted into weights.

[0095] Cross-modal edge association: This is the most critical and complex edge. For example, connecting a Region_Node in an image and a PointCluster_Node in a point cloud. Technical methods include:

[0096] Geometric projection consistency: Using the projection in step S12, determine whether the point cloud cluster has enough points projected into the image region.

[0097] Feature similarity: Calculate the cosine similarity of the initial feature vectors of the nodes.

[0098] Learnable interconnected networks: Designing a small neural network Output an association score and association type (e.g., "description", "contains", "nearby"). The network can be trained end-to-end.

[0099] Uncertainty modeling: Assign an uncertainty score u to each edge. ij The edge weights ∈ [0,1] can be calculated based on the variance of the matching score, the confidence level of the association network prediction, or expert rules. The final edge weights are... Reduce the impact of unreliable connections.

[0100] Step S22 establishes various connections (edges) between nodes. An uncertainty metric is introduced to assign credibility weights to the edges.

[0101] S23. Based on the current task and node importance, the graph is sparsified, preset connections are retained based on edge weights, and dynamic update rules for the graph structure are defined to obtain a dynamic heterogeneous graph.

[0102] K-Nearest Neighbor (K-NN) Graph: For each relation type, each node retains only its K most relevant connections (based on edge weights). ).

[0103] Threshold truncation: Discard weights below the threshold θ edge The edge.

[0104] Topology maintenance strategies include node lifecycle management and edge lifecycle management. Node lifecycle management includes: creating new nodes for newly appearing objects / regions; marking nodes that have not been updated for a long time (e.g., lost tracking) as "inactive" or removing them from the current computation graph. Edge lifecycle management includes: dynamically updating the existence and weight of edges based on changes in node state (e.g., significant changes in location or features).

[0105] The purpose of step S23 is to address the computational overhead of a complete graph in real-time systems, which is prohibitive. This step aims to sparsify the graph based on the current task and node importance, preserve key connections, and define dynamic update rules for the graph structure.

[0106] Step S2 abstracts the multimodal data instances processed in Step S1 (such as regions in images, point clusters in point clouds, and entities in text) into graph nodes, and constructs edges based on spatiotemporal, semantic, and intermodal relationships to form a dynamically changing heterogeneous graph. This graph is the direct input to the Dynamic Graph Neural Network (DGNN). This step uses the alignment features output from Step S1 as input to construct the initial graph structure. This graph structure will serve as the input graph for the DGNN in Step S3. Meanwhile, the dynamic updating of the graph (addition and deletion of nodes and edges) is one of the core tasks of Step S5.

[0107] Step S3: Input the dynamic heterogeneous graph into the dynamic graph neural network, aggregate neighbor information through a multi-layer message passing mechanism, learn the high-level, context-aware feature representation of nodes, and obtain the global graph representation.

[0108] In this embodiment, step S3, which involves inputting the dynamic heterogeneous graph into a dynamic graph neural network, aggregating neighbor information through a multi-layer message passing mechanism, and learning high-level, context-aware feature representations of nodes to obtain a global graph representation, may specifically include the following steps:

[0109] S31, perform attention message passing on the dynamic heterogeneous graph, distinguish the importance of different node types and edge types to message aggregation on the dynamic heterogeneous graph, and realize information fusion.

[0110] An improved Heterogeneous Graph Attention Network (HAN) layer is used.

[0111] Meta-path definition: Define a meaningful sequence of node types (e.g., image region -> object -> point cloud cluster) to capture higher-order semantic associations.

[0112] Hierarchical attention mechanism:

[0113] Node-level attention: For a meta-path Φ, calculate the attention coefficient from node j to node i. Among them, h i ,h j : The current features of nodes i and j. W Φ : The learnable linear transformation matrix for the metapath Φ. Φ : Attention vector for the meta-path Φ. ‖: Concatenation operation. σ: LeakyReLU activation function. The set of neighbors of node i under the metapath Φ.

[0114] This formula calculates the importance score of each neighbor node to the central node under a specific meta-path, thereby achieving differentiated information aggregation.

[0115] Semantic-level attention: learning the importance of different meta-paths to the target node (β) Φ The node representations under each metapath are weighted and merged.

[0116] Message aggregation: Aggregate information along the meta-path Φ for node i:

[0117] S32, perform time-series graph convolution and memory enhancement on the dynamic heterogeneous graph.

[0118] Combine Graph Convolutional Networks (GCNs) with Recurrent Neural Networks (RNNs) or Temporal Convolutional Networks (TCNs).

[0119] Spatiotemporal graph convolutional block: At time step t, for the current graph G t Perform graph convolution operations while incorporating the node features from the previous moment as context:

[0120] Among them, A t : The graph adjacency matrix at time step t (which may include edge weights). The feature matrix of all nodes in layer l at time t. GCN (l) : Convolution operation on the l-th layer graph, for example in It is a normalized adjacency matrix. Θ (l) : Parameters of the l-th layer GCN. GRU(·): Gated recurrent unit, representing the features of this layer at the previous time step. Features of the previous layer at the current moment The input is the time series context information, and the output is the fusion weight.

[0121] This operation simultaneously aggregates the spatial neighbor information at the current moment (via GCN) and its own historical evolution information (via GRU), generating node features with spatiotemporal dependencies.

[0122] Graph Memory Module: Introduces an updatable graph-level memory matrix M t This is used to store long-term important patterns (such as frequently occurring scene layouts). This memory is read and updated through an attention mechanism, and its information is injected into node features.

[0123] The purpose of step S32 is to explicitly model the evolution of the graph structure over time and capture the dynamic patterns of node features.

[0124] S33 implements pooling of graph structures in a multi-layer dynamic graph neural network to capture structural information at different scales and generate a global graph representation in the final layer for graph-level tasks.

[0125] Node clustering pooling: using a learnable clustering assignment matrix (N l N represents the number of nodes in the current layer. l+1 Group the nodes into supernodes (to the number of nodes in the next layer).

[0126] H (l+1) =S (l)T H (l) A (l+1) =S (l)T A (l) S (l) S (l) The elements in the array can be generated by soft clustering of the nodes using another GNN layer. Where S... (l) Clustering assignment matrix, whose elements S ij H represents the probability or membership degree of a node i in layer 1 being assigned to a supernode j in layer l+1. (l) : The node feature matrix of layer 1. A (l) H: The adjacency matrix of the first layer. (l+1) : The feature matrix of the supernodes in the (l+1)th layer after pooling. A (l+1) : Adjacency matrix of supernodes in layer l+1 after pooling.

[0127] By aggregating fine-grained node features and adjacency relationships into coarse-grained supernode features and adjacency relationships, graph downsampling and hierarchical abstraction are achieved.

[0128] Global graph readout: For graph classification or prediction tasks, in the last layer L, all node features are aggregated into a global graph feature vector h. G :

[0129] in, Vector concatenation. V: The set of all nodes in the graph. The feature vector of node i in the last layer L. |V|: Total number of nodes. MaxPool(·): Takes the maximum value of each element of the features of all nodes. AttentionPool(·): Summes the features of all nodes by weighting them with attention weights.

[0130] By combining mean pooling, max pooling, and attention pooling, a global representation is generated that includes both overall statistical information and highlights the contributions of key nodes.

[0131] Step S33 is used to perform hierarchical graph pooling and readout.

[0132] Step S4: Design an attention fusion layer that focuses on intermodal information interaction, selectively and weightedly integrate node features from different modalities to generate a fusion feature representation for each entity.

[0133] In this embodiment, step S4, designing an attention fusion layer focused on intermodal information interaction, selectively and weightedly integrates node features from different modalities to generate a fused feature representation for each entity, may specifically include the following steps:

[0134] S41, calculate the importance of the features from each modality source to the target node during fusion, i.e., intra-modal attention, and the relative importance between different modalities, i.e., inter-modal attention.

[0135] Intramodal attention: For a target node i, assume it has feature sets from M modalities. (These might be neighbor features derived from different edges, or the node's own representation in different modalities). First, for each modality m, calculate an importance score. (If the modality has multiple associated features, then perform intra-modality aggregation first).

[0136] in, Node i is derived from the features of mode m. W m : The learnable weight matrix specific to mode m. b m: A learnable bias vector specific to mode m. v m : The learnable weight vector specific to mode m.

[0137] Evaluate the information richness of the features derived from modality m itself or their relevance to the current context.

[0138] Intermodal attention: Based on the importance scores of each modality, calculate the normalized weights between modalities.

[0139] in, Attention score for modality m. M: Total number of modalities.

[0140] A soft selection mechanism determines which modality's information is more trusted during fusion. For example, in low light conditions, the weight of the visual modality may decrease, while the weight of the lidar modality may increase.

[0141] S42 uses a gating mechanism to control how information from different modalities flows into and updates the fused representation of the target node.

[0142] Design fusion units that treat each modality as an information source and iteratively or in parallel fuse them into a master representation. An effective approach is to use multi-head cross-modal attention gates.

[0143] Multi-head cross-modal attention gate: For modality m, compute its relation to the current fusion state. (Initially, it can be a query, key, and value vector, which may be a dominant modality feature or a zero vector): in, The current fusion characteristics (old state) of node i. Features derived from mode m. A modality-specific learnable weight matrix is ​​used to generate query, key, and value vectors.

[0144] Calculate attention score: Where d k It is the dimension of the key vector.

[0145] Contribution of generating mode m:

[0146] The computation is performed in parallel using multiple heads, and then the outputs of each head are concatenated.

[0147] Perform gating updates and calculate the update gate z. i and reset door r i (Similar to GRU) to control the fusion of new and old information: in, The aggregation of all modal contributions (e.g., a weighted sum, with weights of ). ). ⊙: Element-wise multiplication. σ: Sigmoid function. W z ,b z Update the gate weights and biases. W f ,b f : Weights and biases for candidate state calculation. i Reset the gate vector (calculated similarly to update the gate z). i (Different parameters)

[0148] Through gating mechanisms, new cross-modal information can be selectively transmitted. Integrate into existing representations In this process, an updated and richer fusion representation is generated.

[0149] The purpose of step S42 is to perform gating (such as gating in GRU) cross-modal feature fusion to prevent interference from invalid or noisy information.

[0150] S43 introduces a loss function during the training phase.

[0151] Features of the same entity in different modalities and the fused features f i Different entities are treated as positive sample pairs, while features of different entities are treated as negative sample pairs. InfoNCE loss is used.

[0152] Where sim(·,·) is the cosine similarity function; τ is the temperature parameter; and B is the batch size.

[0153] By narrowing the distance between multimodal features of the same entity and its fused representation, and widening the distance between features of different entities, a well-aligned common representation space can be learned.

[0154] Encourage integration features f i It can reconstruct the original features of each modality to a certain extent through a decoder (small neural network). Ensure that no key modal information is lost during the fusion process.

[0155] Among them, Decoder m (·): Decoder network for mode m. M: Total number of modes.

[0156] Total loss: in It is the main loss of the downstream task (such as classification cross-entropy). α and β are hyperparameters.

[0157] The purpose of step S43 is to encourage the consistency of features from different modalities after fusion by fusing consistency constraints and losses, and to improve the discriminative power of the fused representation.

[0158] Step S4 addresses the issue that while the node features learned in Step S3 contain information from neighbors, the deep cross-modal fusion may still be insufficient. This step specifically designs an attention fusion layer focused on intermodal information interaction, selectively and weightedly integrating node features from different modalities to generate a unified and robust fused representation for each entity (e.g., an object). The high-level node features output from Step S3 are used as an example. The input is the object node, particularly for nodes that are associated with multiple modalities (e.g., an object node is connected to an image region, a point cloud cluster, and a text description). The output is the final fused feature vector f for each entity node. i This is for direct use in step S6 or for further processing.

[0159] Step S5: Perform real-time incremental updates and optimizations.

[0160] In this embodiment, step S5, performing real-time incremental updates and optimization, may specifically include the following steps:

[0161] S51, when a new data frame arrives, update the graph nodes and edges of the multimodal graph structure.

[0162] Lightweight multi-object tracking algorithms (such as greedy algorithms based on IOU or feature matching, or simplified SORT / DeepSORT) are used to associate newly detected nodes with existing nodes in the graph. Nodes that are successfully matched update their attributes (such as location), new nodes that are not matched are added, and nodes that have not been matched for a long time are removed.

[0163] For existing edges, their weights are smoothly updated based on new node characteristics and spatial relationships, for example, using an exponential moving average: Where η is the forgetting factor. Edge weights at time step t. Edge weights at time step t-1. Edge weights calculated based on the new data.

[0164] Edges are recalculated only in local subgraphs affected by new nodes or that have undergone significant changes (K-NN lookup or association network inference), not in the global graph.

[0165] The purpose of step S51 is to perform incremental updates to the streaming graph structure.

[0166] S52 performs lightweight dynamic graph neural network inference and caching.

[0167] Using techniques such as knowledge distillation, pruning, and quantization, the trained DGNN model is converted into a lightweight version. For example, floating-point weights are quantized into 8-bit integers (INT8).

[0168] For nodes and substructures in the graph that have not changed or have changed only slightly, their intermediate layer features from the previous time step are cached. At the current time step, only the changed parts (incremental subgraphs) undergo complete message passing computation, and then the cached features are fused with the newly added incremental features. This requires designing a change detection mechanism to determine whether changes in node features or adjacency relationships exceed a threshold.

[0169] Tasks such as graph updating, DGNN inference, and fusion decision-making are assigned to different processing units or threads and executed in parallel in a pipeline manner, thus hiding some computational latency.

[0170] The purpose of step S52 is to accelerate the forward propagation process of DGNN at each time step.

[0171] S53 enables online learning and parameter adaptation.

[0172] Maintain a small meta-learner that quickly fine-tunes key parameters (such as the weights of the attention layer) in the DGNN or fusion network based on the performance of recent data (stored in the replay buffer).

[0173] During the inference phase, the mean and variance calculated online are used to update the statistics of the batch normalization layer, so that the feature normalization is adapted to the current data stream.

[0174] Monitor the uncertainty of the model's predictions (e.g., calculate the prediction variance through multiple inferences via Dropout). When the uncertainty persists above a threshold, trigger a more intensive local retraining or parameter update process using recently collected data with (automatic or manual) labels.

[0175] The purpose of step S53 is to enable the model to adapt to slow changes in the environment or data distribution (concept drift).

[0176] Step S5 addresses the real-time scenario where the system needs to process continuously arriving data with extremely low latency. This step aims to design an efficient incremental algorithm, avoiding recalculation from scratch at each step to meet real-time requirements. It involves online updates of the graph structure, node features, and model parameters. This step is not an independent sequential step but a supporting step closely intertwined with steps S2, S3, and S4. It provides key technologies for updating the dynamic graph (step S2), fast inference of the DGNN (step S3), and online adaptation of the fusion module (step S4).

[0177] Step S6: Output the fusion result based on the fusion feature representation and / or global graph representation.

[0178] In this embodiment, step S6, outputting the fusion result based on the fusion feature representation and / or global graph representation, may specifically include the following steps:

[0179] S61, based on application requirements, designs fully connected networks or multilayer perceptrons to map fused features to the output space required by the task.

[0180] Node-level tasks (such as object classification and attribute recognition): fusion features f for each node i i Use a shared MLP: y i =MLP node (f i ).

[0181] Edge-level tasks (such as relation prediction): Features [f] of a pair of nodes (i,j) i ||f j Using MLP: y ij =MLP edge ([f i ||f j ]).

[0182] Graph-level tasks (such as activity recognition): For the global graph representation h... G Using MLP: y G =MLP graph (h G ).

[0183] The purpose of step S61 is to design a simple fully connected network or multilayer perceptron (MLP) based on different application requirements (such as 3D object detection, behavior recognition, scene understanding, trajectory prediction) and map the fused features to the output space required by the task.

[0184] S62 performs multi-task joint learning and output fusion.

[0185] Shared backbone, dedicated head: The DGNN and fusion mechanism in steps S3 and S4 serve as a shared feature extraction backbone, with an independent output head for each task (step S61).

[0186] Task weight adaptation: Dynamically adjusts the importance of different task losses during training and inference. For example, using uncertainty weighting: total loss in It is a learnable, task-related uncertainty parameter. K: Total number of tasks. The loss of the kth task. The uncertainty (variance) of the k-th task.

[0187] Inter-task information exchange: Introducing a lightweight cross-task attention module before or between the output heads allows information from one task to assist decision-making in another. For example, predictions of object categories can be used to constrain their possible behaviors.

[0188] Step S62 is necessary because many practical applications require multiple tasks to be performed simultaneously (such as simultaneously detecting objects, predicting their trajectories, and identifying relationships between them). This step coordinates the outputs of multiple tasks and handles the dependencies between them.

[0189] S63 performs post-processing on the original model output to generate the final result, which is then presented to the user or sent to the control system.

[0190] Temporal smoothing involves applying temporal filters (such as Kalman filtering or exponential smoothing) to outputs like bounding boxes and classification scores to reduce jitter.

[0191] Non-maximum suppression (NMS) is used to eliminate duplicate detection boxes.

[0192] Trajectory association connects discrete detection boxes into a continuous motion trajectory.

[0193] Visualization overlays detection boxes, classification labels, trajectory lines, relationship arrows, etc. onto the raw sensor data (such as images, point cloud top views) for rendering.

[0194] The feedback loop includes:

[0195] Performance monitoring: Calculate online metrics (such as accuracy and latency), and trigger alarms or adaptive adjustments when metrics decline (step S53).

[0196] The human feedback interface allows users to provide correction labels on keyframes, which are added to the playback buffer for incremental fine-tuning of the model.

[0197] The confidence feedback feeds the confidence of the final decision back to the edge uncertainty model in step S2 or the attention mechanism in step S4, reducing the impact of low-confidence results in the next inference.

[0198] Step S63 performs post-processing on the original model output, including smoothing, filtering, and correlation, to generate a stable and consistent final result, which is then presented to the user or sent to the control system. Simultaneously, performance evaluations or user feedback are sent back to the system for online optimization.

[0199] The purpose of step S6 is to utilize the deep fusion feature representation {f} generated in step S4. i} and / or the global graph representation h generated by step S3 GThis step executes specific downstream tasks and generates a final, user-understandable output. It consumes the high-quality fusion features produced by preceding steps. Its output (such as decision results) can be fed back into the system to optimize preceding steps (such as adjusting graph construction strategies and attention weights), forming a closed loop. Downstream specifically refers to control systems, human-computer interfaces, etc.

[0200] The beneficial effects of implementing this embodiment are:

[0201] (1) The spatiotemporal alignment and dynamic graph modeling capabilities significantly improve data correlation. Through spatiotemporal alignment preprocessing of multimodal data, the differences in temporal resolution and spatial scale of heterogeneous data are effectively solved; the construction of dynamic heterogeneous graph structures can capture the dynamic correlation between data in real time, which is especially suitable for dynamic scenarios such as traffic and industrial monitoring, and enhances the adaptability of the model to complex environmental changes;

[0202] (2) Multi-layer message passing mechanism and context-aware feature learning enhance feature representation capabilities. Dynamic graph neural networks can mine high-order relationships between nodes by aggregating multi-layer neighbor information, generating context-aware feature representations containing global semantics. This structure not only preserves the local features of single-modal data, but also achieves feature complementarity through cross-modal interaction, significantly improving feature discriminability.

[0203] (3) The introduction of the attention fusion layer enables intelligent integration of information between modalities. By dynamically allocating the weights of different modalities, this layer can adaptively focus on key information and suppress noise interference. Especially when some modal data is missing or degraded, it can still maintain stable fusion performance. This selective integration mechanism significantly improves the robustness of the fusion results.

[0204] (4) The real-time incremental update mechanism ensures the system's low latency and scalability. Through local updates and parameter optimization of the dynamic graph structure, the model can adapt to changes in data distribution without full retraining, significantly reducing computational overhead while maintaining high accuracy, and providing efficient technical support for real-time decision-making scenarios (such as autonomous driving and intelligent security).

[0205] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0206] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0207] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0208] Example 2

[0209] Further reference Figure 2 As a response to the above Figure 1 The present invention provides an embodiment of a real-time multimodal data fusion device based on a dynamic graph neural network, which is implemented in accordance with the method shown. Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0210] likeFigure 2 As shown, the real-time multimodal data fusion device 70 based on dynamic graph neural networks described in this embodiment includes: a preprocessing module 71, a construction module 72, a learning module 73, a generation module 74, an update module 75, and an output module 76. Wherein:

[0211] Preprocessing module 71 is used to perform spatiotemporal alignment and preprocessing on the multimodal data to obtain multimodal data instances;

[0212] Construction module 72 is used to construct a multimodal graph structure based on the multimodal data instance to obtain a dynamic heterogeneous graph;

[0213] Learning module 73 is used to input the dynamic heterogeneous graph into a dynamic graph neural network, aggregate neighbor information through a multi-layer message passing mechanism, learn the high-level, context-aware feature representation of nodes, and obtain a global graph representation.

[0214] The generation module 74 is used to design an attention fusion layer that focuses on intermodal information interaction, selectively and weightedly integrating node features from different modalities to generate a fusion feature representation for each entity;

[0215] Update module 75, used for real-time incremental updates and optimizations;

[0216] Output module 76 is used to output the fusion result based on the fusion feature representation and / or global graph representation.

[0217] The beneficial effects of implementing this embodiment are:

[0218] (1) The spatiotemporal alignment and dynamic graph modeling capabilities significantly improve data correlation. Through spatiotemporal alignment preprocessing of multimodal data, the differences in temporal resolution and spatial scale of heterogeneous data are effectively solved; the construction of dynamic heterogeneous graph structures can capture the dynamic correlation between data in real time, which is especially suitable for dynamic scenarios such as traffic and industrial monitoring, and enhances the adaptability of the model to complex environmental changes;

[0219] (2) Multi-layer message passing mechanism and context-aware feature learning enhance feature representation capabilities. Dynamic graph neural networks can mine high-order relationships between nodes by aggregating multi-layer neighbor information, generating context-aware feature representations containing global semantics. This structure not only preserves the local features of single-modal data, but also achieves feature complementarity through cross-modal interaction, significantly improving feature discriminability.

[0220] (3) The introduction of the attention fusion layer enables intelligent integration of information between modalities. By dynamically allocating the weights of different modalities, this layer can adaptively focus on key information and suppress noise interference. Especially when some modal data is missing or degraded, it can still maintain stable fusion performance. This selective integration mechanism significantly improves the robustness of the fusion results.

[0221] (4) The real-time incremental update mechanism ensures the system's low latency and scalability. Through local updates and parameter optimization of the dynamic graph structure, the model can adapt to changes in data distribution without full retraining, significantly reducing computational overhead while maintaining high accuracy, and providing efficient technical support for real-time decision-making scenarios (such as autonomous driving and intelligent security).

[0222] Example 3

[0223] To address the aforementioned technical problems, embodiments of the present invention also provide a computer device. Please refer to [link / reference needed]. Figure 3 , Figure 3 This is a basic structural block diagram of the computer device in this embodiment.

[0224] The aforementioned computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected via a system bus. It should be noted that only the computer device 8 with components 81, 82, and 83 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0225] The aforementioned computer devices can be desktop computers, laptops, handheld computers, and cloud servers, among other computing devices. These devices can facilitate human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0226] The aforementioned memory 81 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the aforementioned memory 81 may be an internal storage unit of the aforementioned computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the aforementioned memory 81 may also be an external storage device of the aforementioned computer device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 8. Of course, the aforementioned memory 81 may also include both the internal storage unit and its external storage device of the aforementioned computer device 8. In this embodiment, the aforementioned memory 81 is typically used to store the operating system and various application software installed on the aforementioned computer device 8, such as computer-readable instructions for a real-time multimodal data fusion method based on dynamic graph neural networks. In addition, the aforementioned memory 81 can also be used to temporarily store various types of data that have been output or will be output.

[0227] In some embodiments, the processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 82 is typically used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to execute computer-readable instructions stored in the memory 81 or to process data, for example, to execute the computer-readable instructions of the real-time multimodal data fusion method based on dynamic graph neural networks.

[0228] The network interface 83 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 8 and other electronic devices.

[0229] The beneficial effects of implementing this embodiment are as follows: the spatiotemporal alignment and dynamic graph modeling capabilities significantly improve data correlation; the multi-layer message passing mechanism and context-aware feature learning enhance feature expression capabilities; the introduction of the attention fusion layer enables intelligent integration of information between modalities; and the real-time incremental update mechanism ensures the system's low latency and scalability.

[0230] Example 4

[0231] The present invention also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the real-time multimodal data fusion method based on dynamic graph neural networks as described above.

[0232] The beneficial effects of implementing this embodiment are as follows: the spatiotemporal alignment and dynamic graph modeling capabilities significantly improve data correlation; the multi-layer message passing mechanism and context-aware feature learning enhance feature expression capabilities; the introduction of the attention fusion layer enables intelligent integration of information between modalities; and the real-time incremental update mechanism ensures the system's low latency and scalability.

[0233] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0234] Obviously, the embodiments described above are merely some embodiments of the present invention, not all embodiments. The accompanying drawings show preferred embodiments of the present invention, but do not limit the patent scope of the present invention. The present invention can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the patent protection scope of this invention.

Claims

1. A real-time multimodal data fusion method based on dynamic graph neural networks, characterized in that, Includes the following steps: The multimodal data is spatiotemporally aligned and preprocessed to obtain multimodal data instances; Based on the aforementioned multimodal data instance, a multimodal graph structure is constructed to obtain a dynamic heterogeneous graph; The dynamic heterogeneous graph is input into a dynamic graph neural network. Through a multi-layer message passing mechanism, neighbor information is aggregated, and high-level, context-aware feature representations of nodes are learned to obtain a global graph representation. The design focuses on an attention fusion layer for intermodal information interaction, which selectively and weightedly integrates node features from different modalities to generate a fused feature representation for each entity. Perform real-time incremental updates and optimizations; The fusion result is output based on the fusion feature representation and / or global graph representation.

2. The real-time multimodal data fusion method based on dynamic graph neural networks according to claim 1, characterized in that, The steps of performing spatiotemporal alignment and preprocessing on the multimodal data to obtain multimodal data instances specifically include: The multimodal data is subjected to time synchronization and interpolation processing; A unified spatial coordinate transformation and projection are performed on the multimodal data after time synchronization and interpolation processing; The multimodal data, after undergoing unified spatial coordinate transformation and projection, is subjected to modality-specific preprocessing and primary feature extraction to obtain multimodal data instances.

3. The real-time multimodal data fusion method based on dynamic graph neural networks according to claim 1, characterized in that, The specific steps for constructing a multimodal graph structure based on the multimodal data instance to obtain a dynamic heterogeneous graph include: Based on the multimodal data instance, the type and granularity of nodes in the multimodal graph structure are determined, and an initial feature vector is assigned to each node; Establish various connection relationships between nodes, and the edge type represents spatial proximity, temporal association, modal complementarity, or semantic similarity; Based on the current task and node importance, the graph is sparsified, preset connections are retained based on edge weights, and dynamic update rules for the graph structure are defined to obtain a dynamic heterogeneous graph.

4. The real-time multimodal data fusion method based on dynamic graph neural networks according to claim 1, characterized in that, The steps of inputting the dynamic heterogeneous graph into a dynamic graph neural network, aggregating neighbor information through a multi-layer message passing mechanism, learning high-level, context-aware feature representations of nodes, and obtaining a global graph representation specifically include: Attention message passing is performed on the dynamic heterogeneous graph. On the dynamic heterogeneous graph, the importance of different node types and edge types to message aggregation is distinguished to achieve information fusion. The dynamic heterogeneous graph is subjected to temporal graph convolution and memory enhancement. In multi-layer dynamic graph neural networks, pooling of graph structures is implemented to capture structural information at different scales, and a global graph representation is generated in the final layer for graph-level tasks.

5. The real-time multimodal data fusion method based on dynamic graph neural networks according to claim 1, characterized in that, The design focuses on an attention fusion layer for intermodal information interaction. The steps involved in selectively and weightedly integrating node features from different modalities to generate a fused feature representation for each entity specifically include: The importance of features from each modality source to the target node during fusion is calculated, i.e., intra-modal attention, and the relative importance between different modalities is calculated, i.e., inter-modal attention. Gating mechanisms are used to control how information from different modalities flows into and updates the fused representation of the target node; A loss function is introduced during the training phase.

6. The real-time multimodal data fusion method based on dynamic graph neural networks according to claim 1, characterized in that, The steps for performing real-time incremental updates and optimizations specifically include: When a new data frame arrives, the graph nodes and edges of the multimodal graph structure are updated; Perform lightweight dynamic graph neural network inference and caching; Conduct online learning and parameter adaptation.

7. The real-time multimodal data fusion method based on dynamic graph neural networks according to any one of claims 1 to 6, characterized in that, The step of outputting the fusion result based on the fusion feature representation and / or global graph representation specifically includes: Based on application requirements, design a fully connected network or a multilayer perceptron to map the fused features to the output space required by the task. Perform multi-task joint learning and output fusion; The original output of the model is post-processed to generate the final result, which is then presented to the user or sent to the control system.

8. A real-time multimodal data fusion device based on a dynamic graph neural network, characterized in that, include: The preprocessing module is used to perform spatiotemporal alignment and preprocessing on the multimodal data to obtain multimodal data instances; The construction module is used to construct a multimodal graph structure based on the multimodal data instance to obtain a dynamic heterogeneous graph; The learning module is used to input the dynamic heterogeneous graph into the dynamic graph neural network, aggregate neighbor information through a multi-layer message passing mechanism, learn the high-level, context-aware feature representation of nodes, and obtain a global graph representation. The generation module is used to design an attention fusion layer that focuses on intermodal information interaction. It selectively and weightedly integrates node features from different modalities to generate a fusion feature representation for each entity. The update module is used for real-time incremental updates and optimizations; The output module is used to output the fusion result based on the fusion feature representation and / or global graph representation.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the real-time multimodal data fusion method based on a dynamic graph neural network as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the real-time multimodal data fusion method based on a dynamic graph neural network as described in any one of claims 1 to 7.

Images