Multi-modal data situation intelligent arrangement system and method

Abstract

The invention provides a multi-modal data situation intelligent arrangement system and method, and relates to the technical field of data arrangement. The system comprises a dynamic heterogeneous data fusion module, an intelligent situation arrangement module and an incremental situation analysis module. And the dynamic heterogeneous data fusion module maps the multi-modal data flow into a hypergraph structure with dynamic weight through a semantic hypergraph fusion method of spatio-temporal context perception. And the situation intelligent arrangement module generates an evolution path of the cross-modal situation according to the node association strength and the semantic similarity in the hypergraph structure. And the incremental situation analysis module performs multi-granularity situation prediction and anomaly detection on an evolution path through hypergraph tensor decomposition and an attention mechanism, and dynamically optimizes an arrangement strategy. The method can be used for rapidly generating a large-screen situation display page of a data track and a situation thermodynamic diagram, high-speed situation arrangement and updating are achieved, and visual and dynamic support is provided for commanding and decision making.

Description

Technical Field

[0001] The present invention relates to the technical field of data orchestration, and specifically to a multi-modal data situation intelligent orchestration system and method. Background Art

[0002] In applications such as smart cities, industrial Internet, and environmental monitoring, multi-modal data usually includes various types such as text, images, videos, and sensor data, and these data have complex correlations. The core of data orchestration is to achieve real-time observation and analysis of the situation through effective organization and scheduling of data.

[0003] With the development of big data and artificial intelligence technologies, existing technologies have been able to perform orchestration processing on single-modal or simple multi-modal data. These methods have obvious limitations when dealing with multi-modal data. Existing technologies are difficult to effectively fuse multi-modal data with different spatio-temporal characteristics, resulting in insufficient mining of the correlations between data and limited accuracy of the orchestration results; traditional methods cannot adapt to the dynamic changes of data sources, resulting in insufficient self-adaptability and comprehensiveness of the orchestration. Summary of the Invention

[0004] To achieve the above objectives, the present invention is realized through the following technical solutions: A multi-modal data situation intelligent orchestration system, including:

[0005] A dynamic heterogeneous data fusion module: maps multi-modal data streams into a hypergraph structure with dynamic weights through a semantic hypergraph fusion method based on spatio-temporal context awareness;

[0006] A situation intelligent orchestration module: generates an evolution path of cross-modal situations according to the node association strength and semantic similarity in the hypergraph structure;

[0007] An incremental situation analysis module: performs multi-granularity situation prediction and anomaly detection on the evolution path through hypergraph tensor decomposition and attention mechanism, and dynamically optimizes the orchestration strategy;

[0008] Among them, the semantic hypergraph fusion method includes dynamic weight calculation, and the dynamic weight is calculated by compounding the creditability C, update rate T, and context association degree R of the data source.

[0009] The dynamic heterogeneous data fusion module includes a spatio-temporal context modeling unit, a semantic association encoding unit, and a hypergraph generation unit; the spatio-temporal context modeling unit compensates for the spatio-temporal offsets of multi-modal data using a bidirectional spatio-temporal gated network; for example, data from different sensors, such as traffic flow data and meteorological data, have offsets due to differences in collection time or spatial location; this unit models the time dimension and the spatial dimension respectively through a bidirectional gating mechanism, and outputs an aligned spatio-temporal feature vector to ensure the spatio-temporal consistency of multi-modal data; the semantic association encoding unit constructs a cross-modal semantic projection space through a contrastive learning framework to generate modality-independent semantic embedding representations; the semantic association encoding unit adopts a contrastive learning algorithm to construct a unified semantic projection space by maximizing the embedding similarity of the same semantic concept in different modalities while minimizing the embedding similarity of different semantic concepts; for example, the text description such as good air quality and the PM2.5 concentration value of sensor data are mapped to the same semantic space through this unit to achieve cross-modal semantic alignment; the hypergraph generation unit performs tensor splicing on the spatio-temporal feature vector and the semantic embedding representation, and dynamically constructs a hypergraph structure through a differentiable hyperedge generation algorithm; the hyperedge weights are dynamically updated according to the formula W = αC + βT + γR, where C is the creditability of the data source, obtained through data quality assessment; T is the update rate, obtained by normalizing the difference between the data collection time and the current time; R is the context correlation degree, obtained through semantic similarity calculation and context feature extraction; the weight coefficients α, β, and γ are used to adjust the relative importance of each factor, for example, the device sensor data and the operation and maintenance log data dynamically construct a hypergraph structure through this unit, and the hyperedge weights reflect the reliability and relevance of the data source in real time, thus supporting intelligent orchestration decisions.

[0010] The differentiable hyperedge generation algorithm constructs a potential hyperedge candidate set, and constructs a cross-modal semantic projection space through a contrastive learning framework to map the node embeddings of different modalities into a unified semantic space, where each node corresponds to data of one modality, such as text, image, sensor data; the cross-modal semantic distance between nodes is calculated by cosine similarity, and candidate hyperedges in the hyperedge candidate set are screened based on a dynamic threshold; the setting of the dynamic threshold is based on the distribution characteristics of semantic similarity in the real-time data stream: continuously monitor the similarity mean and variance of the current data batch, and adjust the threshold range in combination with the statistical results of the historical window; for example, when the noise interference of the input data is strong, the threshold is increased to the upper quartile of the similarity distribution, and only the node combinations with semantic distances less than this threshold are retained as candidate hyperedges; the state of reinforcement learning is defined as the topological features of the current hypergraph, including the dispersion of the node degree distribution, the density of the spatio-temporal regions covered by hyperedges, and the semantic consistency of cross-modal associations; the relationship between topological optimization and path generation performance is dynamically balanced through gradient updates. If a certain hyperedge, such as intersection camera data - weather report, improves the accuracy of path generation, its selection probability will be increased; the hyperedge weights and node embedding representations are jointly optimized through gradient backpropagation; during the forward propagation process, the situation evolution path is generated according to the current hyperedge weights and node embedding representations; then the loss function between the generated result and the ground truth is calculated, and the system propagates the gradient of the loss function to the hyperedge weights and node embedding representations through gradient backpropagation, and updates their values according to these gradient information to achieve joint optimization; for example, the hyperedge weights are dynamically optimized according to the coefficients α, β, γ in the formula W = αC + βT + γR. If the high update rate T of real-time weather data is important for the generation of the evolution path, gradient backpropagation will automatically increase the weight of β, making the hyperedge more inclined to fuse data sources with high update rates; the optimized hypergraph structure connection is fed back to the dynamic heterogeneous data fusion module to continuously improve semantic consistency and path generation accuracy.

[0011] In the situation intelligent orchestration module, the node association strength is quantified by the dynamic weight W of the belonging hyper-edge. For example, if nodes A and B belong to a hyper-edge with a weight of 0.8, then the association strength between them is 0.8; if they belong to multiple hyper-edges simultaneously, the weights are accumulated to enhance the association. The semantic similarity is obtained by calculating the cosine similarity through the semantic association coding unit. The situation evolution stage is divided based on the hypergraph structure, and each stage corresponds to a certain spatio-temporal semantic pattern. The system analyzes the characteristics of nodes and hyper-edges in the hypergraph, identifies regions or time periods with similar spatio-temporal characteristics and semantic meanings, and divides them into a situation evolution stage. For example, the system divides the morning rush hour, flat peak period, and evening rush hour into different situation evolution stages according to the change law of traffic flow and road conditions, and each stage corresponds to a spatio-temporal semantic pattern. For example, the morning rush hour stage corresponds to the pattern of large traffic flow and road congestion. The evolution path generation is constructed through logical reasoning rules and real-time data matching mechanisms. The logical reasoning rules are extracted based on the causal relationships of domain knowledge and historical data statistical characteristics, and the expert knowledge is transformed into logical expressions. Domain experts define a series of causal relationship rules according to their own experience and domain knowledge. For example, if the traffic flow suddenly increases and there is road construction, it will lead to traffic congestion. At the same time, the system extracts statistical characteristics from historical data, such as the distribution law of traffic flow in different time periods, the frequency of accidents, etc., to further improve the logical reasoning rules. The real-time data matching mechanism preprocesses the real-time monitoring data, extracts features and matches them with the logical reasoning rules. The system collects data in real time, performs operations such as data cleaning, noise reduction, and normalization on the data, extracts the features of the data, and matches the features with the conditions in the logical reasoning rules. When the conditions of a certain rule are met, the corresponding path generation action is triggered, and the node sequence with high association strength and consistent semantics is preferentially selected as the initial path. For example, if it is real-time monitored that the traffic flow of a certain road section suddenly increases and road construction is underway at this section, the system matches the above logical reasoning rule and triggers the generation of an evolution path of traffic congestion. When implementing path optimization, the system compares and adjusts the generated evolution path with the physical constraint conditions to ensure the feasibility and safety of the path.

[0012] In the incremental situation analysis module, a hierarchical hypergraph tensor decomposition framework is designed to decompose the original hypergraph into a global situation tensor, a local pattern tensor, and an abnormal residual tensor; the hierarchical hypergraph tensor decomposition framework performs an asymmetric Tucker decomposition on the hypergraph adjacency tensor to retain high-order interaction features; the system first initializes the decomposition parameters, including the dimensions of the core tensor and the scales of the factor matrices; the factor matrices and the core tensor are iteratively updated by the alternating least squares method until convergence; for example, according to the time series and spatial distribution of the data, the system decomposes the hypergraph into a global situation tensor that captures the overall trend, a local pattern tensor that reflects the patterns in specific regions or time periods, and an abnormal residual tensor that represents the deviation from the normal pattern; among them, the core tensor is a low-dimensional tensor, and each of its dimensions corresponds to a specific semantic concept, which is used to explicitly describe the complex interaction patterns between modalities such as nodes and hyperedges in the hypergraph; for example, a certain dimension of the core tensor represents the correlation strength between modality A and modality B, and another dimension represents the importance of modality C in a certain context; through the asymmetric Tucker decomposition, the core tensor decomposes the hypergraph adjacency tensor into the product form of the core tensor and a set of factor matrices; during the decomposition process, an interpretability constraint is imposed on the core tensor to force each of its dimensions to align with the semantic concepts.

[0013] During the decomposition process, the system automatically determines the decomposition levels and dimensions according to the distribution characteristics and semantic associations of the data to ensure that each tensor can accurately represent the corresponding situation features; a dual-channel attention mechanism is developed. Channel one focuses on the pattern mutations in the spatio-temporal dimensions, and channel two detects the abnormal associations in the semantic dimensions; for channel one, the system calculates the attention weights of nodes in the spatio-temporal sequence through graph attention; the attention weight of each node is determined by its spatio-temporal correlation with adjacent nodes, and nodes with higher correlation have larger weights; for example, if a certain node shows a sharp fluctuation in the time series, its corresponding attention weight will increase, and thus it will be recognized as a potential mutation point by the system; for channel two, the system detects the abnormal changes in the semantic associations between different modalities based on the similarity calculation in the semantic embedding space; the system calculates the similarity of different modality data in the semantic embedding space, and when the similarity suddenly increases or decreases, an anomaly detection is triggered; an online learning pipeline is constructed. When it is detected that the situation deviates from the predicted path, the weight recalibration and orchestration strategy of the semantic hypergraph fusion method are triggered for real-time update; the situation changes are monitored in real time, and by comparing the current situation with the predicted path, it is judged whether to trigger the update mechanism; for example, when the deviation between the actual situation and the predicted path exceeds the preset threshold, the system transmits the decomposition result to the dynamic heterogeneous data fusion module through feedback, recalculates the credibility, update rate, and context association degree of the data sources, calibrates the hyperedge weights, and adjusts the orchestration strategy according to the new weights.

[0014] The present invention provides a multi-modal data situation intelligent orchestration system and method, which has the following beneficial effects:

[0015] 1. Through the semantic hypergraph fusion method with spatio-temporal context awareness, the present invention dynamically calculates the credibility, update rate, and context correlation degree of data sources, solves the problem of orchestration deviation caused by relying on static weights or single indicators, and significantly improves the accuracy and robustness of multi-modal data orchestration.

[0016] 2. Through the node association strength and semantic similarity of the hypergraph structure, combined with logical reasoning rules and real-time data matching mechanisms, the present invention dynamically generates and optimizes the situation evolution path, solves the limitations of artificial rules or offline statistical features, and significantly improves the real-time performance and self-adaptability of situation orchestration.

[0017] 3. Through the hierarchical hypergraph tensor decomposition framework and dual-channel attention mechanism, the present invention conducts multi-granularity analysis on the orchestration path, and triggers dynamic weight recalibration and orchestration strategy update when a situation deviation is detected, solves the problems of update lag and isolated module optimization, and significantly improves the real-time performance and interpretability of the overall system orchestration performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0020] Embodiment 1:

[0021] The dynamic heterogeneous data fusion module uses a semantic hypergraph fusion method with spatio-temporal context awareness to map multi-modal data streams into a hypergraph structure with dynamic weights, solving the problems of heterogeneity of multi-modal data in terms of time, space, and semantics. The spatio-temporal context modeling unit uses a bidirectional spatio-temporal gated network to compensate for the spatio-temporal offsets of multi-modal data. The time gating unit uses a long short-term memory network to model time series, and the space gating unit captures spatial correlations through a graph convolutional network. After the input data is processed by the bidirectional gated network, an aligned spatio-temporal feature vector is output, and the spatio-temporal feature dimension is set to 128 dimensions. The semantic association encoding unit constructs a cross-modal semantic projection space through a contrastive learning framework, uses cosine similarity as the loss function, the dimension of the embedding vector is 64 dimensions, the batch size is set to 256, and the learning rate is 0.001 during training. The hypergraph generation unit performs tensor splicing on the spatio-temporal feature vector and the semantic embedding representation, with an input dimension of 192 dimensions, and dynamically constructs hyperedge weights through a differentiable hyperedge generation algorithm. The dynamic weight calculation formula is W = 0.4C + 0.3T + 0.3R, where the credibility C is calculated through data quality assessment. For example, when the signal-to-noise ratio of sensor data ≥ 30dB, C = 1.0. The update rate T is normalized to the [0, 1] interval according to the time difference between data acquisition time and the current time. For example, when the time difference ≤ 5 minutes, T = 1.0. The context correlation degree R is calculated through cross-modal semantic similarity. For example, when the similarity between text and sensor data ≥ 0.8, R = 1.0. Reinforcement learning takes the node association strength and situation prediction accuracy in the hypergraph structure as the optimization objectives, and optimizes hyperedge generation by adjusting the weight coefficients α, β, and γ. During implementation, the system obtains data in real time. When a logical inference rule is matched, it triggers an adjustment of hyperedge weights. For example, it increases the β value of high-update-rate data. The optimized hypergraph structure updates the node embedding representation through gradient backpropagation and feeds it back to the incremental situation analysis module for situation prediction. It significantly improves the semantic consistency of multi-modal data fusion, reduces the credibility error, and improves the orchestration efficiency.

[0022] The semantic association encoding unit constructs a cross-modal semantic projection space through a contrastive learning framework to generate modality-independent semantic embedding representations. For example, for text descriptions and corresponding sensor data, by maximizing the embedding similarity of the same semantic concept in different modalities while minimizing the embedding similarity of different semantic concepts, a unified semantic projection space is constructed. Assuming the initial embedding dimension of text data is 300 and the initial embedding dimension of sensor data is 200, after being processed by the contrastive learning framework, the generated modality-independent semantic embedding representation dimension is unified to 256, achieving cross-modal semantic alignment and significantly improving the accuracy of semantic association.

[0023] The hypergraph generation unit performs tensor concatenation on the spatio-temporal feature vector and the semantic embedding representation, and dynamically constructs a hypergraph structure through a differentiable hyperedge generation algorithm; the hyperedge weights are dynamically updated according to W = αC + βT + γR, and the weight coefficients α, β, and γ are used to adjust the relative importance of each factor. For example, in a certain data processing, α = 0.4, β = 0.3, γ = 0.3, and the hyperedge weights are calculated through this formula, dynamically reflecting the reliability and relevance of the data source, thereby supporting intelligent orchestration decisions; assume that in a data fusion process, there are a total of 1000 data nodes, and a hypergraph structure is dynamically constructed through the hypergraph generation algorithm, and the number of hyperedges is automatically determined according to the data association situation, effectively improving the flexibility and accuracy of data fusion.

[0024] The situation intelligent orchestration module divides the situation evolution stages based on the hypergraph structure, and each stage corresponds to a certain spatio-temporal semantic pattern; the system analyzes the characteristics of nodes and hyperedges in the hypergraph, identifies regions or time periods with similar spatio-temporal characteristics and semantic meanings, and divides them into a situation evolution stage; for example, according to the change rules of traffic flow and road conditions, the morning rush hour, the off-peak period, and the evening rush hour are divided into different situation evolution stages, and each stage corresponds to a spatio-temporal semantic pattern, such as the morning rush hour stage corresponding to the pattern of large traffic flow and road congestion, the off-peak period corresponding to the pattern of moderate traffic flow and smooth roads, and the evening rush hour corresponding to the pattern of increasing traffic flow again and partial road congestion.

[0025] The evolution path generation is constructed through logical inference rules and a real-time data matching mechanism; the logical inference rules are based on the causal relationship modeling of domain knowledge and the extraction of historical data statistical features, and transform expert knowledge into logical expressions; for example, domain experts define a series of causal relationship rules based on their own experience and domain knowledge, such as if the traffic flow suddenly increases and there is road construction, then traffic congestion will be caused; at the same time, the system extracts statistical features from historical data, such as the distribution law of traffic flow in different time periods, the frequency of accidents, etc., to further improve the logical inference rules; the real-time data matching mechanism preprocesses the real-time monitoring of data, extracts features and matches them with the logical inference rules; the system collects data in real time, performs operations such as data cleaning, noise reduction, and normalization on the data, extracts the features of the data, and matches the features with the conditions in the logical inference rules; when the conditions of a certain rule are met, the corresponding path generation action is triggered; for example, if it is real-time monitored that the traffic flow of a certain section suddenly increases and the section is under construction, the system matches the above logical inference rule and triggers the generation of an evolution path of traffic congestion; when implementing path optimization, the system compares and adjusts the generated evolution path with the physical constraint conditions to ensure the feasibility and safety of the path, such as avoiding generating paths that violate traffic rules, significantly improving the accuracy and reliability of situation evolution.

[0026] The incremental situation analysis module designs a hierarchical hypergraph tensor decomposition framework, which decomposes the original hypergraph into a global situation tensor, a local pattern tensor, and an abnormal residual tensor; the hierarchical hypergraph tensor decomposition framework performs an asymmetric Tucker decomposition on the hypergraph adjacency tensor to retain high-order interaction features; the system first initializes the decomposition parameters, including the dimensions of the core tensor and the scales of the factor matrices; the factor matrices and the core tensor are iteratively updated by the alternating least squares method until convergence; for example, according to the time series and spatial distribution of the data, the system decomposes the hypergraph into a global situation tensor that captures the overall trend, a local pattern tensor that reflects the patterns in specific regions or time periods, and an abnormal residual tensor that represents the deviation from the normal pattern; among them, the core tensor is a low-dimensional tensor, and each of its dimensions corresponds to a specific semantic concept, which is used to explicitly describe the complex interaction patterns between modalities such as nodes and hyperedges in the hypergraph; for example, a certain dimension of the core tensor represents the correlation strength between modality A and modality B, and another dimension represents the importance of modality C in a certain context; through the asymmetric Tucker decomposition, the core tensor decomposes the hypergraph adjacency tensor into the product form of the core tensor and a set of factor matrices; during the decomposition process, interpretability constraints are imposed on the core tensor to force each of its dimensions to align with the semantic concepts, effectively improving the interpretability and accuracy of the situation analysis.

[0027] Develop a dual-channel attention mechanism. Channel one focuses on the pattern mutations in the spatio-temporal dimension, and channel two detects the correlation anomalies in the semantic dimension; for channel one, the system calculates the attention weights of nodes in the spatio-temporal sequence through graph attention; the attention weight of each node is determined by its spatio-temporal correlation with adjacent nodes, and nodes with higher correlation have larger weights; for example, if a certain node shows a sharp fluctuation in the time series, its corresponding attention weight will increase, and thus it will be recognized as a potential mutation point by the system; for channel two, based on the similarity calculation in the semantic embedding space, the system detects the abnormal changes in the semantic correlation between different modalities; the system calculates the similarity of different modality data in the semantic embedding space, and when the similarity suddenly increases or decreases, it triggers anomaly detection; construct an online learning pipeline. When it is detected that the situation deviates from the predicted path, it triggers the real-time update of the weight recalibration and orchestration strategy of the semantic hypergraph fusion method; monitor the situation changes in real time, and judge whether to trigger the update mechanism by comparing the difference between the current situation and the predicted path; for example, when the deviation between the actual situation and the predicted path exceeds the preset threshold, the system passes the decomposition result to the dynamic heterogeneous data fusion module through feedback, recalculates the credibility, update rate, and context correlation degree of the data source, calibrates the hyperedge weights, and adjusts the orchestration strategy according to the new weights, significantly improving the real-time performance and interpretability of the overall system orchestration performance.

[0028] Example two:

[0029] For example, applying the present invention to a military exercise scenario, by accessing multi-source heterogeneous data such as satellite remote sensing data, ship AIS signals, aircraft ADS-B data, radar monitoring streams, meteorological sensors, tactical instruction texts, and 3D geographic information models, dynamic situation choreography is achieved, and an interactive large-screen display page is generated.

[0030] The dynamic heterogeneous data fusion module first performs spatio-temporal alignment and semantic association on multi-modal data; there are spatio-temporal offsets between satellite images and radar monitoring data due to different acquisition times, and the system uses a bidirectional spatio-temporal gating network for compensation; the time gating unit models the time series based on the long short-term memory network, and the spatial gating unit corrects the geographical coordinate deviation through the graph convolutional network, and finally outputs a 128-dimensional spatio-temporally aligned feature vector; at the same time, the semantic association encoding unit uses a contrastive learning framework to map the text instructions and 3D model coordinates to a unified 256-dimensional semantic space to ensure cross-modal semantic consistency.

[0031] The hypergraph generation unit concatenates the spatio-temporal features and semantic embeddings into a 192-dimensional tensor, and dynamically constructs a hypergraph structure through a differentiable hyperedge generation algorithm; for example, the hyperedge weights between aircraft carrier nodes and escort ship nodes are calculated by compounding the credibility, update rate, and context correlation degree of the data sources; the credibility is determined through data quality assessment, such as when the signal-to-noise ratio of radar data is higher than 35 dB, the credibility is 1.0; the update rate is normalized according to the data acquisition time difference, and when the time difference of real-time AIS signals is less than 2 minutes, the update rate is 1.0; the context correlation degree is calculated through semantic similarity, such as when the semantic similarity between meteorological data and the navigation path is higher than 0.85, the correlation degree is 0.9; the final hyperedge weight calculation formula is W = 0.4C + 0.3T + 0.3R, where C, T, and R represent the credibility, update rate, and context correlation degree respectively;

[0032] The situation intelligent choreography module divides the exercise stage based on the hypergraph structure, and each stage corresponds to a specific spatio-temporal semantic pattern; for example, the system identifies a node cluster with a hyperedge weight higher than 0.8 in the aircraft carrier group, combines the statistical characteristics of historical data and expert rules, and matches the current sensor data in real time; when an abnormal target is detected by the shipborne radar, the system triggers a logical rule, generates an evolution path for the aircraft carrier formation to turn into a defensive formation, and highlights the threat area on the large-screen page; physical constraint conditions, such as the minimum safe distance between ships, are introduced in the path optimization stage to ensure that the generated formation conforms to the actual navigation specifications.

[0033] The incremental situation analysis module decomposes the original hypergraph into a global situation tensor, a local pattern tensor, and an abnormal residual tensor through a hierarchical hypergraph tensor decomposition framework; the global situation tensor captures the overall navigation trend, the local pattern tensor reflects the cooperation pattern of escort ships, and the abnormal residual tensor identifies data that deviates from the normal pattern, such as abnormal radar signals.

[0034] The dual-channel attention mechanism monitors the situational changes in real time; the spatio-temporal channel focuses on the mutations of nodes in the spatio-temporal sequence. For example, when the speed of a certain ship suddenly changes, the corresponding attention weight surges; the semantic channel detects the abnormal correlations between different modalities. For example, when the tactical instruction deviates from the actual course, the semantic similarity drops sharply; when the detected situation deviates from the predicted path, the system triggers the weight recalibration mechanism, dynamically adjusts the hyperedge weights and updates the choreography strategy; for example, the credibility of an abnormal radar signal drops to 0.6, the system recalculates the hyperedge weights and synchronously refreshes to the abnormal target tracking mode on the large-screen page, and highlights the suspicious target trajectory in the three-dimensional view.

[0035] The final large-screen situational page generated by the system integrates a two-dimensional nautical chart, a three-dimensional exercise scenario, a real-time data panel and a text briefing; users can freely combine the data layers through the interactive interface. For example, overlay the meteorological cloud chart to predict the route risk, or associate the news text to mark the exercise stage; the layout and visualization styles of all elements are intelligently choreographed by the system to ensure the balance between information density and readability; when the exercise switches to the night mode, the system automatically switches to the infrared thermal layer and dynamically adjusts the color contrast to reflect the all-weather adaptive ability.

[0036] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-modal data situation intelligent orchestration system, characterized in that Including: Dynamic heterogeneous data fusion module: Maps multi-modal data streams into a hypergraph structure with dynamic weights through a semantic hypergraph fusion method based on spatio-temporal context awareness; Situation intelligent orchestration module: Generates the evolution path of cross-modal situations according to the node association strength and semantic similarity in the hypergraph structure; Incremental situation analysis module: Performs multi-granularity situation prediction and anomaly detection on the evolution path through hypergraph tensor decomposition and attention mechanism, and dynamically optimizes the orchestration strategy; Among them, the semantic hypergraph fusion method includes dynamic weight calculation, and the dynamic weight is calculated by compounding the credibility, update rate, and context association degree of the data source.

2. The multimodal data situation intelligent orchestration system according to claim 1, wherein: The dynamic heterogeneous data fusion module includes: Spatio-temporal context modeling unit: Compensates the spatio-temporal offset of multi-modal data using a bidirectional spatio-temporal gated network, and outputs the aligned spatio-temporal feature vector; Semantic association encoding unit: Constructs a cross-modal semantic projection space through a contrastive learning framework, and generates modality-independent semantic embedding representations; Hypergraph generation unit: Performs tensor splicing on the spatio-temporal feature vector and the semantic embedding representation, and dynamically constructs a hypergraph structure through a differentiable hyperedge generation algorithm, where the hyperedge weight is dynamically updated according to the following formula: W = αC + βT + γR, where W is the hyperedge weight, C is the credibility, T is the update rate, and R is the context association degree; α, β, and γ are weight coefficients used to adjust the relative importance of each factor.

3. The multimodal data situation intelligent orchestration system according to claim 2, characterized in that: When constructing the potential hyperedge candidate set, the differentiable hyperedge generation algorithm introduces a semantic similarity threshold constraint, and only retains the node combinations with a cross-modal semantic distance less than the dynamic threshold; uses reinforcement learning to optimize hyperedge generation, considering both the minimization of topological structure entropy and the improvement of situation prediction accuracy; the differentiable hyperedge generation algorithm jointly optimizes the hyperedge weight and the node embedding representation through gradient backpropagation.

4. A multimodal data situation intelligent orchestration system according to claim 1, characterized in that: The node association strength of the situation intelligent orchestration module is obtained from the hyperedge weight to which it belongs. If two nodes belong to the same hyperedge, the association strength between them is the same hyperedge weight; If they belong to multiple hyperedges at the same time, the weights are accumulated to obtain it; The semantic similarity is obtained by calculating the cosine similarity between nodes through the modality-independent semantic embedding representation generated by the semantic association encoding unit; divides the situation evolution stage based on the hypergraph structure, and each stage corresponds to a certain spatio-temporal semantic pattern; constructs the evolution path generation through a logical reasoning rule and a real-time data matching mechanism. The logical reasoning rule is based on the causal relationship modeling of domain knowledge and the extraction of historical data statistical features, transforms expert knowledge into logical expressions, and uses the node association strength and semantic similarity as trigger conditions. Through the real-time data matching mechanism, the sensor data is monitored and preprocessed in real time, features are extracted and dynamically matched with the logical rules, and the candidate node sequence that meets the association strength threshold and semantic consistency requirements is screened. If the conditions are met, the path generation is triggered; physical constraint conditions are used during path optimization.

5. The multimodal data situation intelligent orchestration system according to claim 1, characterized in that: The incremental situation analysis module designs a hierarchical hypergraph tensor decomposition framework to decompose the original hypergraph into a global situation tensor, a local pattern tensor, and an anomaly residual tensor; develops dual-channel attention, where channel one focuses on pattern mutations in the spatio-temporal dimension, and channel two detects correlation anomalies in the semantic dimension; constructs an online learning pipeline, and when a situation deviates from the predicted path, triggers real-time updates of the weight recalibration of the semantic hypergraph fusion method and the orchestration strategy.

6. The multimodal data situation intelligent orchestration system according to claim 5, wherein: The hierarchical hypergraph tensor decomposition framework performs asymmetric Tucker decomposition on the hypergraph adjacency tensor to retain high-order interaction features; embeds interpretability constraints in the core tensor to force different dimensions to correspond to specific semantic concepts; feeds the decomposition result back to the dynamic heterogeneous data fusion module through residual connections; where the core tensor is jointly optimized by asymmetric Tucker decomposition and semantic constraints and is used to describe the high-order interaction relationships between different modalities in the hypergraph.

7. The method proposed by a multi-modal data situation intelligent orchestration system according to any one of claims 1-6, characterized in that, It includes the following steps: S1. Through a spatio-temporal context-aware semantic hypergraph fusion method, map the multi-modal data stream into a hypergraph structure with dynamic weights, where the dynamic weights are calculated by a composite of the credit C, update rate T, and context relevance R of the data source, specifically: W = αC + βT + γR, which is used to balance the contributions of various factors; S2. Divide the situation evolution stage according to the association strength of nodes and cross-modal semantic similarity in the hypergraph structure, and construct the generation of the evolution path through logical inference rules and real-time data matching mechanisms, and at the same time introduce physical constraint conditions to optimize the path generation; S3. Perform multi-granularity prediction and anomaly detection on the evolution path through the hierarchical hypergraph tensor decomposition framework, including: Decompose the hypergraph into a global situation tensor, a local pattern tensor, and an anomaly residual tensor; Adopt a dual-channel attention mechanism to detect spatio-temporal dimension mutations and semantic dimension anomalies respectively; When it is detected that the situation deviates from the predicted path, trigger real-time updates of the dynamic weight recalibration and the orchestration strategy; S4. Feed the decomposition result back to the dynamic heterogeneous data fusion module through residual connections to continuously improve the semantic consistency of the hypergraph structure and the situation prediction accuracy.

Images