Multimodal spatiotemporal data visualization method, device and equipment

Abstract

The present invention relates to the field of data visualization technology, and discloses a multimodal spatiotemporal data visualization method, apparatus, and device. The method comprises: performing time alignment and spatial registration on original multimodal spatiotemporal data to generate a grid feature vector set and a spatiotemporal adjacency matrix; performing multidimensional anomaly assessment to generate a multimodal spatiotemporal anomaly detection feature matrix; establishing decision tree nodes and determining abnormal areas; inputting the spatiotemporal feature sequence of the abnormal area into a Transformer decoder, calculating the correlation strength between the abnormal area and normal grid cells, and performing recurrent neural network iterative calculation on the correlation strength to generate an abnormal diffusion intensity sequence; performing interval mapping operation on the abnormal diffusion intensity sequence, establishing a time series evolution axis and a retrieval interaction component, and outputting a multi-level visualization interface. The present invention constructs a multi-level visualization interface, supports real-time monitoring and interactive query of abnormal areas, and enhances the comprehensibility and usability of anomaly detection results.

Description

Technical Field

[0001] The present invention relates to the field of data visualization technology, and in particular to a multimodal spatiotemporal data visualization method, device, and equipment. Background Art

[0002] With the rapid development of Internet of Things (IoT) technology and sensor devices, the collection and accumulation of multimodal spatiotemporal data has exploded. These data contain rich temporal variation characteristics and spatial distribution characteristics. In areas such as urban management, environmental monitoring, and transportation, timely detection and location of abnormal events are crucial for ensuring the safe operation of these systems. However, due to issues such as inconsistent sampling frequencies, uneven spatial distribution, and variable data quality in multimodal data, traditional single-modal anomaly detection methods struggle to effectively leverage the complementarity and synergy between these data.

[0003] Current anomaly detection methods have three main limitations: the lack of a unified processing framework for multimodal data makes it impossible to fully explore the correlation characteristics between different modal data; secondly, existing anomaly detection algorithms often ignore the continuity and local correlation of spatiotemporal data, resulting in instability in detection results; and the visualization of anomaly detection results is single, making it difficult to intuitively display the evolution process and cause analysis of abnormal events. Summary of the Invention

[0004] The present invention provides a multimodal spatiotemporal data visualization method, apparatus, and device. The present invention constructs a multi-level visualization interface, supports real-time monitoring and interactive query of abnormal areas, and enhances the comprehensibility and usability of anomaly detection results.

[0005] In a first aspect, the present invention provides a multimodal spatiotemporal data visualization method, the multimodal spatiotemporal data visualization method comprising:

[0006] Perform temporal alignment and spatial registration on the original multimodal spatiotemporal data to generate a set of grid feature vectors and a spatiotemporal adjacency matrix;

[0007] Performing multi-dimensional anomaly assessment on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multimodal spatiotemporal anomaly detection feature matrix;

[0008] Establishing decision tree nodes based on the multimodal spatiotemporal anomaly detection feature matrix and determining abnormal areas;

[0009] Input the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, calculate the correlation strength between the abnormal area and the normal grid unit, and perform recurrent neural network iterative calculation on the correlation strength to generate an abnormal diffusion intensity sequence;

[0010] An interval mapping operation is performed on the abnormal diffusion intensity sequence, a time series evolution axis and a retrieval interaction component are established, and a multi-level visualization interface is output.

[0011] In a second aspect, the present invention provides a multimodal spatiotemporal data visualization device, the multimodal spatiotemporal data visualization device comprising:

[0012] The registration module is used to perform temporal and spatial registration on the original multimodal spatiotemporal data to generate a set of grid feature vectors and a spatiotemporal adjacency matrix;

[0013] An evaluation module, configured to perform multi-dimensional anomaly evaluation on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multimodal spatiotemporal anomaly detection feature matrix;

[0014] An establishment module is used to establish decision tree nodes based on the multimodal spatiotemporal anomaly detection feature matrix and determine abnormal areas;

[0015] A calculation module is used to input the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, calculate the correlation strength between the abnormal area and the normal grid unit, and perform recurrent neural network iterative calculation on the correlation strength to generate an abnormal diffusion intensity sequence;

[0016] The output module is used to perform interval mapping operation on the abnormal diffusion intensity sequence, establish a time series evolution axis and retrieve interactive components, and output a multi-level visualization interface.

[0017] A third aspect of the present invention provides a computer device comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the above-mentioned multimodal spatiotemporal data visualization method.

[0018] The technical solution provided by this invention uniformly maps heterogeneous multimodal spatiotemporal data into a regular grid structure through time alignment and spatial registration, thus resolving the data heterogeneity issue. A multidimensional anomaly assessment mechanism is adopted, comprehensively considering data integrity, modal synergy, and spatiotemporal coverage, effectively reducing the problem of misjudgment caused by uneven data quality. A hierarchical detection strategy based on decision trees and random forests fully utilizes historical data and spatial correlation information to improve the robustness and accuracy of anomaly detection. A hybrid architecture combining a Transformer decoder and a recurrent neural network enables dynamic propagation analysis of anomaly events, accurately capturing their diffusion patterns. A multi-level visualization interface is constructed to support real-time monitoring and interactive query of anomaly areas, enhancing the understandability and usability of anomaly detection results. Through the quantitative calculation and visual expression of feature contributions, the causes of anomalies are accurately located and intuitively displayed, providing strong support for the early warning and handling of anomaly events. A unified framework design based on digital twin technology achieves full automation of the entire process from data processing, anomaly detection, to visualization, significantly improving the practicality and scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 Schematic diagram of the steps of the multimodal spatiotemporal data visualization method in an embodiment of the present invention;

[0021] Figure 2 Schematic diagram of the structure of a multimodal spatiotemporal data visualization device in an embodiment of the present invention;

[0022] Figure 3 It is a schematic block diagram of the structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] Embodiments of the present invention provide a multimodal spatiotemporal data visualization method, apparatus, and device. The terms "first," "second," "third," "fourth," and so on (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0024] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 An embodiment of a multimodal spatiotemporal data visualization method according to an embodiment of the present invention includes:

[0025] Step S1: performing time alignment and spatial registration on the original multimodal spatiotemporal data to generate a grid feature vector set and a spatiotemporal adjacency matrix;

[0026] It is understandable that the execution subject of the present invention may be a multimodal spatiotemporal data visualization device, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0027] Specifically, the time series in the multimodal spatiotemporal data are sampled and normalized at a fixed time interval Δt, unifying the time scales of different data sources so that subsequent operations can be performed on the same time basis. Simultaneously, spatial coordinates are projected and geometrically corrected to map the original spatial coordinates into a unified coordinate system, creating a unified geographical reference, and generating spatiotemporally aligned normalized data. Data type identification and format conversion are then performed on the aligned normalized data, converting the data from different modalities into numerical features to form a well-regulated and consistent multimodal dataset. Targeted transformation methods are designed based on data characteristics, such as one-hot encoding for categorical data, periodic transformation for timestamp data, and appropriate imputation of missing values, to generate a structured, normalized multimodal dataset. Based on the normalized dataset, the monitoring area is spatially partitioned according to a preset grid size, and a grid index matrix is established. Each geographic location within the monitoring area is mapped to a grid cell with a unique spatiotemporal identifier, forming a regular grid structure. Within the framework of a regular grid structure, the multimodal data in each grid cell is aggregated. By filtering and aggregating data in time and space, all modal features of each grid cell at a specific time t are extracted and integrated into a complete feature vector. The feature vectors of all grid cells are spatiotemporally organized, and feature vectors with the same spatiotemporal index are merged into a grid feature vector set to ensure that data at the same spatiotemporal location can be clearly identified in the structured dataset. A spatial relationship graph between grid cells is constructed based on the regular grid structure. By calculating the connection weight between each pair of adjacent grid cells, adjacency relationships are defined based on their geometric distance, data similarity, or other spatiotemporal interaction features. The connection strength between adjacent grids is calculated using a spatial distance function, and the connection weights are organized in the form of an adjacency matrix to generate a spatiotemporal adjacency matrix with clear spatiotemporal interaction information.

[0028] Step S2: performing multi-dimensional anomaly assessment on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multimodal spatiotemporal anomaly detection feature matrix;

[0029] Specifically, a statistical analysis of the integrity of the multimodal data in the grid feature vector set is performed. The proportion of each of the K modal data types in each grid cell is calculated to quantify the degree of data integrity in each modal dimension. This results in a data integrity score, a metric reflecting the data quality of each grid cell. This score numerically represents the degree of missing or imbalanced data in the modal dimension. Normalized mutual information (MI) is calculated for each pair of modal data in the grid feature vector set. Mutual information is an effective method for measuring the degree of statistical correlation between modalities, quantitatively describing the degree of information sharing between modal data. By calculating the MI of all modal pairs and organizing them into an intermodal correlation matrix, a multidimensional data structure reflecting intermodal synergy is obtained. A weighted average operation is performed on the intermodal correlation matrix. By incorporating modal importance weights, the complex relationships in the matrix are refined into a single modal synergy score, quantifying the degree of collaboration between multimodal data. Spatial adjacency relationships are combined to perform spatiotemporal correlation analysis. Based on the spatiotemporal adjacency matrix, the feature similarity between the target grid cell and its neighboring grid cells is calculated, reflecting the consistency of the target cells in spatial relationships. Through normalization, the similarity values are standardized into a spatiotemporal coverage score, which reflects the degree of consistency in the spatiotemporal distribution of the target grid cell and surrounding grid cells, helping to identify the spatial location of abnormal distributions. A weighted fusion calculation is performed on the data integrity score, modal synergy score, and spatiotemporal coverage score to generate a comprehensive anomaly detection score. The weighted fusion process sets appropriate weights for each score based on the needs of the application scenario. The comprehensive anomaly detection score is compared with a preset anomaly threshold, and grid cells with scores above the threshold are marked as anomalous cells. For anomalous cells, their corresponding multimodal features are extracted and organized in matrix form into a multimodal spatiotemporal anomaly detection feature matrix.

[0030] Step S3: establishing decision tree nodes based on the multimodal spatiotemporal anomaly detection feature matrix and determining abnormal areas;

[0031] Specifically, a feature decomposition operation is performed on the multimodal spatiotemporal anomaly detection feature matrix, breaking its complex multidimensional structure into a temporal feature submatrix and a spatial feature submatrix, thereby extracting key multidimensional feature components. The temporal feature submatrix reflects the temporal variation of abnormal behavior, while the spatial feature submatrix reveals the local and global characteristics of the anomaly's spatial distribution. A decision tree root node is constructed for the multidimensional feature components. A splitting threshold for each feature is calculated based on the information gain criterion to ensure that the splitting node can maximize the distinction between anomalous and non-anomalous data. This calculation generates a branching structure for the decision tree, maximizing the information provided by each feature at a specific split point. The feature value of the target grid cell is compared with the feature splitting threshold, and a depth-first search strategy is used to traverse the branches of the decision tree to obtain the path probability of the target grid cell in the decision tree, representing the probability of the target grid cell being an anomaly in the decision path. Furthermore, to fully utilize the temporal information in the historical data, a pattern matching operation is performed on feature sequences similar to the target grid cell in the historical data. This operation calculates the degree of match between the target grid cell and historical anomaly patterns, generating a time-series anomaly probability value that reflects the degree of correlation between the current anomaly and known historical anomaly patterns. Using the path probability and time-series anomaly probability values as input features, a random forest classifier is constructed. Random forests combine multiple independent decision trees to more accurately identify anomaly areas. During the training and prediction process of a random forest, each decision tree learns based on an independent feature subset, thereby assessing the anomaly probability of the target grid cell from different perspectives. By voting and counting the outputs of all decision trees, a comprehensive anomaly probability distribution is obtained, reflecting the degree of anomaly of the target grid cell across multiple feature dimensions. Cluster analysis is performed on the comprehensive anomaly probability distribution, using clustering methods to group grid cells with similar anomaly characteristics into the same region. Clustering is implemented using density clustering or hierarchical clustering algorithms. Based on the similarity of the comprehensive anomaly characteristics between grid cells, anomaly regions with distinct boundaries are automatically formed, ultimately identifying anomaly regions within the entire system.

[0032] Step S4: Input the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, calculate the correlation strength between the abnormal area and the normal grid cells, and perform recurrent neural network iterative calculation on the correlation strength to generate an abnormal diffusion intensity sequence;

[0033] Specifically, the spatiotemporal feature sequence of the abnormal region undergoes positional encoding and temporal encoding to preserve and enhance the temporal and spatial information in the sequence. Positional encoding uses sine and cosine functions to construct periodic features of different dimensions, while temporal encoding maps temporal information into a high-dimensional vector, embedding it into the feature sequence along with spatial information. After encoding, a sequence of feature embedding vectors containing both positional and temporal context information is obtained. This sequence of feature embedding vectors is input into the multi-head attention layer of the Transformer decoder for feature dimension mapping and feature interaction analysis. The multi-head attention mechanism projects and interacts each feature vector in the embedding sequence using multiple independent attention heads, capturing global dependencies between features from multiple perspectives. Through feature dimension mapping, the Transformer decoder generates mapped features containing rich spatiotemporal interaction information, reflecting the comprehensive representation of the spatiotemporal features within the abnormal region across different modalities and temporal dimensions. The mapped features are then compared with the encoded features of normal grid cells to calculate cross-attention scores, resulting in a correlation matrix between the abnormal region and normal grid cells. This correlation matrix, generated based on the attention mechanism, reflects the strength of spatiotemporal interactions between the abnormal and normal grid cells by measuring the degree of match between them. To ensure that the weight values in the matrix accurately reflect the true correlation strength, the generated correlation matrix is normalized to construct a normalized correlation strength matrix. This correlation strength matrix is then fed into a recurrent neural network to capture the temporal dynamics of anomaly diffusion. In the recurrent neural network, each time step of the correlation strength matrix triggers a state update, generating a hidden layer state sequence. This hidden layer state sequence records the internal state of the correlation strength over time, reflecting the dynamics of anomaly diffusion through gradual state updates. During each propagation step, the hidden layer state of the recurrent neural network updates the anomaly's impact range through forward propagation. This newly calculated state information is incorporated into the temporal modeling, gradually generating an anomaly diffusion strength sequence that reflects the gradual diffusion pattern and impact range of the anomaly region on surrounding normal grid cells over time. Through the aforementioned steps of position and time encoding, multi-head attention mechanism calculation, and recurrent neural network state updates, the anomaly diffusion strength sequence is ultimately obtained, reflecting the impact of the anomaly region and its dynamic changes.

[0034] Step S5: perform interval mapping operation on the abnormal diffusion intensity sequence, establish a time series evolution axis and a retrieval interaction component, and output a multi-level visualization interface.

[0035] Specifically, the anomaly diffusion intensity sequence is normalized to its maximum and minimum values to generate a standardized intensity sequence. A linear normalization algorithm is used to map the original intensity values to a standardized range of [0, 1], allowing subsequent visualization operations to uniformly process intensity values across different ranges. Furthermore, to transform the standardized intensity values into visual representations, a mapping matrix is established between intensity values and color gradient intervals. This mapping matrix encodes intensity values into corresponding color values using a color gradient. Based on this mapping matrix, the standardized intensity sequence is color-coded and the grid cells are colored to generate an initial heatmap. The intensity value of each grid cell is mapped to a corresponding color and presented in a spatial distribution map, visually demonstrating the distribution characteristics of the anomaly diffusion intensity. To enhance the initial heatmap's ability to identify anomaly regions, a boundary extraction operation is performed on the anomaly regions in the heatmap. Morphological algorithms or contour detection techniques are used to extract the boundary information of the anomaly regions, and a contour annotation layer is constructed. This contour annotation layer is overlaid on the initial heatmap to generate a boundary-enhanced heatmap. The contribution of multimodal features to anomaly detection is quantified to generate a feature weight vector. By evaluating the weights of different modal features in anomaly detection, the influence of each modal feature on anomaly results is determined. The feature weight vector is converted into graphic symbols, such as circles of varying sizes or markers of varying shapes, and these symbols are overlaid on the boundary-enhanced heatmap to generate a feature annotation heatmap. The feature annotation heatmap effectively visualizes the causes and contributions of anomaly regions. A time-series sliding control is constructed based on a time window mechanism. By associating the feature annotation heatmap with a timeline, a dynamic heatmap display interface is generated. In the dynamic heatmap, users can view the anomaly diffusion state at different time points by dragging the slider or clicking on a specific time point on the timeline, tracking the changes in the anomaly region in real time. An interactive query module is added to the dynamic heatmap display interface. Through this interactive module, users can click on a grid cell in the heatmap to view detailed information, including its feature value, anomaly cause analysis, and corresponding warning information. The multi-level visualization interface constructed through these steps enables the display of anomaly information from macro to micro, and from static to dynamic.

[0036] By analyzing the contributions of different modal features to anomaly determination in the multimodal spatiotemporal anomaly detection feature matrix, the importance of each modal feature in the anomaly region is scored. Based on interpretation methods of the classification model (such as SHAP values or feature importance scores) or statistical methods (such as analysis of variance), the discriminative power of each modal feature within the anomaly region is calculated, generating a feature importance score sequence that reflects the role each modal feature plays in anomaly detection. A feature correlation network is constructed based on the feature importance score sequence, and a feature correlation matrix is generated by calculating the mutual information coefficient between features. The mutual information coefficient quantifies the degree of information sharing between two modal features. A non-negative matrix factorization operation is performed on the feature correlation matrix, decomposing it into multiple non-negative low-dimensional matrices to extract potential combination patterns between features and converting them into feature weight vectors. This feature weight vector numerically quantifies the contribution of each feature to overall anomaly detection. The feature weight vectors are then piecewise normalized. By dividing the weight values into multiple level intervals, the weight data becomes more intuitive and suitable for subsequent symbol mapping operations. To convert the normalized weight values into visual symbols, a correspondence table between weight levels and graphical symbols is established. In this relationship table, different weight levels correspond to graphic symbols of different shapes and sizes, such as circular, rectangular or triangular symbols filled with different colors, so that the weight levels can be distinguished through intuitive visual elements. Based on the above correspondence table, the numerical values in the feature weight vector are mapped to the corresponding graphic symbols one by one to generate a feature identification symbol set. The feature identification symbol set is superimposed on the corresponding grid position of the boundary enhancement heat map to complete the construction of the feature annotation heat map. During the superposition process, the grid cell of each abnormal area will display the corresponding feature identification symbol, thereby visually enhancing the distribution and contribution information of each modal feature in the abnormal area. The final generated feature annotation heat map integrates the visual expression of feature weights on the basis of the boundary enhancement heat map.

[0037] In this embodiment of the present invention, heterogeneous multimodal spatiotemporal data is uniformly mapped into a regular grid structure through time alignment and spatial registration, addressing data heterogeneity. A multidimensional anomaly assessment mechanism comprehensively considers data integrity, modal synergy, and spatiotemporal coverage, effectively reducing misjudgments caused by uneven data quality. A hierarchical detection strategy based on decision trees and random forests fully leverages historical data and spatial correlation information, improving the robustness and accuracy of anomaly detection. A hybrid architecture combining a Transformer decoder and a recurrent neural network enables dynamic propagation analysis of anomaly events, accurately capturing their diffusion patterns. A multi-level visualization interface supports real-time monitoring and interactive querying of anomaly areas, enhancing the understandability and usability of anomaly detection results. Through the quantitative calculation and visualization of feature contributions, the cause of anomalies is precisely located and intuitively displayed, providing strong support for early warning and handling of anomaly events. A unified framework design based on digital twin technology automates the entire process from data processing and anomaly detection to visualization, significantly improving the system's practicality and scalability.

[0038] In a specific embodiment, the process of executing step S1 may specifically include the following steps:

[0039] The time series in the multimodal spatiotemporal data are sampled and standardized according to a fixed time interval △t, and the spatial coordinates are projected and geometrically corrected to obtain standardized data after spatiotemporal alignment.

[0040] Perform data type identification and format conversion on the standardized data after spatiotemporal alignment, uniformly convert different modal data into numerical features, and obtain a standardized multimodal dataset;

[0041] The normalized multimodal dataset is spatially divided according to the preset grid size, a grid index matrix is established, and the monitoring area is mapped into grid cells with unique spatiotemporal identifiers to obtain a regular grid structure.

[0042] Perform multimodal data aggregation operations on each grid cell in the regular grid structure, extract all modal features of the grid cell at time t, and generate a feature vector;

[0043] Perform spatiotemporal organization on the feature vectors and merge the feature vectors with the same spatiotemporal index into a grid feature vector set;

[0044] Based on the regular grid structure, a spatial relationship graph between grid cells is constructed, the connection weights between adjacent grid cells are calculated, and the connection weights are organized into an adjacency matrix form to obtain a spatiotemporal adjacency matrix.

[0045] Specifically, the time series in the multimodal spatiotemporal data are sampled and standardized according to a fixed time interval △t, unifying the resolution of different data sources in the time dimension to ensure the consistency of data at each time point. Suppose the original time series is , the time interval is Δt, then the new sampling time series Expressed as:

[0046] ;

[0047] In this process, if the data of some time points are missing, the data are supplemented by linear interpolation, nearest neighbor interpolation or mean filling. After completing the standardization of the time dimension, the spatial coordinates are projected and geometrically corrected to ensure that the geographic spatial information of different data sources can be mapped to a unified spatial reference system. For example, if the original data uses different geographic coordinate systems (such as WGS84 and UTM), they are unified into the same coordinate system through projection transformation. Let the original coordinates be , the transformed coordinates are , the projection transformation is expressed as:

[0048] ;

[0049] in is the projection matrix, and its parameters are determined according to the definition of the target coordinate system. After completing the spatiotemporal alignment, the standardized data is identified and format converted, and the multimodal data is uniformly converted into numerical features. Assuming that the multimodal data includes numerical features , classification features and timestamp , and process them separately. Use the unique hot encoding method to map it into a numerical vector, such as a classification feature Converted to [1,0,0],[0,1,0],[0,0,1]. For timestamp , extract periodic features such as hours, months, etc., and map them into numerical variables Represents the time characteristics. After obtaining the normalized multimodal dataset, according to the preset grid size The monitoring area is spatially divided, and a grid index matrix is established to divide the geographical space in the monitoring area into grid cells with unique spatiotemporal identifiers. Assume that the boundary of the monitoring area is , the grid size is and , then the grid index matrix Expressed as:

[0050] ;

[0051] in and is the row and column index of the grid. Through this process, the space is divided into a regular grid structure. For each grid cell in the regular grid structure, the multimodal data contained in it is aggregated. Let the grid cell The set of data points contained in , each data point has a modal eigenvector , then at time right Aggregate all modal data to obtain the feature vector :

[0052] ;

[0053] in Indicates at time belong The aggregated feature vector represents the grid cell at time After feature extraction, the feature vectors with the same spatiotemporal index are organized and merged into a grid feature vector set. In order to analyze the spatial relationship, a spatial relationship diagram between grid cells is constructed based on the regular grid structure. Assume that the geometric distance between grid cells is , defines the connection weights between adjacent grid cells for:

[0054] ;

[0055] in is the distance attenuation coefficient, is the connection radius. By calculating the weights between all grid cells, we get the adjacency matrix , whose elements Represents the connection relationship between grid cells.

[0056] In a specific embodiment, the process of executing step S2 may specifically include the following steps:

[0057] Perform integrity statistics on the multimodal data of each grid cell in the grid feature vector set, calculate the proportion of the number of K types of modal data, and obtain the data integrity score;

[0058] The normalized mutual information of each pair of modal data in the grid feature vector set is calculated to construct the inter-modal correlation matrix, and the weighted average operation of the inter-modal correlation matrix is performed to obtain the modal synergy score;

[0059] The feature similarity between the target grid cell and the adjacent grid cells is calculated based on the spatiotemporal adjacency matrix, and the feature similarity is normalized to obtain the spatiotemporal coverage score.

[0060] The data integrity score, modal synergy score, and spatiotemporal coverage score are weighted and fused to obtain a comprehensive anomaly detection score.

[0061] The comprehensive anomaly detection score is compared with the preset threshold, and the grid cells with scores higher than the threshold are screened out. The multimodal features of the grid cells are organized into a matrix form to obtain the multimodal spatiotemporal anomaly detection feature matrix.

[0062] Specifically, in order to calculate the multimodal data integrity score of each grid unit, the proportion of each modal data is counted. The modal set of , Indicates modality exist The actual number of data points in express The total number of all modal data points in the modal The proportion of existence is:

[0063] ;

[0064] Based on this, the grid cells are defined Data integrity score is the mean of the proportion of all modes:

[0065] ;

[0066] This score reflects the degree of integrity of the data in the grid cell. In order to evaluate the synergy between modes, the normalized mutual information is calculated for each pair of modal data. and The characteristic distributions of and , the joint distribution is , mutual information Defined as:

[0067] ;

[0068] In order to facilitate comparison, the mutual information is normalized to obtain the normalized mutual information :

[0069] ;

[0070] in For modal By calculating the information entropy between all modal pairs , construct the inter-modal correlation matrix ,in . For the matrix Perform weighted average operation to obtain the modal synergy score :

[0071] ;

[0072] The modal synergy score reflects the collaborative relationship between different modal features within a grid cell. At the same time, in order to evaluate the similarity between the target grid cell and its neighboring grid cells, the spatiotemporal adjacency matrix is used to Calculate feature similarity. Assume that the grid unit and its adjacent units The eigenvectors of and , its Euclidean distance for:

[0073] ;

[0074] Defining similarity based on Euclidean distance :

[0075] ;

[0076] in is the distance attenuation coefficient. Through normalization, we can get the spatiotemporal coverage score. :

[0077] ;

[0078] Neighbor Represents a grid cell The set of neighboring units of . Score the data integrity , modality synergy score and spatiotemporal coverage scores Perform weighted fusion calculations to generate a comprehensive anomaly detection score :

[0079] ;

[0080] in is the weight coefficient of the score, satisfying By adjusting the weights, the role of different scores can be highlighted according to the needs of the application scenario. With preset threshold Compare and select the grid cells with scores higher than the threshold. For these abnormal cells, extract their corresponding multimodal features and organize these features into a matrix form to finally generate a multimodal spatiotemporal anomaly detection feature matrix. , whose elements Represents a grid cell Medium mode Abnormal feature values.

[0081] In this embodiment, before performing multi-dimensional anomaly evaluation on the grid feature vector set and establishing a decision tree node based on the multimodal spatiotemporal anomaly detection feature matrix, it also includes: constructing multiple heterogeneous base learners for the multimodal spatiotemporal anomaly detection feature matrix, performing probability distribution calculation on the output results of each base learner, and obtaining a model output distribution set; performing consistency analysis on the prediction results of each base learner in the model output distribution set, calculating the variance and entropy of the prediction results, and generating a model consistency score matrix; performing a distance measurement operation on the model consistency score matrix and a pre-constructed standard sample library, calculating the degree of deviation of each grid unit feature vector from the standard sample, and obtaining an outlier degree DOO feature matrix; constructing a kernel function mapping based on the DOO feature matrix, performing maximum mean difference calculation on samples in the high-dimensional feature space, and generating O-MMD Feature sequence is obtained, and the O-MMD feature sequence is sorted to obtain a sample difference ranking table; an adaptive threshold is set according to the sample difference ranking table, and high-difference samples are screened out to construct a dynamic anomaly sample training set, and the dynamic anomaly sample training set is subjected to data enhancement processing to obtain an enhanced training sample set; the enhanced training sample set is input into a pre-trained deep neural network, and a credibility model is trained through transfer learning to obtain a credibility assessment model; the credibility assessment model is used to calculate the credibility score of each eigenvector in the multimodal spatiotemporal anomaly detection feature matrix, and a weight matrix is constructed based on the credibility score to generate a credibility weighted feature matrix; the credibility weighted feature matrix and the multimodal spatiotemporal anomaly detection feature matrix are adaptively fused, the fusion coefficient is calculated based on the weighted sum criterion, and the enhanced anomaly detection feature matrix is output.

[0082] In a specific embodiment, the process of executing step S3 may specifically include the following steps:

[0083] Perform eigendecomposition on the multimodal spatiotemporal anomaly detection feature matrix to extract the time dimension feature submatrix and the space dimension feature submatrix to obtain multi-dimensional feature components;

[0084] Construct a decision tree root node for the multi-dimensional feature components, calculate the feature splitting threshold based on the information gain criterion, and generate a decision tree branch structure;

[0085] The characteristic value of the target grid cell is compared with the characteristic splitting threshold, and the decision tree branch structure is traversed based on the depth-first search strategy to obtain the path probability value of the target grid cell;

[0086] Perform pattern matching operations on similar feature sequences in historical time series data, calculate the matching degree between the target grid unit and the historical anomaly pattern, and generate the time series anomaly probability value;

[0087] A random forest classifier is constructed based on the path probability value and the time series anomaly probability value. The output results of multiple decision trees of the random forest classifier are voted and counted to obtain the comprehensive anomaly probability distribution.

[0088] Cluster analysis is performed on the comprehensive anomaly probability distribution, and grid cells with similar anomaly characteristics are divided into the same area to determine the anomaly area.

[0089] Specifically, the feature decomposition operation is performed on the multimodal spatiotemporal anomaly detection feature matrix to extract the time dimension feature submatrix and the space dimension feature submatrix to obtain the multi-dimensional feature components. Assume that the anomaly detection feature matrix is ,in represents the number of grid cells, Represents the feature dimension (including time and space dimension features), and the matrix decomposition is expressed as:

[0090] ;

[0091] in is the time dimension feature sub-matrix, is the spatial dimension feature sub-matrix, Represents the number of time features. By decomposing, the temporal dynamics and spatial distribution characteristics in anomaly detection are analyzed separately. For example, principal component analysis is used to extract the main feature components in the time and space dimensions, so that each submatrix captures the key feature information. The root node of the decision tree is constructed using the extracted time and space feature components. and Respectively represent The temporal and spatial features of the grid cells are used to select the split features and calculate the split threshold using the information gain criterion. The information gain is defined as:

[0092] ;

[0093] in is the entropy before partitioning, By feature The entropy of the subset after partitioning, is a dataset, By feature After division, the value is A subset of the data, is a set of possible values of the feature. Select the feature with the largest information gain as the basis for splitting and determine the optimal splitting threshold At this point, the branch structure of the decision tree begins to form, and each node is divided into two parts according to whether the feature value is less than or greater than Further splitting is performed until the tree reaches the specified depth or the sample cannot be split any further. The characteristic value of is input into the decision tree, compared with the split threshold of each node, and the depth-first search strategy is used to traverse the branch structure of the tree to determine the path probability value of the target grid unit in the tree. . Assume that each node The probability of , the path probability is calculated by multiplying the probabilities of the nodes on the path:

[0094] ;

[0095] At the same time, a pattern matching operation is performed on the historical time series data to calculate the matching degree between the target grid unit and the historical anomaly pattern. Assume that the characteristic sequence of the target unit is , the historical abnormal pattern is , the dynamic time warping method is used to calculate the matching degree:

[0096] ;

[0097] in It's time The distance from the upper eigenvalue, Is the matching path. Generate the time series anomaly probability value based on the matching degree , which indicates the abnormality of the target unit feature sequence. Based on the path probability value and the probability value of time series anomaly Construct a random forest classifier. Random forests are composed of multiple independent decision trees, and the classification performance is improved through ensemble learning methods. For each target unit, its feature value is input into all decision trees to obtain the classification results of each tree. :

[0098] ;

[0099] in It is The classification function of the tree, is the number of trees in the random forest. The final comprehensive anomaly probability distribution is obtained by voting statistics:

[0100] ;

[0101] Perform cluster analysis on the comprehensive anomaly probability distribution and divide grid cells with similar anomaly characteristics into the same area. Use density clustering algorithm (such as DBSCAN) or K-means algorithm to cluster according to the comprehensive anomaly probability and spatial location of grid cells. For example, and grid coordinates as input features, and all grid cells are divided into clusters, each cluster represents an abnormal area.

[0102] In this embodiment, after determining the abnormal area and before inputting the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, the following steps are further included: performing spatiotemporal constraint modeling on the state features of the abnormal area at T consecutive time points, representing the feature state at each time point t as St, constructing the state transition constraint matrix At,t+1, and establishing the equation group based on the KVL and KCL principles: St+1 = At,t+1·St + Bt, where Bt is the state offset vector, and T-1 state transfer equations are obtained; the T-1 state transfer equations are combined to construct the time series state matrix M, the feature difference matrix D=M(t+1)-M(t) is calculated for the time series state matrix M, and the feature difference matrix D is subjected to singular value decomposition D=USV', and the eigenvectors corresponding to the first k singular values are extracted as the main change patterns; the main change patterns are reorganized in the time dimension to construct a third-order tensor, and the Tucker decomposition algorithm is used to decompose the third-order tensor into X=G×1U×2V×3W, where G is the core tensor, to obtain the spatial propagation sub-tensor Us and the time evolution sub-tensor Ut; the spatial attention matrix SA is constructed for the spatial propagation sub-tensor Us, the attention score is calculated, and the propagation feature map Gs is constructed based on the spatial propagation eigenvector; the dynamic time warping algorithm is applied to the time evolution sub-tensor Ut to calculate the registration distance moment within the time window The matrix W is used to identify the set of key time points {t1, t2, ..., tm} that meet the threshold conditions and generate a time feature sequence Vt. The spatial propagation feature vector Vs and the time feature sequence Vt are constructed into a joint representation tensor through the outer product operation, and the tensor network compression algorithm is used to obtain a compact anomaly propagation feature representation F. The similarity between the anomaly propagation feature representation and the preset propagation pattern library is calculated to obtain a similarity vector S = [s1, s2, ..., sn]. The anomaly type is determined based on the maximum similarity criterion, and the threshold for sudden anomalies is set to α, and the threshold for gradual anomalies is set to β. Based on the anomaly type, an LSTM network is used to perform sequence prediction in the time dimension to obtain a predicted sequence of anomaly development at the next L time points. The error between the predicted sequence and the actual observation value Y is calculated. When e exceeds the preset threshold γ, the dynamic update mechanism of the anomaly region boundary is triggered to generate a new anomaly region boundary B't.

[0103] In a specific embodiment, the process of executing step S4 may specifically include the following steps:

[0104] The spatiotemporal feature sequence of the abnormal area is processed by position encoding and time encoding to generate a feature embedding vector sequence, which is then input into the multi-head attention layer of the Transformer decoder for feature dimension mapping to obtain the mapped features.

[0105] The mapped features are then combined with the encoded features of the normal grid cells to calculate the cross-attention score, obtaining the abnormal-normal grid cell association matrix. The attention weight distribution is then constructed based on the abnormal-normal grid cell association matrix, and the attention weight distribution is normalized to obtain the association strength matrix.

[0106] The correlation strength matrix is input into the recurrent neural network, and the state update calculation is performed on the correlation strength of each time step to generate a hidden layer state sequence. The hidden layer state sequence is forward propagated, and the anomaly influence range is updated in each propagation step to generate an anomaly diffusion intensity sequence.

[0107] Specifically, the spatiotemporal feature sequence of the abnormal region is processed by position coding and time coding to generate a feature embedding vector sequence. Suppose the spatiotemporal feature sequence of the abnormal region is ,in Indicates the number of time steps, each is the spatiotemporal feature vector with dimension In order to explicitly model the position information in the sequence, the position encoding method is used to embed the position information of each time step in the sequence into the feature vector. Position encoding vector Generated using sine and cosine functions, the formula is:

[0108] ;

[0109] in is the index of the feature dimension, is the time step, is the dimension of the feature vector. Time encoding similarly embeds time information, for example, mapping timestamps to periodic features (such as hours, minutes) and adding them to the features in numerical form. After adding the position code and time code, the enhanced feature embedding vector sequence is obtained ,in Embed the generated features into a vector sequence The input is fed into the multi-head attention layer of the Transformer decoder for feature dimension mapping. The multi-head attention mechanism captures the global dependencies between different features in the sequence through the self-attention mechanism. For each attention head, the input embedding vector Generate query vector through linear transformation , key vector Sum vector :

[0110] ;

[0111] in is the weight matrix of the linear transformation, and its dimensions are The attention score is calculated by the dot product of the query vector and the key vector and normalized to the attention weight:

[0112] ;

[0113] in Is a scaling factor to prevent the dot product value from being too large and causing the gradient to disappear. Multi-head attention calculates multiple sets of attention scores in parallel and then splices them to generate a mapped feature sequence The mapped features Encoded features with normal grid cells Calculate the cross attention score and get the abnormal-normal grid cell association matrix .set up is a feature in the abnormal feature sequence, is the feature in the normal feature sequence, and the cross attention score is calculated by dot product and normalized to weight:

[0114] ;

[0115] in Indicates abnormal unit and normal units The strength of association. Based on the association matrix , construct the attention weight distribution, and perform normalization operation on it to obtain the final correlation strength matrix , whose elements . The correlation strength matrix Input the recurrent neural network (RNN), perform state update calculation on the association strength of each time step, and generate the hidden layer state sequence .set up is the time step The hidden state of is the corresponding association strength, and the RNN state update formula is:

[0116] ;

[0117] in and is the weight matrix, is the bias vector, Is the activation function (such as tanh). Hidden layer state sequence Record the dynamic characteristics of abnormal areas. Perform forward propagation operations, update the anomaly impact range in each propagation step, and generate anomaly diffusion intensity sequence Diffusion intensity Defined as the hidden state Some transformation of , such as summing or taking the norm:

[0118] .

[0119] In a specific embodiment, the process of executing step S5 may specifically include the following steps:

[0120] Perform maximum and minimum normalization on the abnormal diffusion intensity sequence to generate a standardized intensity sequence, and establish a mapping relationship matrix between intensity values and color gradient intervals;

[0121] Perform color coding on the standardized intensity sequence based on the mapping relationship matrix, colorize the grid cells, and generate an initial heat map;

[0122] Perform boundary extraction on the abnormal areas in the initial heat map, construct a contour annotation layer, and superimpose the contour annotation layer on the initial heat map to obtain a boundary enhanced heat map;

[0123] The contribution of multimodal features in anomaly detection is quantified and calculated to generate feature weight vectors. The feature weight vectors are converted into graphic symbols and superimposed on the boundary enhancement heat map to obtain the feature annotation heat map.

[0124] Build a time series sliding control based on the time window mechanism, associate the feature annotation heat map with the time axis, and generate a dynamic heat map display interface;

[0125] An interactive query module is added to the dynamic heat map display interface, and detailed information, abnormal causes and warning information of grid units are associated with the interactive interface to build a multi-level visualization interface.

[0126] Specifically, the maximum and minimum values of the abnormal diffusion intensity sequence are normalized to generate a standardized intensity sequence so that its range is unified to [0,1]. Suppose the original abnormal diffusion intensity sequence is ,in Indicates that at time step The diffusion intensity of is normalized as follows:

[0127] ;

[0128] in and They are The minimum and maximum values of Represents the normalized intensity value. Through this process, all intensity values are compressed to between 0 and 1. A mapping matrix between intensity values and color gradient intervals is established based on the normalized intensity sequence. Color gradient mapping uses linear interpolation to map intensity values from low to high to a specific color space (such as from blue to red). Let the color gradient interval be ,in Indicates the color corresponding to the lowest intensity, Indicates the color corresponding to the highest intensity, color mapping matrix The elements of are represented as:

[0129] ;

[0130] Among them, interpolate is an interpolation function, and the corresponding color is determined by the intensity value. The mapping relationship matrix is used to perform color coding operations on the standardized intensity sequence, and the color information is applied to the coloring of the grid unit to generate an initial heat map. In the heat map, each grid unit is filled with a corresponding color to reflect its intensity level, thereby intuitively showing the spatial distribution of abnormal diffusion. In order to highlight the boundary characteristics of the abnormal area, the boundary extraction operation is performed on the abnormal area in the initial heat map. This is achieved through morphological methods or contour detection algorithms, such as using image gradient methods to extract boundaries. Assume that the heat map matrix is , the boundary extraction algorithm is expressed as:

[0131] ;

[0132] in Represents the boundary matrix. Perform contour annotation, construct a boundary annotation layer, and superimpose it on the initial heat map to generate a boundary enhancement heat map. The boundary enhancement heat map can show the range of the abnormal area and improve the visual effect. Quantify the contribution of multimodal features in anomaly detection. Suppose the modal feature set of the grid unit is , the feature contribution is calculated by feature importance score or normalized weighted calculation, the formula is:

[0133] ;

[0134] in Features weight, importance is the feature importance score. All feature weights are normalized piecewise and converted into graphic symbols, for example, the weight size is represented by the symbol size or shape. The generated feature identification symbol set is superimposed on the grid unit position corresponding to the boundary enhancement heat map to generate the feature annotation heat map. After completing the static heat map, a time series sliding control is constructed through the time window mechanism, and the feature annotation heat map is associated with the time axis to generate a dynamic heat map display interface. Assume that the time axis is Each time point corresponds to a heat map slice, and the time series sliding control allows users to select a specific time point or time range to display the corresponding heat map status in real time. The dynamic heat map shows the spatiotemporal evolution process of abnormal diffusion. An interactive query module is added to the dynamic heat map display interface. By clicking or selecting a grid cell, the user can view the detailed information of the cell, including intensity value, multimodal characteristic value, abnormal cause and warning information. All this information is integrated into a multi-level visualization interface, providing users with multi-dimensional analysis capabilities from global to local, from static to dynamic.

[0135] In a specific embodiment, the execution step of quantifying the contribution of multimodal features in anomaly detection, generating a feature weight vector, converting the feature weight vector into a graphic symbol, and superimposing it on the boundary enhancement heat map to obtain the feature annotation heat map can specifically include the following steps:

[0136] The importance of each modal feature in the abnormal area is scored, and the discrimination of each feature in anomaly judgment is calculated based on the multimodal spatiotemporal anomaly detection feature matrix to obtain a feature importance score sequence;

[0137] Based on the feature importance score sequence, a feature correlation network is constructed, the mutual information coefficient between features is calculated, a feature correlation matrix is generated, and a non-negative matrix decomposition operation is performed on the feature correlation matrix to extract the feature combination pattern and obtain the feature weight vector;

[0138] Perform piecewise normalization on the feature weight vector, divide the weight value into multiple level intervals, and establish a corresponding relationship table between weight levels and graphic symbols;

[0139] Based on the correspondence table, the numerical values in the feature weight vector are converted into graphic symbols of different shapes and sizes to generate a feature identification symbol set, which is then superimposed on the corresponding grid positions of the boundary enhancement heat map to construct a feature annotation heat map.

[0140] Specifically, for each modal feature in the abnormal area, its discrimination in abnormal judgment is calculated based on the multimodal spatiotemporal anomaly detection feature matrix to obtain the feature importance score sequence. Assume that the anomaly detection feature matrix is ,in represents the number of grid cells in the abnormal area, Represents the number of modal features. Features, its discrimination It is obtained by variance analysis or statistical distribution difference calculation. Assume that the abnormal unit set is , the normal unit set is , No. The discrimination of a feature is defined as:

[0141] ;

[0142] in and Represents the abnormal unit and normal unit The mean of the features, represents the variance of the feature, is a smoothing term to prevent the denominator from being zero. By calculating the , construct feature importance score sequence , reflecting the contribution of each modal feature to anomaly detection. After obtaining the feature importance score sequence, the correlation between features is analyzed and a feature correlation network is constructed. Feature correlation is calculated by the mutual information coefficient, which quantifies the degree of information sharing between two features. and The probability distributions of and , the joint distribution is , the mutual information between the two Defined as:

[0143] ;

[0144] For ease of comparison, the mutual information coefficient needs to be standardized to :

[0145] ;

[0146] in It is a feature The information entropy of all modal features is calculated pairwise. , and obtain the feature correlation matrix ,in Indicates the Hedi In order to extract the feature combination pattern, the feature correlation matrix Perform non-negative matrix decomposition operation to decompose it into two low-dimensional non-negative matrices and ,satisfy:

[0147] ;

[0148] in is the number of feature combinations, Represents the weight matrix of features and combinations, Represents the contribution of the combined pattern. By decomposing, the weight vector of each feature is extracted , reflecting its contribution in different combination modes. After merging, the feature weight vector is generated Perform segmented normalization on the feature weight vector and divide the weight value into multiple level intervals. Assume the weight value range is , the formula for piecewise normalization is:

[0149] ;

[0150] in Represents the normalized weight value. According to the weight level, Map to different symbol shapes and sizes, and establish a correspondence table between weight levels and graphic symbols. Using the correspondence table, the values in the feature weight vector are converted into a feature identification symbol set. ,in Features The generated feature identifier symbol set is superimposed on the corresponding grid locations of the boundary enhancement heat map to construct a feature annotation heat map. Each grid cell displays the anomaly diffusion intensity, and the superimposed graphic symbol visually displays the characteristic contribution of the cell, providing interpretability of the anomaly cause.

[0151] The above describes the multimodal spatiotemporal data visualization method in the embodiment of the present invention. The following describes the multimodal spatiotemporal data visualization device in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, a multimodal spatiotemporal data visualization device includes:

[0152] The registration module is used to perform temporal and spatial registration on the original multimodal spatiotemporal data to generate a set of grid feature vectors and a spatiotemporal adjacency matrix;

[0153] An evaluation module is used to perform multi-dimensional anomaly evaluation on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multimodal spatiotemporal anomaly detection feature matrix;

[0154] Establish a module for establishing decision tree nodes based on the multimodal spatiotemporal anomaly detection feature matrix and determining the anomaly area;

[0155] The calculation module is used to input the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, calculate the correlation strength between the abnormal area and the normal grid cells, and perform recurrent neural network iterative calculation on the correlation strength to generate the abnormal diffusion intensity sequence;

[0156] The output module is used to perform interval mapping operations on the abnormal diffusion intensity series, establish the time series evolution axis and retrieve interactive components, and output a multi-level visualization interface.

[0157] Through the collaborative efforts of the aforementioned components, time alignment and spatial registration processes are used to uniformly map heterogeneous multimodal spatiotemporal data into a regular grid structure, addressing data heterogeneity. A multi-dimensional anomaly assessment mechanism, comprehensively considering data integrity, modal coherence, and spatiotemporal coverage, effectively reduces misjudgments caused by uneven data quality. A hierarchical detection strategy based on decision trees and random forests fully leverages historical data and spatial correlation information, improving the robustness and accuracy of anomaly detection. A hybrid architecture combining a Transformer decoder and a recurrent neural network enables dynamic propagation analysis of anomaly events, accurately capturing their diffusion patterns. A multi-level visualization interface supports real-time monitoring and interactive querying of anomaly areas, enhancing the understandability and usability of anomaly detection results. Through the quantitative calculation and visualization of feature contributions, the cause of anomalies is precisely located and intuitively displayed, providing strong support for early warning and resolution of anomaly events. A unified framework design based on digital twin technology automates the entire process from data processing, anomaly detection, to visualization, significantly improving the system's practicality and scalability.

[0158] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, memory, display screen, input device, network interface and database connected via a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0159] Those skilled in the art will understand that Figure 3The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0160] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0161] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0162] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal spatiotemporal data visualization method, characterized in that: The method comprises: Perform temporal alignment and spatial registration on the original multimodal spatiotemporal data to generate a set of grid feature vectors and a spatiotemporal adjacency matrix; A multi-dimensional anomaly assessment is performed on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multimodal spatiotemporal anomaly detection feature matrix; specifically, the method includes: performing integrity statistics on the multimodal data of each grid cell in the grid feature vector set, calculating the proportion of the number of K types of modal data, and obtaining a data integrity score; performing standardized mutual information calculation on each pair of modal data in the grid feature vector set, constructing an inter-modal correlation matrix, and performing weighted average operation on the inter-modal correlation matrix to obtain a modal synergy score; calculating the feature similarity between the target grid cell and the adjacent grid cells based on the spatiotemporal adjacency matrix, normalizing the feature similarity, and obtaining a spatiotemporal coverage score; performing weighted fusion calculation on the data integrity score, the modal synergy score, and the spatiotemporal coverage score to obtain a comprehensive anomaly detection score; performing a comparison operation on the comprehensive anomaly detection score and a preset threshold, screening out grid cells with scores higher than the threshold, and organizing the multimodal features of the grid cells into a matrix form to obtain a multimodal spatiotemporal anomaly detection feature matrix; Establishing decision tree nodes based on the multimodal spatiotemporal anomaly detection feature matrix and determining abnormal areas; The spatiotemporal feature sequence of the abnormal area is input into the Transformer decoder, the association strength between the abnormal area and the normal grid cells is calculated, and the association strength is iteratively calculated by a recurrent neural network to generate an abnormal diffusion intensity sequence, which reflects the gradual diffusion pattern and impact range of the abnormal area on the surrounding normal grid cells over time; An interval mapping operation is performed on the abnormal diffusion intensity sequence, a time series evolution axis and a retrieval interaction component are established, and a multi-level visualization interface is output.

2. The multimodal spatiotemporal data visualization method according to claim 1, characterized in that: The method of performing time alignment and spatial registration on the original multimodal spatiotemporal data to generate a grid feature vector set and a spatiotemporal adjacency matrix includes: The time series in the multimodal spatiotemporal data are sampled and standardized according to a fixed time interval △t, and the spatial coordinates are projected and geometrically corrected to obtain standardized data after spatiotemporal alignment. Performing data type identification and format conversion processing on the standardized data after the spatiotemporal alignment, uniformly converting the different modal data into numerical features, and obtaining a standardized multimodal data set; The normalized multimodal dataset is spatially divided according to a preset grid size, a grid index matrix is established, and the monitoring area is mapped into grid cells with unique spatiotemporal identifiers to obtain a regular grid structure; Performing a multimodal data aggregation operation on each grid cell in the regular grid structure, extracting all modal features of the grid cell at time t, and generating a feature vector; Performing spatiotemporal organization processing on the feature vectors, and merging feature vectors with the same spatiotemporal index into a grid feature vector set; A spatial relationship graph between grid cells is constructed based on the regular grid structure, connection weights between adjacent grid cells are calculated, and the connection weights are organized into an adjacency matrix form to obtain a spatiotemporal adjacency matrix.

3. The multimodal spatiotemporal data visualization method according to claim 1, characterized in that: The step of establishing a decision tree node based on the multimodal spatiotemporal anomaly detection feature matrix and determining an abnormal area includes: Performing an eigendecomposition operation on the multimodal spatiotemporal anomaly detection feature matrix to extract a time dimension feature submatrix and a space dimension feature submatrix to obtain a multidimensional feature component; Constructing a decision tree root node for the multi-dimensional feature components, and calculating a feature splitting threshold based on an information gain criterion to generate a decision tree branch structure; Comparing the characteristic value of the target grid cell with the characteristic splitting threshold, and traversing the decision tree branch structure based on a depth-first search strategy to obtain a path probability value of the target grid cell; Perform pattern matching operations on similar feature sequences in historical time series data, calculate the matching degree between the target grid unit and the historical anomaly pattern, and generate the time series anomaly probability value; Building a random forest classifier based on the path probability value and the time series anomaly probability value, and performing voting statistics on the output results of multiple decision trees of the random forest classifier to obtain a comprehensive anomaly probability distribution; Cluster analysis is performed on the comprehensive anomaly probability distribution, grid cells with similar anomaly characteristics are divided into the same area, and the anomaly area is determined.

4. The multimodal spatiotemporal data visualization method according to claim 1, characterized in that: The process of inputting the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, calculating the association strength between the abnormal area and the normal grid cells, and performing recurrent neural network iterative calculation on the association strength to generate an abnormal diffusion intensity sequence includes: Performing position encoding and time encoding on the spatiotemporal feature sequence of the abnormal area to generate a feature embedding vector sequence, and inputting the feature embedding vector sequence into the multi-head attention layer of the Transformer decoder for feature dimension mapping to obtain the mapped features; The mapped features and the encoded features of the normal grid cells are then combined to calculate the cross-attention scores to obtain the abnormal-normal grid cell association matrix. An attention weight distribution is constructed based on the abnormal-normal grid cell association matrix, and the attention weight distribution is normalized to obtain the association strength matrix. The association strength matrix is input into a recurrent neural network, and a state update calculation is performed on the association strength of each time step to generate a hidden layer state sequence. A forward propagation operation is performed on the hidden layer state sequence, and the abnormal influence range is updated in each propagation step to generate an abnormal diffusion intensity sequence.

5. The multimodal spatiotemporal data visualization method according to claim 1, characterized in that: The process of performing interval mapping operation on the abnormal diffusion intensity sequence, establishing a time series evolution axis and a retrieval interaction component, and outputting a multi-level visualization interface includes: Performing maximum and minimum value normalization processing on the abnormal diffusion intensity sequence to generate a standardized intensity sequence, and establishing a mapping relationship matrix between intensity values and color gradient intervals; Performing a color coding operation on the standardized intensity sequence based on the mapping relationship matrix, performing coloring processing on the grid cells, and generating an initial heat map; Performing boundary extraction operations on abnormal areas in the initial heat map, constructing a contour annotation layer, and superimposing the contour annotation layer onto the initial heat map to obtain a boundary enhanced heat map; Quantifying the contribution of multimodal features in anomaly detection to generate a feature weight vector, converting the feature weight vector into a graphic symbol, and superimposing it on the boundary enhancement heat map to obtain a feature annotation heat map; Building a time series sliding control based on a time window mechanism, associating the feature annotation heat map with the time axis, and generating a dynamic heat map display interface; An interactive query module is added to the dynamic heat map display interface, and detailed information, abnormal causes and warning information of grid units are associated with the interactive interface to build a multi-level visualization interface.

6. The multimodal spatiotemporal data visualization method according to claim 5, characterized in that: The contribution of multimodal features to anomaly detection is quantified and calculated to generate a feature weight vector, which is then converted into a graphic symbol and superimposed on the boundary enhancement heat map to obtain a feature annotation heat map, including: Scoring the importance of each modal feature in the abnormal area, calculating the discrimination of each feature in abnormality judgment based on the multimodal spatiotemporal anomaly detection feature matrix, and obtaining a feature importance score sequence; Constructing a feature correlation network based on the feature importance score sequence, calculating the mutual information coefficient between features, generating a feature correlation matrix, and performing a non-negative matrix decomposition operation on the feature correlation matrix to extract feature combination patterns and obtain a feature weight vector; Performing piecewise normalization processing on the feature weight vector, dividing the weight value into multiple level intervals, and establishing a corresponding relationship table between the weight level and the graphic symbol; Based on the correspondence table, the numerical values in the feature weight vector are converted into graphic symbols of different shapes and sizes to generate a feature identification symbol set, and the feature identification symbol set is superimposed on the corresponding grid positions of the boundary enhancement heat map to construct a feature annotation heat map.

7. A multimodal spatiotemporal data visualization device, characterized in that: For executing the multimodal spatiotemporal data visualization method according to any one of claims 1 to 6, the multimodal spatiotemporal data visualization device comprises: The registration module is used to perform temporal and spatial registration on the original multimodal spatiotemporal data to generate a set of grid feature vectors and a spatiotemporal adjacency matrix; An evaluation module, configured to perform multi-dimensional anomaly evaluation on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multimodal spatiotemporal anomaly detection feature matrix; An establishment module is used to establish decision tree nodes based on the multimodal spatiotemporal anomaly detection feature matrix and determine abnormal areas; A calculation module is used to input the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, calculate the correlation strength between the abnormal area and the normal grid unit, and perform recurrent neural network iterative calculation on the correlation strength to generate an abnormal diffusion intensity sequence; The output module is used to perform interval mapping operation on the abnormal diffusion intensity sequence, establish a time series evolution axis and retrieve interactive components, and output a multi-level visualization interface.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the multimodal spatiotemporal data visualization method according to any one of claims 1 to 6 is implemented.

Images