Multi-modal spatio-temporal data visualization method, device and equipment

Abstract

The invention relates to the technical field of data visualization, and discloses a multi-modal spatio-temporal data visualization method, device and equipment, and the method comprises the steps: carrying out the time alignment and space registration of original multi-modal spatio-temporal data, and generating a grid feature vector set and a spatio-temporal adjacency matrix; performing multi-dimensional anomaly evaluation, and generating a multi-modal space-time anomaly detection feature matrix; establishing a decision tree node, and determining an abnormal region; inputting the spatial-temporal feature sequence of the abnormal region into a Transform decoder, calculating the correlation intensity of the abnormal region and a normal grid unit, and performing recurrent neural network iterative calculation on the correlation intensity to generate an abnormal diffusion intensity sequence; and performing interval mapping operation on the abnormal diffusion intensity sequence, establishing a time sequence evolution axis and a retrieval interaction component, and outputting a multi-level visual interface. According to the method, a multi-level visual interface is constructed, real-time monitoring and interactive query of an abnormal region are supported, and the understandability and availability of an abnormal detection result are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of data visualization technology, and in particular to a multimodal spatiotemporal data visualization method, device and equipment. Background Art

[0002] With the rapid development of Internet of Things technology and sensor equipment, the collection and accumulation of multimodal spatiotemporal data has shown explosive growth. These data contain rich temporal variation characteristics and spatial distribution characteristics. In the fields of urban management, environmental monitoring, transportation, etc., timely detection and location of abnormal events are of great significance to ensure the safe operation of the system. However, due to the problems of inconsistent sampling frequency, uneven spatial distribution, and uneven data quality of multimodal data, traditional single-modal anomaly detection methods are difficult to effectively utilize the complementarity and synergy between data.

[0003] Current anomaly detection methods have three main limitations: the lack of a unified processing framework for multimodal data makes it impossible to fully explore the correlation features between different modal data; secondly, existing anomaly detection algorithms often ignore the continuity and local correlation of spatiotemporal data, resulting in instability in detection results; the visualization of anomaly detection results is single, making it difficult to intuitively display the evolution process and cause analysis of abnormal events. Summary of the invention

[0004] The present invention provides a multimodal spatiotemporal data visualization method, device and equipment. The present invention constructs a multi-level visualization interface, supports real-time monitoring and interactive query of abnormal areas, and enhances the comprehensibility and usability of abnormal detection results.

[0005] In a first aspect, the present invention provides a multimodal spatiotemporal data visualization method, the multimodal spatiotemporal data visualization method comprising: Perform temporal alignment and spatial registration on the original multimodal spatiotemporal data to generate a set of grid feature vectors and a spatiotemporal adjacency matrix; Performing multi-dimensional anomaly evaluation on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multimodal spatiotemporal anomaly detection feature matrix; Establishing decision tree nodes based on the multimodal spatiotemporal anomaly detection feature matrix and determining abnormal areas; Input the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, calculate the correlation strength between the abnormal area and the normal grid unit, and perform recurrent neural network iterative calculation on the correlation strength to generate an abnormal diffusion intensity sequence; An interval mapping operation is performed on the abnormal diffusion intensity sequence, a time series evolution axis and a retrieval interaction component are established, and a multi-level visualization interface is output.

[0006] In a second aspect, the present invention provides a multimodal spatiotemporal data visualization device, the multimodal spatiotemporal data visualization device comprising: The registration module is used to perform temporal alignment and spatial registration on the original multimodal spatiotemporal data to generate a set of grid feature vectors and a spatiotemporal adjacency matrix; An evaluation module, used for performing multi-dimensional anomaly evaluation on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multimodal spatiotemporal anomaly detection feature matrix; An establishment module is used to establish decision tree nodes based on the multimodal spatiotemporal anomaly detection feature matrix and determine abnormal areas; A calculation module is used to input the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, calculate the association strength between the abnormal area and the normal grid unit, and perform recurrent neural network iterative calculation on the association strength to generate an abnormal diffusion strength sequence; The output module is used to perform interval mapping operation on the abnormal diffusion intensity sequence, establish a time series evolution axis and retrieve interactive components, and output a multi-level visualization interface.

[0007] The third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the above-mentioned multimodal spatiotemporal data visualization method.

[0008] In the technical solution provided by the present invention, heterogeneous multimodal spatiotemporal data are uniformly mapped to a regular grid structure through time alignment and spatial registration processing, thereby solving the problem of data heterogeneity. A multi-dimensional anomaly assessment mechanism is adopted, which comprehensively considers data integrity, modal synergy and spatiotemporal coverage, and effectively reduces the problem of misjudgment caused by uneven data quality. The hierarchical detection strategy based on decision trees and random forests makes full use of historical data and spatial correlation information to improve the robustness and accuracy of anomaly detection. The hybrid architecture of Transformer decoder and recurrent neural network is introduced to realize the dynamic propagation analysis of abnormal events and accurately capture the diffusion law of abnormal events. A multi-level visualization interface is constructed to support real-time monitoring and interactive query of abnormal areas, and enhance the comprehensibility and usability of anomaly detection results. Through the quantitative calculation and visual expression of feature contribution, the accurate positioning and intuitive display of abnormal causes are realized, providing strong support for the early warning and disposal of abnormal events. The unified framework design based on digital twin technology realizes the automation of the entire process from data processing, anomaly detection to visual display, which significantly improves the practicality and scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0010] Figure 1 Schematic diagram of the steps of the multimodal spatiotemporal data visualization method in an embodiment of the present invention; Figure 2 Schematic diagram of the structure of a multimodal spatiotemporal data visualization device in an embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0011] Embodiments of the present invention provide a multimodal spatiotemporal data visualization method, apparatus and device. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0012] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of a multimodal spatiotemporal data visualization method in an embodiment of the present invention includes: Step S1, performing time alignment and spatial registration on the original multimodal spatiotemporal data to generate a grid feature vector set and a spatiotemporal adjacency matrix; It is understandable that the execution subject of the present invention may be a multimodal spatiotemporal data visualization device, or a terminal or a server, which is not specifically limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0013] Specifically, the time series in the multimodal spatiotemporal data are sampled and standardized according to a fixed time interval Δt, and the time scales of different data sources are unified so that subsequent operations can be performed on the same time basis. At the same time, the spatial coordinates are projected and geometrically corrected, and the original spatial coordinates are mapped to a unified coordinate system so that they have a unified reference base in terms of geographical location, and standardized data after spatiotemporal alignment are obtained. The standardized data after spatiotemporal alignment is subjected to data type identification and format conversion, and the data of different modes are uniformly converted into numerical features to form a multimodal data set with good standardization and consistency. Targeted conversion methods are designed in combination with data characteristics, such as unique hot encoding of categorical data, periodic conversion of timestamp data, and reasonable filling of missing values ​​to generate a structured normalized multimodal data set. On the basis of the normalized data set, the monitoring area is spatially divided according to the preset grid size, and a grid index matrix is ​​established to map each geographical location in the monitoring area to a grid unit with a unique spatiotemporal identifier to form a regular grid structure. In the framework of the regular grid structure, the multimodal data in each grid cell are aggregated. Through the screening and aggregation of data in time and space, all modal features of each grid cell at a specific time t are extracted, and these features are integrated into a complete feature vector. The feature vectors of all grid cells are spatiotemporally organized, and the feature vectors with the same spatiotemporal index are merged into a grid feature vector set to ensure that the data at the same spatiotemporal location can be clearly identified in the structured data set. A spatial relationship graph between grid cells is constructed based on the regular grid structure. By calculating the connection weights between each pair of adjacent grid cells, the adjacency relationship is defined according to their geometric distance, data similarity or other spatiotemporal interaction features. The connection strength between adjacent grids is calculated using the spatial distance function, and the connection weights are organized in the form of an adjacency matrix to generate a spatiotemporal adjacency matrix with clear spatiotemporal interaction information.

[0014] Step S2, performing multi-dimensional anomaly evaluation on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multimodal spatiotemporal anomaly detection feature matrix; Specifically, the integrity statistical analysis of the multimodal data in the grid feature vector set is performed. The K types of modal data in each grid unit are calculated to calculate their proportion of existence, quantify the degree of integrity of the data in each modal dimension, and obtain an indicator reflecting the data quality of each grid unit - the data integrity score, which is expressed in numerical form as the missing or unbalanced situation of the data in the modal dimension. The standardized mutual information value is calculated for each pair of modal data in the grid feature vector set. Mutual information is an effective method to measure the degree of statistical correlation between modalities, and quantitatively describes the degree of information sharing between different modal data. By calculating the mutual information of all modal pairs and organizing them into an inter-modal correlation matrix, a multidimensional data structure reflecting the synergy between modalities is obtained. The inter-modal correlation matrix is ​​weighted averaged. By combining the importance weights of the modalities, the complex relationships in the matrix are refined into a single modal synergy score to quantify the degree of collaboration between multimodal data. Spatiotemporal correlation analysis is performed in combination with spatial adjacency. Based on the spatiotemporal adjacency matrix, the feature similarity between the target grid unit and its adjacent grid units is calculated to reflect the consistency characteristics of the target unit in the spatial relationship. Through normalization, the similarity value is standardized into a spatiotemporal coverage score, which reflects the consistency between the target grid cell and the surrounding grid cells in spatiotemporal distribution, and helps to identify the spatial location of abnormal distribution. The data integrity score, modal synergy score and spatiotemporal coverage score are weighted and fused to generate a comprehensive anomaly detection score. The weighted fusion process sets appropriate weights for each score according to the needs of the application scenario. The comprehensive anomaly detection score is compared with the preset anomaly threshold, and the grid cells with scores higher than the threshold are marked as abnormal cells. For abnormal cells, their corresponding multimodal features are extracted, and these features are organized in matrix form as a multimodal spatiotemporal anomaly detection feature matrix.

[0015] Step S3, establishing decision tree nodes based on the multimodal spatiotemporal anomaly detection feature matrix, and determining the abnormal area; Specifically, the feature decomposition operation is performed on the multimodal spatiotemporal anomaly detection feature matrix, and its complex multidimensional structure is disassembled into the time dimension feature submatrix and the space dimension feature submatrix, and the key multidimensional feature components are extracted. The time dimension feature submatrix reflects the temporal change pattern of abnormal behavior, while the space dimension feature submatrix reveals the local and global characteristics of the anomaly in the spatial distribution. The root node of the decision tree is constructed for the multidimensional feature components. The splitting threshold of each feature is calculated based on the information gain criterion to ensure that the splitting node can distinguish abnormal and non-abnormal data to the greatest extent. Through calculation, the branch structure of the decision tree is generated so that each feature can maximize the information at a specific split point. The feature value of the target grid unit is compared with the feature splitting threshold, and the depth-first search strategy is used to traverse the branch structure of the decision tree to obtain the path probability value of the target grid unit in the decision tree, which characterizes the abnormal possibility of the target grid unit in the decision path. At the same time, in order to make full use of the time series information in the historical data, the feature sequence similar to the target grid unit in the historical time series data is pattern matched. Through this operation, the matching degree between the target grid unit and the historical abnormal pattern is calculated, and the time series abnormal probability value is generated to reflect the correlation degree between the current abnormality and the known historical abnormal pattern. The path probability value and the time series abnormal probability value are used as input features to construct a random forest classifier. Random forest achieves more accurate judgment of abnormal areas by combining multiple independent decision trees. In the training and prediction process of random forest, each decision tree learns based on an independent feature subset, so as to evaluate the abnormal possibility of the target grid unit from different perspectives. By voting and counting the output results of all decision trees, the comprehensive abnormal probability distribution is obtained, which reflects the abnormal degree of the target grid unit under multiple feature dimensions. Cluster analysis is performed on the comprehensive abnormal probability distribution, and grid units with similar abnormal features are divided into the same area by clustering methods. The clustering operation is implemented by density clustering or hierarchical clustering algorithm. According to the similarity of the comprehensive abnormal features between grid units, abnormal areas with significant boundaries are automatically formed, and finally the abnormal areas in the entire system are determined.

[0016] Step S4, input the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, calculate the association strength between the abnormal area and the normal grid unit, and perform recurrent neural network iterative calculation on the association strength to generate an abnormal diffusion intensity sequence; Specifically, the spatiotemporal feature sequence of the abnormal area is processed by position encoding and time encoding to retain and strengthen the temporal and spatial information in the sequence. Position encoding uses sine and cosine functions to construct periodic features of different dimensions, while time encoding embeds the temporal information into the feature sequence together with the spatial information by mapping the temporal information into a high-dimensional vector. After encoding, a feature embedding vector sequence containing position and temporal context information is obtained. The feature embedding vector sequence is input into the multi-head attention layer of the Transformer decoder for feature dimension mapping and feature interaction analysis. The multi-head attention mechanism projects and interacts each feature vector in the embedding sequence through multiple sets of independent attention heads to capture the global dependencies between features from multiple perspectives. Through feature dimension mapping, the Transformer decoder generates mapping features containing rich spatiotemporal interaction information, reflecting the comprehensive performance of spatiotemporal features in the abnormal area in different modalities and time dimensions. The mapped features are cross-attention scores calculated with the encoded features of normal grid cells to obtain the correlation matrix between the abnormal area and the normal grid cells. This correlation matrix is ​​generated based on the attention mechanism and reflects the strength of their spatiotemporal interaction by measuring the matching degree between the abnormal and normal grid cells. In order to ensure that the weight values ​​in the matrix can correctly reflect the true correlation strength, the generated correlation matrix is ​​normalized to construct the normalized correlation strength matrix. The correlation strength matrix is ​​input into the recurrent neural network to capture the temporal dynamic characteristics of abnormal diffusion. In the recurrent neural network, each time step of the correlation strength matrix triggers the state update calculation and generates a hidden layer state sequence. The hidden layer state sequence records the internal state of the correlation strength changing over time, and reflects the dynamic process of abnormal diffusion by gradually updating the state. In each propagation step, the hidden layer state of the recurrent neural network updates the abnormal influence range through the forward propagation operation, and integrates the newly calculated state information into the time series modeling, gradually generating an abnormal diffusion intensity sequence, reflecting the gradual diffusion pattern and influence range of the abnormal area on the surrounding normal grid cells over time. Through the above-mentioned steps of position and time encoding, multi-head attention mechanism calculation, recurrent neural network state update, etc., the abnormal diffusion intensity sequence is finally obtained, reflecting the degree of influence of the abnormal area and its dynamic change process.

[0017] Step S5: perform interval mapping operation on the abnormal diffusion intensity sequence, establish a time series evolution axis and a retrieval interaction component, and output a multi-level visualization interface.

[0018] Specifically, the maximum and minimum values ​​of the abnormal diffusion intensity sequence are normalized to generate a standardized intensity sequence. The original intensity value is mapped to the standardized range of [0,1] through a linear normalization algorithm, so that the subsequent visualization operation can uniformly process the intensity values ​​in different ranges. At the same time, in order to convert the standardized intensity value into a visual expression, a mapping relationship matrix between the intensity value and the color gradient interval is established. The mapping relationship matrix encodes the intensity value into the corresponding color value in a color gradient manner. Based on the mapping relationship matrix between the intensity value and the color, the standardized intensity sequence is color-coded, and the grid unit is colored to generate an initial heat map. The intensity value of each grid unit is mapped to the corresponding color and presented in the spatial distribution map, thereby displaying the distribution characteristics of the abnormal diffusion intensity in an intuitive way. In order to improve the recognition ability of the initial heat map for the abnormal area, the boundary extraction operation is performed on the abnormal area in the heat map, the boundary information of the abnormal area is extracted by the morphological algorithm or contour detection technology, and the contour annotation layer is constructed. The contour annotation layer is superimposed on the initial heat map to obtain a boundary enhanced heat map. The contribution of multimodal features in anomaly detection is quantified and calculated to generate feature weight vectors. By evaluating the weights of different modal features in anomaly detection, the influence of each modal feature on the anomaly results is clarified. The feature weight vector is converted into graphic symbols, such as circles of different sizes or marker symbols of different shapes, and these symbols are superimposed on the boundary enhancement heat map to generate a feature annotation heat map. The feature annotation heat map effectively displays the causes and contributions of the abnormal area in a visual way. A time series sliding control is constructed based on the time window mechanism. A dynamic heat map display interface is generated by associating the feature annotation heat map with the time axis. In the dynamic heat map, users can view the state of anomaly diffusion at different time points by dragging the slider or clicking on a specific time point on the time axis, and track the changes in the abnormal area in real time. An interactive query module is added to the dynamic heat map display interface. Through the interactive module, users click on the grid unit in the heat map to view its detailed information, including the characteristic value of the grid unit, the analysis of the cause of the anomaly, and the corresponding warning information. The multi-level visualization interface constructed by the above steps realizes the display of abnormal information from macro to micro and from static to dynamic.

[0019] By analyzing the contribution of different modal features in the multimodal spatiotemporal anomaly detection feature matrix to anomaly judgment, the importance of each modal feature in the anomaly area is scored. Based on the interpretation method of the classification model (such as SHAP value or feature importance score) or statistical method (such as variance analysis), the discrimination of each modal feature in the anomaly area is calculated to generate a feature importance score sequence, which reflects the role of each modal feature in anomaly detection. Based on the feature importance score sequence, a feature correlation network is constructed, and a feature correlation matrix is ​​generated by calculating the mutual information coefficient between features. The mutual information coefficient quantifies the degree of information sharing between two modal features. The feature correlation matrix is ​​subjected to non-negative matrix decomposition operation. By decomposing the matrix into multiple non-negative low-dimensional matrices, the potential combination pattern between features is extracted and converted into a feature weight vector. The feature weight vector quantifies the contribution of each feature in the overall anomaly detection in numerical form. The feature weight vector is piecewise normalized. By dividing the weight value into multiple level intervals, the weight data is made more intuitive and suitable for subsequent symbol mapping operations. In order to convert the normalized weight value into a visual symbol, a corresponding relationship table between the weight level and the graphic symbol is established. In this relationship table, different weight levels correspond to graphic symbols of different shapes and sizes, such as circular, rectangular or triangular symbols filled with different colors, so as to distinguish the weight levels through intuitive visual elements. Based on the above correspondence table, the numerical values ​​in the feature weight vector are mapped to the corresponding graphic symbols one by one to generate a feature identification symbol set. The feature identification symbol set is superimposed on the corresponding grid position of the boundary enhancement heat map to complete the construction of the feature annotation heat map. During the superposition process, the grid cell of each abnormal area will display the corresponding feature identification symbol, thereby visually enhancing the distribution and contribution information of each modal feature in the abnormal area. The final generated feature annotation heat map integrates the visual expression of feature weights on the basis of the boundary enhancement heat map.

[0020] In the embodiment of the present invention, heterogeneous multimodal spatiotemporal data are uniformly mapped into a regular grid structure through time alignment and spatial registration processing, thereby solving the problem of data heterogeneity. A multi-dimensional anomaly assessment mechanism is adopted, which comprehensively considers data integrity, modal synergy and spatiotemporal coverage, and effectively reduces the problem of misjudgment caused by uneven data quality. The hierarchical detection strategy based on decision trees and random forests makes full use of historical data and spatial correlation information to improve the robustness and accuracy of anomaly detection. The hybrid architecture of Transformer decoder and recurrent neural network is introduced to realize the dynamic propagation analysis of abnormal events and accurately capture the diffusion law of abnormal events. A multi-level visualization interface is constructed to support real-time monitoring and interactive query of abnormal areas, and enhance the comprehensibility and usability of anomaly detection results. Through the quantitative calculation and visual expression of feature contribution, the accurate positioning and intuitive display of abnormal causes are realized, which provides strong support for the early warning and disposal of abnormal events. The unified framework design based on digital twin technology realizes the automation of the whole process from data processing, anomaly detection to visual display, which significantly improves the practicality and scalability of the system.

[0021] In a specific embodiment, the process of executing step S1 may specifically include the following steps: The time series in the multimodal spatiotemporal data are sampled and standardized according to a fixed time interval △t, and the spatial coordinates are projected and geometrically corrected to obtain standardized data after spatiotemporal alignment. Perform data type identification and format conversion on the standardized data after spatiotemporal alignment, convert different modal data into numerical features, and obtain a standardized multimodal data set; The normalized multimodal data set is spatially divided according to the preset grid size, a grid index matrix is ​​established, and the monitoring area is mapped into grid cells with unique spatiotemporal identifiers to obtain a regular grid structure; Perform multimodal data aggregation operations on each grid unit in the regular grid structure, extract all modal features of the grid unit at time t, and generate a feature vector; Performing spatiotemporal organization processing on the feature vectors, merging the feature vectors with the same spatiotemporal index into a grid feature vector set; Based on the regular grid structure, a spatial relationship graph between grid cells is constructed, the connection weights between adjacent grid cells are calculated, and the connection weights are organized into an adjacency matrix form to obtain a spatiotemporal adjacency matrix.

[0022] Specifically, the time series in the multimodal spatiotemporal data are sampled and standardized according to a fixed time interval △t, unifying the resolution of different data sources in the time dimension to ensure the consistency of data at each time point. Suppose the original time series is , the time interval is Δt, then the new sampling time series It is expressed as: ; In this process, if the data of some time points are missing, linear interpolation, nearest neighbor interpolation or mean filling methods are used to complete the data. After completing the standardization of the time dimension, the spatial coordinates are projected and geometrically corrected to ensure that the geographic spatial information of different data sources can be mapped to a unified spatial reference system. For example, if the original data uses different geographic coordinate systems (such as WGS84 and UTM), they are unified into the same coordinate system through projection transformation. Let the original coordinates be , the transformed coordinates are , the projection transformation is expressed as: ; in is the projection matrix, and its parameters are determined according to the definition of the target coordinate system. After completing the spatiotemporal alignment, the standardized data is identified and format converted, and the multimodal data is uniformly converted into numerical features. Assume that the multimodal data includes numerical features , classification features and timestamp , respectively. Use the unique hot encoding method to map it into a numerical vector, such as a classification feature Converted to [1,0,0],[0,1,0],[0,0,1]. For timestamp , extract periodic features, such as hours, months, etc., and map them into numerical variables Represents the time characteristics. After obtaining the normalized multimodal dataset, according to the preset grid size The monitoring area is spatially divided, and a grid index matrix is ​​established to divide the geographic space in the monitoring area into grid cells with unique spatiotemporal identifiers. Assume that the boundary of the monitoring area is , the grid size is and , then the grid index matrix It is expressed as: ; in and is the row and column index of the grid. Through this process, the space is divided into a regular grid structure. For each grid cell in the regular grid structure, the multimodal data contained in it is aggregated. Let the grid cell The set of data points contained in , each data point has a modal eigenvector , then at time right Aggregate all modal data to get the feature vector : ; in Indicates at time belong The aggregated feature vector represents the grid unit at time After feature extraction, feature vectors with the same spatiotemporal index are organized and merged into a grid feature vector set. In order to analyze the spatial relationship, a spatial relationship diagram between grid cells is constructed based on the regular grid structure. Assume that the geometric distance between grid cells is , defines the connection weights between adjacent grid cells for: ; in is the distance attenuation coefficient, is the connection radius. By calculating the weights between all grid cells, we get the adjacency matrix , whose elements Represents the connection relationship between grid cells.

[0023] In a specific embodiment, the process of executing step S2 may specifically include the following steps: Perform integrity statistics on the multimodal data of each grid cell in the grid feature vector set, calculate the proportion of the number of K types of modal data, and obtain the data integrity score; The standardized mutual information of each pair of modal data in the grid feature vector set is calculated to construct the inter-modal correlation matrix, and the inter-modal correlation matrix is ​​weighted averaged to obtain the modal synergy score; The feature similarity between the target grid cell and the adjacent grid cells is calculated based on the spatiotemporal adjacency matrix, and the feature similarity is normalized to obtain the spatiotemporal coverage score; The data integrity score, modal synergy score and spatiotemporal coverage score are weighted and fused to obtain a comprehensive anomaly detection score. The comprehensive anomaly detection score is compared with the preset threshold, the grid cells with scores higher than the threshold are screened out, and the multimodal features of the grid cells are organized into a matrix form to obtain a multimodal spatiotemporal anomaly detection feature matrix.

[0024] Specifically, in order to calculate the multimodal data integrity score of each grid unit, the proportion of each modal data is counted. The modal set of , Representing modality exist The number of data points actually present in express The total number of all modal data points in the The existence ratio is: ; Based on this, the grid cells are defined Data integrity score is the mean of the proportion of all modes: ; This score reflects the completeness of the data in the grid cell. In order to evaluate the synergy between modes, the normalized mutual information is calculated for each pair of modal data. and The characteristic distributions of and , the joint distribution is , mutual information Defined as: ; In order to facilitate comparison, the mutual information is standardized to obtain the standardized mutual information : ; in For modal By calculating the information entropy between all modal pairs , construct the inter-modal correlation matrix ,in . For the matrix Perform weighted average operation to obtain the modal synergy score : ; The modal synergy score reflects the collaborative relationship between different modal features within a grid cell. At the same time, in order to evaluate the similarity between the target grid cell and its neighboring grid cells, the spatiotemporal adjacency matrix Calculate feature similarity. Let grid unit and its adjacent units The eigenvectors of and , its Euclidean distance for: ; Defining similarity based on Euclidean distance : ; in is the distance attenuation coefficient. Through normalization, we get the spatiotemporal coverage score : ; Neighbor Represents a grid cell The set of neighboring units of . Score the data integrity , modality synergy score and spatiotemporal coverage scores Perform weighted fusion calculations to generate a comprehensive anomaly detection score : ; in is the weight coefficient of the score, satisfying By adjusting the weights, the role of different scores can be highlighted according to the needs of the application scenario. With preset threshold Compare and select the grid cells with scores higher than the threshold. For these abnormal cells, extract their corresponding multimodal features and organize these features into a matrix form to finally generate a multimodal spatiotemporal anomaly detection feature matrix. , whose elements Represents a grid cell Medium Mode Abnormal feature value.

[0025] In this embodiment, before performing multi-dimensional anomaly evaluation on the grid feature vector set and establishing a decision tree node based on the multimodal spatiotemporal anomaly detection feature matrix, it also includes: constructing multiple heterogeneous base learners for the multimodal spatiotemporal anomaly detection feature matrix, performing probability distribution calculation on the output result of each base learner, and obtaining a model output distribution set; performing consistency analysis on the prediction results of each base learner in the model output distribution set, calculating the variance and entropy value of the prediction results, and generating a model consistency score matrix; performing distance measurement operation on the model consistency score matrix and a pre-constructed standard sample library, calculating the degree of deviation of each grid unit feature vector from the standard sample, and obtaining an outlier degree DOO feature matrix; constructing a kernel function mapping based on the DOO feature matrix, performing maximum mean difference calculation on samples in the high-dimensional feature space, and generating O-MMD Feature sequence, and sort the O-MMD feature sequence to obtain a sample difference sorting table; set an adaptive threshold according to the sample difference sorting table, screen out high-difference samples to construct a dynamic anomaly sample training set, and perform data enhancement processing on the dynamic anomaly sample training set to obtain an enhanced training sample set; input the enhanced training sample set into the pre-trained deep neural network, and train the credibility model through transfer learning to obtain a credibility evaluation model; use the credibility evaluation model to calculate the credibility score for each eigenvector in the multimodal spatiotemporal anomaly detection feature matrix, and construct a weight matrix based on the credibility score to generate a credibility weighted feature matrix; perform adaptive feature fusion on the credibility weighted feature matrix and the multimodal spatiotemporal anomaly detection feature matrix, calculate the fusion coefficient based on the weighted sum criterion, and output the enhanced anomaly detection feature matrix.

[0026] In a specific embodiment, the process of executing step S3 may specifically include the following steps: Perform feature decomposition on the multimodal spatiotemporal anomaly detection feature matrix, extract the time dimension feature submatrix and the space dimension feature submatrix, and obtain multi-dimensional feature components; Construct the root node of the decision tree for the multi-dimensional feature components, calculate the feature splitting threshold based on the information gain criterion, and generate the decision tree branch structure; The characteristic value of the target grid cell is compared with the characteristic splitting threshold, and the decision tree branch structure is traversed based on the depth-first search strategy to obtain the path probability value of the target grid cell; Perform pattern matching operations on similar feature sequences in historical time series data, calculate the matching degree between the target grid unit and the historical anomaly pattern, and generate the time series anomaly probability value; A random forest classifier is built based on the path probability value and the time series anomaly probability value, and the output results of multiple decision trees of the random forest classifier are voted and counted to obtain the comprehensive anomaly probability distribution; Cluster analysis is performed on the comprehensive anomaly probability distribution, and grid cells with similar anomaly characteristics are divided into the same area to determine the anomaly area.

[0027] Specifically, the feature decomposition operation is performed on the multimodal spatiotemporal anomaly detection feature matrix to extract the time dimension feature submatrix and the space dimension feature submatrix to obtain the multi-dimensional feature components. Assume that the anomaly detection feature matrix is ,in represents the number of grid cells, Represents the feature dimension (including time and space dimension features), and the matrix decomposition is expressed as: ; in is the time dimension feature sub-matrix, is the spatial dimension feature submatrix, Represents the number of time features. By decomposing, the time dynamics and spatial distribution characteristics in anomaly detection are analyzed separately. For example, principal component analysis is used to extract the main feature components in the time and space dimensions, so that each submatrix captures the key feature information. The root node of the decision tree is constructed using the extracted time and space feature components. and Respectively represent The temporal and spatial features of the grid cells are used to select the split features and calculate the split threshold through the information gain criterion. The information gain is defined as: ; in is the entropy before partitioning, By feature The entropy of the subset after partitioning, is a dataset, By feature After division, the value is A subset of the data, is a set of possible values ​​of the feature. Select the feature with the largest information gain as the basis for splitting and determine the optimal splitting threshold At this point, the branching structure of the decision tree begins to form, and each node is divided into two parts according to whether the feature value is less than or greater than Further splitting is performed until the tree reaches a specified depth or the sample cannot be split any further. The characteristic value of is input into the decision tree, compared with the split threshold of each node, and the branch structure of the tree is traversed using the depth-first search strategy to determine the path probability value of the target grid unit in the tree. . Assume that each node The probability of , the path probability is calculated by multiplying the probabilities of the nodes on the path: ; At the same time, a pattern matching operation is performed on the historical time series data to calculate the matching degree between the target grid unit and the historical anomaly pattern. Suppose the characteristic sequence of the target unit is , the historical abnormal pattern is , the dynamic time warping method is used to calculate the matching degree: ; in It's time point The distance from the upper eigenvalue, is the matching path. Generates the probability value of timing anomaly based on the matching degree , indicating the abnormality of the target unit feature sequence. Based on the path probability value and the probability value of time series anomaly Construct a random forest classifier. Random forests are composed of multiple independent decision trees, and the classification performance is improved through ensemble learning methods. For each target unit, its feature value is input into all decision trees to obtain the classification results of each tree. : ; in It is The classification function of a tree, is the number of trees in the random forest. The final comprehensive anomaly probability distribution is obtained by voting statistics: ; Perform cluster analysis on the comprehensive anomaly probability distribution and divide grid cells with similar anomaly characteristics into the same area. Use density clustering algorithm (such as DBSCAN) or K-means algorithm to cluster grid cells according to their comprehensive anomaly probability and spatial location. and grid coordinates as input features, and all grid cells are divided into clusters, each cluster represents an abnormal area.

[0028] In this embodiment, after determining the abnormal area and before inputting the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, it also includes: performing spatiotemporal constraint modeling on the state features of the abnormal area at T consecutive time points, representing the feature state at each time point t as St, constructing a state transfer constraint matrix At,t+1, and establishing a set of equations based on the KVL and KCL principles: St+1 = At,t+1·St + Bt, where Bt is the state offset vector, and T-1 state transfer equations are obtained; the T-1 state transfer equations are combined to construct the time series state matrix M, the feature difference matrix D=M(t+1)-M(t) is calculated for the time series state matrix M, and the feature difference matrix D is subjected to singular value decomposition D=USV', and the eigenvectors corresponding to the first k singular values ​​are extracted as the main change mode; the main change mode is reorganized in the time dimension to construct a third-order tensor, and the Tucker decomposition algorithm is used to decompose the third-order tensor into X=G×1U×2V×3W, where G is the core tensor, and the spatial propagation sub-tensor Us and the time evolution sub-tensor Ut are obtained; the spatial attention matrix SA is constructed for the spatial propagation sub-tensor Us, the attention score is calculated, and the propagation feature map Gs is constructed based on the spatial propagation feature vector; the dynamic time warping algorithm is applied to the time evolution sub-tensor Ut to calculate the registration distance moment within the time window Matrix W, identify the key time point set {t1, t2, ..., tm} that meets the threshold condition, and generate the time feature sequence Vt; construct the joint representation tensor of the spatial propagation feature vector Vs and the time feature sequence Vt through the outer product operation, and use the tensor network compression algorithm to obtain the compact abnormal propagation feature representation F; calculate the similarity between the abnormal propagation feature representation and the preset propagation mode library to obtain the similarity vector S=[s1, s2, ..., sn], judge the abnormal type based on the maximum similarity criterion, and set the threshold of sudden abnormality to α, and the threshold of gradual abnormality to β; based on the abnormal type, use the LSTM network to perform sequence prediction in the time dimension to obtain the abnormal development prediction sequence of the next L time points, calculate the error between the predicted sequence and the actual observation value Y, and when e exceeds the preset threshold γ, trigger the dynamic update mechanism of the abnormal area boundary to generate a new abnormal area boundary B't.

[0029] In a specific embodiment, the process of executing step S4 may specifically include the following steps: The spatiotemporal feature sequence of the abnormal area is processed by position encoding and time encoding to generate a feature embedding vector sequence, and the feature embedding vector sequence is input into the multi-head attention layer of the Transformer decoder for feature dimension mapping to obtain the mapped features; The mapped features and the encoded features of the normal grid cells are used to calculate the cross attention scores to obtain the abnormal-normal grid cell association matrix. The attention weight distribution is constructed based on the abnormal-normal grid cell association matrix, and the attention weight distribution is normalized to obtain the association strength matrix. The correlation strength matrix is ​​input into the recurrent neural network, the correlation strength of each time step is updated and calculated to generate a hidden layer state sequence, the hidden layer state sequence is forward propagated, and the abnormal influence range is updated in each propagation step to generate an abnormal diffusion intensity sequence.

[0030] Specifically, the spatiotemporal feature sequence of the abnormal area is processed by position encoding and time encoding to generate a feature embedding vector sequence. Suppose the spatiotemporal feature sequence of the abnormal area is ,in represents the number of time steps, each is the spatiotemporal feature vector with dimension In order to explicitly model the position information in the sequence, the position encoding method is used to embed the position information of each time step in the sequence into the feature vector. Position encoding vector Generated using sine and cosine functions, the formula is: ; in is the index of the feature dimension, is the time step, is the dimension of the feature vector. Time encoding similarly embeds time information, for example, mapping timestamps to periodic features (such as hours and minutes) and adding them to the features in numerical form. After adding the position code and time code, the enhanced feature embedding vector sequence is obtained ,in Embed the generated features into a vector sequence The input is fed into the multi-head attention layer of the Transformer decoder for feature dimension mapping. The multi-head attention mechanism captures the global dependencies between different features in the sequence through the self-attention mechanism. For each attention head, the input embedding vector Generate query vector through linear transformation , key vector Sum value vector : ; in is the weight matrix of the linear transformation, and its dimensions are The attention score is calculated by the dot product of the query vector and the key vector and normalized to the attention weight: ; in is a scaling factor to prevent the dot product value from being too large and causing the gradient to disappear. Multi-head attention generates a mapped feature sequence by calculating multiple sets of attention scores in parallel and then concatenating them. The mapped features Encoded features with normal grid cells Calculate the cross attention score to get the abnormal-normal grid cell association matrix .set up is a feature in the abnormal feature sequence, is the feature in the normal feature sequence, and the cross attention score is calculated by dot product and normalized to weight: ; in Indicates abnormal unit and normal unit The strength of association. Based on the association matrix , construct the attention weight distribution, and perform normalization operation on it to obtain the final association strength matrix , whose elements . The correlation strength matrix Input the recurrent neural network (RNN), perform state update calculations on the association strength of each time step, and generate a hidden layer state sequence .set up is the time step The hidden state of is the corresponding association strength, and the state update formula of RNN is: ; in and is the weight matrix, is the bias vector, is the activation function (such as tanh). The hidden state sequence Record the dynamic characteristics of abnormal areas. Perform forward propagation operations, update the anomaly impact range in each propagation step, and generate anomaly diffusion intensity sequence . Diffusion intensity Defined as the hidden state Some transformation of , such as summing or taking the norm: .

[0031] In a specific embodiment, the process of executing step S5 may specifically include the following steps: The abnormal diffusion intensity sequence is normalized to the maximum and minimum values ​​to generate a standardized intensity sequence, and a mapping relationship matrix between intensity values ​​and color gradient intervals is established; Perform color coding operation on the standardized intensity sequence based on the mapping relationship matrix, color the grid cells, and generate an initial heat map; Perform boundary extraction operation on the abnormal area in the initial heat map, construct a contour annotation layer, and superimpose the contour annotation layer on the initial heat map to obtain a boundary enhanced heat map; The contribution of multimodal features in anomaly detection is quantified and calculated to generate feature weight vectors, which are then converted into graphic symbols and superimposed on the boundary enhancement heat map to obtain a feature annotation heat map. Build a time series sliding control based on the time window mechanism, associate the feature annotation heat map with the time axis, and generate a dynamic heat map display interface; An interactive query module is added to the dynamic heat map display interface, and the detailed information, abnormal causes and warning information of the grid units are associated with the interactive interface to build a multi-level visualization interface.

[0032] Specifically, the maximum and minimum values ​​of the abnormal diffusion intensity sequence are normalized to generate a standardized intensity sequence so that its range is unified to [0,1]. Suppose the original abnormal diffusion intensity sequence is ,in Indicates that at time step The diffusion intensity of is normalized as follows: ; in and They are The minimum and maximum values ​​of Represents the standardized intensity value. Through this process, all intensity values ​​are compressed between 0 and 1. A mapping matrix between intensity values ​​and color gradient intervals is established based on the standardized intensity sequence. Color gradient mapping uses linear interpolation to map intensity values ​​from low to high to a specific color space (such as from blue to red). Suppose the color gradient interval is ,in Indicates the color corresponding to the lowest intensity, Indicates the color corresponding to the highest intensity, the color mapping matrix The elements of are represented as: ; Among them, interpolate is an interpolation function, and the corresponding color is determined by the intensity value. The mapping relationship matrix is ​​used to perform color coding operations on the standardized intensity sequence, and the color information is applied to the coloring process of the grid unit to generate an initial heat map. In the heat map, each grid unit is filled with a corresponding color to reflect its intensity level, thereby intuitively showing the spatial distribution of abnormal diffusion. In order to highlight the boundary characteristics of the abnormal area, the boundary extraction operation is performed on the abnormal area in the initial heat map. It is achieved through morphological methods or contour detection algorithms, such as using image gradient methods to extract boundaries. Assume that the heat map matrix is , the boundary extraction algorithm is expressed as: ; in represents the boundary matrix. Contour annotation is performed to construct a boundary annotation layer, which is then superimposed on the initial heat map to generate a boundary enhancement heat map. The boundary enhancement heat map can show the range of the abnormal area and improve the visual effect. The contribution of multimodal features in anomaly detection is quantified. Assume that the modal feature set of the grid unit is , the feature contribution is calculated by feature importance score or normalized weighted calculation, the formula is: ; in Features The weight, importance is the feature importance score. All feature weights are piecewise normalized and converted into graphic symbols, for example, the weight size is represented by the symbol size or shape. The generated feature identification symbol set is superimposed on the grid unit position corresponding to the boundary enhancement heat map to generate a feature annotation heat map. After completing the static heat map, a time series sliding control is constructed through the time window mechanism, and the feature annotation heat map is associated with the time axis to generate a dynamic heat map display interface. Assume that the time axis is , each time point corresponds to a heat map slice, and the time series sliding control allows users to select a specific time point or time range to display the corresponding heat map status in real time. The dynamic heat map shows the spatiotemporal evolution of abnormal diffusion. An interactive query module is added to the dynamic heat map display interface. By clicking or selecting a grid cell, the user views the detailed information of the cell, including intensity value, multimodal characteristic value, abnormal cause and warning information. All this information is integrated into a multi-level visualization interface, providing users with multi-dimensional analysis capabilities from global to local, from static to dynamic.

[0033] In a specific embodiment, the execution step quantifies the contribution of the multimodal features in anomaly detection, generates a feature weight vector, converts the feature weight vector into a graphic symbol, and superimposes it on the boundary enhancement heat map. The process of obtaining the feature annotation heat map can specifically include the following steps: The importance of each modal feature in the abnormal area is scored, and the discrimination of each feature in abnormal judgment is calculated based on the multimodal spatiotemporal anomaly detection feature matrix to obtain a feature importance score sequence; Based on the feature importance score sequence, a feature correlation network is constructed, the mutual information coefficient between features is calculated, a feature correlation matrix is ​​generated, and a non-negative matrix decomposition operation is performed on the feature correlation matrix to extract the feature combination pattern and obtain the feature weight vector; Perform piecewise normalization on the feature weight vector, divide the weight value into multiple level intervals, and establish a corresponding relationship table between the weight level and the graphic symbol; Based on the correspondence table, the numerical values ​​in the feature weight vector are converted into graphic symbols of different shapes and sizes to generate a feature identification symbol set, and the feature identification symbol set is superimposed on the corresponding grid position of the boundary enhancement heat map to construct a feature annotation heat map.

[0034] Specifically, for each modal feature in the abnormal area, its discrimination in abnormal judgment is calculated based on the multimodal spatiotemporal anomaly detection feature matrix to obtain the feature importance score sequence. Suppose the anomaly detection feature matrix is ,in represents the number of grid cells in the abnormal area, Represents the number of modal features. Features, its distinguishing It is calculated by variance analysis or statistical distribution difference. Assume that the abnormal unit set is , the normal unit set is , No. The discrimination of a feature is defined as: ; in and The abnormal unit and the normal unit are represented by The mean of the features, represents the variance of the feature, is a smoothing term to prevent the denominator from being zero. , construct feature importance score sequence , reflecting the contribution of each modal feature to anomaly detection. After obtaining the feature importance score sequence, the correlation between features is analyzed and a feature correlation network is constructed. Feature correlation is calculated by the mutual information coefficient, which quantifies the degree of information sharing between two features. and The probability distributions are and , the joint distribution is , the mutual information between the two Defined as: ; For ease of comparison, the mutual information coefficient needs to be standardized to : ; in It is a feature The information entropy of all modal features is calculated pairwise. , and obtain the feature correlation matrix ,in Indicates and In order to extract the feature combination pattern, the feature association matrix Perform non-negative matrix decomposition to decompose it into two low-dimensional non-negative matrices and ,satisfy: ; in is the number of feature combinations, Represents the weight matrix of features and combinations, Represents the contribution of the combined mode. By decomposing, the weight vector of each feature is extracted , reflecting its contribution in different combination modes. After merging, the feature weight vector is generated The feature weight vector is normalized piecewise, and the weight value is divided into multiple level intervals. Assume that the weight value range is , the formula for piecewise normalization is: ; in Represents the normalized weight value. According to the weight level, Map to different symbol shapes and sizes, and establish a correspondence table between weight levels and graphic symbols. Using the correspondence table, convert the values ​​in the feature weight vector into a feature identifier symbol set. ,in Features The generated feature identification symbol set is superimposed on the corresponding grid position of the boundary enhancement heat map to construct a feature annotation heat map. Each grid unit shows the anomaly diffusion intensity, and the feature contribution of the unit is intuitively displayed through superimposed graphic symbols, providing interpretability of the cause of the anomaly.

[0035] The above describes the multimodal spatiotemporal data visualization method in the embodiment of the present invention. The following describes the multimodal spatiotemporal data visualization device in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, a multimodal spatiotemporal data visualization device includes: The registration module is used to perform temporal alignment and spatial registration on the original multimodal spatiotemporal data to generate a set of grid feature vectors and a spatiotemporal adjacency matrix; An evaluation module is used to perform multi-dimensional anomaly evaluation on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multi-modal spatiotemporal anomaly detection feature matrix; Establish a module for establishing decision tree nodes based on the multimodal spatiotemporal anomaly detection feature matrix and determining the abnormal area; The calculation module is used to input the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, calculate the correlation strength between the abnormal area and the normal grid unit, and perform recurrent neural network iterative calculation on the correlation strength to generate an abnormal diffusion intensity sequence; The output module is used to perform interval mapping operations on the abnormal diffusion intensity sequence, establish the time series evolution axis and retrieve interactive components, and output a multi-level visualization interface.

[0036] Through the collaboration of the above components, the heterogeneous multimodal spatiotemporal data are uniformly mapped into a regular grid structure through time alignment and spatial registration processing, solving the problem of data heterogeneity. A multi-dimensional anomaly assessment mechanism is adopted, which comprehensively considers data integrity, modal synergy and spatiotemporal coverage, effectively reducing the problem of misjudgment caused by uneven data quality. The hierarchical detection strategy based on decision trees and random forests makes full use of historical data and spatial correlation information to improve the robustness and accuracy of anomaly detection. The hybrid architecture of Transformer decoder and recurrent neural network is introduced to realize the dynamic propagation analysis of abnormal events and accurately capture the diffusion law of abnormal events. A multi-level visualization interface is constructed to support real-time monitoring and interactive query of abnormal areas, enhancing the comprehensibility and usability of anomaly detection results. Through the quantitative calculation and visualization expression of feature contribution, the accurate positioning and intuitive display of the cause of the anomaly are realized, providing strong support for the early warning and disposal of abnormal events. The unified framework design based on digital twin technology realizes the automation of the entire process from data processing, anomaly detection to visualization, significantly improving the practicality and scalability of the system.

[0037] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0038] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0039] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0040] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0041] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal spatiotemporal data visualization method, characterized in that: The method comprises: Perform temporal alignment and spatial registration on the original multimodal spatiotemporal data to generate a set of grid feature vectors and a spatiotemporal adjacency matrix; Performing multi-dimensional anomaly evaluation on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multimodal spatiotemporal anomaly detection feature matrix; Establishing decision tree nodes based on the multimodal spatiotemporal anomaly detection feature matrix and determining abnormal areas; Input the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, calculate the correlation strength between the abnormal area and the normal grid unit, and perform recurrent neural network iterative calculation on the correlation strength to generate an abnormal diffusion intensity sequence; An interval mapping operation is performed on the abnormal diffusion intensity sequence, a time series evolution axis and a retrieval interaction component are established, and a multi-level visualization interface is output.

2. The multimodal spatiotemporal data visualization method according to claim 1, characterized in that: The method of performing time alignment and spatial registration on the original multimodal spatiotemporal data to generate a grid feature vector set and a spatiotemporal adjacency matrix includes: The time series in the multimodal spatiotemporal data are sampled and standardized according to a fixed time interval △t, and the spatial coordinates are projected and geometrically corrected to obtain standardized data after spatiotemporal alignment. Performing data type identification and format conversion processing on the standardized data after the spatiotemporal alignment, uniformly converting the different modal data into numerical features, and obtaining a standardized multimodal data set; The normalized multimodal data set is spatially divided according to a preset grid size, a grid index matrix is ​​established, and the monitoring area is mapped into grid cells with unique spatiotemporal identifiers to obtain a regular grid structure; Performing a multimodal data aggregation operation on each grid unit in the regular grid structure, extracting all modal features of the grid unit at time t, and generating a feature vector; Performing spatiotemporal organization processing on the feature vectors, merging feature vectors with the same spatiotemporal index into a grid feature vector set; A spatial relationship diagram between grid units is constructed based on the regular grid structure, connection weights between adjacent grid units are calculated, and the connection weights are organized into an adjacency matrix form to obtain a spatiotemporal adjacency matrix.

3. The multimodal spatiotemporal data visualization method according to claim 1, characterized in that: The performing multi-dimensional anomaly evaluation on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multi-modal spatiotemporal anomaly detection feature matrix includes: Performing integrity statistics on the multimodal data of each grid unit in the grid feature vector set, calculating the proportion of the number of K types of modal data, and obtaining a data integrity score; Performing standardized mutual information calculation on each pair of modal data in the grid feature vector set, constructing an inter-modal correlation matrix, and performing a weighted average operation on the inter-modal correlation matrix to obtain a modal synergy score; Calculating feature similarities between a target grid cell and adjacent grid cells based on the spatiotemporal adjacency matrix, and normalizing the feature similarities to obtain a spatiotemporal coverage score; Performing weighted fusion calculation on the data integrity score, the modal synergy score, and the spatiotemporal coverage score to obtain a comprehensive anomaly detection score; The comprehensive anomaly detection score is compared with a preset threshold, grid cells with scores higher than the threshold are screened out, and the multimodal features of the grid cells are organized into a matrix form to obtain a multimodal spatiotemporal anomaly detection feature matrix.

4. The multimodal spatiotemporal data visualization method according to claim 1, characterized in that: The step of establishing a decision tree node based on the multimodal spatiotemporal anomaly detection feature matrix and determining an abnormal area includes: Performing a feature decomposition operation on the multimodal spatiotemporal anomaly detection feature matrix, extracting a time dimension feature submatrix and a space dimension feature submatrix, and obtaining a multidimensional feature component; Constructing a decision tree root node for the multi-dimensional feature components, and calculating a feature splitting threshold based on an information gain criterion to generate a decision tree branch structure; Comparing the characteristic value of the target grid unit with the characteristic splitting threshold, and traversing the decision tree branch structure based on a depth-first search strategy to obtain a path probability value of the target grid unit; Perform pattern matching operations on similar feature sequences in historical time series data, calculate the matching degree between the target grid unit and the historical anomaly pattern, and generate the time series anomaly probability value; A random forest classifier is constructed based on the path probability value and the time series anomaly probability value, and voting statistics are performed on the output results of multiple decision trees of the random forest classifier to obtain a comprehensive anomaly probability distribution; A cluster analysis is performed on the comprehensive abnormal probability distribution, grid cells with similar abnormal characteristics are divided into the same area, and the abnormal area is determined.

5. The multimodal spatiotemporal data visualization method according to claim 1, characterized in that: The step of inputting the spatiotemporal feature sequence of the abnormal area into a Transformer decoder, calculating the association strength between the abnormal area and the normal grid unit, and performing recurrent neural network iterative calculation on the association strength to generate an abnormal diffusion intensity sequence includes: Performing position encoding and time encoding on the spatiotemporal feature sequence of the abnormal area to generate a feature embedding vector sequence, and inputting the feature embedding vector sequence into the multi-head attention layer of the Transformer decoder for feature dimension mapping to obtain mapped features; The mapped features and the encoded features of the normal grid cells are used to calculate the cross attention scores to obtain the abnormal-normal grid cell association matrix, and the attention weight distribution is constructed based on the abnormal-normal grid cell association matrix, and the attention weight distribution is normalized to obtain the association strength matrix; The association strength matrix is ​​input into a recurrent neural network, a state update calculation is performed on the association strength of each time step to generate a hidden layer state sequence, a forward propagation operation is performed on the hidden layer state sequence, and the abnormal influence range is updated in each propagation step to generate an abnormal diffusion strength sequence.

6. The multimodal spatiotemporal data visualization method according to claim 1, characterized in that: The process of performing interval mapping operation on the abnormal diffusion intensity sequence, establishing a time series evolution axis and a search interaction component, and outputting a multi-level visualization interface includes: Performing maximum and minimum value normalization processing on the abnormal diffusion intensity sequence to generate a standardized intensity sequence, and establishing a mapping relationship matrix between intensity values ​​and color gradient intervals; Performing color coding operation on the standardized intensity sequence based on the mapping relationship matrix, coloring the grid units, and generating an initial heat map; Performing boundary extraction operation on the abnormal area in the initial thermal map, constructing a contour annotation layer, and superimposing the contour annotation layer on the initial thermal map to obtain a boundary enhanced thermal map; Quantifying the contribution of multimodal features in anomaly detection to generate a feature weight vector, converting the feature weight vector into a graphic symbol, and superimposing it on the boundary enhancement heat map to obtain a feature annotation heat map; Building a time series sliding control based on a time window mechanism, associating the feature annotation heat map with the time axis, and generating a dynamic heat map display interface; An interactive query module is added to the dynamic heat map display interface, and detailed information, abnormal causes and warning information of the grid unit are associated with the interactive interface to build a multi-level visualization interface.

7. The multimodal spatiotemporal data visualization method according to claim 6, characterized in that: The contribution of the multimodal features in anomaly detection is quantified and calculated to generate a feature weight vector, and the feature weight vector is converted into a graphic symbol and superimposed on the boundary enhancement heat map to obtain a feature annotation heat map, including: Scoring the importance of each modal feature in the abnormal area, calculating the discrimination of each feature in abnormality determination based on the multimodal spatiotemporal anomaly detection feature matrix, and obtaining a feature importance score sequence; Based on the feature importance score sequence, a feature correlation network is constructed, the mutual information coefficient between features is calculated, a feature correlation matrix is ​​generated, and a non-negative matrix decomposition operation is performed on the feature correlation matrix to extract feature combination patterns and obtain a feature weight vector; Performing piecewise normalization processing on the feature weight vector, dividing the weight value into multiple level intervals, and establishing a corresponding relationship table between the weight level and the graphic symbol; Based on the correspondence table, the numerical values ​​in the feature weight vector are converted into graphic symbols of different shapes and sizes to generate a feature identification symbol set, and the feature identification symbol set is superimposed on the corresponding grid positions of the boundary enhancement heat map to construct a feature annotation heat map.

8. A multimodal spatiotemporal data visualization device, characterized in that: Used to perform the multimodal spatiotemporal data visualization method according to any one of claims 1 to 7, the multimodal spatiotemporal data visualization device comprising: The registration module is used to perform temporal alignment and spatial registration on the original multimodal spatiotemporal data to generate a set of grid feature vectors and a spatiotemporal adjacency matrix; An evaluation module, used for performing multi-dimensional anomaly evaluation on the grid feature vector set and the spatiotemporal adjacency matrix to generate a multimodal spatiotemporal anomaly detection feature matrix; An establishment module is used to establish decision tree nodes based on the multimodal spatiotemporal anomaly detection feature matrix and determine abnormal areas; A calculation module is used to input the spatiotemporal feature sequence of the abnormal area into the Transformer decoder, calculate the association strength between the abnormal area and the normal grid unit, and perform recurrent neural network iterative calculation on the association strength to generate an abnormal diffusion strength sequence; The output module is used to perform interval mapping operation on the abnormal diffusion intensity sequence, establish a time series evolution axis and retrieve interactive components, and output a multi-level visualization interface.

9. A computer device, characterized in that: It comprises a memory and a processor, the memory stores a computer program that can be run on the processor, and the processor implements the multimodal spatiotemporal data visualization method described in any one of claims 1 to 7 when executing the computer program.

Images