A marine ecological anomaly early warning method and system based on multimodal perception and spatiotemporal reasoning

Abstract

The present invention relates to the field of marine anomaly detection technology, and in particular to a marine ecological anomaly early warning method and system based on multimodal perception and spatiotemporal reasoning. The method comprises acquiring original multimodal data, including image data and time series data; performing global context encoding on the acquired image data, dynamically evolving the time series data, and aligning multimodal features through feature normalization; fusing multimodal features based on a dynamic ecological knowledge graph fusion mechanism; predicting fused features based on spatiotemporal graph changes; and performing attribution and diagnosis based on the prediction results. The present invention enables the model to automatically amplify key signals and suppress irrelevant noise, greatly improving the ability to capture early, weak anomaly signals in complex marine environments.

Description

Technical Field

[0001] The present invention relates to the technical field of marine anomaly detection, and in particular to a marine ecological anomaly early warning method and system based on multimodal perception and spatiotemporal reasoning. Background Art

[0002] As changes in the global marine ecosystem become increasingly significant, abnormal evolution of biome structures has become a key indicator for assessing marine health, identifying local ecological crises, and monitoring the spread of eutrophication and biodiversity degradation. Especially in typical eutrophication-sensitive areas such as nearshore areas, semi-enclosed bays, and estuaries, phenomena such as marine phytoplankton blooms, jellyfish invasions, and frequent red and green tides often cause structural imbalances in the entire community, leading to species migration, local extinction, and even the collapse of fisheries. Therefore, accurately capturing abnormal changes in marine biome structures and enabling early warning and identification are key tasks of smart ocean monitoring systems.

[0003] Although current marine ecological monitoring has entered the era of multi-source observation, which allows the acquisition of massive amounts of multimodal data such as images, acoustics, hydrology, and biochemistry, existing technologies still face serious bottlenecks in data utilization. On the one hand, traditional multimodal data fusion methods mostly adopt static or shallow fusion strategies, such as simple splicing of feature vectors or weighted averaging of fixed weights. This approach ignores the dynamic importance of different modalities under different ecological events and fails to capture the coordinated response relationship between modalities, resulting in redundant fusion feature expressions and the submergence of key signals. On the other hand, existing methods are generally insufficient in modeling the complex spatiotemporal dependencies of ecological anomalies. The evolution of marine anomalies usually has both spatial propagation and diffusion and long-term temporal laws. Traditional models may have difficulty processing the non-Euclidean spatial graph structure formed by monitoring stations, or have limitations in capturing long-term temporal dependencies, making it impossible to effectively portray the complete spatiotemporal dynamic panorama of anomaly events. More critically, most current anomaly detection models are "diagnostic" models, which tend to focus on post-identification and lack the ability to predict the future state of the ecosystem, resulting in delayed early warning. At the same time, their "black box" decision-making process cannot provide a clear explanation for the causes and dominant factors of anomalies, greatly limiting their application value in actual ocean management decision-making.

[0004] Therefore, there is an urgent need to develop a new marine ecological anomaly detection method that can dynamically perceive the importance of multiple modalities, deeply model complex spatiotemporal dependencies, and provide explainable predictive warnings, so as to break through the bottleneck of existing technologies and improve the intelligence level of smart ocean monitoring systems. Summary of the Invention

[0005] In order to solve the above-mentioned problems, the present invention provides a marine ecological anomaly early warning method and system based on multimodal perception and spatiotemporal reasoning.

[0006] In a first aspect, the present invention provides a marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning, which adopts the following technical solutions:

[0007] A marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning, including:

[0008] Acquire raw multimodal data, including image data and time series data;

[0009] Perform global context encoding on the acquired image data, dynamically evolve the time series data, and perform multimodal feature alignment through feature normalization;

[0010] Multimodal features are fused based on the dynamic ecological knowledge graph fusion mechanism;

[0011] Predict fusion features based on spatiotemporal map changes;

[0012] Make attributions and diagnoses based on the predicted results.

[0013] Furthermore, the global context encoding of the acquired image data includes encoding the image using a visual transformer, directly modeling the relationship between any two regions in the image using a self-attention mechanism, and obtaining true global context features, wherein at time t Image data of a certain monitoring area obtained Split into a space N The grid consists of image blocks, each block is of size P×P, Mapped to D feat The dimensional feature space forms an image block embedding sequence, and then the query Q, key K, and value V matrices are obtained through linear transformation. The attention weight is obtained by calculating the dot product of Q and K, and is scaled to avoid too small gradients. Finally, it is normalized by the softmax function and applied to V. The multi-head attention mechanism projects Q, K, and V h times through different linear transformations, calculates the attention separately, concatenates the results, and performs linear transformation again. The Transformer encoder layer applies residual connection and layer normalization LN after each sub-layer to promote gradient propagation and stabilize the training process, which is expressed as:

[0014] ,

[0015] ,

[0016] ,

[0017] in, and are the input and output of the lth layer respectively. FFN is a network composed of two layers of linear transformation and ReLU activation function. After all L layers of encoding, the final output sequence is taken The vector corresponding to the category token in is used as the global feature representation of the entire image at time t .

[0018] Furthermore, the dynamic evolution encoding of time series data includes encoding using a time convolutional network TCN. For a one-dimensional input sequence And a convolution kernel f of size m, the convolution operation at sequence time t is defined as:

[0019] ,

[0020] Where d is the expansion factor; multiple residual blocks of TCN are used to dilate the input It is directly added to the output of the residual block through a shortcut connection. The internal calculation process of the residual block is expressed as:

[0021] ,

[0022] Among them, Conv1D1 and Conv1D2 are dilated causal convolutional layers; represents the weight normalization layer; is the ReLU activation function layer; Dropout is the Dropout layer.

[0023] Furthermore, the multimodal feature alignment by feature normalization includes introducing feature normalization alignment before feature fusion to eliminate the imbalance between modalities caused by encoder differences. For the feature vector output by any modality, , first calculate the mean of all elements in the vector and variance , then the feature vector is normalized using the calculated mean and variance, and an affine transformation is performed through two learnable parameters including gain g and bias b to maintain and enhance the expressive power of the model, which is expressed as:

[0024] ,

[0025] in, is a very small stabilizing constant added to prevent the denominator from being zero; Learnable affine transformation parameters, with the same dimension as the feature vector; Represents element-wise multiplication to form aligned features , , .

[0026] Furthermore, the dynamic ecological knowledge graph fusion mechanism is used to fuse multimodal features, including introducing domain expert knowledge and encoding it into a static ecological knowledge adjacency matrix. , represents the graph of the prior correlation strength between modalities, where the nodes of the graph represent each modality, and the matrix , where the number of modalities M=3, including image, acoustic, and biochemical, and its elements The value range is [0, 1], which represents the strength of the association between modality i and modality j in marine ecology theory. The data-driven cross-modal dynamic attention mechanism is introduced to combine the three modal features aligned at time t into a feature matrix , using the self-attention mechanism to calculate the dynamic attention weight matrix , whose elements It represents how much attention modality i pays to modality j in the data scenario at the current time t, expressed as:

[0027] ,

[0028] in, , are the learnable query and key projection matrices, respectively; is the dimension of the query and key vectors.

[0029] Furthermore, the dynamic ecological knowledge graph fusion mechanism is used to fuse multimodal features, and also includes an organic combination of static prior knowledge and dynamic data-driven attention, through a learnable gated scalar , dynamically weighing expert knowledge The relationship with learning The proportion of , and calculate the fusion guidance matrix :

[0030] ,

[0031] in, A simple neural network based on the current modal feature matrix Generate, use the fusion guidance matrix to weight the original alignment feature matrix to achieve interaction and enhancement of information between modalities, expressed as: , and finally the enhanced feature matrix Perform integration and dimensionality reduction, perform nonlinear transformation and dimensionality compression through a multi-layer perceptron network, and obtain the final fusion state vector:

[0032] ,

[0033] in, The structure is a bottleneck structure that includes an activation function to force the model to learn a more informative and compact representation:

[0034] ,

[0035] Where x is the flattened vector of the input (i.e. ); , is the weight matrix of MLP; , is the bias vector of MLP.

[0036] Furthermore, the prediction of fusion features based on spatiotemporal map changes includes A discrete observation point is abstracted as a set of nodes in the graph , construct the connection relationship between nodes, i.e. edge set, based on their spatial and ecological correlation , forming a static space graph , at each time step t, the state vectors of all N regions form the feature matrix: ; Use the graph attention network GAT to aggregate spatial information. For any node in the graph , which is related to neighbor nodes The attention coefficient between Calculated by a single-layer feedforward network, the weight vector Parameterize the network:

[0037] ,

[0038] in, is a learnable linear transformation matrix; Represents the concatenation operation of vectors; Transpose of the vector; It is a leaky rectified linear unit activation function, and the softmax function is used to activate the nodes The attention coefficients of all neighbors are normalized to obtain the final attention weight :

[0039] ,

[0040] Normalized attention weights Indicates at time , neighbor nodes Information about nodes Importance of nodes Feature vector updated by the GAT layer That is, it is the weighted sum of all its neighbor node features based on the attention weight.

[0041] Furthermore, the prediction of fusion features based on spatiotemporal graph changes also includes using a temporal Transformer encoder to model time evolution, time-step spatially enhanced state sequence As the input of the Temporal Transformer, the model uses a multi-layer multi-head self-attention mechanism to calculate the correlation between the states of any two historical moments. The linear prediction head is then used to decode the output of the Temporal Transformer at the last time step to predict the ecological state at the future time t+1:

[0042] ,

[0043] in, It represents the feature matrix of the last time step of the temporal Transformer output sequence, and the predicted state matrix of the final output Is the model based on the past The spatiotemporal evolution law of each moment can be used to comprehensively predict the ecological status of the entire monitoring network at the next moment.

[0044] Furthermore, the attribution and diagnosis based on the prediction results include the following: The real observation data is encoded into the real state vector Then, calculate the vector The reconstruction error between the two regions is , its raw anomaly score It is defined as the square of the Euclidean distance between two things: , and a dynamic normalization method based on Z-score is used to calculate the final anomaly score, where for each node Maintain a length of A sliding time window is used to calculate the mean of the historical reconstruction error within the window and standard deviation :

[0045] ,

[0046] in, Node in the past Reconstruction error within the time window The mean and standard deviation of is a very small stabilizing constant added to prevent the denominator from being zero, and the final abnormal score is Indicates the extent to which the current forecast error deviates from its historical norm.

[0047] The second aspect is a marine ecological anomaly early warning system based on multimodal perception and spatiotemporal reasoning, including:

[0048] The data acquisition module is configured to acquire raw multimodal data, including image data and time series data;

[0049] The encoding module is configured to perform global context encoding on the acquired image data, dynamically evolve the time series data, and perform multimodal feature alignment through feature normalization;

[0050] The alignment module is configured to perform feature fusion on multimodal features based on the dynamic ecological knowledge graph fusion mechanism;

[0051] The prediction module is configured to predict the fused features based on the spatiotemporal graph changes;

[0052] The diagnosis module is configured to perform attribution and diagnosis based on the prediction results.

[0053] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, for providing a marine ecological anomaly warning method based on multimodal perception and spatiotemporal reasoning.

[0054] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded and executed by the processor to prepare a marine ecological anomaly warning method based on multimodal perception and spatiotemporal reasoning.

[0055] In summary, the present invention has the following beneficial technical effects:

[0056] Compared to existing marine ecological anomaly identification methods, which face technical bottlenecks such as static and blind modal information fusion, insufficient depth in spatiotemporal evolution modeling, delayed anomaly perception, and difficult-to-interpret results, this invention addresses the need for predictive and interpretable intelligent early warning of "abnormal changes in marine ecosystems" and builds a new end-to-end solution based on spatiotemporal graph converters and dynamic knowledge fusion. The beneficial effects of this invention are mainly reflected in the following aspects:

[0057] First, this invention fundamentally addresses the challenge of multi-source, heterogeneous data fusion through its innovative Dynamic Ecological Knowledge Graph Fusion module (D-EGF). By combining a data-driven cross-modal attention mechanism with a domain expert knowledge graph, modal fusion transcends fixed, unchanging feature concatenation and instead enables dynamic, adaptive, and focused intelligent weighting based on real-time data quality and content. This enables the model to automatically amplify key signals and suppress irrelevant noise, significantly improving its ability to detect early, weak, and anomalous signals in complex marine environments.

[0058] Secondly, the Spatiotemporal Graph Transformer Prediction module (ST-GTP) designed in this paper achieves a significant breakthrough in the depth and breadth of spatiotemporal dependency modeling. This module innovatively combines the Graph Attention Network (GAT) with the Transformer architecture. GAT dynamically captures irregular, time-varying spatial influences between regions, while the Transformer, with its long-range self-attention mechanism, effectively overcomes the limitations of traditional time series models (such as RNNs / LSTMs) in handling long-term dependencies. Working together, the model achieves unprecedented insights into and predictions of the complex propagation and evolution of marine anomalies across a broad range of spatiotemporal dimensions.

[0059] Furthermore, the core advantage of this invention lies in achieving a paradigm shift from "diagnostic detection" to "predictive early warning." The entire framework is trained with the goal of predicting future states, and anomaly detection is based on the deviation between predicted and true values. This "prediction-verification" model enables the system to issue early warnings, indicating deviations from normal evolutionary trajectory before abnormal conditions fully manifest. This provides a valuable window for marine management authorities to implement preventative interventions. Its early warning value far exceeds that of traditional post-event alerting methods.

[0060] Finally, this invention perfectly addresses the "black box" problem of deep learning models through its Predictive Anomaly Diagnosis and Interpretation (PADI) module. The system not only accurately identifies anomalies but also leverages the model's inherent attention weights to clearly attribute the driving factors of anomalies across modal, spatial, and temporal dimensions, automatically generating structured diagnostic reports. This inherent interpretability transforms early warning results into "white box" intelligence with scientific evidence and decision-making support capabilities, significantly enhancing the technology's credibility, reliability, and usability in practical management applications.

[0061] In summary, under moderate disturbance conditions, the proposed method improves the accuracy (ACC) of ecological anomaly warnings to 94.6% and an F1 score of 93.8%. The average warning lead time (lead time) reaches 8.2 hours, and the factor consistency assessment for diagnostic interpretation reaches 92.5%. The average system inference time is 185 milliseconds. This method demonstrates excellent performance across multiple core dimensions, including warning accuracy, timeliness, and interpretability, and possesses strong practical deployment value and engineering potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Schematic diagram of a marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning according to Example 1 of the present invention;

[0063] Figure 2 1 is a schematic diagram comparing the core performance indicators of the model of Example 1 of the present invention;

[0064] Figure 3 1 is a schematic diagram comparing high-level functional indicators of the model according to Example 1 of the present invention;

[0065] Figure 4 Schematic diagram of the model comprehensive performance radar of Example 1 of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further described in detail below with reference to the accompanying drawings.

[0067] Example 1

[0068] Reference Figure 1 The present embodiment provides a marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning, including:

[0069] The present invention proposes a "marine ecological anomaly early warning method based on multimodal perception and spatiotemporal graph reasoning" (hereinafter referred to as STEAD-Net). Its overall technical framework includes four core modules: (1) multimodal hierarchical feature encoding module (MFE); (2) dynamic ecological knowledge graph fusion module (D-EGF); (3) spatiotemporal graph transformer prediction module (ST-GTP); (4) predictive anomaly diagnosis and interpretation module (PADI). The specific scheme is as follows:

[0070] (1) Multimodal hierarchical feature encoding module (MFE)

[0071] This module is the perception foundation of the entire system. Its core task is to transform raw multimodal data from different sensors, with vastly different structures and semantics, into deep feature vectors with uniform dimensions and condensed information. Ocean observation data is unique. For example, remote sensing images are often obscured by factors such as clouds and solar flares, and anomalies (such as red tides) are large-scale and irregular in shape. Meanwhile, time-series data such as biochemical and acoustic data contain a mixture of long-term trends, cyclical fluctuations, and random noise. Therefore, this module must be highly robust and targeted. We have designed optimized encoder structures for different modalities to maximize the extraction of effective information related to ecological anomalies.

[0072] 1) Image modality global context encoding,

[0073] In ocean monitoring, image data is the most intuitive basis for identifying large-scale anomalies, such as red tides and oil spills. Traditional convolutional neural networks expand their receptive range by stacking local receptive fields, but their ability to process non-local, long-range macroscopic structures, such as "ribbon-like" red tides or large areas of turbid water, is limited. To overcome this limitation, the present invention employs a Vision Transformer (ViT) to encode images. Its self-attention mechanism directly models the relationship between any two regions in the image, thereby capturing truly global contextual features.

[0074] First, at the time t Image data of a certain monitoring area obtained Split into a space N The grid consists of image blocks, each block is of size P×P These image patches are flattened and projected by a learnable linear matrix Map to D feat dimensional feature space to form an image block embedding sequence.

[0075] To enable the model to perceive the original spatial position of the image patch, a learnable position encoding matrix is added to the sequence of image patch embeddings. In addition, a special, learnable “category token” is concatenated at the beginning of the sequence. This token does not correspond to any specific image patch, but it will interact with all image patches through the self-attention mechanism, and its output state in the last layer of the Transformer encoder will be used as the aggregated feature representation of the entire image. After this preprocessing, the input image is converted into a feature sequence that can be processed by the Transformer:

[0076] ,

[0077] in, Indicates time The input image, 、 、 are the height, width and number of channels of the image respectively; For the a vector of flattened image patches; is the total number of image blocks; represents a learnable linear projection matrix. represents the learnable initial embedding of category tokens. is the learnable position encoding matrix.

[0078] The feature sequence It is then fed into a network consisting of L layers of stacked Transformer encoders with the same structure. Each encoder layer consists of two core sub-layers: a multi-head self-attention (MHSA) sub-layer and a position feed-forward network (FFN) sub-layer.

[0079] The core of the HSA sublayer is scaled dot-product attention. Given an input sequence, it first performs a linear transformation to obtain three matrices: query (Q), key (K), and value (V). Attention weights are calculated by computing the dot product of Q and K, scaled to minimize gradients, and finally normalized using a softmax function before being applied to V:

[0080] ,

[0081] in, The dimension of the key vector in a single attention head.

[0082] The multi-head attention mechanism projects Q, K, and V h times (i.e., h heads) through different linear transformations, calculates attention separately, concatenates the results, and then performs another linear transformation. This allows the model to simultaneously focus on information from different locations in different representation subspaces.

[0083] ,

[0084] ,

[0085] in, is the input sequence of the Transformer encoder layer; is the number of attention heads. are all learnable weight matrices of this layer.

[0086] A full Transformer encoder layer applies residual connections and layer normalization (LN) after each sub-layer to facilitate gradient propagation and stabilize the training process:

[0087] ,

[0088] ,

[0089] ,

[0090] in, and are the input and output of the lth layer respectively. FFN is a network consisting of two layers of linear transformation and ReLU activation function.

[0091] After all L layers of encoding, the final output sequence is taken The vector corresponding to the category token in (i.e., the first vector) is used as the global feature representation of the entire image at time t .

[0092] 2) Temporal modal dynamic evolution coding,

[0093] Marine biochemical (such as chlorophyll, dissolved oxygen, and nutrients) and acoustic (such as backscatter intensity in specific frequency bands) data are inherently time series. Their dynamic trends—whether long-term slow growth or short-term sharp fluctuations—are key indicators of ecological anomalies. To effectively capture these long- and short-term patterns, this paper employs a temporal convolutional network (TCN) for encoding. By combining causal convolution with dilated convolution, this approach achieves efficient parallel computation while possessing long-term memory capabilities comparable to or even exceeding those of recurrent neural networks.

[0094] The core building block of TCN is dilated causal convolution. And a convolution kernel f of size m, the convolution operation at sequence time t is defined as:

[0095] ,

[0096] Where d is the dilation factor. The "causal" property ensures that when computing the output at time t, the convolution kernel only accesses the input at time t or before, and does not utilize future information. The "dilation" property, by sampling with an exponentially increasing step size d, enables a single node in the top-level network to cover a very long time range at the bottom level, effectively capturing long-range dependencies.

[0097] The overall architecture of TCN is composed of multiple stacked residual blocks. Each residual block contains two layers of dilated causal convolution, as well as weight normalization, ReLU activation function and Dropout layer. It is directly added to the output of the residual block through a shortcut connection (if the input and output dimensions are different, it is transformed through a 1×1 convolution), which greatly stabilizes the training of deep networks. The internal calculation process of a residual block can be expressed as:

[0098] ,

[0099] Among them, Conv1D1 and Conv1D2 are dilated causal convolutional layers; represents the weight normalization layer; is the ReLU activation function layer; Dropout is the Dropout layer.

[0100] For input historical sequences, such as biochemical indicator sequences , after the TCN network processes it, the output of the last time step is taken as the summary feature vector of the time series mode at time t Similarly, the independent TCN network transforms the acoustic feature sequence Encoded as a feature vector These eigenvectors condense the evolution trend, fluctuation pattern and periodic characteristics of each mode over a period of time.

[0101] 3) Multimodal feature normalization and alignment,

[0102] Due to the different structures and activation functions of different encoders (ViT, TCN), the feature vectors they output are , , Although the dimensions are uniform, their numerical distributions (i.e., the mean and variance of their statistical properties) can vary significantly. If fusion proceeds without further processing, modal features with larger numerical ranges will dominate the fusion calculations (such as weighted summation and inner product calculations), while modal information with smaller numerical ranges may be overwhelmed. This defeats the original intention of the model to fairly and dynamically weigh the contributions of each modality.

[0103] To eliminate the inter-modality imbalance caused by encoder differences, this paper introduces a feature normalization and alignment step before feature fusion. This step independently performs layer normalization (LN) on the feature vectors output by each modality encoder. Unlike batch normalization, LN operates on the feature dimensions of a single sample and is unaffected by batch size, making it more suitable for single-sample or small-batch inference scenarios, as may be encountered in this paper.

[0104] For any modal output eigenvector , the layer normalization process is as follows:

[0105] First calculate the mean of all elements in the vector and variance :

[0106] ,

[0107] in, D feat Represents the total dimensional feature space.

[0108] The feature vector is then normalized using the calculated mean and variance, and affine transformed using two learnable parameters — gain g and bias b — to maintain and enhance the expressive power of the model:

[0109] ,

[0110] in, is a very small stabilizing constant added to prevent the denominator from being zero; Learnable affine transformation parameters, with the same dimension as the feature vector; Represents element-wise multiplication.

[0111] After this step, the eigenvectors of all modes are adjusted to similar numerical scales, forming "aligned" features. , , , laying the foundation for fair and effective dynamic integration in the next step.

[0112] (2) Dynamic Ecological Knowledge Graph Fusion Module (D-EGF)

[0113] After obtaining high-quality feature representations of each modality aligned through the MFE module , , Finally, the core task of this module is to effectively connect these information islands to form a unified, comprehensive, and profound understanding of the current ecological state. Simple feature concatenation or averaging cannot capture the complex nonlinear dependencies and dynamic changes in importance between modalities. Therefore, this paper proposes a dynamic ecological knowledge graph fusion mechanism that innovatively combines domain expert knowledge with a data-driven attention mechanism to achieve dynamic, adaptive, and interpretable fusion of multimodal features.

[0114] 1) Construction of ecological prior knowledge graph,

[0115] In order to make the model fusion process more robust and consistent with ecological common sense, the present invention first introduces domain expert knowledge. This knowledge is encoded into a static "ecological knowledge adjacency matrix" This is a graph representing the strength of the prior associations between modalities, where the nodes of the graph represent the individual modalities.

[0116] The matrix , where the number of modalities M = 3 (image, acoustic, biochemical), and its elements The range of is [0, 1], which represents the strength of the correlation between mode i and mode j in marine ecology theory.

[0117] Strong correlation: Water color (image modality) is directly related to chlorophyll concentration (biochemical modality), so and Can be set to 0.9; Moderate correlation: Large aggregations of large plankton such as jellyfish (identifiable by the specific scattering intensity of the acoustic mode) usually lead to a decrease in dissolved oxygen in the local water column (biochemical mode), with an indirect but significant connection. Can be set to 0.6; Weak correlation: Background noise (acoustic mode) and sea surface temperature (biochemical mode) have no direct causal relationship in most cases. Can be set to a lower value of 0.1.

[0118] The construction of this knowledge matrix provides a robust foundation for the model, preventing it from learning spurious associations that violate ecological common sense from the data when the data are sparse or the noise interference is severe.

[0119] 2) Data-driven cross-modal dynamic attention,

[0120] Static knowledge alone is insufficient for coping with the complex and ever-changing ocean environment. For example, the reliability of the image modality decreases dramatically when obscured by cloud cover, and the acoustic modality introduces strong noise when a ship passes. The model must be able to dynamically adjust its reliance on each modality based on the real-time quality and content of the current data. To this end, this paper introduces a data-driven cross-modal dynamic attention mechanism.

[0121] Combine the three modal features aligned at time t into a feature matrix , using the self-attention mechanism, the model can calculate a dynamic attention weight matrix , whose elements Indicates how much attention modality i should pay to modality j in the data scenario at the current time t. The calculation process is as follows:

[0122] ,

[0123] in, , They are respectively the learnable query and key projection matrices; is the dimension of the query and key vectors; softmax is performed independently on each row of the matrix to ensure that the sum of the attention weights of each modality to all other modalities is 1.

[0124] This dynamic attention matrix This gives the model great flexibility, enabling it to autonomously discover and enhance collaborative signals between modalities, or suppress conflicting and noisy signals based on the instantaneous characteristics of the input data.

[0125] 3) Feature fusion of knowledge and attention guidance,

[0126] The static prior knowledge mentioned above is organically combined with dynamic data-driven attention. Through a learnable gated scalar , the model can dynamically weigh whether to trust expert knowledge more ( ), or does it rely more on relationships learned from the current data ( ). The final fusion guidance matrix Calculated by the following formula:

[0127] ,

[0128] in, A simple neural network based on the current modal feature matrix Generate it so that it has data dependency:

[0129] ,

[0130] in, This design allows the model to rely more on high-quality data and clear signals. When the data is noisy or the pattern is ambiguous, it can fall back more on robust expert knowledge. ;MLP is used for nonlinear feature transformation; A Sigmoid activation function is applied to the MLP output to limit its output to the (0,1) interval.

[0131] After obtaining the fusion guidance matrix, it is used to weight the original alignment feature matrix to achieve interaction and enhancement of information between modalities, which can be expressed as a matrix multiplication:

[0132] ,

[0133] 4) Fusion feature dimensionality reduction and output,

[0134] Finally, in order to obtain a single final state vector representing the monitoring area at time t, it is necessary to transform the enhanced feature matrix Integration and dimensionality reduction are performed. First, it is flattened into a long vector, and then a multi-layer perceptron (MLP) network is used for nonlinear transformation and dimension compression to obtain the final fused state vector:

[0135] ,

[0136] in, The structure is a bottleneck structure (that is, the hidden layer dimension is smaller than the input layer and output layer) that includes an activation function to force the model to learn a more informative and compact representation:

[0137] ,

[0138] Where x is the flattened vector of the input (i.e. ); , is the weight matrix of MLP; , is the bias vector of MLP.

[0139] The final output fusion state vector This highly condenses mutually reinforcing, collaborative information from all modalities, guided by both knowledge and data. It is no longer a simple accumulation of modal features, but a holistic, structural description of the current ecological state, serving as input for the next module, the "Spatiotemporal Graph Converter."

[0140] (3) Spatiotemporal Graph Transformer Prediction Module (ST-GTP)

[0141] This module is the core of the present invention, which aims to model and predict the complex evolutionary patterns of the ecological status of multiple monitoring areas in both time and space. After that, this module will The key to achieving "early warning" rather than "alarm" is to construct a spatiotemporal map of the region as a whole and learn its dynamic laws to achieve accurate prediction of future status.

[0142] 1) Construction of regional ecological map,

[0143] First, the entire monitoring sea area Discrete observation points (such as buoy stations, drone aerial photography centers) are abstracted into a set of nodes in the graph The connection relationship between nodes, that is, the edge set , is constructed based on their spatial and ecological connections. Connections are defined based on various criteria, such as geographic Euclidean distance, connectivity predicted by ocean circulation models, or historically observed correlations. This creates a static spatial map that can characterize the potential interactions between regions. .

[0144] At each time step t, the state vectors of all N regions form a graph signal, which is represented as a feature matrix:

[0145] ,

[0146] The matrix Complete description of the moment The ecological status of all nodes in the entire monitoring network is the starting point for spatial information dissemination in this module.

[0147] 2) spatially dependent dynamic modeling,

[0148] The spatial propagation of marine ecological anomalies, such as the spread of red tides, is not uniform. It is affected by factors such as local ocean currents and water mass boundaries, resulting in dynamic variations in the influence of different neighboring regions on the central region. To capture this dynamic spatial dependency, this paper employs a graph attention network (GAT) to aggregate spatial information. Unlike traditional graph convolutional networks (GCNs), which use a fixed Laplacian matrix for neighborhood aggregation, GAT dynamically learns an attention weight for each neighboring node.

[0149] For any node in the graph , which is related to neighbor nodes ( is a node The attention coefficient between the first-order neighbor set of It is calculated by a single-layer feed-forward network consisting of a learnable weight vector Parameterization:

[0150] ,

[0151] in, It is a learnable linear transformation matrix that acts on each node feature and is used to map the input features to a higher-level feature space; Represents the concatenation operation of vectors; Transpose of the vector; is the activation function of the rectified linear unit with leakage.

[0152] In order to make the attention coefficients of different nodes comparable, the softmax function is used to calculate the attention coefficients of nodes. The attention coefficients of all neighbors are normalized to obtain the final attention weight :

[0153] ,

[0154] Normalized attention weights Indicates at time , neighbor nodes Information about nodes Importance of Node Feature vector updated by the GAT layer That is, the weighted sum of all its neighbor node features based on the attention weight:

[0155] ,

[0156] in, Is a nonlinear activation function. To enhance the expressiveness and stability of the model, multiple independent attention heads can be used in parallel and their outputs can be concatenated or averaged. After processing by the GAT layer, the state representation matrix that integrates the dynamic spatial neighborhood information is obtained. .

[0157] 3) Long-term time evolution prediction,

[0158] After modeling the spatial dependencies within each time slice, it is necessary to further capture the evolution of these spatial states over time. Changes in marine ecosystems often exhibit long-term dependencies. For example, El Niño events may not significantly alter the marine ecosystem within their affected area until months have passed. Traditional recurrent neural networks (RNNs) are prone to vanishing or exploding gradients when processing such long sequences. Therefore, this paper employs a temporal Transformer encoder to model temporal evolution.

[0159] will pass time-step spatially enhanced state sequence As the input of the temporal transformer. Similar to ViT in image modality coding, this module uses a multi-layer multi-head self-attention mechanism to enable the model to calculate the correlation between the states of any two historical moments. For example, the model can learn The continuous high temperature state at all times is the cause The output of the temporal transformer is a key precursor to the algae bloom, even though the two moments are far apart in time. A deep representation that integrates the entire historical spatiotemporal context is provided for each node at each historical moment.

[0160] 4) Future ecological status prediction output,

[0161] Finally, in order to achieve "predictive" anomaly detection, the present invention uses a linear prediction head to decode the output of the temporal Transformer at the last time step to predict the ecological state at the future time t+1.

[0162] ,

[0163] in, Represents the feature matrix of the last time step of the temporal Transformer output sequence.

[0164] The final output prediction state matrix Is the model based on the past The spatial and temporal evolution patterns of each moment provide a comprehensive forecast of the ecological status of the entire monitoring network at the next moment. This forecast is the cornerstone for the subsequent anomaly diagnosis and interpretation modules to make forward-looking judgments.

[0165] (4) Predictive Abnormal Diagnosis and Interpretation Module (PADI)

[0166] This module is the final decision-making and interpretation outlet for the entire intelligent early warning system. Simply outputting a label of "abnormal" or "normal," or a raw prediction of future conditions, is far from sufficient for ocean managers and researchers. The core task of this module is to compare the prediction results of the previous module with the actual observation data. It not only accurately determines the occurrence of abnormal events and locates their spatiotemporal location, but also deeply analyzes the basis for model decisions, answering key questions such as "why is it abnormal" and "what drives it" in a "white box" manner, thereby generating an information-rich structured early warning report that can be directly used by decision makers.

[0167] 1) Anomaly scoring based on prediction error,

[0168] The core idea of ​​anomaly detection in this invention is that a well-trained spatiotemporal prediction model under normal evolutionary laws should have a prediction of future states that is highly consistent with the actual observations; if there is a huge deviation between the two, it is very likely that the ecosystem has deviated from the known normal pattern, that is, an anomaly has occurred.

[0169] Getting the next moment The real observation data is encoded into the real state vector through modules (1) and (2) After that, this module calculates the vector predicted by module (3) For each regional node , its raw anomaly score It is defined as the square of the Euclidean distance (L2 norm) between two variables:

[0170] ,

[0171] However, the stability of the background environment in different sea areas is different, and the prediction error baseline under normal conditions is also different. For example, the normal error fluctuation of an estuary area with intense hydrological activity may be higher than that of a calm inner bay. If a global unified fixed threshold is adopted, it is very easy to generate a large number of false alarms in the former, and may miss early weak abnormal signals in the latter. In order to make the abnormality score adaptive to the historical fluctuation characteristics of each area, the present invention adopts a dynamic normalization method based on Z-score to calculate the final abnormality score. Specifically, the system will generate a Z-score for each node. Maintain a length of A sliding time window is used to calculate the mean of the historical reconstruction error within the window and standard deviation :

[0172] ,

[0173] in, Node in the past Reconstruction error within the time window The mean and standard deviation of is a very small stabilizing constant added to prevent the denominator from being zero.

[0174] The final anomaly score It visually indicates the degree to which the current forecast error deviates from its historical normal level. When the score exceeds a preset, statistically significant threshold , the system can determine the node exist Abnormal events occur at all times, thus achieving adaptive and highly robust anomaly detection.

[0175] 2) Multi-dimensional attribution and explainability diagnosis,

[0176] After determining an anomaly, answering the question "why" is the key to improving the practical value of the system. The framework design of this invention inherently retains rich process information, making it possible to conduct attribution analysis from the three dimensions of modality, space, and time.

[0177] Modality attribution: Which type of observation data (image, acoustic, or biochemical) is the anomaly dominated by the drastic changes? The answer is contained in the dynamic ecological knowledge graph fusion module (D-EGF) in module (2). Fusion guidance matrix The row vector represents the ability of each mode to absorb the information of other modes after fusion. Define a modal contribution score , calculate the matrix The norm of the sum of the rows is used to quantify the relative importance of each modality at the time of the anomaly. The higher the contribution score of a modality, the greater the influence of its features on the final fusion representation in the current scenario, and the more likely it is to be the main driver of the anomaly.

[0178] Spatial attribution: Is the anomaly in a region caused by its own internal factors, or is it strongly affected by neighboring regions (such as the spread of pollutants)? The answer can be found in the Graph Attention Network (GAT) of module (3). At the moment of anomaly occurrence, for the node that is judged to be abnormal, You can check its The attention weight obtained at , if the weight corresponding to a neighbor node j If the weight of attention is much higher than that of other neighbors, there is good reason to believe that node j is the key spatial driving force that causes node abnormality. By tracing back such high attention weight connections, the spatial propagation path of abnormal events can be outlined.

[0179] Temporal attribution: Is the current anomaly a sudden event, or is it a gradual evolution of key events at one or more moments in the past? The answer is hidden in the temporal Transformer encoder of module (3). By analyzing its internal self-attention map, especially the attention weights connecting the final prediction output with each historical time step, we can identify the key historical moments that contribute most to the current anomaly prediction. For example, if the model predicts When the time is abnormal, If we pay extremely high attention to the status at that moment, it means that the event that occurred 48 hours ago (such as a surge in terrestrial input caused by heavy rainfall) is an important precursor to this anomaly.

[0180] 3) Generation of structured early warning reports,

[0181] The final output of this module is not an isolated anomaly score or attribution value, but a structured warning report that integrates all the above information and can be read and understood by humans. Exceeding the threshold The system will automatically generate a report. The report includes:

[0182] Basic information: warning time, location of abnormal area (such as number, longitude and latitude).

[0183] Anomaly level: Based on anomaly score The values ​​are graded ( "No abnormality" is “mildly abnormal”; is “moderately abnormal”; is “highly abnormal”).

[0184] Dominant mode: List the most contributing modes and their key indicator changes (e.g., "The biochemical mode is dominant, with the dissolved oxygen indicator decreasing by 30% within 6 hours").

[0185] Spatial origin: Indicate the most affected neighboring area and its status (e.g., "mainly affected by the diffusion of high-concentration nutrient-salt water masses in area A02").

[0186] Key precursors: Describe the most influential historical events (e.g., "highly correlated with the typhoon passing two days ago")

[0187] By generating such a comprehensive, in-depth and explainable early warning report, the present invention transforms complex model outputs into intuitive and clearly guiding decision-making basis, thereby playing a huge role in practical applications such as prevention, control and tracing of marine ecological disasters.

[0188] Experimental verification:

[0189] To systematically validate the performance advantages of our proposed method for detecting marine ecological anomalies, we constructed a multimodal ecological observation dataset based on a representative coastal area of ​​the Yellow Sea. The dataset consists of three modalities: ① Imagery: Daily MODIS and GOCI satellite remote sensing imagery, from which spatially distributed products such as chlorophyll a concentration, suspended sediment, and sea surface temperature are extracted; ② Acoustic: Acoustic Doppler Current Profiler (ADCP) data deployed in key aquaculture areas, reflecting water stratification, current profiles, and plankton aggregation; and ③ Biochemical: High-frequency time-series data such as temperature, salinity, dissolved oxygen, pH, and nutrient concentrations collected hourly by multiple fixed buoy stations. A total of 680 multimodal sample events were annotated in the dataset, divided into a training set (500) and a test set (180), ensuring coverage of four typical ecological states: "normal state," "early algal bloom," "jellyfish aggregation," and "bottom-level hypoxia."

[0190] To comprehensively evaluate the performance of our method, we compared it with four representative mainstream methods from academia and industry: ① LSTM: A classic time series detection method that uses only multi-channel biochemical modality data for training and prediction, representing the benchmark for single-modality time series analysis. ② Concat-MLP: A basic multimodal fusion method that simply concatenates the feature vectors of all modalities at the same time point and inputs them into a multi-layer perceptron (MLP) for classification, representing a shallow static fusion approach. ③ GCN-GRU: An advanced spatiotemporal graph network method that uses a graph convolutional network (GCN) to process the spatial relationships between monitoring sites and a gated recurrent unit (GRU) to handle time series evolution. This method is a strong contender in the current field of spatiotemporal anomaly detection. Our method (STEAD-Net): This is the complete framework proposed in this paper, integrating dynamic knowledge fusion, graph attention networks, and a predictive model based on temporal transformers.

[0191] All methods were evaluated using the same training and test sets. Evaluation metrics included: ① Accuracy (ACC): the proportion of samples correctly classified as normal / abnormal; ② F1-Score (F1-Score): the harmonic mean of precision and recall, a comprehensive measure of the ability to identify various types of abnormal events; ③ Lead Time (h): the average number of hours before the actual occurrence of an anomaly, for predictive models; ④ Interpretation Consistency (Interp-Acc, %): the consistency between the attribution explanations provided by the model (e.g., dominant mode, spatial origin) and key influencing factors annotated by marine ecology experts; and ⑤ Inference Time (Time, milliseconds): the average time required to process a single time step, a measure of system deployment efficiency.

[0192] Table 1 Data comparison of different methods under five indicators

[0193] Method Name ACC(%) F1-Score (%) Lead Time (h) Interp-Acc (%) Time (ms) LSTM 80.5 77.2 N / A N / A 130 Concat-MLP 79.2 75.1 N / A 45.8 98 GCN-GRU 88.7 87.2 2.5 71.3 210 Method of the present invention 94.6 93.8 8.2 92.5 185

[0194] Because LSTM and Concat-MLP models are designed as "diagnostic" classifiers, their task is to determine the category of the current sample, rather than predicting future states. Therefore, the "lead time" metric for evaluating predictive performance is not applicable to them. Furthermore, as a unimodal time series model, LSTM's internal state is difficult to directly map into a multidimensional, quantifiable causal explanation. While Concat-MLP is multimodal, it is essentially a "black box" fully connected network, lacking inherent interpretability mechanisms and unable to generate structured attribution reports. Therefore, the "Interpretation Consistency (Interp-Acc)" metric is either inapplicable or extremely low.

[0195] Because the five metrics of "Accuracy," "F1 score," "Alert Lead Time," "Explanation Consistency," and "Inference Time" have different dimensions and evaluation methods, they cannot be directly displayed in a single radar chart. To ensure a fair and intuitive comparison of comprehensive performance, this paper first normalizes the metrics. This paper retains the positive metrics (higher values ​​are better, namely, accuracy, F1 score, alert lead time, and explanation consistency) unchanged. For the negative metrics (lower values ​​are better, namely, inference time), this paper first normalizes all metrics to their maximum and minimum values, then applies the inverse process of "1 - normalized value" to convert all metrics to a "higher values ​​are better" format, allowing for a clear display of performance in a single chart. However, after this process, the worst-performing model (such as the LSTM model with zero alert lead time and explanation consistency) will have normalized values ​​for multiple metrics of 0 or very close to 0. When multiple such indicators appear, the corresponding multiple directions on the radar chart will shrink back to the center point, and its performance profile will "collapse" into a straight line or a point, losing the visual significance of comparing the profile with other models. To avoid this problem, the present invention introduces a paranoid offset This mechanism converts all values ​​from the original Range, force linear compression to a new range In this way, even the worst performing metrics will not be displayed as 0 on the graph, ensuring that all methods have at least some visual display space in the graph, making the overall performance profile of the model clearer and easier to interpret.

[0196] The experimental results are shown in Table 1. Figure 2 、 Figure 3 、 Figure 4As shown, the method of the present invention achieves significant advantages in overall performance. Traditional unimodal methods, such as LSTM, completely ignore spatial information and other modalities such as images and acoustics, resulting in limited performance and unsatisfactory accuracy and F1 scores. The simple multimodal fusion method, Concat-MLP, although integrating multi-source information, fails to capture dynamic intermodal correlations due to its static splicing. Its performance is even slightly lower than that of the unimodal LSTM, and its interpretability is extremely poor. GCN-GRU, as an excellent spatiotemporal model, significantly improves performance by explicitly modeling spatial graph structures, demonstrating the necessity of joint spatiotemporal modeling. However, the GRU it uses is weaker than the Transformer in capturing long-range temporal dependencies, and its fixed spatial aggregation weights and insufficient dynamic adaptability result in significant gaps in accuracy, warning lead time, and interpretability compared to the method of the present invention. Because the present invention utilizes multiple complex deep learning modules, including a visual transformer, a temporal convolutional network, a graph attention network, and a temporal transformer, this increased complexity is a necessary price to pay for achieving advanced features such as high accuracy, predictability, and interpretability. This makes the proposed method (185ms) more time-consuming than the simplest model, Concat-MLP. However, it is worth noting that the inference speed of the proposed method (185ms) is significantly faster than that of the GCN-GRU (210ms), which is also a complex spatiotemporal model. This is because the Transformer architecture used in the proposed method for processing time series has a high degree of parallel computing capability and can process the entire time series at once; whereas the GRU units in the GCN-GRU are inherently recursive and must be calculated sequentially in time steps. This serial dependency becomes a computational bottleneck when processing long sequences. Therefore, the proposed method achieves better computational efficiency despite the higher model complexity.

[0197] In comparison, the method of the present invention (STEAD-Net) has achieved systematic breakthroughs in multiple dimensions: the accuracy (94.6%) and F1 score (93.8%) are both the highest, proving the effectiveness of its dynamic knowledge fusion and deep spatiotemporal modeling; the warning lead time is as long as 8.2 hours, far exceeding the comparative methods, highlighting the great practical value of its "predictive" framework; the explanation consistency is as high as 92.5%, indicating that its endogenous attribution mechanism can provide reliable decision support; and the inference time, although slightly higher than the simplest model, is still at the millisecond level, fully meeting the near-real-time business needs.

[0198] In summary, the experimental results fully verify the comprehensive leadership of the method of the present invention in key capabilities such as diagnostic accuracy, warning timeliness, and decision interpretability, and prove its great potential and application value in solving the problem of multimodal intelligent anomaly detection in complex marine environments.

[0199] Example 2

[0200] This embodiment provides a marine ecological anomaly early warning system based on multimodal perception and spatiotemporal reasoning, including:

[0201] The data acquisition module is configured as follows:

[0202] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, for example, a marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning.

[0203] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded and executed by the processor to describe a marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning.

[0204] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning, characterized by: include: Acquire raw multimodal data, including image data and time series data; Perform global context encoding on the acquired image data, dynamically evolve the time series data, and perform multimodal feature alignment through feature normalization; Multimodal features are fused based on the dynamic ecological knowledge graph fusion mechanism; Predict fusion features based on spatiotemporal map changes; Make attributions and diagnoses based on the predicted results; The dynamic ecological knowledge graph fusion mechanism is used to fuse multimodal features, including introducing domain expert knowledge and encoding it into a static ecological knowledge adjacency matrix. , represents the graph of the prior correlation strength between modalities, where the nodes of the graph represent each modality, and the matrix , where the number of modalities M=3, including image, acoustic, and biochemical, and its elements The value range is [0, 1], which represents the correlation strength between modality i and modality j in marine ecology theory. The data-driven cross-modal dynamic attention mechanism is introduced to combine the three modal features aligned at time t into a feature matrix , using the self-attention mechanism to calculate the dynamic attention weight matrix , whose elements It represents how much attention modality i pays to modality j in the data scenario at the current time t, expressed as: ， in, , are the learnable query and key projection matrices, respectively; is the dimension of query and key vectors; The dynamic ecological knowledge graph fusion mechanism is used to fuse multimodal features, and also includes an organic combination of static prior knowledge and dynamic data-driven attention, through a learnable gated scalar. , dynamically weighing expert knowledge The relationship with learning The proportion of , and calculate the fusion guidance matrix : ， in, A simple neural network based on the current modal feature matrix Generate, use the fusion guidance matrix to weight the original alignment feature matrix to achieve interaction and enhancement of information between modalities, expressed as: , and finally the enhanced feature matrix Perform integration and dimensionality reduction, and perform nonlinear transformation and dimensionality compression through a multi-layer perceptron network to obtain the final fusion state vector ,in, The structure is a bottleneck structure that includes an activation function to force the model to learn a more informative and compact representation: ， Where x is the flattened vector of the input (i.e. ); , is the weight matrix of MLP; , is the bias vector of MLP.

2. The marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning according to claim 1 is characterized in that: The global context encoding of the acquired image data includes encoding the image using a visual transformer, directly modeling the relationship between any two regions in the image using a self-attention mechanism, and obtaining true global context features, wherein at time t Image data of a certain monitoring area obtained Split into a space N The grid consists of image blocks, each block is of size P×P, Mapped to D feat The dimensional feature space forms an image block embedding sequence, and then obtains the query Q, key K, and value V matrices through linear transformation. The attention weight is obtained by calculating the dot product of Q and K, and is scaled to avoid too small gradients. Finally, it is normalized by the softmax function and applied to V. The multi-head attention mechanism projects Q, K, and V h times through different linear transformations, calculates the attention separately, concatenates the results, and performs linear transformation again. The Transformer encoder layer applies residual connection and layer normalization LN after each sub-layer to promote gradient propagation and stabilize the training process, which is expressed as: ， ， ， in, and are the input and output of the lth layer respectively. FFN is a network composed of two layers of linear transformation and ReLU activation function. After all L layers of encoding, the final output sequence is taken The vector corresponding to the category token in is used as the global feature representation of the entire image at time t .

3. The marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning according to claim 2 is characterized in that: The dynamic evolution encoding of time series data includes encoding using a time convolutional network TCN. For a one-dimensional input sequence And a convolution kernel f of size m, the convolution operation at sequence time t is defined as: ， Where d is the expansion factor; multiple residual blocks of TCN are used to dilate the input It is directly added to the output of the residual block through a shortcut connection. The internal calculation process of the residual block is expressed as: ， Among them, Conv1D1 and Conv1D2 are dilated causal convolutional layers; represents the weight normalization layer; is the ReLU activation function layer; Dropout is the Dropout layer.

4. The marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning according to claim 3 is characterized in that: The multimodal feature alignment by feature normalization includes introducing feature normalization alignment before feature fusion to eliminate the imbalance between modalities caused by encoder differences. For the feature vector output by any modality, , first calculate the mean of all elements in the vector and variance , then the feature vector is normalized using the calculated mean and variance, and an affine transformation is performed through two learnable parameters including gain g and bias b to maintain and enhance the expressive power of the model, which is expressed as: ， in, is a very small stabilizing constant added to prevent the denominator from being zero; Learnable affine transformation parameters, with the same dimension as the feature vector; Represents element-wise multiplication to form aligned features , , .

5. The marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning according to claim 4 is characterized in that: The prediction of fusion features based on spatiotemporal map changes includes the prediction of A discrete observation point is abstracted as a set of nodes in the graph , construct the connection relationship between nodes, i.e. edge set, based on their spatial and ecological correlation , forming a static space graph , at each time step t, the state vectors of all N regions form the feature matrix: ; Use the graph attention network GAT to aggregate spatial information. For any node in the graph , which is related to neighbor nodes The attention coefficient between Calculated by a single-layer feedforward network, the weight vector Parameterize the network: ， in, is a learnable linear transformation matrix; Represents the concatenation operation of vectors; Transpose of the vector; It is a leaky rectified linear unit activation function, and the softmax function is used to activate the nodes The attention coefficients of all neighbors are normalized to obtain the final attention weight : ， Normalized attention weights Indicates at time , neighbor nodes Information about nodes Importance of nodes Feature vector updated by the GAT layer That is, it is the weighted sum of all its neighbor node features based on the attention weight.

6. The marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning according to claim 5 is characterized in that: The prediction of fusion features based on spatiotemporal graph changes also includes using a temporal Transformer encoder to model time evolution, time-step spatially enhanced state sequence As the input of the Temporal Transformer, the model uses a multi-layer multi-head self-attention mechanism to calculate the correlation between the states of any two historical moments. The linear prediction head is then used to decode the output of the Temporal Transformer at the last time step to predict the ecological state at the future time t+1: ， in, It represents the feature matrix of the last time step of the temporal Transformer output sequence, and the predicted state matrix of the final output Is the model based on the past The spatiotemporal evolution law of each moment can be used to comprehensively predict the ecological status of the entire monitoring network at the next moment.

7. The marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning according to claim 6 is characterized in that: The attribution and diagnosis based on the prediction results, including the next moment The real observation data is encoded into the real state vector Then, calculate the vector The reconstruction error between the two regions is , its raw anomaly score It is defined as the square of the Euclidean distance between two things: , and a dynamic normalization method based on Z-score is used to calculate the final anomaly score, where for each node Maintain a length of A sliding time window is used to calculate the mean of the historical reconstruction error within the window and standard deviation : ， in, Node in the past Reconstruction error within the time window The mean and standard deviation of is a very small stabilizing constant added to prevent the denominator from being zero, and the final abnormal score is Indicates the extent to which the current forecast error deviates from its historical norm.

8. A marine ecological anomaly early warning system based on multimodal perception and spatiotemporal reasoning, which implements the marine ecological anomaly early warning method based on multimodal perception and spatiotemporal reasoning as claimed in claim 1, characterized in that: include: The data acquisition module is configured to acquire raw multimodal data, including image data and time series data; The encoding module is configured to perform global context encoding on the acquired image data, dynamically evolve the time series data, and perform multimodal feature alignment through feature normalization; The alignment module is configured to perform feature fusion on multimodal features based on the dynamic ecological knowledge graph fusion mechanism; The prediction module is configured to predict the fused features based on the spatiotemporal graph changes; The diagnosis module is configured to perform attribution and diagnosis based on the prediction results.

Images