Cyanobacterial bloom monitoring method based on multi-modal data fusion and deep learning

Abstract

The embodiment of the invention is suitable for the field of environment monitoring of computer vision, and provides a cyanobacterial bloom monitoring method based on multi-modal data fusion and deep learning, in the monitoring method, after obtained initial data is preprocessed, variation point detection is introduced to reduce human factor interference, reduce the calculation amount and enhance the model adaptability; according to the method, a convolutional neural network (CNN) is utilized to extract multi-layer features, the features are input into a long short-term memory (LSTM) network for time sequence modeling, the model performance is improved, on the basis, the features of the CNN and the LSTM are fused, and the model accuracy is further improved; finally, variation point detection is introduced again, rich feature information and multi-mode complementarity are utilized, image key information is enhanced, and detection precision and model robustness are improved. The monitoring method provided by the embodiment of the invention breaks through the limitation of spatial-temporal characteristic splitting of a traditional method, significantly improves the prediction continuity and accuracy, and provides high-reliability support for early warning of cyanobacterial bloom.

Description

Technical Field

[0001] The embodiments of the present application belong to the technical field of environmental monitoring in computer vision, and particularly relate to a cyanobacterial bloom monitoring method based on multi-modal data fusion and deep learning. Background Art

[0002] In recent years, the frequency and intensity of cyanobacterial blooms have increased significantly, becoming a major environmental problem threatening the safety of water ecosystems, the quality of drinking water sources, and human health. Therefore, timely and accurate monitoring and identification of cyanobacterial blooms have become an urgent and key technical issue in the current research field for preventing and controlling cyanotoxin poisoning incidents.

[0003] The existing methods for improving the accuracy of cyanobacteria target detection and prediction accuracy can be divided into two categories: deep learning-based image processing methods (DL-IPM) and deep learning methods based on multi-spectral data (DL-MDM); the former mainly relies on RGB image data, and directly extracts features from images and classifies or detects them through technologies such as convolutional neural networks (CNNs) and attention mechanisms; the latter combines multi-spectral data (such as visible light, near-infrared, etc.) and uses deep learning models (such as Transformers) to process multi-band information, which can better capture the spectral characteristics of cyanobacteria.

[0004] Specifically, DL-IPM mainly relies on visible light image data. Therefore, in low visibility environments such as rain, snow, and haze, light scattering and spectral distortion will cause the degradation of imaging quality, affecting the extraction and recognition of cyanobacteria features. In addition, DL-IPM usually processes spatial features and time series independently, and it is difficult to effectively capture the dynamic coupling relationship between the spatial distribution pattern evolution and the time diffusion trend of cyanobacterial blooms, thus limiting the accuracy of the prediction model.

[0005] In related technologies, although DL-MDM utilizes the rich spectral information of multi-spectral data, the data has high dimensions and complex processing, requiring more complex preprocessing and feature extraction methods. At the same time, although multi-spectral data alleviates the impact of low visibility environments to a certain extent, under extreme weather conditions, spectral distortion and scattering may still cause a decline in imaging quality. In addition, DL-MDM has high requirements for data integrity and quality, and data missing or uneven acquisition intervals may lead to a decline in model prediction accuracy and it is difficult to adaptively correct. Summary of the Invention

[0006] The purpose of the embodiments of the present application is to provide a cyanobacterial bloom monitoring method based on multi-modal data fusion and deep learning to at least solve the technical problem of poor model prediction accuracy existing in related cyanobacterial bloom monitoring technologies.

[0007] To achieve the above purpose, the embodiments of the present application provide the following technical solutions.

[0008] According to an embodiment of the present application, a cyanobacterial bloom monitoring method based on multimodal data fusion and deep learning is provided;

[0009] The method includes the following steps:

[0010] Step S1: Use a multispectral camera to collect the reflected light and scattered light in the cyanobacteria distribution area, and combine infrared thermal imaging technology for multispectral fusion to generate image data;

[0011] Step S2: Take the image data as the input of the improved FAST algorithm to obtain candidate mutation points, screen the candidate mutation points based on a dynamic threshold, and remove isolated noise points through DBSCAN clustering, and output the key areas with dense spatial distribution;

[0012] Step S3: Based on the screened key areas, fuse the visible light texture features and the infrared thermal imaging temperature gradient to generate a joint feature map, and screen out the abnormal areas;

[0013] Step S4: Use a convolutional neural network to extract spatial features, and combine a long short-term memory network to model the time series to output temporal features, combine the spatial-temporal attention mechanism to dynamically calculate weights, and weighted-fuse the spatial features and temporal features based on the calculated weights;

[0014] Step S5: Introduce a one-class support vector machine to identify abnormal points in the fused features, and construct a bidirectional spatio-temporal prediction network based on the attention mechanism. In the constructed prediction network, extract the global spatio-temporal dependence relationship through the multi-head self-attention mechanism, decode the features in the forward / backward time sequence, generate the predicted value of the cyanobacteria density distribution, and obtain the prediction result;

[0015] Step S6: Combine the historical data of the cyanobacteria distribution area with the prediction result to generate a spatio-temporal evolution map of the cyanobacteria diffusion trend for visual analysis.

[0016] Further, in step S4, the step of using a convolutional neural network to extract spatial features includes:

[0017] Construct a two-branch convolutional neural network based on the HSV color space for extracting multimodal features of cyanobacteria;

[0018] By constructing a three-level feature pyramid of low-level features, middle-level features, and high-level features, fuse spatial information at different scales; among them, in the low-level feature layer, capture the basic visual information including hue boundaries and local textures in the image; in the middle-level feature layer, extract the algal patch morphology and saturation gradient through 3×3 transposed convolutional upsampling to balance the detail and semantic information; in the high-level feature layer, compress the channels after global average pooling of the middle-level features to generate a compact global feature vector, enhancing the generalization ability of the model to complex scenes;

[0019] After processing the data through a three - level pyramid, the data is fused across levels. The high - level features are upsampled to 64×64, concatenated with the middle - level features and low - level features to form 64×64×864, and then compressed to 64×64×512 data through 1×1 convolution, eliminating redundant information and greatly reducing the computational amount at the same time.

[0020] Further, in the constructed dual - branch convolutional neural network, the hue branch layer uses 5×5 dilated convolution to expand the receptive field, capture the hue boundary features, and outputs a 128×128×32 feature map after passing through a 3×3 pooling layer; the saturation - lightness joint branch layer extracts texture features through 3×3 convolution and introduces an SE attention module to dynamically enhance the key regions, where the weight matrix 、 , and outputs a 128×128×64 feature map; the features of the two branches are concatenated into 128×128×96 and compressed to 128×128×128 through 1×1 convolution to enhance cross - modal interaction.

[0021] Further, in step S4, the step of modeling the time series by combining the long short - term memory network to output the temporal features includes:

[0022] The 64×64×512 spatio - temporal feature map sequence is unfolded along the spatial dimension into 64×64 independent time series and input into a bidirectional LSTM network: the forward LSTM layer processes the sequence in the positive time order to generate a forward hidden state ; the backward LSTM layer processes the sequence in the reverse time order to generate a backward hidden state ;

[0023] Finally, the forward and backward hidden states at each spatial position are concatenated along the channel dimension to output a 64×64×1024 spatio - temporal coupling feature matrix, completely retaining the spatio - temporal evolution law of cyanobacteria diffusion.

[0024] Further, in step S4, the step of weighted - fusing the spatial features and temporal features based on the calculated weights includes:

[0025] The 64×64×512 spatial features output by the CNN are extended to 1024 channels through 1×1 convolution to align with the 64×64×1024 temporal feature dimension of the LSTM;

[0026] The joint features are extracted through a 3×3 convolution with 512 channels and ReLU activation, and then compressed in channels through 1×1 convolution to generate an attention map. After Softmax normalization, the fusion weight α∈[0,1] at each position is obtained, and the weight α is expressed as:

[0027] ;

[0028] Fuse the CNN features and LSTM features with weights α and 1 - α, and the features after weighted fusion are expressed as:

[0029] ;

[0030] In the formula, represents the spatial feature map output by the CNN, with a dimension of 64×64×512; represents the temporal feature map output by the LSTM, with a dimension of 64×64×1024; represents concatenating the CNN and LSTM features along the channel dimension to generate a joint feature map of 64×64×1536; represents a 3×3 convolutional kernel with 512 output channels, used to extract spatio - temporal joint features; represents a 1×1 convolutional kernel with 1 output channel, used to compress features to generate an attention weight map; represents a normalization operation, which maps the weights to the interval [0, 1], ensures that the sum of the weights is equal to 1, α represents the dynamic weight map with a dimension of 64×64×1, and ⊙ represents element - wise multiplication, used to weight the CNN and LSTM features at each position.

[0031] Furthermore, in step S5, the step of introducing a one - class support vector machine to identify outliers in the fused features includes:

[0032] Take the output after the CNN - LSTM spatio - temporal feature fusion as the data of the mutation points;

[0033] Construct an outlier detection algorithm based on a one - class support vector machine. Consider each position in the 64×64 grid as an independent sample, generating a total of 4096 feature vectors, and the dimension of each feature vector is 1024;

[0034] Use principal component analysis (PCA) to reduce the feature vectors to 32 dimensions, and use the decision function for mutation point detection. When f(x)= - 1, it is marked as a mutation point;

[0035] Based on the outlier probability heat map output by the mutation point detection , extract the set of candidate region coordinates that satisfy ;

[0036] Among them, the constructed outlier detection algorithm is expressed as: ;

[0037] The decision function is expressed as: f(x)=sgn(w·Φ(x)-ρ);

[0038] Wherein, (x + 1, y) represents the coordinates of the adjacent pixel on the right side of the current pixel (x, y), (x - 1, y) represents the coordinates of the adjacent pixel on the left side of the current pixel (x, y), w is the weight vector of the hyperplane, Φ(x) is the mapping of the data point x in the high-dimensional space, ρ is a bias term; sgn() represents the sign function.

[0039] Furthermore, in step S5, the global spatio-temporal dependence relationship is extracted through the multi-head self-attention mechanism, expressed as:

[0040] ;

[0041] Wherein, Q, K, and V are the query matrix, key matrix, and value matrix respectively; d k represents the feature dimension;

[0042] The predicted value of the cyanobacteria density distribution generated is expressed as .

[0043] Furthermore, in step S6, the cyanobacteria bloom outbreak threshold is defined and expressed as: , where is the mean value of the historical cyanobacteria density, is the standard deviation; when the predicted value , mark this grid as a high-risk area, output the warning heat map , and mark the risk level.

[0044] Compared with the prior art, the beneficial effects of the cyanobacteria bloom monitoring method based on multi-modal data fusion and deep learning of the present invention are:

[0045] First, through the design of the multi-modal data fusion and deep learning architecture, the present invention achieves a breakthrough improvement in environmental adaptability and feature extraction efficiency; aiming at the spectral distortion problem of traditional methods in low visibility scenarios, through the multi-spectral fusion technology of infrared thermal imaging and visible light data, combined with radiation correction and atmospheric correction algorithms, the scattering interference in extreme weather such as rain, snow, and haze is effectively suppressed; the HSV color space is used to independently process the brightness and chromaticity features, and complementary information fusion is achieved through dynamic weight allocation, significantly improving the effective feature extraction rate and reducing the feature loss rate;

[0046] Second, through the dual mutation point detection mechanism and spatio-temporal feature fusion architecture, the present invention overcomes the dual bottlenecks of efficiency and accuracy of traditional models; introduces improved corner detection and texture analysis algorithms to quickly locate key areas and eliminate noise interference, significantly reducing redundant computational amounts; combines the CNN-LSTM cascade architecture, models the time series dynamics through bidirectional LSTM, and performs attention-weighted fusion with the CNN spatial features to achieve a holographic capture of the spatio-temporal evolution law of cyanobacteria diffusion; this design breaks through the limitation of the spatio-temporal feature fragmentation of traditional methods, significantly improves the prediction coherence and accuracy, and provides high-reliability support for cyanobacteria bloom early warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0048] Figure 1 It is a system architecture diagram of the cyanobacteria bloom monitoring method based on multi-modal data fusion and deep learning of the present invention;

[0049] Figure 2 It is an implementation flowchart of the cyanobacteria bloom monitoring method based on multi-modal data fusion and deep learning of the present invention;

[0050] Figure 3 It is a sub-flowchart of the cyanobacteria bloom monitoring method based on multi-modal data fusion and deep learning of the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] In order to make the objectives, technical solutions and advantages of the present invention clearer, the following further elaborates on the present invention in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0052] The following describes the specific implementation of the present invention in detail in conjunction with specific embodiments.

[0053] Please refer to Figure 1 and Figure 2 , in an embodiment of the present invention, a cyanobacteria bloom monitoring method based on multi-modal data fusion and deep learning is provided. The cyanobacteria bloom monitoring method includes the following steps:

[0054] Step S1: Use a multi-spectral camera to collect the reflected light and scattered light in the cyanobacteria distribution area, and perform multi-spectral fusion in combination with infrared thermal imaging technology to generate image data;

[0055] In step S1 of the present invention, the reflected light and scattered light in the cyanobacteria distribution area are collected by the multispectral cameras carried by satellites and drones, and multispectral fusion is performed in combination with infrared thermal imaging technology to generate high-quality image data; then, through preprocessing steps such as radiometric correction, atmospheric correction, and data enhancement, the image quality and the generalization ability of the model are significantly improved;

[0056] Among them, the present invention uses high-precision multispectral cameras carried on satellite or drone platforms to obtain reflected light and scattered light data in the cyanobacteria distribution area from different angles and time periods, and generates remote sensing data containing image and spectral information and labels . Multispectral fusion is performed in combination with infrared thermal imaging technology to generate a multispectral image that contains both temperature information and retains a clear contour and labels ;

[0057] Furthermore, the obtained initial data is preprocessed. The preprocessing process includes data enhancement, data normalization, data size unification, image correction, and color space conversion to improve the image quality and the generalization ability of the model;

[0058] Specifically, in the data collection step, high-precision multispectral cameras are carried on satellite or drone platforms to obtain reflected light and scattered light data in the cyanobacteria distribution area from different angles and time periods, and generate remote sensing data containing image and spectral information. These data are transmitted to the ground receiving station through signal modulation, and after demodulation and decompression processing, a preliminary optical image is obtained and labels . At the same time, uncooled infrared thermal imagers carried by satellites or drones are used to synchronously collect infrared thermal imaging data in the cyanobacteria bloom area.

[0059] Based on the principle of thermal radiation, this device generates a grayscale image reflecting the temperature distribution by detecting the infrared radiation with a wavelength of 8-14μm emitted from the water surface and labels ;

[0060] Then image fusion is performed. The specific process of image fusion includes:

[0061] First, through SIFT feature point matching, the visible light image and the infrared thermal image are spatially aligned to ensure that the pixel points under the same geographical coordinates correspond one by one. The SIFT feature point matching is expressed as:

[0062]

[0063]

[0064] Among them, G(x,y,σ) is the Gaussian kernel, is the original image, and σ is the scale parameter;

[0065] For the obtained optical image, for each color channel (R, G, B), the gray-level co-occurrence matrix (GLCM) is calculated respectively, and typical texture features are extracted to characterize the morphological contour of cyanobacteria. The process of extracting typical texture features is expressed as:

[0066]

[0067] where, represents the joint probability of gray levels i and j at direction θ and distance d, and the square term (i - j) 2 represents magnifying the gray difference to highlight the drastic change of texture in the cyanobacteria aggregation area;

[0068] For the obtained grayscale image, temperature gradient features are extracted, and the abnormal high-temperature area CTAZ is located through the Otsu algorithm to reflect the surface temperature change caused by cyanobacteria aggregation. The Otsu algorithm is expressed as:

[0069]

[0070] where, represents maximizing the between-class variance, respectively represent the pixel proportions of the foreground (class C0) and background (class C1) after threshold t segmentation, 、 respectively represent the average gray values of the two types of pixels;

[0071] Then, the weights of the two images are assigned through the dynamic weight allocation formula, and the two images are fused together using the pixel-level fusion formula to generate a multispectral image and label , which is expressed as:

[0072]

[0073]

[0074]

[0075] where, represents the spectral response intensity of the visible light sensor at wavelength λ, represents the spectral response intensity of the infrared sensor at wavelength λ; represents the fusion weight of the visible light image, represents the fusion weight of the infrared image, represents the pixel value of the visible light image at coordinates (x, y), represents the pixel value of the infrared thermal imaging image at coordinates (x, y), Represents the pixel value of the visible light image at the coordinates (x, y).

[0076] Furthermore, after obtaining the fused image, perform radiometric correction on the obtained fused image to eliminate the sensor's own noise, environmental interference, and radiation distortion during the imaging process, and restore the true radiation value of the cyanobacteria;

[0077] Among them, the steps of performing radiometric correction on the fused image are expressed as:

[0078]

[0079] Among them, represents the radiance of band λ, DN represents the original value recorded by the sensor, represents the gain coefficient, represents the offset value;

[0080] After radiometric correction, perform atmospheric correction processing. Specifically, use Dark Object Subtraction (DOS) to eliminate the influence of atmospheric scattering and absorption, and invert the true surface reflectance, which is expressed as:

[0081]

[0082] Among them, represents the surface reflectance, represents the top-of-atmosphere radiance, represents the path radiance, represents the solar irradiance, represents the solar zenith angle;

[0083] Furthermore, in the steps of preprocessing the initial data obtained by the present invention:

[0084] Data augmentation: Expand the dataset by means of inversion, rotation, random cropping, etc., to enhance the generalization ability of the model;

[0085] Data standardization: For the obtained RGB images, calculate the mean and standard deviation of the R, G, and B channels, and use the mean-variance method for standardization to effectively avoid the problem of data gradient explosion or disappearance. The standardization process is expressed as:

[0086]

[0087] Among them, is the standardized pixel value, x is the original pixel value, μ is the mean of this channel, and σ is the standard deviation of this channel;

[0088] Unify the data size: Unify the dataset to a size of 256×256 for convenient subsequent operations on the dataset;

[0089] Image correction: The image is divided into multiple regions, and the corresponding gamma value is calculated according to the brightness of each region. The gamma value of each region is calculated for correction to enhance the contrast of image details and reduce the dependence of the model on illumination. The expression for calculating the gamma value of each region for correction is:

[0090]

[0091] Among them, represents the pixel value of the transformed image, represents the pixel value of the original image, and γ represents the calculated gamma value;

[0092] Color space conversion: Convert the RGB image to the HSV color space so that color features and brightness features can be processed independently, facilitating the extraction and analysis of color information under different illumination conditions, thereby enhancing the robustness of the model.

[0093] Therefore, in terms of data acquisition and data processing, the present invention adopts a mode of combining remote sensing with infrared remote sensing and the HSV color space, effectively solving the limitations of the traditional method on light and shooting angle in bad weather such as rain, snow, haze, etc., and improving the robustness and adaptability of the model.

[0094] Please continue to refer to Figure 1 and Figure 2 , the monitoring method provided by the embodiment of the present invention further includes:

[0095] Step S2: Take the image data as the input of the improved FAST algorithm to obtain candidate mutation points, screen the candidate mutation points based on a dynamic threshold, and eliminate isolated noise points through DBSCAN clustering, and output a key region with a dense spatial distribution;

[0096] In step S2, it is the first mutation point detection. Through the mutation point detection technology, the key region can be quickly located and the calculation amount can be reduced, while enhancing the self - adaptability of the model;

[0097] Specifically, in step S2, first adopt the improved FAST corner detection algorithm combined with GLCM to extract local texture features: input the data into the improved FAST algorithm to obtain candidate mutation points V, and then quantify the brightness difference between regions through contrast, characterize the texture complexity with entropy value, and describe the structural regularity with correlation. The texture feature Fi = {Contrasti, Entropyi, Correlationi}, where Contrasti represents the contrast feature, Entropyi represents the entropy value feature, and Correlationi represents the correlation feature;

[0098] Among them:

[0099] The contrast is expressed as:

[0100]

[0101] The entropy value is expressed as:

[0102]

[0103] The correlation is expressed as:

[0104]

[0105] In the formula, represents the joint probability of gray levels i and j at direction θ and distance d; the square term represents amplifying the gray level difference and highlighting the drastic change in the texture of the cyanobacteria aggregation area; respectively represent the standard deviations of the x-axis and y-axis; 、 respectively represent the means of gray levels i and j at direction θ and distance d;

[0106] Subsequently, based on the dynamic threshold τ, V is screened and isolated noise points are removed by DBSCAN clustering, and the key regions with dense spatial distribution are output 。

[0107] Step S3: Based on the screened key regions, fuse the visible light texture features and the infrared thermal imaging temperature gradient to generate a joint feature map, and screen out the abnormal regions;

[0108] Specifically, in step S3 of the present invention, based on the key regions screened for the first time , fuse the visible light texture features and the infrared thermal imaging temperature gradient to generate a joint feature map M(x, y), and screen out the abnormal regions;

[0109] Among them, the infrared thermal imaging temperature gradient is expressed as:

[0110]

[0111] In the formula, represents the gradient of the temperature field in the horizontal direction, which is used to quantify the local temperature change rate; T represents the temperature; (x + 1, y) and (x - 1, y) represent the coordinates of the adjacent pixels on the right and left sides of the current pixel (x, y).

[0112] Step S4: Use a convolutional neural network to extract spatial features, combine a long short-term memory network to model the time series to output temporal features, combine the spatial-temporal attention mechanism to dynamically calculate weights, and weight and fuse the spatial features and temporal features based on the calculated weights;

[0113] In step S4 of the present invention, for the abnormal region obtained in step S3, that is, excluding the part outside the abnormal region, spatial features are extracted based on the abnormal region using a convolutional neural network;

[0114] Step S5: Introduce a one-class support vector machine to identify abnormal points for the fused features, and construct a bidirectional spatio-temporal prediction network based on the attention mechanism. In the constructed prediction network, global spatio-temporal dependence relationships are extracted through the multi-head self-attention mechanism, and features are decoded in the forward / backward time sequence to generate predicted values of the cyanobacteria density distribution, obtaining the prediction result;

[0115] Step S6: Combine the historical data of the cyanobacteria distribution area with the prediction result to generate a spatio-temporal evolution map of the cyanobacteria diffusion trend for visual analysis.

[0116] Please refer to Figure 3 , in an implementation manner of the present invention, the steps of using a convolutional neural network to extract spatial features in step S4 specifically include:

[0117] Step S41: Construct a dual-branch convolutional neural network based on the HSV color space for extracting multi-modal features of cyanobacteria;

[0118] Specifically, when extracting spatial features, the input is an HSV image of 256×256×3, and the parameters are , and the H (hue), S (saturation), and V (value) channels are respectively normalized to [0, 1], where the normalization process is expressed as:

[0119]

[0120] In the formula, represents the hue feature after normalization processing, represents the saturation feature after normalization processing, represents the value feature after normalization processing;

[0121] Furthermore, in the constructed dual-branch convolutional neural network, the hue branch layer (H channel) uses 5×5 dilated convolution (dilation rate = 2) to expand the receptive field and capture hue boundary features. After passing through a 3×3 pooling layer (stride = 2), a 128×128×32 feature map is output; the saturation-value joint branch layer (S+V channel) extracts texture features through 3×3 convolution and introduces an SE attention module to dynamically enhance key regions, where the weight matrices , , R indicates that the element data in the matrix is real, and a 128×128×64 feature map is output; the features of the two branches are concatenated into 128×128×96 and compressed to 128×128×128 through 1×1 convolution to enhance cross-modal interaction;

[0122] Among them, in the step of introducing the SE attention module to dynamically enhance the key area, it is expressed as:

[0123]

[0124] In the formula, W1 and W2 represent weight matrices; S c represents the attention weight map processed by the SE attention module, which is used to dynamically enhance the features of the key area; F SV represents the feature map of the saturation-brightness joint branch layer;

[0125] Step S42: By constructing a three-level feature pyramid of low-level features, middle-level features, and high-level features, fuse spatial information at different scales; among them, in the low-level feature layer, capture the basic visual information including hue boundaries and local textures in the image; in the middle-level feature layer, through 3×3 transposed convolution upsampling, extract the morphology of algal patches and saturation gradients, balancing details and semantic information; in the high-level feature layer, after global average pooling of the middle-level features, compress the channels to generate a compact global feature vector, enhancing the generalization ability of the model to complex scenes;

[0126] Specifically, in the low-level feature layer (128×128×128), capture the basic visual information including hue boundaries and local textures in the image, which is expressed as:

[0127]

[0128] In the formula, represents the weight matrix for extracting low-level features;

[0129] Specifically, in the middle-level feature layer (64×64×256), through 3×3 transposed convolution (stride = 2) upsampling, extract the morphology of algal patches and saturation gradients, which is expressed as:

[0130]

[0131] Specifically, in the high-level feature layer (32×32×512), after global average pooling (GAP) of the middle-level features, compress the channels, which is expressed as: ;

[0132] After processing the data through the three-level pyramid, fuse the data across levels. Upsample the high-level features to 64×64, concatenate them with the middle-level features and low-level features to form 64×64×864, and then compress them to 64×64×512 data through 1×1 convolution, eliminating redundant information and greatly reducing the computational amount at the same time.

[0133] Specifically, the cross-level fusion of the data is expressed as:

[0134]

[0135] In the formula, is the fused feature, is the weight matrix, BN represents batch normalization, which accelerates training and prevents gradient vanishing; ReLU is the activation function, which enhances the non-linear expression ability; F mid is the low-level feature input, is a 3×3 transposed convolution with a stride of 2, which upsamples the resolution from 64×64 to 128×128; represents global average pooling of the intermediate features (64×64×256), and the output is 1×1×256; represents a 1×1 convolution kernel, which expands the number of channels from 256 to 512 to enhance the feature expression ability; represents upsampling the high-level features 32×32×512 to 64×64×512; Concat represents concatenating three groups of features along the channel dimension.

[0136] Further, in step S4, the step of modeling the time series by combining the long short-term memory network to output the temporal features includes: expanding the 64×64×512 spatio-temporal feature map sequence along the spatial dimension into 64×64 independent time series, and inputting them into the bidirectional LSTM network: the forward LSTM layer processes the sequence in the positive time order to generate the forward hidden state ; the backward LSTM layer processes the sequence in the reverse time order to generate the backward hidden state ;

[0137] Finally, the forward and backward hidden states at each spatial position are concatenated along the channel dimension to output a spatio-temporal coupling feature matrix of 64×64×1024, completely retaining the spatio-temporal evolution law of the cyanobacteria diffusion.

[0138] Further, in step S4, the step of weighted fusion of the spatial feature and the temporal feature based on the calculated weight includes:

[0139] Expanding the 64×64×512 spatial feature output by the CNN to 1024 channels through a 1×1 convolution to align the dimensions with the 64×64×1024 temporal features of the LSTM;

[0140] Combining the spatio-temporal attention mechanism to dynamically calculate the weight : Extracting the joint features through a 3×3 convolution with 512 channels and ReLU activation, then compressing the channels through a 1×1 convolution to generate the attention map, and obtaining the fusion weight α∈[0,1] at each position after Softmax normalization. The weight α is expressed as:

[0141]

[0142] Fuse the CNN features and LSTM features with weights α and 1 - α, and the fused features are expressed as:

[0143] ; In the formula, represents the spatial feature map output by the CNN, with a dimension of 64×64×512; represents the temporal feature map output by the LSTM, with a dimension of 64×64×1024; represents concatenating the CNN and LSTM features along the channel dimension to generate a joint feature map of 64×64×1536; represents a 3×3 convolution kernel with 512 output channels, used to extract spatio - temporal joint features; represents a 1×1 convolution kernel with 1 output channel, which compresses the features to generate an attention weight map; Softmax represents a normalization operation that maps the weights to the [0,1] interval to ensure that the sum of the weights is equal to 1, α represents a dynamic weight map with a dimension of 64×64×1, and ⊙ represents element - wise multiplication, which is used to weight the CNN and LSTM features at each position.

[0144] Furthermore, in step S5, the step of introducing a one - class support vector machine to identify outliers in the fused features includes:

[0145] Take the output after the CNN - LSTM spatio - temporal feature fusion as the data of the mutation points, integrating the spatial form and temporal dynamics of the multi - modal data to form a joint feature map containing spatio - temporal coupling information; in addition, the fused features also retain the temperature gradient information of the infrared thermal imaging and the texture features of the visible light image, providing a data basis for multi - modal complementary analysis;

[0146] Construct an outlier detection algorithm based on the one - class support vector machine (One - Class SVM). Treat each position in the 64×64 grid as an independent sample, generating a total of 4096 feature vectors, and the dimension of each feature vector is 1024;

[0147] Use principal component analysis PCA to reduce the feature vectors to 32 dimensions, and use the decision function for mutation point detection. When f(x)= - 1, it is marked as a mutation point;

[0148] Based on the outlier probability heat map output by the mutation point detection , extract the set of candidate region coordinates that satisfy . The role of the candidate region coordinate set C is to locate high - risk regions, further analyze and process them, and better understand the diffusion trend;

[0149] Among them, the constructed anomaly detection algorithm is expressed as:

[0150] The decision function is expressed as: ;

[0151] In the formula, represents the coordinates of the candidate region; represents the gradient of the temperature field in the horizontal direction, T represents temperature; (x + 1, y) represents the coordinates of the adjacent pixel on the right of the current pixel (x, y), (x - 1, y) represents the coordinates of the adjacent pixel on the left of the current pixel (x, y), w is the weight vector of the hyperplane, is the mapping of the data point x in the high-dimensional space, ρ is a bias term, and sgn() represents the sign function.

[0152] In steps S2 and S4 of the present invention, mutation point detection is adopted, and a double mutation point detection mechanism is used, which not only reduces the amount of calculation and the calculation cost, but also reduces the interference of non-natural factors, removes the mutation points, and improves the accuracy of the model.

[0153] Further, in step S5, based on the spatio-temporal coupling feature matrix , a bidirectional spatio-temporal prediction network based on the attention mechanism is constructed; the spatio-temporal feature sequences of 30 consecutive days are input in chronological order , and a cascaded Transformer-LSTM architecture is used for prediction;

[0154] Further, in step S5, the global spatio-temporal dependence relationship is extracted through the multi-head self-attention mechanism, which is expressed as:

[0155] ;

[0156] In the formula, Q, K, and V are the query matrix, key matrix, and value matrix respectively; represents the feature dimension; T represents the transpose of the matrix;

[0157] The predicted value of the cyanobacteria density distribution generated is expressed as .

[0158] In step S6, based on the historical data statistics and ecological carrying capacity assessment, the cyanobacteria bloom outbreak threshold is defined and expressed as: , where is the historical cyanobacteria density mean value, is the standard deviation; when the predicted value , mark this grid as a high-risk area, output the warning heat map , and mark the risk level.

[0159] Furthermore, a comparison mechanism is introduced into the prediction module in step S5 of the embodiment of the present invention, and the images obtained through the up and down time differences are continuously corrected for accurate prediction, reducing the data requirements of the traditional prediction method.

[0160] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and embodiments. It can be fully applied to various fields suitable for the present invention. For those skilled in the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrated examples herein.

Claims

1. A method for monitoring cyanobacterial blooms based on multimodal data fusion and deep learning, characterized in that It includes the following steps: Step S1: Use a multispectral camera to collect the reflected light and scattered light in the cyanobacteria distribution area, and perform multispectral fusion in combination with infrared thermal imaging technology to generate image data; Step S2: Use the image data as the input of the improved FAST algorithm to obtain candidate mutation points, screen the candidate mutation points based on a dynamic threshold, and remove isolated noise points through DBSCAN clustering, and output the key areas with dense spatial distribution; Step S3: Based on the screened key areas, fuse the visible light texture features and the infrared thermal imaging temperature gradient to generate a joint feature map, and screen out the abnormal areas; Step S4: Use a convolutional neural network to extract spatial features, and combine a long short-term memory network to model the time series to output temporal features, combine the spatial-temporal attention mechanism to dynamically calculate weights, and weighted fuse the spatial features and temporal features based on the calculated weights; Step S5: Introduce a one-class support vector machine to identify abnormal points in the fused features, and construct a bidirectional spatio-temporal prediction network based on the attention mechanism. In the constructed prediction network, extract the global spatio-temporal dependence relationship through the multi-head self-attention mechanism, decode the features in the forward / backward time sequence, generate the predicted value of the cyanobacteria density distribution, and obtain the prediction result; Step S6: Combine the historical data of the cyanobacteria distribution area with the prediction result to generate a spatio-temporal evolution map of the cyanobacteria diffusion trend for visual analysis.

2. The cyanobacteria bloom monitoring method based on multi-modal data fusion and deep learning according to claim 1, characterized in that, In step S4, the steps of using a convolutional neural network to extract spatial features include: Construct a dual-branch convolutional neural network based on the HSV color space for extracting multi-modal features of cyanobacteria; By constructing a three-level feature pyramid of low-level features, middle-level features, and high-level features, fuse spatial information at different scales; In the low-level feature layer, capture the basic visual information including hue boundaries and local textures in the image; In the middle-level feature layer, through 3×3 transposed convolution upsampling, extract the algal spot morphology and saturation gradient, and balance the detail and semantic information; In the high-level feature layer, after global average pooling of the middle-level features, compress the channels to generate a global feature vector; After processing the data through the three-level pyramid, fuse the data across levels, upsample the high-level features to 64×64, splice them with the middle-level features and low-level features to form 64×64×864, and then compress them to 64×64×512 through 1×1 convolution.

3. The cyanobacteria bloom monitoring method based on multi-modal data fusion and deep learning according to claim 2, characterized in that, In the constructed dual-branch convolutional neural network, the hue branch layer uses 5×5 dilated convolution to expand the receptive field, capture the hue boundary features, and output a 128×128×32 feature map after passing through a 3×3 pooling layer; the saturation-value joint branch layer extracts texture features through 3×3 convolution, and introduces an SE attention module to dynamically enhance the key areas, expressed as: ； Among them, W1 and W2 represent weight matrices; S c represents the attention weight map processed by the SE attention module, which is used to dynamically enhance the features of key regions; represents the feature map of the saturation-lightness joint branch layer; the weight matrix , , output a feature map of 128×128×64, and R represents that the element data in the matrix is a real number; Splice the features of the two branches into 128×128×96, and compress them to 128×128×128 through 1×1 convolution.

4. The method for monitoring cyanobacterial blooms based on multi-modal data fusion and deep learning according to claim 3, characterized in that, In step S4, the steps of combining a long short-term memory network to model the time series to output temporal features include: The 64×64×512 spatio-temporal feature map sequence is unfolded along the spatial dimension into 64×64 independent time series and input into a bidirectional LSTM network: the forward LSTM layer processes the sequence in the forward order of time to generate a forward hidden state ; the backward LSTM layer processes the sequence in the reverse order of time to generate a backward hidden state ; Splice the forward and backward hidden states at each spatial position along the channel dimension to output a 64×64×1024 spatio-temporal coupling feature matrix.

5. The method for monitoring cyanobacterial blooms based on multi-modal data fusion and deep learning according to claim 4, characterized in that, In step S4, the steps of weighted fusing the spatial features and temporal features based on the calculated weights include: The 64×64×512 spatial features output by the CNN are extended to 1024 channels through 1×1 convolution, aligning with the 64×64×1024 temporal feature dimension of the LSTM; The joint features are extracted through 3×3 convolution with 512 channels and ReLU activation, and then compressed through 1×1 convolution to generate an attention map. After Softmax normalization, the fusion weight α ∈ [0,1] at each position is obtained. The weight α is expressed as: ； Fuse the CNN features and LSTM features by weighting with weights α and 1 - α, and the features after weighted fusion are expressed as: ； where, F cnn represents the spatial feature map output by the CNN, with a dimension of 64×64×512; represents the temporal feature map output by the LSTM, with a dimension of 64×64×1024; represents concatenating the CNN and LSTM features along the channel dimension to generate a joint feature map of 64×64×1536; represents a 3×3 convolutional kernel with 512 output channels, used to extract spatio-temporal joint features; represents a 1×1 convolutional kernel with 1 output channel, compressing the features to generate an attention weight map; Softmax represents a normalization operation that maps the weights to the [0,1] interval to ensure that the sum of the weights equals 1, α represents the dynamic weight map with a dimension of 64×64×1, and ⊙ represents element-wise multiplication, used to weight the CNN and LSTM features at each position.

6. The cyanobacteria bloom monitoring method based on multi-modal data fusion and deep learning according to claim 5, wherein In step S5, the step of introducing a one-class support vector machine to identify outliers in the fusion features includes: The output after fusing the spatio-temporal features of CNN-LSTM Data serving as mutation points; Constructing an outlier detection algorithm based on the one-class support vector machine, regarding each position of the 64×64 grid as an independent sample, a total of 4096 feature vectors are generated, and the dimension of each feature vector is 1024; Using principal component analysis PCA to reduce the feature vectors to 32 dimensions, and using the decision function for anomaly point detection. When f(x) = -1, it is marked as an anomaly point; Abnormal probability heat map based on the output of mutation point detection , extract the candidate region coordinate set that satisfies ; ; Among them, the constructed anomaly detection algorithm is expressed as: ; The decision function is expressed as: f(x) = sgn(w·Φ(x) - ρ); In the formula, represents the coordinates of the candidate region; represents the gradient of the temperature field in the horizontal direction, T represents temperature; (x + 1, y) represents the coordinates of the adjacent pixel on the right side of the current pixel (x, y), (x - 1, y) represents the coordinates of the adjacent pixel on the left side of the current pixel (x, y), w is the weight vector of the hyperplane, Φ(x) is the mapping of the data point x in the high-dimensional space, and ρ is a bias term; sgn() represents the sign function.

7. The monitoring method of cyanobacterial bloom based on multi-modal data fusion and deep learning according to claim 6, characterized in that In step S5, the global spatio-temporal dependence is extracted through the multi-head self-attention mechanism, expressed as: ； Where Q is the query matrix, K is the key matrix, and V is the value matrix; d k represents the feature dimension; T represents the transpose of the matrix; The predicted value of the generated cyanobacteria density distribution, expressed as .

8. The method for monitoring cyanobacterial blooms based on multi-modal data fusion and deep learning according to claim 7, characterized in that, In step S6, a threshold for the outbreak of cyanobacterial blooms is defined , which is expressed as: ； Among them, is the mean value of historical cyanobacteria density, is the standard deviation; when the predicted value , mark this grid as a high-risk area and output a warning heat map , and mark the risk level.

Images