Water body environment abnormal change detection method based on multi-source data space-time fusion network

Abstract

The invention relates to the technical field of water body detection, in particular to a water body environment abnormal change detection method based on a multi-source data space-time fusion network. Comprising the steps of obtaining an optical image and a radar image of a to-be-detected water body; preprocessing the optical image and the radar image; extracting data features of the optical image and the radar image, and performing data alignment fusion on the extracted data features of the optical image and the radar image; and inputting the data features of the aligned and fused optical image and radar image into the 3D convolution-LSTM mixed structure to obtain a water body environment abnormal change detection result. The optical image and the radar image are comprehensively utilized, the optical image is rich in spectral information, the radar image can acquire data all day long and all weather, and the defect of a single data source is overcome through combination of the optical image and the radar image. And the data complementarity reduces the monitoring blind area, so that the water body change monitoring is more comprehensive.

Description

Technical Field

[0001] The present invention relates to the technical field of water body detection, and in particular to a method for detecting abnormal changes in water body environment based on a multi-source data spatiotemporal fusion network. Background Art

[0002] Accurate and timely monitoring of abnormal changes in the water environment is crucial in water resource management and ecological protection. However, existing water change detection technologies still have many shortcomings. Traditional detection methods often rely on single-source image data. Although optical images can present rich spectral information, they are restricted by cloud cover and the nighttime environment. It is difficult to obtain clear and effective water information during cloudy or nighttime periods, resulting in monitoring blind spots. Synthetic aperture radar (SAR) imagery, while offering the advantage of all-day, all-weather observation, is susceptible to interference from factors such as terrain shadows and surface roughness, resulting in extremely high false detection rates and seriously compromising the accuracy of water detection. In the spatiotemporal analysis dimension, existing methods often treat the spatial and temporal dimensions in isolation, failing to systematically grasp the spatiotemporal variations of water bodies. It is difficult to capture the gradual changes in time and the continuous evolution of water bodies in space. As a result, detection results fail to accurately reflect the true dynamic changes in water bodies and cannot provide comprehensive and accurate decision-making support for water resource management. The current mainstream water body detection models focus on the current status of water bodies and lack the ability to predict future trends in water body changes. Against the backdrop of climate change and intensified human activities, water body changes are uncertain and dynamic. If the direction of water body changes cannot be predicted in a forward-looking manner, it will seriously restrict the timeliness of decisions such as water resource scheduling and disaster warning. Summary of the Invention

[0003] The present invention discloses a method for detecting abnormal changes in water environment based on a multi-source data spatiotemporal fusion network. The specific method is as follows:

[0004] Obtain optical images and radar images of the water body to be detected;

[0005] Preprocessing of optical and radar images;

[0006] Extracting data features of optical images and radar images, and performing data alignment and fusion on the extracted data features of optical images and radar images;

[0007] The data features of the aligned and fused optical image and radar image are input into the 3D convolution-LSTM hybrid structure to obtain the detection results of abnormal changes in the water environment.

[0008] Furthermore, the optical image and radar image of the water body to be detected are obtained as follows:

[0009] Using satellite remote sensing technology, periodic multispectral images of the water body to be detected are collected as optical images at fixed time intervals;

[0010] With the help of synthetic aperture radar equipment, the SAR image data of the same water body to be detected is obtained as the radar image when the deviation from the optical image acquisition time does not exceed the preset time.

[0011] Furthermore, the optical image and radar image are preprocessed as follows:

[0012] Correct the optical image, the correction formula is:

[0013]

[0014] Among them, L λ is the radiation value of the top of the atmosphere, d is the distance between the sun and the earth, ESUN λ is the solar irradiance, θ s is the solar incidence angle;

[0015] The radar image is corrected, and the correction formula is:

[0016]

[0017] Where DN is the digital value, A is the absolute calibration coefficient, θ inc is the angle of incidence;

[0018] For the spatiotemporal alignment of the corrected optical image and radar image, the high-resolution SAR image is first subjected to a bilinear downsampling algorithm to reduce its resolution to the same level as the optical image. The phase correlation method is then used to calculate the image offsets Δx and Δy in the x and y directions, achieving pixel-level alignment of the optical image and the SAR image. The formula is:

[0019]

[0020] Among them, F(I1) and F(I2) are the frequency domain information of SAR image and optical image after Fourier transform, respectively;

[0021] For optical images and radar images with missing dates, cubic spline interpolation is used to complete them, and the preprocessed optical image sequence is recorded as The radar image sequence is recorded as

[0022] Furthermore, the optical image data features are extracted. The specific method is as follows:

[0023] Preprocessed optical image sequence As input, Perform global average pooling operation to obtain the feature F after global average pooling global_avg ;

[0024] The feature F after global average pooling global_avgThe input is sent to the multi-layer perceptron for nonlinear transformation, and then processed by the Softmax function to obtain the weight W of each spectral band. spectral , the calculation formula is:

[0025] W spectral =Softmax(MLP(F global_avg ))

[0026] Use SwinBlock in Swin Transformer Perform feature extraction to obtain preliminary feature representation

[0027] The initial features and the weight W of the spectral band spectral Perform element-wise multiplication (⊙) to obtain the features enhanced by spectral attention:

[0028]

[0029] Furthermore, the features of radar image data are extracted. The specific method is as follows:

[0030] Radar image sequence As input, parallel hole convolution groups are used to output fusion features for radar image sequences at different scales. Perform feature extraction and suppress noise;

[0031] Construct an adaptive weight matrix and modulate it through the Sigmoid function to achieve multi-scale anti-noise feature fusion. The specific formula is as follows:

[0032]

[0033] Among them, DConv(X sar ,k,r) is the input radar image sequence The dilated convolution operation is performed, k is the kth dilated convolution layer in the parallel dilated convolution group, r is the dilation rate of the dilated convolution, W k is the corresponding weight matrix.

[0034] Furthermore, the extracted data features of the optical image and radar image are aligned and fused. The specific method is as follows:

[0035] Extracted optical image data features and extracted radar image data features Perform alignment;

[0036] The aligned features are fused and then normalized by layers. The specific formula is:

[0037]

[0038] Furthermore, the 3D convolution-LSTM hybrid structure has the following specific data processing methods:

[0039] Use the 3D convolution layer to slide the window along the time dimension to extract spatiotemporal features. The formula is:

[0040]

[0041] Then connect ConvLSTM to model the temporal dependency relationship, the formula is:

[0042]

[0043] Among them, H t and C t are the hidden state and memory state at the current moment, H t-1 and C t-1 is the corresponding state at the previous moment;

[0044] Based on the feature representation obtained by spatiotemporal modeling, the output change probability map is output through convolution operation and Sigmoid activation:

[0045] M prob =Sigmoid(Conv2D(H T ))

[0046] After binarization, the final map M is generated. change And the water changes at the next Δ moment are calculated through a multi-layer perceptron. The formula is:

[0047]

[0048] Among them, Δ is the prediction step size, τ is the time window size, and MLP is a multi-layer perceptron.

[0049] Furthermore, the 3D convolution-LSTM hybrid structure is trained, and the objective function of its end-to-end training is:

[0050]

[0051] Where N is the number of observation samples, λ1, λ2 and λ3 are the hyperparameters of the multi-source data spatiotemporal fusion network model;

[0052] The first term of the objective function is the change map loss term, is the cross entropy loss function, the specific formula is:

[0053]

[0054] in, Is a binary variable, which is the real water body change label of the i-th sample. Indicates that the water body in the area corresponding to the sample has changed. Indicates no change. is the probability of water body change predicted by the model for the i-th sample, and its value range is between [0,1];

[0055] The second term of the objective function is the prediction result loss term, is the mean square error loss function;

[0056] The third term of the objective function is the regularization term, and the calculation formula is:

[0057]

[0058] where w j are the weights in the neural network.

[0059] Furthermore, the parameters of the 3D convolution-LSTM hybrid structure are updated using the following formula:

[0060]

[0061] Where η is the learning rate, which is the parameter value at the tth iteration. is the gradient of the objective function with respect to the parameter θ

[0062] Furthermore, the abnormal change detection result of the water environment includes a water change map M change and water status prediction results Calculate the water area change rate ΔA using the following formula:

[0063]

[0064] A risk threshold T is set in advance, and an early warning is triggered when the calculated water area change rate exceeds the threshold T.

[0065] Due to the adoption of the above technical solution, the present invention has the following beneficial effects:

[0066] 1. This invention utilizes both optical and radar images. Optical images are rich in spectral information, while radar images can acquire data around the clock and in all weather conditions. This combination overcomes the shortcomings of a single data source. The complementary nature of the data reduces monitoring blind spots and enables more comprehensive monitoring of water changes.

[0067] 2. This invention preprocesses multi-source data using a system that includes radiometric correction, spatiotemporal alignment, and time series interpolation. Radiometric correction ensures the data quality of optical and SAR images, ensuring uniform physical dimensions and accurate radiometric information. Spatiotemporal alignment achieves precise pixel-level matching of the two images, providing a foundation for subsequent feature fusion. Time series interpolation ensures data continuity and integrity, providing reliable data support for subsequent analysis.

[0068] 3. The optical branch utilizes multiple techniques to extract spectral and texture features, enhancing the representation of water features in optical imagery. The radar branch utilizes parallel dilated convolution groups and adaptive weight modulation to achieve multi-scale noise-resistant feature fusion, improving the ability to extract water features from radar imagery. Multi-source data alignment and spatiotemporal modeling further integrate the features of both imagery types, capturing the evolution of water bodies in both time and space, providing deeper information for water change detection.

[0069] 4. Based on the water body change map and prediction results output by the model, the water body area change rate is calculated and the early warning threshold is set, which can timely detect abnormal changes in water bodies, provide effective decision-making support for water resources management, and improve the efficiency and scientific nature of water resources management.

[0070] 5. This invention is highly adaptable to optical and radar images of varying sources and characteristics, and can process image data of varying resolutions and wavelengths. Its network model structure is highly scalable and can be adjusted and optimized based on actual needs to accommodate diverse water change detection tasks and application scenarios.

[0071] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings of the present invention are described below.

[0073] Figure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0074] The present invention will be further described below with reference to the accompanying drawings and examples.

[0075] A method for detecting abnormal changes in water environment based on multi-source data spatiotemporal fusion network, such as Figure 1 The specific steps are as follows:

[0076] S1. Obtain optical images and radar images of the water body to be detected.

[0077] Step S1 includes step S11 optical image acquisition and step S12 radar image acquisition, which are specifically as follows:

[0078] S11. Use satellite remote sensing technology to collect periodic multispectral images of the monitoring area at fixed time intervals based on the needs of water body change monitoring.

[0079] S12. Using synthetic aperture radar (SAR) equipment, obtain SAR image data of the area within a time deviation of no more than 12 hours from the time of optical image acquisition.

[0080] S2. Preprocess the optical image and radar image. The specific steps are as follows:

[0081] S21: Perform radiation correction on the multi-source data to convert the optical image data into apparent reflectance data with a unified physical dimension, providing standardized input for subsequent processing. Calculate its apparent reflectance TOA using the formula:

[0082]

[0083] Among them, L λ is the radiation value of the top of the atmosphere, d is the distance between the sun and the earth, ESUN λ is the solar irradiance, θ s is the solar incidence angle.

[0084] SAR images are calibrated for radiation and corrected for terrain to eliminate radiation differences in radar images caused by sensor characteristics and terrain undulations. The formula is:

[0085]

[0086] Where DN is the digital value, A is the absolute calibration coefficient, θ inc is the angle of incidence;

[0087] S22: Perform spatiotemporal alignment. First, the high-resolution SAR image is subjected to a bilinear downsampling algorithm to reduce its resolution to the same level as the optical image. Then, the phase correlation method is used to calculate the image offsets Δx and Δy in the x and y directions. Pixel-level alignment is achieved between the optical image and the SAR image to ensure the accuracy of subsequent feature fusion. The formula is:

[0088]

[0089] Among them, F(I1) and F(I2) are the frequency domain information of SAR image and optical image after Fourier transform, respectively.

[0090] S23: Perform time series interpolation. For data with missing dates, use cubic spline interpolation to fill in the gaps, ensuring the continuity and integrity of the entire time series data.

[0091] S24: After the preprocessing operation, the preprocessed optical image sequence is recorded as The radar image sequence is recorded as

[0092] S3. Extract data features of the optical image and the radar image, and perform data alignment and fusion on the extracted data features of the optical image and the radar image.

[0093] S31, extracting optical image features, specifically:

[0094] The aligned optical image sequence obtained after preprocessing As input, Perform global average pooling to extract global statistical information of each spectral band and obtain the feature F after global average pooling globall_avg Then, the global average pooled feature F globall_avg The input is sent to the multi-layer perceptron MLP for nonlinear transformation. The output of the MLP is then processed by the Softmax function to obtain the weight W of each spectral band. spectral , the calculation formula is:

[0095] W spectral =Softmax(MLP(F globall_avg ))

[0096] Use SwinBlock in Swin Transformer to input optical images Perform feature extraction to obtain preliminary feature representation Then this preliminary feature representation is combined with the band weight W calculated previously spectral Perform element-by-element multiplication (⊙) to achieve weighted features of different bands, thereby obtaining features enhanced by spectral attention:

[0097]

[0098] Optical image input Extract the Histogram of Oriented Gradients (HOG) feature H t , the features obtained above after spectral attention enhancement And the extracted HOG feature H t Perform splicing to obtain the spliced features The concatenated features contain both spectral and texture features. The convolution operation is performed on the concatenated features to obtain the final optical features:

[0099]

[0100] S32. Extract radar image features, specifically:

[0101] Preprocessed SAR image sequence As input. Using parallel dilated convolution groups to output fused features, we extract features from SAR images at different scales and suppress noise, fully capturing the rich information in the image. By setting different dilation rates, the convolution kernel can capture features of objects at different scales. At the same time, we construct an adaptive weight matrix and modulate it using a Sigmoid function to enhance the expression of features at different scales, achieving multi-scale noise-resistant feature fusion. The formula is:

[0102]

[0103] Among them, DConv(X sar ,k,r) is the input SAR image The dilated convolution operation is performed, k is the kth dilated convolution layer in the parallel dilated convolution group, r is the dilation rate of the dilated convolution, W k is the corresponding weight matrix.

[0104] S33, Multi-source data alignment

[0105] The cross-data attention (CMA) mechanism is used to align the features extracted from the optical branch and the radar branch. The aligned features are fused and then layer normalization (LayerNorm) is performed to make the fused features more stable and accelerate model convergence. The formula is:

[0106]

[0107] S4. Input the data features of the aligned and fused optical image and radar image into the 3D convolution-LSTM hybrid structure to obtain the detection results of abnormal changes in the water environment.

[0108] S41. Use 3D convolution-LSTM hybrid structure to integrate features in time and space dimensions. Modeling is performed to capture the evolution of water bodies in time and space.

[0109] Use the 3D convolution layer to slide the window along the time dimension to extract spatiotemporal features. The formula is:

[0110]

[0111] Then connect ConvLSTM to model temporal dependencies to capture the evolution of water bodies in the temporal and spatial dimensions. The formula is:

[0112]

[0113] Among them, H t and C t are the hidden state and memory state at the current moment, H t-1 and C t-1 The corresponding state at the previous moment.

[0114] S42. Generate change maps and prediction results to visually display the change areas and extent of water bodies during the monitoring period.

[0115] Based on the feature representation obtained by spatiotemporal modeling, the output change probability map is output through convolution operation and Sigmoid activation:

[0116] M prob =Sigmoid(Conv2D(H T ))

[0117] After binarization, the final map M is generated. change And the water changes at the next Δ time are calculated through a multi-layer perceptron. The formula is:

[0118]

[0119] Among them, Δ is the prediction step size, τ is the time window size, and MLP is a multi-layer perceptron.

[0120] S43. The pre-processed optical image and radar image sequence data pairs are combined with the manually labeled true values of water body changes to construct a training data set.

[0121] S44. Construct an objective function for end-to-end training;

[0122]

[0123] Where N is the number of observation samples, λ1, λ2, and λ3 are the hyperparameters of the multi-source data spatiotemporal fusion network model.

[0124] The first term of the objective function is the change map loss term, is the cross entropy loss function, specifically

[0125]

[0126] in, Is a binary variable, which is the real water body change label of the i-th sample. Indicates that the water body in the area corresponding to the sample has changed. Indicates no change. is the probability of water body change predicted by the model for the i-th sample, and its value range is between [0,1];

[0127] The second term of the objective function is the prediction result loss term, is the mean square error loss function;

[0128] The third term of the objective function is a regularization term, which is used to constrain the parameters of the model to prevent the parameter values from being too large and to avoid overfitting of the model. The calculation formula is:

[0129]

[0130] where w j are the weights in the neural network.

[0131] S45, parameter update of multi-source data spatiotemporal fusion network model, the formula is

[0132]

[0133] Where η is the learning rate, which is the parameter value at the tth iteration. is the gradient of the objective function with respect to the parameter θ.

[0134] S46, extract new multi-source data observation samples and output water body change map M change and water status prediction results Assist water resources management decision-making.

[0135] Jointly output water body change map M change and water status prediction results Calculate the water area change rate ΔA, the calculation formula is

[0136]

[0137] S47. Pre-set a risk threshold T, and trigger an early warning when the calculated water area change rate exceeds the threshold T.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for detecting abnormal changes in water environment based on a multi-source data spatiotemporal fusion network, characterized in that: The specific method is as follows: Obtain optical images and radar images of the water body to be detected; Preprocessing of optical and radar images; Extracting data features of optical images and radar images, and performing data alignment and fusion on the extracted data features of optical images and radar images; The data features of the aligned and fused optical image and radar image are input into the 3D convolution-LSTM hybrid structure to obtain the detection results of abnormal changes in the water environment.

2. The method for detecting abnormal changes in water environment based on a multi-source data spatiotemporal fusion network according to claim 1, characterized in that: Obtain the optical image and radar image of the water body to be detected. The specific method is as follows: Using satellite remote sensing technology, periodic multispectral images of the water body to be detected are collected as optical images at fixed time intervals; With the help of synthetic aperture radar equipment, the SAR image data of the same water body to be detected is obtained as the radar image when the deviation from the optical image acquisition time does not exceed the preset time.

3. The method for detecting abnormal changes in water environment based on a multi-source data spatiotemporal fusion network according to claim 1, characterized in that: Preprocess the optical and radar images. The specific methods are as follows: Correct the optical image, the correction formula is: Among them, L λ is the radiation value of the top of the atmosphere, d is the distance between the sun and the earth, ESUN λ is the solar irradiance, θ s is the solar incidence angle; The radar image is corrected, and the correction formula is: Where DN is the digital value, A is the absolute calibration coefficient, θ inc is the angle of incidence; For the spatiotemporal alignment of the corrected optical image and radar image, the high-resolution SAR image is first subjected to a bilinear downsampling algorithm to reduce its resolution to the same level as the optical image. The phase correlation method is then used to calculate the image offsets Δx and Δy in the x and y directions, achieving pixel-level alignment of the optical image and the SAR image. The formula is: Among them, F(I1) and F(I2) are the frequency domain information of SAR image and optical image after Fourier transform, respectively; For optical images and radar images with missing dates, cubic spline interpolation is used to complete them, and the preprocessed optical image sequence is recorded as The radar image sequence is recorded as 4. The method for detecting abnormal changes in water environment based on a multi-source data spatiotemporal fusion network according to claim 3, characterized in that: Extract optical image data features. The specific method is as follows: Preprocessed optical image sequence As input, Perform global average pooling operation to obtain the feature F after global average pooling global_avg ; The feature F after global average pooling global_avg The input is sent to the multi-layer perceptron for nonlinear transformation, and then processed by the Softmax function to obtain the weight W of each spectral band. spectral , the calculation formula is: W spectral =Softmax(MLP(F global_avg )) Use SwinBlock in Swin Transformer Perform feature extraction to obtain preliminary feature representation The initial features and the weight W of the spectral band spectral Perform element-wise multiplication (⊙) to obtain the features enhanced by spectral attention:

5. The method for detecting abnormal changes in water environment based on a multi-source data spatiotemporal fusion network according to claim 4, characterized in that: Extract radar image data features. The specific method is as follows: Radar image sequence As input, parallel hole convolution groups are used to output fusion features for radar image sequences at different scales. Perform feature extraction and suppress noise; Construct an adaptive weight matrix and modulate it through the Sigmoid function to achieve multi-scale anti-noise feature fusion. The specific formula is as follows: Among them, DConv(X sar ,k,r) is the input radar image sequence The dilated convolution operation is performed, k is the kth dilated convolution layer in the parallel dilated convolution group, r is the dilation rate of the dilated convolution, W k is the corresponding weight matrix.

6. The method for detecting abnormal changes in water environment based on a multi-source data spatiotemporal fusion network according to claim 5, characterized in that: The data features of the extracted optical image and radar image are aligned and fused. The specific method is as follows: Extracted optical image data features and extracted radar image data features Perform alignment; The aligned features are fused and then normalized by layers. The specific formula is:

7. The method for detecting abnormal changes in water environment based on a multi-source data spatiotemporal fusion network according to claim 6, characterized in that: 3D convolution-LSTM hybrid structure, the specific data processing method is as follows: Use the 3D convolution layer to slide the window along the time dimension to extract spatiotemporal features. The formula is: Then connect ConvLSTM to model the temporal dependency relationship, the formula is: Among them, H t and C t are the hidden state and memory state at the current moment, H t-1 and C t-1 is the corresponding state at the previous moment; Based on the feature representation obtained by spatiotemporal modeling, the output change probability map is output through convolution operation and Sigmoid activation: M prob =Sigmoid(Conv2D(H T )) After binarization, the final map M is generated. change And the water changes at the next Δ moment are calculated through a multi-layer perceptron. The formula is: Among them, Δ is the prediction step size, τ is the time window size, and MLP is a multi-layer perceptron.

8. The method for detecting abnormal changes in water environment based on a multi-source data spatiotemporal fusion network according to claim 6, characterized in that: Training the 3D convolution-LSTM hybrid structure, the objective function of its end-to-end training is: Where N is the number of observation samples, λ1, λ2 and λ3 are the hyperparameters of the multi-source data spatiotemporal fusion network model; The first term of the objective function is the change map loss term, is the cross entropy loss function, the specific formula is: in, Is a binary variable, which is the real water body change label of the i-th sample. Indicates that the water body in the area corresponding to the sample has changed. Indicates no change. is the probability of water body change predicted by the model for the i-th sample, and its value range is between [0,1]; The second term of the objective function is the prediction result loss term, is the mean square error loss function; The third term of the objective function is the regularization term, and the calculation formula is: where w j are the weights in the neural network.

9. The method for detecting abnormal changes in water environment based on a multi-source data spatiotemporal fusion network according to claim 6, characterized in that: The parameter update formula of 3D convolution-LSTM hybrid structure is: Where η is the learning rate, which is the parameter value at the tth iteration. is the gradient of the objective function with respect to the parameter θ.

10. The method for detecting abnormal changes in water environment based on a multi-source data spatiotemporal fusion network according to claim 6, characterized in that: The detection results of abnormal changes in water environment include water body change map M change and water status prediction results Calculate the water area change rate ΔA using the following formula: A risk threshold T is set in advance, and an early warning is triggered when the calculated water area change rate exceeds the threshold T.

Images