Deep learning method for monitoring river and lake water area shoreline based on multi-layer confidence features

Abstract

The application discloses a deep learning river and lake water area shoreline remote sensing monitoring method based on multilayer confidence features, relates to the field of river and lake water area shoreline monitoring, and comprises the following steps: firstly, acquiring a remote sensing image dataset of a river and lake water area shoreline; then, performing image processing based on Opencv to generate a multi-source heterogeneous remote sensing sample set of the river and lake water area shoreline; then, performing segmentation on sample data in the multi-source heterogeneous remote sensing sample set based on a multilayer confidence convolutional neural network; again, constructing a deep learning model based on multilayer confidence features; then, completing deep learning model training; again, monitoring the river and lake water area shoreline through the deep learning model, and analyzing and comparing the monitoring result with a historically acquired river and lake shoreline monitoring result; finally, based on the analysis and comparison result, describing 'four disorder' point information through data statistical analysis and data visualization means; the application can effectively distinguish 'four disorder' targets and interference substances, and significantly improves the efficiency and accuracy of target interpretation.

Description

Technical Field

[0001] This invention relates to the field of river and lake shoreline monitoring, specifically to a deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-layer confidence features. Background Technology

[0002] The statements in this section are provided only as background information in connection with this disclosure and may not constitute prior art.

[0003] With socio-economic development, the planning and utilization of river and lake shorelines have been continuously improving. However, problems such as illegal occupation, illegal construction, illegal dumping, and illegal mining have also emerged. The effective protection and rational utilization of river shorelines have a significant impact on regional ecological civilization construction and socio-economic development. Therefore, effective monitoring of river and lake shorelines is of great importance.

[0004] Currently, remote sensing image recognition relies primarily on manual labor, and the volume of remote sensing data is typically large, making it difficult to promptly identify "four types of disorderly" remote sensing targets. Therefore, there is an urgent need to improve monitoring methods and develop detection algorithm models using new technologies such as artificial intelligence and big data. Thus, utilizing deep learning algorithms with real-time detection capabilities for suspected targets to improve the accuracy and speed of remote sensing image recognition is a key issue that needs to be addressed in river and lake shoreline monitoring. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the prior art by providing a deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-level confidence features. This method utilizes advanced technologies such as deep learning to process high-resolution satellite remote sensing images and low-altitude UAV remote sensing images. Through OpenCV-based river and lake shoreline remote sensing image processing technology, multi-source heterogeneous image data segmentation technology based on multi-level confidence convolutional neural networks, and deep learning model training based on multi-level confidence features, the accuracy and speed of remote sensing image recognition are improved. This further enhances the technical level of the integrated air-space-ground monitoring system for rivers and lakes, combining satellite remote sensing, UAVs, and manual ground inspections.

[0006] The technical solution of the present invention is as follows:

[0007] Deep learning-based remote sensing methods for river and lake shoreline monitoring based on multi-level confidence features include:

[0008] Step S1: Obtain a remote sensing image dataset of river and lake shorelines; the remote sensing image dataset includes: historical remote sensing images, high-resolution satellite remote sensing images, and low-to-medium altitude remote sensing images from UAVs;

[0009] Step S2: Based on OpenCV, perform image processing on the images in the remote sensing image dataset to generate a multi-source heterogeneous remote sensing sample set of river and lake shorelines;

[0010] Step S3: Segment the sample data in the multi-source heterogeneous remote sensing sample set based on a multi-layer belief convolutional neural network;

[0011] Step S4: Construct a deep learning model based on multi-layer confidence features;

[0012] Step S5: Based on the segmented sample data, complete the deep learning model training;

[0013] Step S6: Monitor the river and lake shoreline using a deep learning model, and analyze and compare the monitoring results with historical river and lake shoreline monitoring results.

[0014] Step S7: Based on the analysis and comparison results, describe the information on the "four disorderly" points through data statistical analysis and data visualization.

[0015] Further, step S2 includes:

[0016] Step S21: Image noise reduction; Denoising the image is performed using Gaussian filtering and median filtering in OpenCV;

[0017] Step S22: Image dehazing; The dark channel prior algorithm reconstructs the image by calculating the dark channel image and using the estimated atmospheric light value and transmittance;

[0018] Step S23: Image correction; Using the camera calibration and correction functions in OpenCV, estimate the camera's intrinsic and extrinsic parameters, and correct the image;

[0019] Step S24: Image fusion; The images are fused using the weighted fusion method and hybrid fusion method provided by OpenCV to obtain a multi-source heterogeneous remote sensing sample set of river and lake shorelines.

[0020] Furthermore, the multi-layer confidence convolutional neural network is a U2net network model based on multi-layer confidence;

[0021] The U2net network model is an improvement on existing convolutional neural networks, residual neural networks, fusion encoder-decoder structures, spatial pyramid pooling modules, multi-layer confidence feature modules, attention mechanism modules, and skip connection modules. It integrates Unet sub-modules and a U-shaped structure, and further improves segmentation performance by adding more skip connections and multi-scale feature fusion. By combining gradient attention modules and inverted attention modules, it can better capture edge information and extract subtle features. Furthermore, by combining the Canny operator for river and lake shorelines, it can recover important detail information and accurately segment the land and water boundaries of remote sensing images of rivers and lakes.

[0022] Further, step S4 includes:

[0023] Step S41: Organize the spectral data;

[0024] Step S42: Spectral feature preprocessing; perform radiometric correction, atmospheric correction, and noise removal on the spectral data to eliminate the influence of environmental factors on the spectral data; and use Rasterio to standardize the spectral data to ensure that the spectral values ​​of different bands are comparable;

[0025] Step S43: Design the spectral feature input branch; transform the spectral features into feature vectors that match the input size of the YOLOv5s network;

[0026] Step S44: Design a feature fusion strategy; perform feature fusion in the P3 layer of the feature pyramid of the YOLOv5s network, using the SEA attention mechanism to enable the model to adaptively adjust the weights of spectral features and visual features.

[0027] Further, step S41 includes:

[0028] Organize the collected remote sensing image dataset containing spectral information to ensure that each satellite image has corresponding spectral feature data and that there is a one-to-one correspondence between the two.

[0029] Further, step S42 includes:

[0030] Standard Scaling is used to standardize the spectral data, as shown in the following formula:

[0031]

[0032] Where X represents the original spectral data. For the standardized data, μ is the mean of the original spectral data, and σ is the standard deviation of the original spectral data.

[0033] Further, step S43 includes:

[0034] If the spectral data is hyperspectral, PCA is used to reduce the spectral features, transforming them into low-dimensional vectors while retaining the main components as input to the fully connected layer. A fully connected layer is added before the YOLOv5s network as the spectral feature input branch. The input dimension of this fully connected layer is the length of the spectral feature vector, and the output dimension can be set according to the network structure adjustment needs.

[0035] Furthermore, the fully connected layer is represented as follows:

[0036] h = W·x + b

[0037] Where h is the output of the fully connected layer, W is the weight matrix, x is the input vector, and b is the bias vector.

[0038] Furthermore, the dimensionality reduction strategy of the PCA is as follows:

[0039]

[0040]

[0041] Z = X·W

[0042] Where X is the original spectral data matrix, Let W be the sample covariance matrix. i Let be the direction vector of the i-th principal component, W be the matrix composed of the direction vectors of the principal components, and Z be the data matrix after dimensionality reduction.

[0043] Furthermore, the SEA attention mechanism is represented as follows:

[0044] s=σ(W2δ(W1z))

[0045] y=s⊙z

[0046] Where z is the input feature vector, W1 and W2 are learnable weight matrices, δ is the activation function, σ is the activation function, ⊙ is element-wise multiplication, s is the attention vector, and y is the weighted feature vector.

[0047] Compared with existing technologies, the advantages of this invention are:

[0048] 1. A deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-level confidence features was developed. An encoding and decoding model based on attention mechanism and multi-level confidence features was established, which can effectively distinguish between "four types of disorder" targets and their interference, and significantly improve the efficiency and accuracy of target interpretation.

[0049] 2. A deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-layer confidence features enables automated interpretation of satellite and UAV remote sensing data. This method can serve as an important supplement to the existing "sky-ground integrated" river and lake shoreline monitoring platform, further supporting precise decision-making in river and lake shoreline management.

[0050] 3. A deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-level confidence features was developed. A neural network model based on multi-level confidence convolution was designed. The multi-level confidence convolutional neural network model based on the encoding and decoding structure, combined with the Canny operator for river and lake shorelines, can better recover important details in the image and more accurately segment the river and lake water-land boundaries in remote sensing images. This effectively distinguishes "four types of disorder" targets and their interference, significantly improving the efficiency and accuracy of target interpretation.

[0051] 4. A deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-layer confidence features, enabling automated interpretation of satellite and UAV remote sensing data.

[0052] 5. A deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-layer confidence features. This method uses an anchor-based detection approach and the YOLOv5 model to directly predict the center point and aspect ratio of the target. This approach can reduce the number of anchor boxes and improve the speed and accuracy of target image detection. Attached Figure Description

[0053] Figure 1 This is an architecture diagram of a deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-level confidence features.

[0054] Figure 2 This is a flowchart illustrating step S2;

[0055] Figure 3 This is a flowchart illustrating step S3;

[0056] Figure 4 This is a schematic diagram of a U2net network model based on multi-level belief.

[0057] Figure 5 This is a deep learning model based on multi-level confidence. Detailed Implementation

[0058] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0059] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0060] Example 1

[0061] Please see Figure 1 A deep learning-based remote sensing monitoring method for river and lake shorelines, based on multi-level confidence features, includes the following steps:

[0062] Step S1: Obtain a remote sensing image dataset of river and lake shorelines; the remote sensing image dataset includes: historical remote sensing images, high-resolution satellite remote sensing images, and low-to-medium altitude remote sensing images from UAVs; that is, by collecting high-resolution satellite remote sensing images and low-to-medium altitude remote sensing images of river and lake shorelines through satellites and UAVs, and combining them with historical remote sensing images to form a remote sensing image dataset; in this embodiment, the proprietary satellite remote sensing data of the Shiting River shoreline in Shifang City, Sichuan Province, and high-resolution remote sensing data from satellites such as Gaofen-1 and Jilin-1 are obtained through the "Mining University Nanhu" SAR remote sensing satellite, the "Hongkou Fuxing" optical remote sensing satellite, and the "China Electronics Agricultural Innovation" thermal infrared remote sensing satellite. Vector data adopts the "Shoreline Protection and Utilization Plan" for 12 important provincial rivers and lakes compiled by the Sichuan Provincial Department of Water Resources. High-resolution satellite remote sensing images and low-to-medium altitude remote sensing images of the Shiting River shoreline are collected and combined with historical remote sensing images to form a remote sensing image dataset;

[0063] Step S2: Based on OpenCV, perform image processing on the images in the remote sensing image dataset to generate a multi-source heterogeneous remote sensing sample set of river and lake shorelines; that is, generate a multi-source heterogeneous remote sensing sample set of river and lake shorelines through remote sensing image processing such as image denoising, image dehazing, geometric correction, image fusion, and image enhancement.

[0064] Step S3: Segment the sample data in the multi-source heterogeneous remote sensing sample set based on the multi-level confidence U2net model; that is, to understand and analyze the image by identifying different regions in the image.

[0065] Step S4: Construct a deep learning model based on multi-layer confidence features;

[0066] Step S5: Train the deep learning model constructed in the previous steps based on the segmented sample data;

[0067] Step S6: Monitor the shoreline of rivers and lakes using a deep learning model (process the satellite high-resolution remote sensing images and UAV low- and medium-altitude remote sensing images of the key river and lake areas to be monitored, and use the trained deep learning model to process the data to obtain the monitoring results of the key river and lake shorelines for analysis and evaluation), and analyze and compare the monitoring results with the historical river and lake shoreline monitoring results, and display the remote sensing monitoring map of the river and lake shoreline.

[0068] Step S7: Based on the analysis and comparison results, describe the location, type, scale, and scope of impact of the "four disorder" points through data statistical analysis and data visualization, and implement hazard assessment and remedial measures.

[0069] In this embodiment, for details, please refer to... Figure 2 Step S2 includes:

[0070] Step S21: Image noise reduction; The image is denoised using Gaussian filtering and median filtering in OpenCV; Gaussian filtering reduces noise by calculating the weighted average of the neighboring pixels around each pixel; Median filtering removes noise by replacing the value of each pixel with the median of its neighboring pixels.

[0071] Step S22: Image dehazing; The dark channel prior algorithm calculates the dark channel image and uses the estimated atmospheric light value and transmittance to restore the image, effectively reducing the impact of haze.

[0072] Step S23: Image correction; Using the camera calibration and correction functions in OpenCV, estimate the camera's intrinsic and extrinsic parameters, and correct the image;

[0073] Step S24: Image fusion; The images are fused using the weighted fusion method and hybrid fusion method provided by OpenCV, thereby further enhancing the pixel information and spatial information of the remote sensing images and obtaining a multi-source heterogeneous remote sensing sample set of river and lake shorelines.

[0074] In this embodiment, for details, please refer to... Figure 3 and Figure 4 The multi-layer confidence convolutional neural network is a U2net network model based on multi-layer confidence. The U2net network model can further enhance the learning ability of the neural network and improve the accuracy and reliability of the model.

[0075] The U2net network model is an improvement upon existing convolutional neural networks, residual neural networks, fusion encoder-decoder structures, spatial pyramid pooling modules, multi-layer confidence feature modules, attention mechanism modules, and skip connection modules. It integrates Unet sub-modules and a U-shaped structure, further improving segmentation performance by adding more skip connections and multi-scale feature fusion. The U2net model enhances the network's attention to important features, specifically by combining gradient attention modules and inverted attention modules, which can better capture edge information and extract subtle features such as details and textures. Furthermore, it can combine the Canny operator for river and lake shorelines to recover important detail information and accurately segment the land and water boundaries of remote sensing images.

[0076] In this embodiment, specifically, step S4 involves optimizing the network structure of the YOLOv5s model. Since spectral information in satellite imagery can reveal the material composition and properties of objects compared to spectral information in traditional images, it provides a deeper level of identification beyond the surface color of traditional images, enhancing the accuracy and anti-interference capabilities of target recognition and classification. The original CSP1_x feature extraction method is changed to a spectral information-based feature extraction method, replacing the BackBone network in the YOLOv5s model. A fully connected layer is constructed to integrate the spectral features into the YOLOv5s neck network structure and is used during the fusion training phase.

[0077] In this embodiment, specifically step S4, please refer to [link / reference needed]. Figure 5 ,include:

[0078] Step S41: Organize the spectral data;

[0079] Step S42: Spectral feature preprocessing; radiometric correction, atmospheric correction, and noise removal are performed on the spectral data to eliminate the influence of environmental factors on the spectral data and improve data quality; and Rasterio is used to standardize the spectral data to ensure that the spectral values ​​of different bands are comparable.

[0080] Step S43: Design the spectral feature input branch; transform the spectral features into feature vectors that match the input size of the YOLOv5s network;

[0081] Step S44: Design a feature fusion strategy; perform feature fusion at layer P3 of the Feature Pyramid Network (FPN) of the YOLOv5s network, and use the Squeeze-and-Excitation Attention (SEA) mechanism to enable the model to adaptively adjust the weights of spectral features and visual features.

[0082] In this embodiment, specifically, step S41 includes:

[0083] Organize the collected remote sensing image dataset containing spectral information to ensure that each satellite image has corresponding spectral feature data and that there is a one-to-one correspondence between the two.

[0084] In this embodiment, specifically, step S42 includes:

[0085] Standard Scaling is used to standardize the spectral data, as shown in the following formula:

[0086]

[0087] Where X represents the original spectral data. For the standardized data, μ is the mean of the original spectral data, and σ is the standard deviation of the original spectral data.

[0088] In this embodiment, specifically, step S43 includes:

[0089] If the spectral data is hyperspectral, PCA is used to reduce the spectral features, transforming them into low-dimensional vectors while retaining the main components as input to the fully connected layer. A fully connected layer is added before the YOLOv5s network as the spectral feature input branch. The input dimension of this fully connected layer is the length of the spectral feature vector, and the output dimension can be set according to the network structure adjustment needs.

[0090] In this embodiment, the fully connected layer is specifically represented as follows:

[0091] h = W·x + b

[0092] Where h is the output of the fully connected layer, W is the weight matrix, x is the input vector, and b is the bias vector.

[0093] In this embodiment, the specific dimensionality reduction strategy of PCA is as follows:

[0094]

[0095]

[0096] Z = X·W

[0097] Where X is the original spectral data matrix, Let W be the sample covariance matrix. i Let W be the direction vector of the i-th principal component, W be the matrix composed of the direction vectors of the principal components, and Z be the data matrix after dimensionality reduction.

[0098] It should be noted that using PCA dimensionality reduction strategy can reduce redundant information, extract the main variation components of spectral data, reduce the burden on the model, improve computational efficiency, and may also enhance the model's ability to identify key spectral features.

[0099] In this embodiment, the SEA attention mechanism is specifically represented as follows:

[0100] s=σ(W2δ(W1z))

[0101] y=s⊙z

[0102] Where z is the input feature vector, W1 and W2 are learnable weight matrices, δ is the activation function, σ is the activation function, ⊙ is element-wise multiplication, s is the attention vector, and y is the weighted feature vector.

[0103] That is, it can adaptively adjust the weights of visual features and spectral features, highlighting features that contribute significantly to object detection and improving the model's discrimination ability in complex scenes, especially when spectral features are highly discriminative for specific target types.

[0104] In this embodiment, specifically, the training includes:

[0105] Based on the collected multi-source heterogeneous remote sensing sample set, the data is input into the deep learning model constructed in the previous step for training by combining the Channel Attention mechanism; and the model parameters are updated and optimized by using a multi-scale deep convolutional neural network training method based on the stochastic gradient descent algorithm and the backpropagation algorithm.

[0106] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

[0107] This background section is provided to generally present the context of the invention. The work of the currently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither expressly nor impliedly acknowledged as prior art to the invention.

Claims

1. A deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-level confidence features, characterized in that... include: Step S1: Obtain a dataset of remote sensing images of river and lake shorelines; The remote sensing image dataset includes: historical remote sensing images, high-resolution satellite remote sensing images, and low- and medium-altitude remote sensing images from UAVs; Step S2: Based on OpenCV, perform image processing on the images in the remote sensing image dataset to generate a multi-source heterogeneous remote sensing sample set of river and lake shorelines; Step S3: Segment the sample data in the multi-source heterogeneous remote sensing sample set based on a multi-layer belief convolutional neural network; Step S4: Construct a deep learning model based on multi-layer confidence features; Step S5: Based on the segmented sample data, complete the deep learning model training; Step S6: Monitor the river and lake shoreline using a deep learning model, and analyze and compare the monitoring results with historical river and lake shoreline monitoring results. Step S7: Based on the analysis and comparison results, describe the information on the "four disorderly" points through data statistical analysis and data visualization. The multi-layer confidence convolutional neural network is a U2net network model based on multi-layer confidence; The U2net network model is an improvement on existing convolutional neural networks, residual neural networks, fusion encoder-decoder structures, spatial pyramid pooling modules, multi-layer confidence feature modules, attention mechanism modules, and skip connection modules. It integrates Unet sub-modules and a U-shaped structure, and further improves segmentation performance by adding more skip connections and multi-scale feature fusion. By combining gradient attention modules and inverted attention modules, it can better capture edge information and extract subtle features. Furthermore, by combining the Canny operator for river and lake shorelines, it can recover important detail information and accurately segment the land and water boundaries of remote sensing images of rivers and lakes. Step S4 includes: Step S41: Organize the spectral data; Step S42: Spectral feature preprocessing; perform radiometric correction, atmospheric correction, and noise removal on the spectral data to eliminate the influence of environmental factors on the spectral data; and use Rasterio to standardize the spectral data to ensure that the spectral values ​​of different bands are comparable; Step S43: Design the spectral feature input branch; change the original CSP1_x feature extraction method to a feature extraction method based on spectral information, replace the BackBone network in the YOLOv5s model, construct a fully connected layer to access the spectral features into the YOLOv5s neck network structure, and use it in the fusion training stage; If the spectral data is hyperspectral, PCA is used to reduce the spectral features, transforming it into a low-dimensional vector that retains the principal components as input to the fully connected layer; a fully connected layer is added before the YOLOv5s network as the spectral feature input branch, and the input dimension of this fully connected layer is the length of the spectral feature vector; Step S44: Design a feature fusion strategy; perform feature fusion in the P3 layer of the feature pyramid of the YOLOv5s network, and use the SEA attention mechanism to perform fusion so that the model can adaptively adjust the weights of spectral features and visual features. The SEA attention mechanism is represented as follows: in, For the input feature vector, and The weight matrix is ​​a learnable matrix. Let be the excitation function. For activation function, For element-wise multiplication, For attention vectors, This is the weighted eigenvector.

2. The deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-layer confidence features according to claim 1, characterized in that, Step S2 includes: Step S21: Image noise reduction; Denoising the image is performed using Gaussian filtering and median filtering in OpenCV; Step S22: Image dehazing; The dark channel prior algorithm reconstructs the image by calculating the dark channel image and using the estimated atmospheric light value and transmittance; Step S23: Image correction; Using the camera calibration and correction functions in OpenCV, estimate the camera's intrinsic and extrinsic parameters, and correct the image; Step S24: Image fusion; The images are fused using the weighted fusion method and hybrid fusion method provided by OpenCV to obtain a multi-source heterogeneous remote sensing sample set of river and lake shorelines.

3. The deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-layer confidence features according to claim 1, characterized in that, Step S41 includes: Organize the collected remote sensing image dataset containing spectral information to ensure that each satellite image has corresponding spectral feature data and that there is a one-to-one correspondence between the two.

4. The deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-layer confidence features according to claim 1, characterized in that, Step S42 includes: Standard Scaling is used to standardize the spectral data, as shown in the following formula: in, For the raw spectral data, For standardized data, The mean of the original spectral data. This represents the standard deviation of the original spectral data.

5. The deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-layer confidence features according to claim 4, characterized in that, The fully connected layer is represented as follows: in, For the output of a fully connected layer, This is the weight matrix. For the input vector, This is the bias vector.

6. The deep learning-based remote sensing monitoring method for river and lake shorelines based on multi-layer confidence features according to claim 1, characterized in that, The dimensionality reduction strategy of PCA is as follows: in, The original spectral data matrix, The sample covariance matrix, For the first Principal component direction vectors It is a matrix composed of the direction vectors of the principal components. This is the data matrix after dimensionality reduction.

Images