Shallow water multispectral remote sensing depth detection method based on multi-resolution scale fusion

Abstract

The present invention discloses a shallow water multispectral remote sensing water depth detection method based on multi-resolution scale fusion, comprising the following steps: cropping and resampling the input pre-processed multispectral remote sensing image through a multi-scale hierarchical sampler to obtain a multi-resolution hierarchical multispectral remote sensing image group corresponding to different spatial scales; an encoder module group corresponding to multiple specific spatial scale resolutions is used to extract features at different spatial scales, and the spectral feature aggregation from a rough large spatial scale to a fine small spatial scale is realized through a subsequent center alignment fusion module; extracting the features of the finest high-resolution branches in the four encoder stages in the backbone model and mapping them after splicing along the channel dimension to obtain the output water depth result; training the model and evaluating its performance by using the measured water depth data as the true value. This method can achieve high-precision water depth detection and has a certain extrapolation capability, and has practical application value in the field of seabed topography detection and mapping.

Description

Technical Field

[0001] The present invention relates to the fields of satellite water depth inversion, ocean mapping, ocean topography and geomorphology detection and deep learning, and in particular to a shallow water multi-spectral remote sensing water depth detection method based on multi-resolution scale fusion. Background Art

[0002] The acquisition of water depth data in coastal areas, especially in shallow sea areas of islands and reefs, involves important fields such as marine engineering construction, marine environmental resource development and management, and sea lane safety. The current mainstream means is to obtain data through ship-borne acoustic measurement equipment or airborne laser radar equipment, but it has great limitations, such as sea conditions in the measurement area, control point layout, and high costs. Especially in remote areas or areas that cannot be reached due to objective reasons, seeking a non-contact, large-area, efficient water depth data acquisition method is an urgent problem that needs to be solved.

[0003] In recent years, the rapid development of optical remote sensing observation technology has led to a rapid improvement in its imaging quality, such as spatial resolution and other technical parameters. In particular, the improvement of multispectral remote sensing technology has greatly promoted the implementation of ground remote sensing observation data in urban spatial planning, land and resources assessment and protection. In shallow sea areas with clear water quality, multispectral sensors can receive the combined reflectivity reflected by the seabed, sea surface and atmosphere, which contains water depth information. Based on this, multispectral satellite water depth inversion technology has been proposed and has made great progress in the past period of time. Its accuracy has improved with the deployment of advanced regression models. For example, it has gradually evolved from early classic but relatively simple semi-empirical algorithms (such as band combination models, logarithmic band ratio models) to models with better fitting capabilities. With the development of machine learning theory and technology, the scheme of directly constructing the mapping function between remote sensing reflectivity and water depth represented by empirical models has significantly improved the accuracy of predicting water depth. In the spatial dimension, spatial features have a certain pattern as a representation of the spatial distribution of underwater terrain. Based on this, spatial feature satellites are used for water depth inversion tasks, and significant accuracy improvements have been achieved. However, the existing technology still has some shortcomings, especially it is difficult to achieve a balance between the breadth of spatial features and model efficiency, resulting in shortcomings in the accuracy and extrapolation generalization capabilities of existing satellite water depth detection models. Summary of the invention

[0004] In view of this, the present invention proposes a shallow water multispectral remote sensing water depth detection method based on multi-resolution scale fusion. The method has a reasonable design and solves the defect of limited spatial receptive field of traditional multispectral remote sensing water depth inversion model to a certain extent. It greatly improves the accuracy and extrapolation generalization ability of satellite water depth detection model while ensuring efficiency.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A shallow water multispectral remote sensing depth detection method based on multi-resolution scale fusion, comprising:

[0007] Step 1: crop and resample the input pre-processed multispectral remote sensing image through a multi-scale hierarchical sampler to obtain a multi-resolution hierarchical multispectral remote sensing image group corresponding to different spatial scales;

[0008] Step 2: construct an encoder module group consisting of a multi-branch convolutional neural network, where each branch corresponds to a specific spatial scale resolution, and integrates feature maps of different spatial resolutions through a subsequent center alignment fusion module. The two parts are combined into a single encoder stage;

[0009] Step 3: a total of 4 encoder stages are set up, and the serial layout is used to achieve the extraction of concrete to abstract features and cross-resolution fusion;

[0010] Step 4: extract the clearest high-resolution branch output features of the four encoder stages and concatenate them along the channel dimension and map them to obtain the predicted water depth result;

[0011] Step 5: Train or fine-tune and evaluate the model using the measured water depth data as the true value.

[0012] Preferably, in step 1, the principle of the multi-scale hierarchical sampler and the method for processing the input multispectral image to obtain a multi-resolution scale hierarchical image combination are as follows:

[0013] Step 1.1: Select a multispectral remote sensing image of the target area under appropriate imaging conditions to reduce the impact of factors such as clouds and human activities on the imaging quality and even the results; if it is not possible to obtain a sufficiently ideal remote sensing image, the median filter can be used to fuse multiple images within a certain time series range. The principle is shown in formula (1):

[0014] (1);

[0015] Here, R represents the reflectivity of a certain band, i and j correspond to the spatial position index of the pixel, and N is the number of images in the time series interval. Here, the remote sensing image is further processed and the missing values ​​are filled in by the bilinear interpolation algorithm.

[0016] Step 1.2: The multi-scale hierarchical sampler first needs to determine the center coordinate i of the corresponding cropping range C With j C , by pre-setting the resampling resolution RES R The corresponding expansion ratio ER (RES) is calculated from the input resolution RES R / RES), and the expanded cropping window size EW (ER×W) is determined by the preset window size parameter W; then, the multispectral image block with a size of EW is cropped around the center coordinate, and a non-overlapping sliding window mean filter is used in the cropped multispectral image fragment to obtain a resolution of RES R By setting multiple resampling resolutions RES R A hierarchical image combination can be obtained, which includes coarse large-scale low-resolution remote sensing images to fine small-scale high-resolution images, where a single image has its corresponding spatial resolution and scale range to cover the spatial feature expression at different scales. Here, since the centers of each cropped image in the multi-resolution image group are the same, the low-resolution branch covers the adjacent high-resolution branch content.

[0017] Preferably, in step 2, multiple convolutional neural networks corresponding to specific spatial scale resolutions form a backbone multi-branch encoder, and each encoder module group is followed by a corresponding center alignment fusion module to achieve the connection of feature information of different resolution scales, and the two constitute a single encoder stage, and the method is as follows:

[0018] Step 2.1: For each resampling branch, configure a suitable convolutional neural network to construct its corresponding encoder module. Each encoder module branch has its corresponding input spatial scale, and there is no weight parameter sharing between different branches. This ensures that each branch can extract underwater terrain features at a specific spatial resolution scale.

[0019] Step 2.2: Each encoder module independently processes the input multispectral remote sensing image of a specific spatial scale. After each group of encoder modules, a corresponding fusion module is configured to realize the transmission of relevant feature information between feature maps of different spatial resolutions, namely, the center alignment fusion module. The fusion process is from the flow direction of the feature from the low-resolution branch (large scale) to the high-resolution branch (small scale). That is, the center alignment fusion module first crops the central part of the low-resolution branch feature map. Let the resolution of the low-resolution branch at this time be RES L The high-resolution branch is RES H , then the size range after clipping is the original RES H / RES L .

[0020] Step 2.3: For the cropped feature map, restore its size to the original size through bilinear interpolation, and its resolution becomes RES with upsampling H, thus completing the alignment of the two feature maps. The resampled features and the high-resolution branch feature maps are spliced ​​along the channel dimension to achieve feature merging, and the subsequent convolutional neural network module further completes feature fusion and updates the output features of the high-resolution branch.

[0021] Step 2.4: The above process proceeds in the direction of resolution refinement to achieve the alignment and fusion of feature maps at different resolution levels, and complete the update of feature maps at other resolution scales except the lowest resolution branch in the image group.

[0022] Preferably, in step 3, the specific method is to form an encoder body consisting of 4 encoder stages consisting of an encoder module group and a center alignment fusion module, and the serial layout is used to realize the extraction and fusion of specific features and abstract features.

[0023] Preferably, in step 4, the method by which the multi-stage feature splicing module splices features of all stages in the encoder and maps them to water depth values ​​is as follows:

[0024] Step 4.1: The backbone multi-branch encoder contains 4 stages, and the center alignment fusion module in each stage aligns and fuses the extracted feature maps across resolution scales and updates the content. The multi-stage feature concatenation module here concatenates the finest high-resolution branch output features in the 4 encoder stages along the channel dimension, and implements feature fusion in the channel dimension through a convolutional neural network module.

[0025] Step 4.2: After feature fusion is completed, the feature dimension is mapped to 1 through two consecutive 1×1 convolution kernels, and to ensure that the output water depth is not negative, a ReLU activation function is added to the end, as shown in formula (2):

[0026] (2);

[0027] Here x is the value of the input ReLU after mapping, and y is the output predicted water depth value.

[0028] Preferably, in step 5, the model is trained or fine-tuned using the existing water depth measurement values ​​by relying on the mask mean square error (MaskMSE), which evaluates the quality of the model output water depth under the premise of sparse distribution of the true value and serves as a supervised training indicator. The calculation method is shown in formula (3):

[0029] (3);

[0030] Where K is the number of available sample points, x is the predicted value, y is the true value, and m is a Boolean matrix where true refers to pixels with real water depth values ​​and false refers to pixels without water depth values.

[0031] Beneficial technical effects brought by the present invention:

[0032] The present invention discloses a shallow water multispectral remote sensing water depth detection method based on multi-resolution scale fusion. Compared with the prior art, the present invention realizes the spectral spatial feature fusion from a rough large spatial scale to a fine small spatial scale by multiple cropping and sampling of multispectral remote sensing images, which solves the problem of limited spatial receptive field of the current traditional multispectral remote sensing water depth inversion model to a certain extent, and significantly improves the accuracy and extrapolation generalization ability of the satellite water depth detection model under the premise of ensuring overall efficiency. The invention can play an important role in the fields of island and reef coastal marine engineering construction, coastal resource management and development, submarine topography and landform detection, and marine surveying and mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is the process of the shallow water multispectral remote sensing depth detection method based on multi-resolution scale fusion in the present invention.

[0034] Figure 2 This is the structure and operating principle of the multi-scale hierarchical sampler in the present invention.

[0035] Figure 3 This is the shallow water multi-spectral remote sensing water depth detection model structure based on multi-resolution scale fusion in the present invention.

[0036] Figure 4 This is the center alignment fusion module structure in the present invention. DETAILED DESCRIPTION

[0037] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0038] Here, a satellite multispectral remote sensing image data is used as the input data source, and the spatial resolution of all channels is sampled to 10 m. The specific process is shown in the attached Figure 1 As shown, it includes: step 1, cropping and resampling the input pre-processed multispectral remote sensing image through a multi-scale hierarchical sampler to obtain a multi-resolution hierarchical multispectral remote sensing image group corresponding to different spatial scales; step 2, constructing an encoder module group composed of a multi-branch convolutional neural network, each branch corresponds to a specific spatial scale resolution, and integrating different spatial resolution feature maps through a subsequent center alignment fusion module, and the two parts are combined into a single encoder stage; step 3, a total of 4 encoder stages are set, and the serial layout is used to realize the extraction of specific to abstract features and cross-resolution fusion; step 4, extracting the clearest high-resolution branch output features of the 4 encoder stages and mapping them along the channel dimension to obtain the predicted water depth result; step 5, training or fine-tuning the model and evaluating it using the measured water depth data as the true value.

[0039] Step 1: The input preprocessed multispectral remote sensing image is cropped and resampled through a multi-scale hierarchical sampler to obtain a multi-resolution hierarchical multispectral remote sensing image group corresponding to different spatial scales.

[0040] Step 1.1: First, select the appropriate multispectral remote sensing image in the target area. It can be manually screened to determine the scene with the best imaging quality as the input data source of the detection model. For some scenes, it is difficult to obtain ideal multispectral remote sensing images due to cloud interference. At this time, better observation data quality can be obtained by suppressing noise through median filtering within a certain period of time. The principle is shown in formula (1):

[0041] (1);

[0042] Here R represents the reflectivity of a certain band, i and j correspond to the spatial position index of the pixel, and N is the number of images in the time series interval. Here, the remote sensing image is further processed and the missing values ​​are filled by the bilinear interpolation algorithm. Due to some reasons, there will be some missing pixels in the multispectral remote sensing image used. These vacancies need to be filled by bilinear interpolation. For some missing pixels in deep water areas, they can be solved by filling with constant values.

[0043] Step 1.2: The multi-scale hierarchical sampler first needs to determine the center coordinate i of the corresponding cropping range C With j C , the resampling resolution RES set here R is [810 m, 270 m, 90 m, 30 m, 10 m], which means that the output remote sensing image group has 5 sets of data; and the input resolution RES is 10 m, and the expansion ratio is calculated to be [81, 27, 9,3, 1]. Here, the length and width of the cropped image are set to 15 respectively. The specific processing flow is as shown in the attached figure. Figure 2 As shown. First, according to the center point coordinates [i C , j C ], crop the spatial range corresponding to the coarsest resolution image, whose length and width are both 12150m, corresponding to each band, that is, the spatial range is [i C – 607: i C + 607, j C – 607: j C + 607], i.e. Figure 2 The lowest branch in the image. Then, the resolution is downsampled from the original 10 m to 810 m, and the image size is changed to 15×15. The next step is to crop a new image block at the center of the aforementioned large-scale cropped image, with a length and width of 4050 m, corresponding to Figure 2 The spatial range of the penultimate branch in the multi-scale layer is calculated, and it is completely consistent with the resampled image size in the previous step by downsampling to 270 m. As the resolution becomes finer, the spatial range is continuously reduced until a cropped image with a final resolution of 10 m is obtained. After being processed by the multi-scale hierarchical sampler, a large-scale multispectral remote sensing image is decomposed into a combination of 5 multispectral remote sensing image fragments with a length and width of only 15 pixels by resampling and cropping. Although this method will lose the accuracy and discrimination of spatial information in a large-scale range, it greatly retains the more significant spatial features in the corresponding spatial range. In the case of a true value premise, the true value water depth map corresponding to the finest range (10 m resolution) is also extracted to provide a supervision signal for subsequent training or fine-tuning.

[0044] Step 2: Construct an encoder module group consisting of a multi-branch convolutional neural network, each branch corresponds to a specific spatial scale resolution, and integrates feature maps of different spatial resolutions through the subsequent center alignment fusion module. The two parts are combined into a single encoder stage. Its purpose is to complete the extraction of spatial features at various spatial scales and the interaction of feature information across resolution scales. Its structure is as follows Figure 3 shown.

[0045] Step 2.1: Construct the corresponding convolutional neural network module for each resampling branch. Each encoder module branch has its corresponding spatial input, and there is no weight parameter sharing between different branches. This ensures that each branch can extract underwater terrain features at a specific spatial resolution. Each encoder module can use a convolutional neural network such as ResNet, DenseNet, or a self-attention module based on ViT. Since the length and width of the multispectral image after processing in this embodiment is only 15×15, the simpler residual structure in ResNet is applied.

[0046] Step 2.2: Each encoder module independently processes each spatial resolution input multispectral remote sensing image. A corresponding module is set after each group of encoder modules to realize the information transmission between feature maps of different spatial resolutions, namely, the center alignment fusion module. Its structure is shown in the attached figure. Figure 4 Its direction is to advance from the coarse low-resolution branch to the fine high-resolution branch. That is, the center alignment fusion module first crops the central part of the low-resolution branch feature map. Let the resolution of the low-resolution branch at this time be RES L The resolution of the high-resolution branch is RES H , then its size after trimming is the original RES H / RES LFor RES-810, the cropping operation extracts a 5-pixel fragment from the center, and combines Figure 2 The operation process of the multi-scale hierarchical sampler corresponds spatially to the range covered by the adjacent RES-270 full image.

[0047] Step 2.3: For the cropped feature map, restore its size to the original size through bilinear interpolation. Here, the resolution of the RES-810 center image cropped in step 2.2 is upsampled to 270 m, thereby aligning it with the spatial range of the RES-270 feature map. By splicing the resampled feature with the high-resolution branch feature map (i.e., RES-270) along the channel dimension, the feature fusion is completed through the subsequent convolutional neural network module, and the output feature of the high-resolution RES-270 branch is updated.

[0048] Step 2.4: If Figure 4 As shown in FIG. 1 , the above process proceeds in the direction of resolution refinement, realizes the alignment and fusion of feature maps of different resolution levels, and completes the update of feature maps of other levels except the lowest resolution branch (i.e., the features of the RES-810 branch are not updated).

[0049] Step 3: A total of 4 encoder stages are set up, and the serial layout is used to achieve the extraction of specific to abstract features and cross-resolution fusion. The structure of the entire encoder is as follows Figure 3 shown.

[0050] Step 4: Extract the clearest high-resolution branch output features of the four encoder stages and concatenate them along the channel dimension to map the predicted water depth results.

[0051] Step 4.1: The backbone multi-branch encoder contains 4 stages, and the center alignment fusion module aligns and fuses the extracted multi-resolution scale features after each stage and updates the content. Here, the multi-stage feature concatenation module concatenates the output features of the high-resolution branch (i.e., RES-10 branch) with the clearest content in all stages along the channel dimension, and realizes feature fusion in the channel dimension through a convolutional neural network module. The specific structure of this part is as follows Figure 3 shown.

[0052] Step 4.2: After feature fusion is completed, the feature dimension is mapped to 1 through two consecutive 1×1 convolution kernels, and to ensure that the output water depth is not negative, a ReLU activation function is added to the end, as shown in formula (2):

[0053] (2);

[0054] Here, x is the value of the input ReLU after mapping, and y is the output predicted water depth value. Figure 3 As shown in the figure, the output of the model is a 15×15 single-channel matrix, in which the value of each point corresponds to the predicted non-negative water depth value, and the resolution is consistent with the finest resolution, which is 10 m.

[0055] Step 5: Use the measured water depth data as the true value to train or fine-tune the model and evaluate its performance. Using the existing water depth measurement values ​​to train or fine-tune the model relies on the mask mean square error (MaskMSE) to evaluate the model output quality under the premise of sparse distribution of the true value and use it as a supervised training indicator. The calculation method is shown in formula (3):

[0056] (3);

[0057] Where K is the number of available sample points, x is the predicted value, y is the true value, and m is a Boolean matrix where true is a pixel with a real water depth value and false is a pixel without a water depth value. The model is trained using the AdamW optimizer with an initial learning rate of 2.5e-4, and a cosine learning rate controller is used to update the learning rate during the entire training process. The number of epochs used for training is 15. The predicted water depth values ​​obtained by the model after training are generally accurate and highly consistent with the actual water depth.

[0058] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, all of which belong to the protection scope of the present invention. The protection scope of the present invention is given by the attached claims and any equivalent technical solutions thereof.

Claims

1. A shallow water multispectral remote sensing depth detection method based on multi-resolution scale fusion, characterized by: Step 1, cropping and resampling the input pre-processed multispectral remote sensing image through a multi-scale hierarchical sampler to obtain a multi-resolution hierarchical multispectral remote sensing image group corresponding to different spatial scales; Step 2: construct an encoder module group consisting of a multi-branch convolutional neural network, where each branch corresponds to a specific spatial scale resolution, and integrates feature maps of different spatial resolutions through a subsequent center alignment fusion module. The two parts are combined into a single encoder stage; Step 3: a total of 4 encoder stages are set up, and the serial layout is used to achieve the extraction of concrete to abstract features and cross-resolution fusion; Step 4: extract the clearest high-resolution branch output features of the four encoder stages and concatenate them along the channel dimension and map them to obtain the predicted water depth result; Step 5, train or fine-tune the model and evaluate it using the measured water depth data as the true value; The step 1 comprises: Step 1.1: Select multispectral remote sensing images of the target area under appropriate imaging conditions to reduce the impact of cloud cover and human activities on the imaging quality and even the results; if it is not possible to obtain a sufficiently ideal remote sensing image, use median filtering to fuse multiple images within a certain time series range, as shown in formula (1): (1); Here R represents the reflectivity of a certain band, i and j correspond to the spatial position index of the pixel, and N is the number of images in the time series interval; the remote sensing image obtained is further processed and the missing values ​​are filled in by the bilinear interpolation algorithm; Step 1.2: The multi-scale hierarchical sampler first needs to determine the center coordinate i of the corresponding cropping range C With j C , and needs to be resampled by the preset resampling resolution RES R The corresponding expansion ratio ER is calculated from the input resolution RES, that is, RES R / RES, and use the preset window size W to determine the extended window size EW, that is, ER×W; then, according to the determined center coordinates, the multispectral image block with a size of EW is cropped, and a non-overlapping sliding window mean filter is used in the cropped image to obtain a resolution of RES R By setting multiple resampling resolutions RES R Obtain a hierarchical image group including coarse large-scale low-resolution remote sensing images to fine small-scale high-resolution images, wherein a single image has its corresponding spatial resolution and scale range to cover the spatial feature content expression at different scales; The step 2 comprises: Step 2.1: Construct a corresponding convolutional neural network encoder module for each resampling branch. Each encoder module branch has an input of its corresponding scale, and there is no weight parameter sharing between different branches. Step 2.2: Each encoder module independently processes the multispectral remote sensing image input of its corresponding spatial resolution. A corresponding feature fusion module is set after each group of encoder modules to realize the information transfer between feature maps of different spatial resolutions, namely, the center alignment fusion module; its fusion process is promoted from the low-resolution branch to the high-resolution branch, namely, the center alignment fusion module first crops the central part of the low-resolution branch feature map, at which time the resolution of the low-resolution branch is RES L The high-resolution branch is RES H , so that the size range after cropping is the original RES H / RES L ; Step 2.3: For the cropped feature map, restore its size to the original size through bilinear interpolation, and the resolution becomes RES with upsampling. H , thus completing the alignment of the two feature maps; further, the resampled features are spliced ​​with the high-resolution branch feature map along the channel dimension to merge the two, and the feature fusion is completed through the subsequent convolutional neural network module, and the output features of the high-resolution branch are updated at the same time; Step 2.4: Advance from the low-resolution branch to the fine high-resolution branch to achieve the alignment and fusion of feature maps at different resolution levels, and complete the update of output features of all levels except the lowest resolution branch.

2. The shallow water multi-spectral remote sensing depth detection method based on multi-resolution scale fusion as claimed in claim 1, characterized in that: In the step 3, four encoder stages consisting of an encoder module group and a center alignment fusion module are set to form the encoder body, and the serial layout is used to realize the extraction and fusion of specific features and abstract features.

3. The shallow water multi-spectral remote sensing depth detection method based on multi-resolution scale fusion as claimed in claim 1, characterized in that: The step 4 comprises: Step 4.1: The backbone multi-branch encoder contains 4 stages, and the center alignment fusion module contained in each stage aligns and fuses the features of different branches across resolution scales and updates the content; Step 4.2: After the feature fusion is completed, its dimension is mapped to 1 through two consecutive 1×1 convolution kernels, and in order to ensure that the output water depth is not negative, a ReLU activation function is added to the end, as shown in formula (2): (2); Here x is the value of the input ReLU after mapping, and y is the output predicted water depth value.

4. The shallow water multi-spectral remote sensing depth detection method based on multi-resolution scale fusion as claimed in claim 1, characterized in that: In step 5, the model is trained or fine-tuned and evaluated using the measured water depth data as the true value. The model training or fine-tuning relies on the mask mean square error (MaskMSE) to evaluate the model output quality under the condition that the true value distribution is sparse and serves as a supervised training indicator. The calculation method is shown in formula (3): (3); Where K is the number of available sample points, x is the predicted value, y is the true value, and m is a Boolean matrix where true refers to pixels with real water depth values ​​and false refers to pixels without water depth values.

Images