A super-resolution depth model construction method based on relay residual diffusion probability

Abstract

The present invention discloses a method for constructing a super-resolution water depth model of relay residual diffusion probability, comprising the following steps: modeling the forward diffusion process in the form of a Markov chain, by continuously adding residuals and noise to the high-resolution image, the image is gradually degraded to the superposition state of the low-resolution image and Gaussian noise, so as to realize the gradual transfer of the target distribution to the known distribution; modeling the conditional probability distribution of reverse diffusion, and the reverse diffusion is defined as a learnable Markov chain; constructing a neural network module SDU-Net, and independently training the residual prediction model and the noise prediction model to predict the unknown quantity residual and noise; using the trained model for relay sampling, and realizing the generation of a high-resolution DEM image according to the low-resolution DEM image. The present invention can achieve a super-resolution of a 5-fold magnification factor, and has practical application value in the process of seabed DEM construction and enhancement.

Description

Technical Field

[0001] The present invention relates to the fields of water depth measurement, ocean mapping, topographic mapping and generative deep learning, and in particular to a method for constructing a super-resolution water depth model of relay residual diffusion probability. Background Art

[0002] The water depth model provides a basic data set for studying climate change, plate movement, marine disasters, seabed natural resource surveys and many other fields. However, water depth measurements in ocean areas mainly use multi-beam echo sounder systems. This ship-borne mobile measurement cannot cover the global ocean in a short period of time. The current coverage of the multi-beam measurement area only accounts for 26.1% of the global ocean area. In addition, although estimating water depth through satellite gravity measurement is more efficient, the spatial resolution is only 1 arc minute, which is difficult to meet the current research needs in the field of marine science. It is urgent to develop an efficient and high-precision water depth model enhancement method.

[0003] In recent years, DEM super-resolution based on deep learning has developed rapidly. This data-driven image processing mode provides a more efficient and accurate way for DEM enhancement. Common methods include CNN-Based, GAN-Based and Transformer-Based. Since 2021, image generation technology based on diffusion models has continued to develop and mature, and has quickly emerged in the field of AIGC large models. It breaks through the limitations of traditional CNN and visual Transformer on single image fitting. As a probabilistic generation model, the goal of the diffusion model is to fit the distribution of the integrated target image domain. At the same time, compared with the GAN model, the diffusion model training is more stable and easier to expand. The diffusion model has achieved excellent results in image editing, image restoration, image enhancement and other fields. It usually takes hundreds or even thousands of steps to build a Markov chain of a diffusion model, and the training time cost is high. At the same time, too many sampling steps will emphasize the diversity of the results and increase the possibility of generating errors. On the other hand, the current single sampling mode of the diffusion model cannot guarantee the optimality of the reverse process at each moment, and lacks a more flexible sampling mechanism. Therefore, the current diffusion model still needs to be improved in the field of DEM super-resolution. Summary of the invention

[0004] In view of this, the present invention proposes a method for constructing a super-resolution water depth model of relay residual diffusion probability. The present invention constructs a Markov chain of diffusion transfer in the residual space of the water depth model, which can shorten the total number of diffusion steps to 50 steps, and significantly improves the efficiency and accuracy of the super-resolution of the diffusion model. At the same time, the present invention proposes a relay sampling mechanism of the diffusion model, which can effectively avoid the disadvantageous positions under different inverse modes and integrate their respective advantageous positions.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A method for constructing a super-resolution water depth model by relaying residual diffusion probability, comprising:

[0007] Step 1: Use the Markov chain to model the forward diffusion process. By continuously adding residuals and noise to the high-resolution image, the image is gradually degraded to the superposition state of the low-resolution image and Gaussian noise, so as to achieve the distribution of the target P ( x 0 ) gradually transfers to the known distribution P ( x T );

[0008] Step 2: Model the conditional probability distribution of reverse diffusion. Reverse diffusion is defined as a learnable Markov chain according to t The distribution of time P( x t ), to model t-1 The conditional probability distribution P( x t-1 | x t ), P( x t-1 | x t ) has two estimation modes: residual prediction and noise prediction;

[0009] Step 3: Construct a neural network model SDU-Net, and train the residual prediction model and the noise prediction model independently to predict the unknown quantities introduced in the reverse diffusion - residual and noise;

[0010] Step 4: Use the trained model for relay sampling. The entire sampling process is designed to be divided into two stages, which are completed by two reverse diffusion modes, and finally a high-resolution DEM image is generated based on the low-resolution DEM image.

[0011] Preferably, in step 1, the forward diffusion process is modeled in the form of a Markov chain, and by continuously adding residuals and noise to the high-resolution image, the image is gradually degraded to a superposition state of a low-resolution image and Gaussian noise, so as to achieve the distribution of the target image. P ( x 0 ) gradually transfers to the known distribution P ( x T ):

[0012] Step 1.1: First, pair the high-resolution DEM (HR) and the low-resolution DEM (LR) according to the geographic coordinates to form a (HR, LR) image pair for training. Then, use the nearest neighbor interpolation to upsample the LR so that the LR and HR have the same size in the following calculation process. For the convenience of subsequent expression, the high-resolution DEM is recorded as x 0 , the interpolated low-resolution image is denoted as y 0 ;

[0013] Step 1.2: Definition t = The conditional distribution from time 0 to any time is shown in formula (1):

[0014] (1);

[0015] In the formula, q represents the conditional distribution of the forward process, N represents the Gaussian distribution, r 0 is the residual between HR and LR, r 0 = y 0 - x 0 , Indicates from x 0 → x t The noise scheduling parameters, k is a hyperparameter that controls the standard deviation of the noise, I is the unit matrix; reparameterize formula (1) to obtain x t Analytical formula:

[0016] (2);

[0017] In the formula, , represents the noise randomly sampled from a standard Gaussian distribution.

[0018] Step 1.3: The noise scheduling function is shown in formula (3):

[0019] (3);

[0020] In the formula, t represents the current time, T Represents the total number of time steps required for the entire diffusion process.

[0021] Preferably, in step 2, the conditional probability distribution of reverse diffusion is modeled, and reverse diffusion is defined as a learnable Markov chain according to t The distribution of time P(x t ), to estimate t-1 The conditional probability distribution P( x t-1 | x t ), P( x t-1 | x t ) has two estimation modes: residual prediction and noise prediction, and the methods are as follows:

[0022] Step 2.1: Reverse conditional probability P( x t-1 | x t ) is a Gaussian distribution, with variance ∑ for:

[0023] (4).

[0024] Step 2.2: In the noise prediction mode, the noise added to the image is first considered as a known quantity to model the inverse conditional probability P( x t-1 | x t ) μ :

[0025] (5);

[0026] In the formula, ε t is the noise added during forward diffusion, .

[0027] Step 2.3: In the residual prediction mode, first convert the residual between HR and LR r 0 As a known quantity, we can model the inverse conditional probability P( x t-1 | x t ) μ :

[0028] (6).

[0029] Preferably, in step 3, a neural network module SDU-Net is constructed, and a residual prediction model and a noise prediction model are trained independently to predict the unknown quantities introduced in the reverse diffusion, namely, residual and noise, in the following manner:

[0030] Step 3.1: The overall network design is a bilaterally symmetrical U-Net architecture. The left side is the encoder, which includes three downsamplings. Each downsampling halves the spatial resolution of the feature map and doubles the number of channels. The right side is the decoder, which includes three upsamplings. Each upsampling doubles the resolution and halves the number of channels. Skip connections are constructed between the same layers of the encoder and decoder.

[0031] Step 3.2: The encoder of SDU-Net accepts a single-channel image input and first uses a convolutional layer to increase the number of image channels to 64; then three layers of feature extraction are performed, each layer of feature extraction contains a sliding window transform module and a downsampling module, and each downsampling module includes a pooling layer and two densely connected modules.

[0032] Step 3.3: The decoder is symmetrical to the encoder. Each layer consists of an upsampling module and a sliding window transform module. A convolutional layer is used at the end of the decoder to fuse the 64-dimensional feature map into a single-channel image, thereby outputting the predicted result.

[0033] Step 3.4: Use forward diffusion to obtain x t and noise ε t To train SDU-Net, the objective function of the training is:

[0034] (7);

[0035] In the formula, θ 1 represents the parameters of the noise prediction neural network, and the input is x t , y 0 and t , Indicates the noise ε t prediction results.

[0036] Step 3.5: Use forward diffusion to obtain x t and residual r 0 To train SDU-Net, the objective function of the training is:

[0037] (8);

[0038] In the formula, θ 2 Represents the parameters of the residual prediction neural network, and the input is x t , y0 and t , Residual r 0 prediction results.

[0039] Preferably, in step 4, the trained model is used for relay sampling, and the entire sampling process is designed to be two stages, which are completed by two reverse diffusion modes, and finally a high-resolution DEM image is generated according to a low-resolution DEM image. The method is as follows:

[0040] Step 4.1: In the length T Add breakpoints to the reverse Markov chain b , so that the whole sampling process is divided into two stages.

[0041] Step 4.2: In the first stage of the sampling process, T Time to b At this moment, the residual prediction mode is used for sampling, and the sampling formula is as follows:

[0042] (9);

[0043] In the formula, z is the random noise sampled from the standard Gaussian distribution, using formula (9), according to x T Gradually extrapolate to x b .

[0044] Step 4.3: In the second stage of the sampling process, b Time to 0 At this moment, the noise prediction mode is used for sampling, and the sampling formula is as follows:

[0045] (10);

[0046] In the formula, z is the random noise sampled from the standard Gaussian distribution; using formula (10), according to x b Gradually extrapolate to x 0 , x 0 This is the super-resolution result of the depth model.

[0047] The method for constructing a super-resolution water depth model of relay residual diffusion probability is used to construct an accurate and efficient diffusion model, which can improve the resolution of the currently most advanced global water depth model, GEBCO, from 15 arc seconds to 3 arc seconds, achieving a 5-fold super-resolution. The water depth model after super-resolution has more complete details and clearer edges, and a more refined underwater terrain expression is achieved on the basis of the current GEBCO, which is used in the fields of water depth model enhancement, seabed resource development, seabed terrain and landform detection, and ocean surveying and mapping.

[0048] Beneficial technical effects brought by the present invention:

[0049] The present invention discloses a method for constructing a super-resolution water depth model of relay residual diffusion probability. Compared with the prior art, we use a residual Markov chain as a conditional introduction mechanism, which allows the transfer of image distribution with fewer steps and has higher efficiency. At the same time, this conditional introduction mechanism has better control capabilities. In addition, the present invention proposes a relay sampling mechanism for the diffusion model, which helps the diffusion model to achieve super-resolution tasks more flexibly and efficiently, and has a beneficial effect on the restoration of texture details of the water depth model. The present invention can play an important role in the fields of water depth model enhancement, seabed resource development, seabed topography and landform detection, and marine surveying and mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The present invention provides a method flow for constructing a super-resolution water depth model based on the relay residual diffusion probability.

[0051] Figure 2 This is the principle of the relay residual diffusion probability model in the present invention.

[0052] Figure 3 This is the overall structure of SDU-Net in the present invention.

[0053] Figure 4 This is the SDU-Net submodule structure in the present invention.

[0054] Figure 5 This is the sampling process of the relay residual diffusion probability model in the present invention.

[0055] Figure 6 These are the results of a 5x super-resolution depth model, where: (a) is located in Hawaiian waters, (b) is located in the Gulf of Mexico, and (c) is located on the eastern continental shelf of the United States. DETAILED DESCRIPTION

[0056] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0057] Here, GEBCO_2024 is used as the low-resolution seafloor DEM with a spatial resolution of 15 arc seconds, and CRM is used as the high-resolution seafloor DEM with a spatial resolution of 3 arc seconds. Figure 1 As shown. It includes: Step 1: Model the forward diffusion process in the form of Markov chain, and gradually degrade the image to the superposition state of low-resolution image and Gaussian noise by continuously adding residuals and noise to the high-resolution image, so as to achieve the distribution of the target P ( x 0 ) gradually transfers to the known distribution P ( x T ); Step 2: Model the conditional probability distribution of reverse diffusion. Reverse diffusion is defined as a learnable Markov chain according to t The distribution of time P( x t ), to model t-1 The conditional probability distribution P( x t-1 | x t ), P( x t-1 | x t ) estimation methods include residual prediction and noise prediction; Step 3: Construct a neural network model SDU-Net, and independently train the residual prediction model and the noise prediction model to predict the unknown quantities introduced in the reverse diffusion - residual and noise; Step 4: Use the trained model for relay sampling. The whole sampling process is designed to be divided into two stages, which are completed by two reverse diffusion modes, and finally realize the generation of high-resolution DEM images based on low-resolution DEM images.

[0058] Step 1: If Figure 2 As shown in Figure 1, noise and residuals are gradually added to the high-resolution DEM to achieve forward diffusion of the model.

[0059] Step 1.1: First, the 15 arc-second low-resolution DEM and the 3 arc-second high-resolution DEM are combined into an image pair (HR, LR). Then, the nearest neighbor interpolation algorithm is used to make the number of pixels of LR the same as that of HR. For the convenience of subsequent expression, the high-resolution DEM is recorded as x 0 , the interpolated low-resolution image is denoted as y 0 ;

[0060] Step 1.2: Definition t = The conditional distribution from time 0 to any time is shown in formula (1):

[0061] (1);

[0062] In the formula, q represents the conditional distribution of the forward process, N represents the Gaussian distribution, r 0 is the residual between HR and LR, r 0 = y 0 - x 0 , Indicates from x 0 → x t The noise scheduling parameters, k is a hyperparameter that controls the standard deviation of the noise, I is the unit matrix. Reparameterizing formula (1) yields x t Analytical formula:

[0063] (2);

[0064] In the formula, , represents the noise randomly sampled from a standard Gaussian distribution.

[0065] Step 1.3: Use the sine function to design the noise scheduling function. The sine function grows slowly near the extreme value, which is conducive to the calculation stability at the beginning of the reverse diffusion. At the same time, the derivative of the sine function is symmetric about the inflection point, which is consistent with the two-stage synthesis design of sampling. The noise scheduling function is shown in formula (3):

[0066] (3);

[0067] In the formula, t represents the current time, T Represents the total number of time steps required for the entire diffusion process.

[0068] Step 2: Model the conditional probability distribution of reverse diffusion. Reverse diffusion is defined as a learnable Markov chain according to t The distribution of time P( x t ), to estimate t-1 The conditional probability distribution P( x t-1 | x t ), P( x t-1 | x t ) has two estimation modes: residual prediction and noise prediction.

[0069] Step 2.1: Reverse conditional probability P( x t-1 | x t ) is a Gaussian distribution, with variance ∑ for:

[0070] (4).

[0071] Step 2.2: In the noise prediction mode, the noise added to the image is first considered as a known quantity to model the inverse conditional probability P( x t-1 | x t ) μ :

[0072] (5);

[0073] In the formula, ε t is the noise added during forward diffusion, .

[0074] Step 2.3: In the residual prediction mode, first convert the residual between HR and LR r 0 As a known quantity, we can model the inverse conditional probability P( x t-1 | x t ) μ :

[0075] (6).

[0076] Step 3: Construct a neural network model SDU-Net, and independently train the residual prediction model and the noise prediction model to predict the unknown quantities introduced in the reverse diffusion - residual and noise.

[0077] Step 3.1: The overall network design is a bilaterally symmetrical U-Net architecture, and its overall structure is as follows: Figure 3 As shown in the figure. The left side is the encoder, which includes three downsamplings. Each downsampling reduces the spatial resolution of the feature map by half and doubles the number of channels. The right side is the decoder, which includes three upsamplings. Each upsampling doubles the resolution and reduces the number of channels by half. Skip connections are constructed between the same layers of the encoder and decoder.

[0078] Step 3.2: The encoder of SDU-Net accepts a single-channel image input and first uses a convolutional layer to increase the number of image channels to 64; then three layers of feature extraction are performed. Each layer of feature extraction contains a sliding window transform module and a downsampling module. Each downsampling module includes a pooling layer and two densely connected modules. The submodule structure is as follows: Figure 4 shown.

[0079] Step 3.3: The decoder is symmetrical to the encoder. Each layer consists of an upsampling module and a sliding window transform module. The decoder uses a convolutional layer at the end to fuse the 64-dimensional feature map into a single-channel image. The predicted result is output. The submodule structure is as follows: Figure 4 shown.

[0080] Step 3.4: Use forward diffusion to obtain x t and noise ε t To train SDU-Net, the objective function of the training is:

[0081] (7);

[0082] In the formula, θ 1 represents the parameters of the noise prediction neural network, and the input is x t , y 0 and t , Indicates the noise ε t prediction results.

[0083] Step 3.5: Use forward diffusion to obtain x t and residual r 0 To train SDU-Net, the objective function of the training is:

[0084] (8);

[0085] In the formula, θ 2 Represents the parameters of the residual prediction neural network, and the input is x t , y 0 and t , Residual r 0 prediction results.

[0086] Step 4: Use the trained model for relay sampling. The entire sampling process is designed to be divided into two stages, which are completed by two reverse diffusion modes, and finally a high-resolution DEM image is generated based on the low-resolution DEM image.

[0087] Step 4.1: In the length T Add breakpoints to the reverse Markov chain b , so that the whole sampling process is divided into two stages. The whole sampling process is as follows Figure 5 As shown, first add Gaussian noise to the low-resolution image to obtain x T , then the residual prediction mode is used for sampling in sampling stage 1, and the noise prediction mode is used for sampling in sampling stage 2. It can be seen that as the sampling proceeds, the noise in the image gradually weakens and the texture gradually recovers until the sampling reaches x 0 , which is the super-resolution DEM.

[0088] Step 4.2: In the first stage of the sampling process, T Time to b At this moment, the residual prediction mode is used for sampling, and the sampling formula is as follows:

[0089] (9);

[0090] In the formula, z is the random noise sampled from the standard Gaussian distribution. Using formula (9), according to x T Gradually extrapolate to x b .

[0091] Step 4.3: In the second stage of the sampling process, b Time to 0 At this moment, the noise prediction mode is used for sampling, and the sampling formula is as follows:

[0092] (10);

[0093] In the formula, z is the random noise sampled from the standard Gaussian distribution. Using formula (10), according to x b Gradually extrapolate to x 0 , x 0 This is the super-resolution result of the water depth model. The super-resolution effect of DEM is as follows: Figure 6As shown, the present invention has an obvious effect on the detailed texture recovery of the water depth model, and has a high degree of conformity with the high-resolution DEM.

[0094] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, all of which belong to the protection scope of the present invention. The protection scope of the present invention is given by the attached claims and any equivalent technical solutions thereof.

Claims

1. A method for constructing a super-resolution water depth model based on relay residual diffusion probability, characterized in that: include: Step 1: Use the Markov chain to model the forward diffusion process. By continuously adding residuals and noise to the high-resolution image, the image is gradually degraded to the superposition state of the low-resolution image and Gaussian noise, so as to achieve the distribution of the target image. P ( x 0 ) gradually transfers to the known distribution P ( x T ); Step 2, model the conditional probability distribution of reverse diffusion. Reverse diffusion is defined as a learnable Markov chain according to t The distribution of time P( x t ), to estimate t-1 The conditional probability distribution P( x t-1 | x t ), P( x t-1 | x t ) estimation methods include residual prediction and noise prediction; Step 3, construct a neural network model SDU-Net, and independently train the residual prediction model and the noise prediction model to predict the unknown quantities introduced in the reverse diffusion, namely residual and noise; Step 4, use the trained model for relay sampling. The whole sampling process is designed to be divided into two stages, which are completed by two reverse diffusion modes, and finally realize the generation of high-resolution DEM images based on low-resolution DEM images; The step 4 comprises: Step 4.1: In the length T Add breakpoints to the reverse Markov chain b , so that the whole sampling process is divided into two stages; Step 4.2: In the first stage of the sampling process, T Time to b At this moment, the residual prediction mode is used for sampling, and the sampling formula is as follows: （9）； In the formula, z is the random noise sampled from the standard Gaussian distribution, using formula (9), according to x T Gradually extrapolate to x b ; Step 4.3: In the second stage of the sampling process, b Time to 0 At this moment, the noise prediction mode is used for sampling, and the sampling formula is as follows: （10）； In the formula, z is the random noise sampled from the standard Gaussian distribution; using formula (10), according to x b Gradually extrapolate to x 0 , x 0 This is the super-resolution result of the depth model.

2. A method for constructing a super-resolution water depth model based on relay residual diffusion probability as claimed in claim 1, characterized in that: The step 1 comprises: Step 1.1: First, pair the high-resolution DEM (HR) and the low-resolution DEM (LR) according to the geographic coordinates to form a (HR, LR) image pair for training. Then, use the nearest neighbor interpolation to upsample the LR so that the LR and HR have the same size in the following calculation process. For the convenience of subsequent expression, the high-resolution DEM is recorded as x 0 , the interpolated low-resolution image is denoted as y 0 ; Step 1.2: Definition t = The conditional distribution from time 0 to any time is shown in formula (1): （1）； In the formula, q represents the conditional distribution of the forward process, N represents a Gaussian distribution, r 0 is the residual between HR and LR, r 0 = y 0 - x 0 , Indicates from x 0 → x t The noise scheduling parameters, k is a hyperparameter that controls the standard deviation of the noise, I is the unit matrix; reparameterize formula (1) to obtain x t Analytical formula: （2）； In the formula, , represents the noise randomly sampled from a standard Gaussian distribution; Step 1.3: The noise scheduling function is shown in formula (3): （3）； In the formula, t represents the current time, T Represents the total number of time steps required for the entire diffusion process.

3. A method for constructing a super-resolution water depth model based on relay residual diffusion probability as claimed in claim 1, characterized in that: The step 2 comprises: Step 2.1: Reverse conditional probability P( x t-1 | x t ) is a Gaussian distribution with variance ∑ for: （4）； Step 2.2: In the noise prediction mode, the noise added to the image is first considered as a known quantity to model the inverse conditional probability P( x t-1 | x t ) μ : （5）； In the formula, ε t is the noise added during forward diffusion, ; Step 2.3: In the residual prediction mode, first convert the residual between HR and LR r 0 As a known quantity, we can model the inverse conditional probability P( x t-1 | x t ) μ : (6).

4. A method for constructing a super-resolution water depth model based on relay residual diffusion probability as claimed in claim 1, characterized in that: The step 3 comprises: Step 3.1: The overall network design is a bilaterally symmetrical U-Net architecture. The left side is the encoder, which includes three downsamplings. Each downsampling reduces the spatial resolution of the feature map by half and doubles the number of channels. The right side is the decoder, which includes three upsamplings. Each upsampling doubles the resolution and reduces the number of channels by half. A skip connection is constructed between the same layers of the encoder and decoder. Step 3.2: The encoder of SDU-Net accepts a single-channel image input and first uses a convolutional layer to increase the number of image channels to 64; then three layers of feature extraction are performed, each of which contains a sliding window transform module and a downsampling module. Each downsampling module includes a pooling layer and two densely connected modules; Step 3.3: The decoder is symmetrical to the encoder. Each layer consists of an upsampling module and a sliding window transform module. A convolutional layer is used at the end of the decoder to fuse the 64-dimensional feature map into a single-channel image, thereby outputting the predicted result. Step 3.4: Use forward diffusion to obtain x t and noise ε t To train SDU-Net, the objective function of the training is: （7）； In the formula, θ 1 represents the parameters of the noise prediction neural network, and the input is x t , y 0 and t , Indicates the noise ε t The prediction results; Step 3.5: Use forward diffusion to obtain x t and residual r 0 To train SDU-Net, the objective function of the training is: （8）； In the formula, θ 2 Represents the parameters of the residual prediction neural network, and the input is x t , y 0 and t , Residual r 0 prediction results.

5. A method for constructing a super-resolution water depth model based on relay residual diffusion probability as claimed in claim 1, characterized in that: It is used to build an accurate and efficient diffusion model, which can achieve a more refined underwater terrain expression based on GEBCO, and can be used in the fields of water depth model enhancement, seabed resource development, seabed topography and landform detection, and marine surveying and mapping.

Images