Super-resolution method and system for river flow and water level remote sensing images based on wavelet domain

Abstract

The present invention discloses a method and system for super-resolution of remote sensing images of river flow and water level based on wavelet domain. It includes: inputting a river flow and water level data set and screening and processing the data set according to the river and time period to be predicted; obtaining the wavelet coefficients of the high-resolution image to convert it into a low-resolution image and extracting its image features; splicing and fusing each of the wavelet coefficient image features with the temporal features of the low-resolution image sequence and the low-resolution image feature map sequence into a new wavelet coefficient feature map; obtaining a low-resolution size wavelet coefficient feature map of the same size as the original low-resolution; and using inverse wavelet transform to generate a corresponding super-resolution image from the wavelet image. The present invention applies a two-dimensional image deep learning algorithm to obtain a high-resolution image sequence, reduces data transmission and storage resources, is low-cost and highly efficient, and makes the generated results more in line with expectations.

Description

Technical Field

[0001] The invention relates to the field of computer vision and remote sensing image super-resolution technology, and in particular to a wavelet-domain-based river flow and water level remote sensing image super-resolution method and system. Background Art

[0002] In recent years, with the development of remote sensing technology, remote sensing images can record the size of electromagnetic waves of various ground objects in real time, becoming a new type of monitoring data, which is applied to various natural detection scenarios, such as land and forest coverage, wetland resources, river flow and water level, etc. In order to improve the efficiency of transmission and processing, remote sensing images are often stored in a compressed form. Therefore, if you want to predict river flow and water level through remote sensing images, super-resolution processing of remote sensing images is a necessary step. With the development of artificial intelligence technology, remote sensing image super-resolution technology has emerged. These technologies mainly generate high-resolution images by inputting low-resolution remote sensing images, running neural networks for training, but are limited by the development of artificial intelligence related technologies. The generated images are prone to problems such as insufficient detail enhancement and local area distortion, which affects the final generation quality and is difficult to be widely used.

[0003] One of the current existing technologies, the patent "A remote sensing image super-resolution reconstruction technology based on wavelet analysis", uses bilinear interpolation, Wiener inverse filtering, convolution and other processing, and uses wavelet decomposition to obtain wavelet coefficients that describe high and low frequency information of different scales in all directions, thereby improving the spatial resolution of optical remote sensing images and achieving the purpose of image super-resolution. The disadvantage of this technology is that the processing object is limited, and the advantages brought by the huge amount of data are not fully utilized, and it is difficult to meet people's requirements for algorithm versatility, robustness, efficiency and superiority in the context of big data.

[0004] The second existing technology, the patent "A single remote sensing image super-resolution reconstruction technology based on deep learning", constructs a four-layer convolutional deep neural network structure, in which the first three layers of convolution are optimized by adding a parameter-corrected linear unit layer and a local response normalization layer, and the remote sensing image super-resolution reconstruction model is obtained after training. The disadvantage of this technology is that although it enhances the super-resolution reconstruction effect of a single remote sensing image, it does not take into account the unique sequence data characteristics of multi-phase images of the same scene of remote sensing images, and it is difficult to meet the effect of real-time monitoring.

[0005] The third existing technology, patent "A deep residual network image super-resolution reconstruction technology based on wavelet domain", uses wavelet transform to separate the "rough" and "detail" features of image content, adopts a flexible and scalable deep neural network with residual nested residuals, and combines the image spatial domain and wavelet domain loss functions for optimization and solution. The disadvantage of this technology is that although it enhances the image super-resolution reconstruction effect and can more effectively capture image detail information, it is difficult to directly apply to remote sensing images, which have relatively less details. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of existing methods and propose a method and system for super-resolution of remote sensing images of river flow and water level based on wavelet domain. The main problems solved by the present invention are: first, the existing traditional remote sensing image super-resolution technology has limitations in processing objects, does not fully utilize the advantages brought by the huge amount of data, and is difficult to meet people's requirements for algorithm versatility, robustness, efficiency and superiority in the context of big data. Second, the existing deep learning image super-resolution reconstruction technology has high requirements for image texture details and is not suitable for remote sensing images, or does not take into account the unique sequence data characteristics of multi-phase images of the same scene of remote sensing images, and is difficult to meet the effect of real-time monitoring. That is, how to obtain high-resolution river flow and water level remote sensing images with prominent details to meet the requirements of real-time monitoring.

[0007] In order to solve the above problems, the present invention proposes a wavelet-based river flow and water level remote sensing image super-resolution method, the method comprising:

[0008] Import a river flow and water level data set consisting of multiple sets of remote sensing image sequences, and filter and input low-resolution image sequences according to the river to be predicted and the time period to be predicted;

[0009] The low-resolution image sequence is first upsampled using a bicubic interpolation algorithm to obtain a high-resolution image of a specified resolution, and then a neural network model of high-dimensional features of a remote sensing image sequence is trained to extract temporal features of the low-resolution image sequence, and finally a convolutional neural network is used to extract features of the low-resolution image to obtain a low-resolution image feature map sequence;

[0010] Using wavelet transform to obtain wavelet coefficients of the high-resolution image into a low-resolution image, obtaining four wavelet coefficient images and performing feature extraction on each of the wavelet coefficient images;

[0011] Each of the wavelet coefficient image features is spliced ​​with the temporal features of the low-resolution image sequence and the low-resolution image feature map sequence, and the three features are fused into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps, and generating a low-resolution wavelet coefficient feature map of the same size as the original low-resolution image by dimensionality reduction and using a convolutional neural network;

[0012] The four groups of low-resolution wavelet coefficient feature maps and the wavelet coefficients are added, and then the wavelet coefficient image is transformed into a corresponding super-resolution image by inverse wavelet transform to obtain a final result, and a loss function is calculated according to the similarity between the super-resolution image and the input high-resolution image, and the neural network is trained and optimized according to the loss until the result of the loss function converges, thereby establishing a super-resolution image generation model;

[0013] By inputting a low-resolution image of the river to be processed in a certain period of time into the super-resolution image generation model, a corresponding high-resolution image is obtained.

[0014] Preferably, the method of importing a river flow and water level data set consisting of a plurality of remote sensing image sequences, and screening and inputting a low-resolution image sequence according to the river to be predicted and the time period to be predicted, is specifically as follows:

[0015] Import the river flow and water level dataset, which is composed of multiple groups of remote sensing image sequences. Each group of image sequences represents multiple low-resolution images of river flow and water level in a certain river in a period of time and the corresponding high-resolution images. Given the river to be predicted and the time period to be predicted, filter and input the low-resolution image sequence.

[0016] Preferably, the low-resolution image sequence is first upsampled using a bicubic interpolation algorithm to obtain a high-resolution image of a specified resolution, and then a neural network model of high-dimensional features of a remote sensing image sequence is trained to extract temporal features of the low-resolution image sequence, and finally a convolutional neural network is used to extract features of the low-resolution image to obtain a low-resolution image feature map sequence, specifically:

[0017] The bicubic interpolation algorithm is used to upsample the low-resolution image to obtain a high-resolution image of a specified resolution. Using the bicubic interpolation image upsampling method, the value of the function f at the point (x, y) can be obtained by weighted averaging the nearest sixteen sampling points in the rectangular grid. Specifically, for the interpolated pixel point (x, y), the 4×4 neighborhood points (x i ,y j ), the interpolation calculation formula is as follows:

[0018]

[0019]

[0020] Among them, x, y represent the coordinates of the pixel to be interpolated, i, j represent the position of the row and column, x i ,y j represents the coordinate values ​​of the 4×4 area points near the pixel to be interpolated, and W represents the basis function of the interpolation cubic function;

[0021] A neural network model of high-dimensional features of a remote sensing image sequence is trained to extract the temporal features of the low-resolution image sequence. The neural network model includes: using a long short-term memory network (LSTM) model, using two LSTM networks as an encoder and a decoder for training respectively, the input of the decoder is the change characteristics of the river flow and water level extracted from the remote sensing image sequence, and the L2 norm loss between the decoder output and input is calculated, and then feedback propagation is performed to optimize the encoder and decoder. The calculation formula is as follows:

[0022]

[0023] Among them, the input of the encoder is the vector P, p i Represents its i-th component, and the output of the decoder is vector Q, q i represents its i-th component;

[0024] The low-resolution image is input, and image features are extracted through a convolutional neural network to obtain a feature map sequence of the low-resolution image. The convolutional neural network includes: 3 feature extraction modules, each module contains 3 convolutional layers, all convolutional layers share the same 3×3 convolution kernel size, the step size and edge padding are set to 1, and the forward propagation is represented as a series of feature maps. The feature extraction network is composed of a plurality of residual blocks cascaded, and the residual block is composed of two convolutions with the same kernel size and the same number of filters. The formula is as follows:

[0025] X=H l ([x0,x1,x3])

[0026] Among them, H(·) represents a non-linear transformation function, using a batch normalization + ReLU activation function + 3×3 convolution structure, and X represents the feature x for each feature extraction module. l , the features of the three modules are fused as the final output features.

[0027] Preferably, the method of using wavelet transform to obtain wavelet coefficients of the high-resolution image into a low-resolution image, obtaining four wavelet coefficient images and performing feature extraction on each of the wavelet coefficient images is specifically as follows:

[0028] The high-resolution image HR is input, and four wavelet coefficient images are obtained through two-dimensional wavelet transform 2DDWT. The specific formula is as follows:

[0029] HRWav={H LL ,H LH ,H HL ,H HH}: =2DDWT{HR}

[0030] Among them, HRWav represents the wavelet of high-resolution image, H LL Represents the wavelet coefficients of the low-frequency component, H LH The wavelet coefficients representing the horizontal low-frequency and vertical high-frequency components, H HL The wavelet coefficients representing the horizontal high-frequency and vertical low-frequency components, H HH The wavelet coefficients representing the diagonal high-frequency components;

[0031] Input the obtained wavelet coefficient H of the low-frequency component LL , the wavelet coefficients H of the horizontal low-frequency and vertical high-frequency components LH , the wavelet coefficients H of the horizontal high frequency and vertical low frequency components HL and the wavelet coefficients H of the diagonal high-frequency components HH , and feature extraction is performed on them respectively. In order to map the features to the same feature space, the feature extraction network same as the convolutional neural network is adopted, that is, the network consists of 3 feature extraction modules, each module contains 3 convolutional layers, all convolutional layers share the same 3×3 convolution kernel size, and the stride and edge padding are set to 1.

[0032] Preferably, each of the wavelet coefficient image features is spliced ​​with the temporal features of the low-resolution image sequence and the low-resolution image feature map sequence, and the three features are fused into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps, and generating a low-resolution wavelet coefficient feature map of the same size as the original low-resolution image by dimensionality reduction and using a convolutional neural network, specifically:

[0033] The four wavelet coefficients H LL , H LH , H HL and H HHEach wavelet coefficient image feature in the image feature is spliced ​​with the extracted feature of the low-resolution image and the temporal feature of the low-resolution image sequence, and the three features are fused into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps;

[0034] The four new wavelet coefficient feature maps are used as input. Since the input wavelet coefficient feature map is formed by splicing three features and has a large dimension, a 1×1 convolution is used to reduce the feature dimension. Then, the convolutional neural network is used to obtain a wavelet coefficient feature map of the same size as the original low-resolution wavelet coefficient feature map. To ensure that the output wavelet coefficient feature map has the same size as the original low-resolution image, the convolutional neural network is composed of three feature extraction modules, each module contains three convolutional layers, and all convolutional layers share the same 3×3 convolution kernel size, and the step size and edge padding are set to 1.

[0035] Preferably, the four groups of low-resolution wavelet coefficient feature maps and the wavelet coefficients are added, and then the wavelet coefficient image is transformed into a corresponding super-resolution image by inverse wavelet transform to obtain the final result, and the loss function is calculated according to the similarity between the super-resolution image and the input high-resolution image, and the neural network is trained and optimized according to the loss until the result of the loss function converges, thereby establishing a super-resolution image generation model, specifically:

[0036] The network adopts a flexible and constrained loss function, and uses two losses, wavelet coefficient loss and image space pixel loss, to robustly calculate the similarity between the output super-resolution image and the input high-resolution image. The specific formula is as follows:

[0037] L=αL w +(1-α)L i

[0038] Among them, L represents the loss function of the entire network, L w represents the wavelet coefficient loss, L i represents the pixel loss in image space, and α represents the weight of the two loss functions;

[0039] Wavelet coefficient loss refers to the similarity comparison between all wavelet coefficient images of the wavelet coefficient feature map of the same size as the original low-resolution wavelet coefficient feature map and the sub-band image obtained by wavelet transforming the input high-resolution image. The most commonly used pixel-by-pixel mean square error loss function in the image space is used to operate the wavelet coefficients in the same way. The loss function is expressed by the L2 norm, and the calculation formula is as follows:

[0040]

[0041] in, represents the wavelet coefficients of a wavelet coefficient feature map of the same size as the original low-resolution wavelet coefficient feature map obtained by the convolutional neural network, represents the wavelet coefficients obtained by wavelet transform of the input high-resolution image, n represents the size of batchsize, i represents the i-th image in the batch processed images, and j represents the j-th coefficient in the wavelet coefficient sequence;

[0042] The loss calculation in the image space refers to a balance between the texture and smoothness of the super-resolution image and the original high-resolution image, expressed using the L2 norm, and the calculation formula is as follows:

[0043]

[0044] Among them, I S represents the super-resolution image, I H represents the input high-resolution image; n represents the size of batchsize, and i represents the i-th image in the batch image;

[0045] By inputting a low-resolution image of the river to be processed in a certain period of time into the super-resolution image generation model, a corresponding high-resolution image is obtained.

[0046] Accordingly, the present invention also provides a river flow and water level remote sensing image super-resolution system based on wavelet domain, comprising:

[0047] A data set screening unit is used to import a river flow and water level data set consisting of multiple sets of remote sensing image sequences, and screen and input a low-resolution image sequence according to the river to be predicted and the time period to be predicted;

[0048] A low-resolution image sequence data processing unit is used to first perform an upsampling operation on the low-resolution image sequence using a bicubic interpolation algorithm to obtain a high-resolution image of a specified resolution, then train a neural network model of high-dimensional features of a remote sensing image sequence to extract temporal features of the low-resolution image sequence, and finally use a convolutional neural network to extract features of the low-resolution image to obtain a low-resolution image feature map sequence;

[0049] A wavelet coefficient image feature extraction unit, used for obtaining wavelet coefficients of the high-resolution image converted into a low-resolution image by using wavelet transform, obtaining four wavelet coefficient images and performing feature extraction on each of the wavelet coefficient images;

[0050] A wavelet coefficient feature map fusion unit is used to respectively splice each of the wavelet coefficient image features with the features and time series features of the low-resolution image, and fuse the three features into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps, and obtaining a wavelet coefficient feature map of the same size as the original low-resolution wavelet coefficient feature map;

[0051] A super-resolution image generation unit is used to add the four groups of low-resolution wavelet coefficient feature maps and the wavelet coefficients, and then use wavelet inverse transform to generate a corresponding super-resolution image from the wavelet coefficient image to obtain a final result, and calculate the loss function according to the similarity between the super-resolution image and the input high-resolution image, and train and optimize the neural network according to the loss until the result of the loss function converges, thereby establishing a super-resolution image generation model, and inputting a low-resolution image of the river to be processed in a certain time period into the super-resolution image generation model to obtain the corresponding high-resolution image.

[0052] The implementation of the present invention has the following beneficial effects:

[0053] The present invention applies a two-dimensional image deep learning algorithm to obtain a high-resolution image sequence, reduces data transmission and storage resources, and has the characteristics of low cost and high efficiency; the input and output between the various parts of the invention are interconnected, and different features are extracted and integrated, which improves efficiency and makes the generated results more in line with people's expectations; the invention uses a neural network method to generate a super-resolution image sequence through a coding and decoding structure, which can maximally guide the neural network to enhance the low-resolution local detail features; when using a convolutional neural network to super-resolution the original low-resolution image sequence, the texture information can be retained to the maximum extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is an overall flow chart of a wavelet-domain-based river flow and water level remote sensing image super-resolution method according to an embodiment of the present invention;

[0055] Figure 2 It is a structural diagram of a river flow and water level remote sensing image super-resolution system based on wavelet domain according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Figure 1 is an overall flow chart of a wavelet-domain-based river flow and water level remote sensing image super-resolution method according to an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0058] S1, importing a river flow and water level data set consisting of multiple sets of remote sensing image sequences, and screening and inputting a low-resolution image sequence according to the river to be predicted and the time period to be predicted;

[0059] S2, firstly upsampling the low-resolution image sequence using a bicubic interpolation algorithm to obtain a high-resolution image of a specified resolution, then training a neural network model of high-dimensional features of a remote sensing image sequence to extract temporal features of the low-resolution image sequence, and finally extracting features of the low-resolution image using a convolutional neural network to obtain a low-resolution image feature map sequence;

[0060] S3, using wavelet transform to obtain wavelet coefficients of the high-resolution image converted into a low-resolution image, obtaining four wavelet coefficient images and performing feature extraction on each of the wavelet coefficient images;

[0061] S4, respectively splicing each of the wavelet coefficient image features with the temporal features of the low-resolution image sequence and the low-resolution image feature map sequence, fusing the three features into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps, and generating a low-resolution wavelet coefficient feature map of the same size as the original low-resolution image by dimensionality reduction and using a convolutional neural network;

[0062] S5, adding the four groups of low-resolution wavelet coefficient feature maps and the wavelet coefficients, and then using inverse wavelet transform to generate a corresponding super-resolution image from the wavelet coefficient image to obtain the final result, and calculating the loss function based on the similarity between the super-resolution image and the input high-resolution image, training and optimizing the neural network based on the loss until the result of the loss function converges, thereby establishing a super-resolution image generation model, and inputting a low-resolution image of the river to be processed in a certain time period into the super-resolution image generation model to obtain the corresponding high-resolution image.

[0063] Step S1 is as follows:

[0064] S1-1, import the river flow and water level dataset, which is composed of multiple groups of remote sensing image sequences. Each group of image sequences represents multiple low-resolution images of river flow and water level in a certain river in a period of time and the corresponding high-resolution images. Given the river to be predicted and the time period to be predicted, filter and input the low-resolution image sequence.

[0065] Step S2 is as follows:

[0066] S2-1, using the bicubic interpolation algorithm, upsampling the low-resolution image to obtain a high-resolution image of a specified resolution. Using the bicubic interpolation image upsampling method, the value of the function f at the point (x, y) can be obtained by weighted averaging the nearest sixteen sampling points in the rectangular grid. Specifically, for the interpolated pixel point (x, y), the 4×4 neighborhood points (x i ,y j ), the interpolation calculation formula is as follows:

[0067]

[0068]

[0069] Among them, x, y represent the coordinates of the pixel to be interpolated, i, j represent the position of the row and column, x i ,y j represents the coordinate values ​​of the 4×4 area points near the pixel to be interpolated, and W represents the basis function of the interpolation cubic function;

[0070] S2-2, training a neural network model of high-dimensional features of remote sensing image sequence, extracting the temporal features of the low-resolution image sequence, the neural network model includes: using a long short-term memory network LSTM model, using two LSTM networks as encoders and decoders for training respectively, the input of the decoder is the change characteristics of the river flow and water level extracted from the remote sensing image sequence, by calculating the L2 norm loss between the decoder output and input, and then performing feedback propagation to optimize the encoder and decoder, the calculation formula is as follows:

[0071]

[0072] Among them, the input of the encoder is the vector P, p i Represents its i-th component, and the output of the decoder is vector Q, q i represents its i-th component;

[0073] S2-3, input the low-resolution image, extract image features through a convolutional neural network, and obtain a feature map sequence of the low-resolution image, wherein the convolutional neural network includes: 3 feature extraction modules, each module includes 3 convolutional layers, all convolutional layers share the same 3×3 convolution kernel size, the step size and edge padding are set to 1, and the forward propagation is represented as a series of feature maps. The feature extraction network is composed of a plurality of residual blocks cascaded, and the residual block is composed of two convolutions with the same kernel size and the same number of filters. The formula is as follows:

[0074] X=H l([x0,x1,x3])

[0075] Among them, H(·) represents the nonlinear transformation function, using the structure of batch normalization + ReLU activation function + 3×3 convolution, and X represents the feature x for each feature extraction module. l , the features of the three modules are fused as the final output features.

[0076] Step S3 is as follows:

[0077] S3-1, input the high-resolution image HR, and obtain four wavelet coefficient images through two-dimensional wavelet transform 2DDWT. The specific formula is as follows:

[0078] HRWav={H LL ,H LH ,H HL ,H HH}: =2DDWT{HR}

[0079] Among them, HRWav represents the wavelet of high-resolution image, H LL Represents the wavelet coefficients of the low-frequency component, H LH The wavelet coefficients representing the horizontal low-frequency and vertical high-frequency components, H HL The wavelet coefficients representing the horizontal high-frequency and vertical low-frequency components, H HH The wavelet coefficients representing the diagonal high-frequency components;

[0080] S3-2, input the obtained wavelet coefficient H of the low-frequency component LL , the wavelet coefficients H of the horizontal low-frequency and vertical high-frequency components LH , the wavelet coefficients H of the horizontal high frequency and vertical low frequency components HL and the wavelet coefficients H of the diagonal high-frequency components HH , and feature extraction is performed on them respectively. In order to map the features to the same feature space, the feature extraction network same as the convolutional neural network is adopted, that is, the network consists of 3 feature extraction modules, each module contains 3 convolutional layers, all convolutional layers share the same 3×3 convolution kernel size, and the step size and edge padding are set to 1.

[0081] Step S4 is as follows:

[0082] S4-1, respectively convert the four wavelet coefficients H LL , H LH , H HL and H HH Each wavelet coefficient image feature in the image feature is spliced ​​with the extracted feature of the low-resolution image and the temporal feature of the low-resolution image sequence, and the three features are fused into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps;

[0083] S4-2, using the four new wavelet coefficient feature maps obtained as input, because the input wavelet coefficient feature map is formed by splicing three features and its dimension is large, a 1×1 convolution is used to reduce the feature dimension, and then the convolutional neural network is used to obtain a wavelet coefficient feature map of the same size as the original low-resolution wavelet coefficient feature map. To ensure that the output wavelet coefficient feature map has the same size as the original low-resolution image, the convolutional neural network used is composed of three feature extraction modules, each module contains three convolution layers, all convolution layers share the same 3×3 convolution kernel size, and the step size and edge padding are set to 1.

[0084] Step S5 is as follows:

[0085] S5-1, the network adopts a flexible and constrained loss function, and uses two losses, wavelet coefficient loss and image space pixel loss, to robustly calculate the similarity between the output super-resolution image and the input high-resolution image. The specific formula is as follows:

[0086] L=αL w +(1-α)L i

[0087] Among them, L represents the loss function of the entire network, L w represents the wavelet coefficient loss, L i represents the image space pixel loss, α represents the weight of the two loss functions, and in this embodiment, it is 0.5;

[0088] S5-2, wavelet coefficient loss refers to the similarity comparison of all wavelet coefficient images of the wavelet coefficient feature map obtained with the same size as the original low-resolution wavelet coefficient feature map and the sub-band image obtained by wavelet transforming the input high-resolution image, using the most commonly used pixel-by-pixel mean square error loss function in the image space, and operating the wavelet coefficients in the same way. The loss function is expressed using the L2 norm, and the calculation formula is as follows:

[0089]

[0090] in, represents the wavelet coefficients of a wavelet coefficient feature map of the same size as the original low-resolution wavelet coefficient feature map obtained by the convolutional neural network, represents the wavelet coefficients obtained by wavelet transform of the input high-resolution image, n represents the size of batchsize, i represents the i-th image in the batch processed images, and j represents the j-th coefficient in the wavelet coefficient sequence;

[0091] S5-3, the loss calculation in the image space refers to a balance between the texture and smoothness of the super-resolution image and the original high-resolution image, expressed using the L2 norm, and the calculation formula is as follows:

[0092]

[0093] Among them, I S represents the super-resolution image, I H represents the input high-resolution image; n represents the size of batchsize, and i represents the i-th image in the batch image;

[0094] S5-4, inputting a low-resolution image of the river to be processed in a certain time period into the super-resolution image generation model, thereby obtaining a corresponding high-resolution image.

[0095] Accordingly, the present invention also provides a river flow and water level remote sensing image super-resolution system based on wavelet domain, such as Figure 2 As shown, including:

[0096] The data set screening unit 1 is used to import a river flow and water level data set consisting of multiple sets of remote sensing image sequences, and screen and input a low-resolution image sequence according to the river to be predicted and the time period to be predicted.

[0097] Specifically, in an embodiment of the present invention, a river flow and water level dataset is imported, and the river flow and water level dataset is composed of multiple groups of remote sensing image sequences, each group of image sequences represents multiple low-resolution images of river flow and water level of a certain river in a time period and corresponding high-resolution images. Given the river to be predicted and the time period to be predicted, the low-resolution image sequence is screened and input.

[0098] The low-resolution image sequence data processing unit 2 is used to first upsample the low-resolution image sequence using a bicubic interpolation algorithm to obtain a high-resolution image of a specified resolution, then train a neural network model of high-dimensional features of a remote sensing image sequence to extract the temporal features of the low-resolution image sequence, and finally use a convolutional neural network to extract the features of the low-resolution image to obtain the low-resolution image feature map sequence.

[0099] Specifically, a bicubic interpolation algorithm is used to upsample the low-resolution image to obtain a high-resolution image of a specified resolution. Using the bicubic interpolation image upsampling method, the value of the function f at the point (x, y) can be obtained by the weighted average of the sixteen nearest sampling points in the rectangular grid; a neural network model of high-dimensional features of a remote sensing image sequence is trained to extract the temporal features of the low-resolution image sequence. The neural network model is implemented using an LSTM model, and two LSTM networks are used as an encoder and a decoder for training respectively. The input of the decoder is the change characteristics of the river flow and water level extracted from the remote sensing image sequence. The solution is calculated by The L2 norm loss between the encoder output and the input is then fed back and propagated to optimize the encoder and decoder; the low-resolution image is input, image features are extracted through a convolutional neural network, and a feature map sequence of the low-resolution image is obtained, the convolutional neural network is composed of three feature extraction modules, each module contains three convolutional layers, all convolutional layers share the same 3×3 convolution kernel size, the step size and edge padding are set to 1, the forward propagation is represented as a series of feature maps, the feature extraction network is composed of a plurality of residual blocks cascaded, the residual block is composed of two convolutions with the same kernel size and the same number of filters, and the features of the three modules are fused as the final output features.

[0100] The wavelet coefficient image feature extraction unit 3 is used for obtaining the wavelet coefficients of the high-resolution image into a low-resolution image by using wavelet transform, obtaining four wavelet coefficient images and performing feature extraction on each of the wavelet coefficient images.

[0101] Specifically, the high-resolution image HR is input, and four wavelet coefficient images are obtained by two-dimensional wavelet transform 2DDWT. The formula is as follows:

[0102] HRWav={H LL ,H LH ,H HL ,H HH}: =2DDWT{HR}

[0103] Among them, HRWav represents the wavelet of high-resolution image, H LL Represents the wavelet coefficients of the low-frequency component, H LH The wavelet coefficients representing the horizontal low-frequency and vertical high-frequency components, H HL The wavelet coefficients representing the horizontal high-frequency and vertical low-frequency components, H HH Represents the wavelet coefficient of the diagonal high-frequency component; input the wavelet coefficient H of the low-frequency component obtained LL , the wavelet coefficients H of the horizontal low-frequency and vertical high-frequency components LH , the wavelet coefficients H of the horizontal high frequency and vertical low frequency components HL and the wavelet coefficients H of the diagonal high-frequency components HH, and feature extraction is performed on them respectively. In order to map the features to the same feature space, the feature extraction network same as the convolutional neural network is adopted, that is, the network consists of 3 feature extraction modules, each module contains 3 convolutional layers, all convolutional layers share the same 3×3 convolution kernel size, and the step size and edge padding are set to 1.

[0104] The wavelet coefficient feature map fusion unit 4 is used to respectively splice each of the wavelet coefficient image features with the features and time series features of the low-resolution image, and fuse the three features into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps, and obtaining a wavelet coefficient feature map of the same size as the original low-resolution wavelet coefficient feature map.

[0105] Specifically, the four wavelet coefficients H are respectively LL , H LH , H HL and H HH Each wavelet coefficient image feature in the image feature is spliced ​​with the extracted feature of the low-resolution image and the temporal feature of the low-resolution image sequence, and the three features are fused into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps; the four new wavelet coefficient feature maps are used as input, because the input wavelet coefficient feature map is spliced ​​by three features and its dimension is large, so a 1×1 convolution is used to reduce the feature dimension, and then the convolutional neural network is used to obtain a wavelet coefficient feature map of the same size as the original low-resolution wavelet coefficient feature map. In order to ensure that the output wavelet coefficient feature map is the same size as the original low-resolution image, the convolutional neural network used is composed of three feature extraction modules, each module contains three convolution layers, all convolution layers share the same 3×3 convolution kernel size, and the step size and edge padding are set to 1.

[0106] The super-resolution image generation unit 5 is used to add the four groups of low-resolution wavelet coefficient feature maps and the wavelet coefficients, and then use the inverse wavelet transform to generate a corresponding super-resolution image from the wavelet coefficient image to obtain the final result, and calculate the loss function according to the similarity between the super-resolution image and the input high-resolution image, and train and optimize the neural network according to the loss until the result of the loss function converges, thereby establishing a super-resolution image generation model, and inputting a low-resolution image of the river to be processed in a certain time period into the super-resolution image generation model to obtain the corresponding high-resolution image.

[0107] Specifically, the network adopts a flexible and constrained loss function, and uses two losses, wavelet coefficient loss and image space pixel loss, to robustly calculate the similarity between the output super-resolution image and the input high-resolution image; wavelet coefficient loss refers to the similarity comparison between all wavelet coefficient images of the wavelet coefficient feature map obtained with the same size as the original low-resolution wavelet coefficient feature map and the sub-band image obtained by wavelet transform of the input high-resolution image; the loss calculation in image space refers to a balance between the texture and smoothness of the super-resolution image and the original high-resolution image.

[0108] Therefore, the present invention applies a two-dimensional image deep learning algorithm to obtain a high-resolution image sequence, reduces data transmission and storage resources, and has the characteristics of low cost and high efficiency; the input and output between the various parts of the invention are interconnected, and different features are extracted and integrated to improve efficiency, so that the generated results are more in line with people's expectations; the invention uses a neural network method to generate a super-resolution image sequence through a coding and decoding structure, which can guide the neural network to enhance the low-resolution local detail features to the maximum extent; when using a convolutional neural network to super-resolution the original low-resolution image sequence, the texture information can be retained to the maximum extent.

[0109] The above is a detailed introduction to the wavelet-domain-based river flow and water level remote sensing image super-resolution method and system provided in the embodiments of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A wavelet-based river flow and water level remote sensing image super-resolution method, characterized in that: The method comprises: Import a river flow and water level data set consisting of multiple sets of remote sensing image sequences, and filter and input low-resolution image sequences according to the river to be predicted and the time period to be predicted; The low-resolution image sequence is first upsampled using a bicubic interpolation algorithm to obtain a high-resolution image of a specified resolution, and then a neural network model of high-dimensional features of a remote sensing image sequence is trained to extract temporal features of the low-resolution image sequence, and finally a convolutional neural network is used to extract features of the low-resolution image to obtain a low-resolution image feature map sequence; Using wavelet transform to obtain wavelet coefficients of the high-resolution image into a low-resolution image, obtaining four wavelet coefficient images and performing feature extraction on each wavelet coefficient image; The features of each of the wavelet coefficient images are spliced ​​with the temporal features of the low-resolution image sequence and the low-resolution image feature map sequence respectively, and the three features are fused into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps, and a low-resolution wavelet coefficient feature map of the same size as the original low-resolution image is generated by dimensionality reduction and using a convolutional neural network; Adding the four groups of low-resolution wavelet coefficient feature maps and the wavelet coefficients, and then using inverse wavelet transform to generate a corresponding super-resolution image from the wavelet coefficient image to obtain a final result, and calculating a loss function based on the similarity between the super-resolution image and the high-resolution image, and training and optimizing the neural network based on the loss until the result of the loss function reaches convergence, thereby establishing a super-resolution image generation model; By inputting a low-resolution image of the river to be processed in a certain period of time into the super-resolution image generation model, a corresponding high-resolution image is obtained.

2. The wavelet-domain-based river flow and water level remote sensing image super-resolution method according to claim 1, characterized in that: The method of importing a river flow and water level data set composed of multiple sets of remote sensing image sequences, and screening and inputting a low-resolution image sequence according to the river to be predicted and the time period to be predicted, is specifically as follows: Import the river flow and water level dataset, which is composed of multiple groups of remote sensing image sequences. Each group of image sequences represents multiple low-resolution images of river flow and water level in a certain river in a period of time and the corresponding high-resolution images. Given the river to be predicted and the time period to be predicted, filter and input the low-resolution image sequence.

3. The wavelet-domain-based river flow and water level remote sensing image super-resolution method according to claim 1, characterized in that: The low-resolution image sequence is first upsampled using a bicubic interpolation algorithm to obtain a high-resolution image of a specified resolution, and then a neural network model of high-dimensional features of a remote sensing image sequence is trained to extract the temporal features of the low-resolution image sequence, and finally a convolutional neural network is used to extract the features of the low-resolution image to obtain the low-resolution image feature map sequence, specifically: The bicubic interpolation algorithm is used to upsample the low-resolution image to obtain a high-resolution image of a specified resolution. The value of the function f at the point (x, y) is obtained by weighted averaging the nearest sixteen sampling points in a rectangular grid using the bicubic interpolation image upsampling method. Specifically, for the interpolated pixel point (x, y), the 4×4 neighboring points (x i ,y j ), the interpolation calculation formula is as follows: Among them, x, y represent the coordinates of the pixel to be interpolated, i, j represent the position of the row and column, x i ,y j represents the coordinate values ​​of the 4×4 area points near the pixel to be interpolated, and W represents the basis function of the interpolation cubic function; A neural network model of high-dimensional features of a remote sensing image sequence is trained to extract the temporal features of the low-resolution image sequence. The neural network model includes: using a long short-term memory network LSTM model, using two LSTM networks as an encoder and a decoder for training respectively, the input of the decoder is the change characteristics of the river flow and water level extracted from the remote sensing image sequence, and the L2 norm loss between the decoder output and input is calculated, and then feedback propagation is performed to optimize the encoder and decoder. The calculation formula is as follows: Among them, the input of the encoder is the vector P, p i Represents its i-th component, and the output of the decoder is vector Q, q i represents its i-th component; The low-resolution image is input, and image features are extracted through a convolutional neural network to obtain a feature map sequence of the low-resolution image. The convolutional neural network includes: 3 feature extraction modules, each module contains 3 convolutional layers, all convolutional layers share the same 3×3 convolution kernel size, the step size and edge padding are set to 1, and the forward propagation is represented as a series of feature maps. The feature extraction network is composed of a plurality of residual blocks cascaded, and the residual block is composed of two convolutions with the same kernel size and the same number of filters. The formula is as follows: X=H([x0,x1,x3]) Among them, H(·) represents the nonlinear transformation function, which adopts the structure of batch normalization + ReLU activation function + 3×3 convolution. X represents the features x0, x1, x3 of each feature extraction module, and the features of the three modules are fused as the final output features.

4. The wavelet-domain-based river flow and water level remote sensing image super-resolution method according to claim 1, characterized in that: The wavelet transform is used to obtain the wavelet coefficients of the high-resolution image into a low-resolution image, four wavelet coefficient images are obtained, and feature extraction is performed on each wavelet coefficient image, specifically: The high-resolution image HR is input, and four wavelet coefficient images are obtained through two-dimensional wavelet transform 2DDWT. The specific formula is as follows: HRWav={H LL ,H LH ,H HL ,H HH }:=2DDWT{HR} Among them, HRWav represents the wavelet of high-resolution image, HLL represents the wavelet coefficient of low-frequency component, and H LH The wavelet coefficients representing the horizontal low-frequency and vertical high-frequency components, H HL The wavelet coefficients representing the horizontal high-frequency and vertical low-frequency components, H HH The wavelet coefficients representing the diagonal high-frequency components; Input the obtained wavelet coefficient H of the low-frequency component LL , the wavelet coefficients H of the horizontal low-frequency and vertical high-frequency components LH , the wavelet coefficients H of the horizontal high frequency and vertical low frequency components HL and the wavelet coefficients H of the diagonal high-frequency components HH , and feature extraction is performed on them respectively. In order to map the features to the same feature space, the feature extraction network same as the convolutional neural network is adopted, that is, the network consists of 3 feature extraction modules, each module contains 3 convolutional layers, all convolutional layers share the same 3×3 convolution kernel size, and the step size and edge padding are set to 1.

5. The wavelet-domain-based river flow and water level remote sensing image super-resolution method according to claim 4, characterized in that: The features of each of the wavelet coefficient images are spliced ​​with the temporal features of the low-resolution image sequence and the low-resolution image feature map sequence, and the three features are fused into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps, and generating a low-resolution wavelet coefficient feature map of the same size as the original low-resolution image by dimensionality reduction and using a convolutional neural network, specifically: The four wavelet coefficients H LL , H LH , H HL and H HH Each wavelet coefficient image feature in the image feature is spliced ​​with the extracted feature of the low-resolution image and the temporal feature of the low-resolution image sequence, and the three features are fused into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps; The four new wavelet coefficient feature maps are used as input. Since the input wavelet coefficient feature map is formed by splicing three features and has a large dimension, a 1×1 convolution is used to reduce the feature dimension. Then, the convolutional neural network is used to obtain a wavelet coefficient feature map of the same size as the original low-resolution wavelet coefficient feature map. To ensure that the output wavelet coefficient feature map has the same size as the original low-resolution image, the convolutional neural network is composed of three feature extraction modules, each module contains three convolutional layers, and all convolutional layers share the same 3×3 convolution kernel size, and the step size and edge padding are set to 1.

6. The wavelet-domain-based river flow and water level remote sensing image super-resolution method according to claim 1, characterized in that: The four groups of low-resolution wavelet coefficient feature maps and the wavelet coefficients are added, and then the wavelet coefficient image is transformed into a corresponding super-resolution image by inverse wavelet transform to obtain the final result, and the loss function is calculated according to the similarity between the super-resolution image and the high-resolution image, and the neural network is trained and optimized according to the loss until the result of the loss function converges, specifically: The network adopts a flexible and constrained loss function, and uses two losses, wavelet coefficient loss and image space pixel loss, to robustly calculate the similarity between the super-resolution image and the high-resolution image. The specific formula is as follows: L=αL w +(1-α)L i Among them, L represents the loss function of the entire network, L w represents the wavelet coefficient loss, L i represents the pixel loss in image space, and α represents the weight of the two loss functions; Wavelet coefficient loss refers to the similarity comparison of all wavelet coefficient images of the wavelet coefficient feature map with the same size as the original low-resolution wavelet coefficient feature map and the sub-band image obtained by wavelet transforming the high-resolution image. The most commonly used pixel-by-pixel mean square error loss function in the image space is used to operate the wavelet coefficients in the same way. The loss function is expressed by the L2 norm, and the calculation formula is as follows: in, represents the wavelet coefficients of a wavelet coefficient feature map of the same size as the original low-resolution wavelet coefficient feature map obtained by the convolutional neural network, represents the wavelet coefficients obtained by wavelet transform of the high-resolution image, n represents the size of batchsize, i represents the i-th image in the batch processed images, and j represents the j-th coefficient in the wavelet coefficient sequence; The loss calculation in the image space refers to a balance between the texture and smoothness of the super-resolution image and the original high-resolution image, expressed using the L2 norm, and the calculation formula is as follows: Among them, I S represents the super-resolution image, I H represents the high-resolution image; n represents the size of batchsize, and i represents the i-th image in the batch processed image; By inputting a low-resolution image of the river to be processed in a certain period of time into the super-resolution image generation model, a corresponding high-resolution image is obtained.

7. A river flow and water level remote sensing image super-resolution system based on wavelet domain, characterized in that: The system comprises: A data set screening unit is used to import a river flow and water level data set consisting of multiple sets of remote sensing image sequences, and screen and input a low-resolution image sequence according to the river to be predicted and the time period to be predicted; A low-resolution image sequence data processing unit is used to first perform an upsampling operation on the low-resolution image sequence using a bicubic interpolation algorithm to obtain a high-resolution image of a specified resolution, then train a neural network model of high-dimensional features of a remote sensing image sequence to extract temporal features of the low-resolution image sequence, and finally use a convolutional neural network to extract features of the low-resolution image to obtain a low-resolution image feature map sequence; A wavelet coefficient image feature extraction unit, used for obtaining the wavelet coefficients of the high-resolution image into a low-resolution image by using wavelet transform, obtaining four wavelet coefficient images and performing feature extraction on each wavelet coefficient image; A wavelet coefficient feature map fusion unit is used to respectively splice the features of each of the wavelet coefficient images with the temporal features of the low-resolution image sequence and the low-resolution image feature map sequence, fuse the three features into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps, and generate a low-resolution wavelet coefficient feature map of the same size as the original low-resolution image by dimensionality reduction and using a convolutional neural network; The super-resolution image generation unit is used to add the four groups of low-resolution wavelet coefficient feature maps and the wavelet coefficients, and then use the inverse wavelet transform to generate a corresponding super-resolution image from the wavelet coefficient image to obtain the final result, and calculate the loss function according to the similarity between the super-resolution image and the high-resolution image, and train and optimize the neural network according to the loss until the result of the loss function converges, thereby establishing a super-resolution image generation model, and inputting the low-resolution image of the river to be processed in a certain time period into the super-resolution image generation model to obtain the corresponding high-resolution image.

8. The wavelet-domain-based river flow and water level remote sensing image super-resolution system according to claim 7, characterized in that: The data set screening unit needs to import a river flow and water level data set, which is composed of multiple groups of remote sensing image sequences. Each group of image sequences represents multiple low-resolution images of river flow and water level in a certain river in a period of time and corresponding high-resolution images. Given a river to be predicted and a time period to be predicted, the low-resolution image sequence is screened and input.

9. The wavelet-domain-based river flow and water level remote sensing image super-resolution system according to claim 7, characterized in that: The low-resolution image sequence data processing unit needs to use a bicubic interpolation algorithm to perform an upsampling operation on the low-resolution image to obtain a high-resolution image of a specified resolution. Using the bicubic interpolation image upsampling method, the value of the function f at the point (x, y) is obtained by weighted averaging the nearest sixteen sampling points in a rectangular grid; A neural network model of high-dimensional features of a remote sensing image sequence is trained to extract temporal features of the low-resolution image sequence. The neural network model comprises: an LSTM model is adopted, two LSTM networks are used as an encoder and a decoder for training respectively, the decoder input is the change feature of the river flow and water level extracted from the remote sensing image sequence, the L2 norm loss between the decoder output and the input is calculated, and then feedback propagation is performed to optimize the encoder and the decoder; the low-resolution image is input, image features are extracted through a convolutional neural network, and a feature map sequence of the low-resolution image is obtained. The convolutional neural network comprises: three feature extraction modules, each module comprises three convolutional layers, all convolutional layers share the same 3×3 convolutional kernel size, the step size and edge padding are set to 1, the forward propagation is represented as a series of feature maps, the feature extraction network is composed of a plurality of residual blocks cascaded, the residual block is composed of two convolutions with the same kernel size and the same number of filters, and the features of the three modules are fused as the final output features.

10. The wavelet-domain-based river flow and water level remote sensing image super-resolution system according to claim 7, characterized in that: The wavelet coefficient image feature extraction unit needs to input the high-resolution image HR, and obtain four wavelet coefficient images through two-dimensional wavelet transform 2DDWT. The specific formula is as follows: HRWav={H LL ,H LH ,H HL ,H HH }:=2DDWT{HR} Among them, HRWav represents the wavelet of high-resolution image, H LL Represents the wavelet coefficients of the low-frequency component, H LH The wavelet coefficients representing the horizontal low-frequency and vertical high-frequency components, H HL The wavelet coefficients representing the horizontal high-frequency and vertical low-frequency components, H HH Represents the wavelet coefficient of the diagonal high-frequency component; input the wavelet coefficient H of the low-frequency component obtained LL , the wavelet coefficients H of the horizontal low-frequency and vertical high-frequency components LH , the wavelet coefficients H of the horizontal high frequency and vertical low frequency components HL and the wavelet coefficients H of the diagonal high-frequency components HH , and feature extraction is performed on them respectively. In order to map the features to the same feature space, the feature extraction network same as the convolutional neural network is adopted, that is, the network consists of 3 feature extraction modules, each module contains 3 convolutional layers, all convolutional layers share the same 3×3 convolution kernel size, and the step size and edge padding are set to 1.

11. The wavelet-domain-based river flow and water level remote sensing image super-resolution system according to claim 10, characterized in that: The wavelet coefficient feature map fusion unit needs to respectively combine the four wavelet coefficients H LL , H LH , H HL and H HH Each wavelet coefficient image feature in the image feature is spliced ​​with the extracted features of the low-resolution image and the temporal features of the low-resolution image sequence, and the three features are fused into a feature map as a new wavelet coefficient feature map, thereby obtaining four new wavelet coefficient feature maps; the four new wavelet coefficient feature maps are used as input, because the input wavelet coefficient feature map is spliced ​​by three features and its dimension is large, so a 1×1 convolution is used to reduce the feature dimension, and then the convolutional neural network is used to obtain a wavelet coefficient feature map of the same size as the original low-resolution wavelet coefficient feature map. In order to ensure that the output wavelet coefficient feature map is the same size as the original low-resolution image, the convolutional neural network used is composed of three feature extraction modules, each module contains three convolution layers, all convolution layers share the same 3×3 convolution kernel size, and the step size and edge padding are set to 1.

12. The wavelet-domain-based river flow and water level remote sensing image super-resolution system according to claim 7, characterized in that: The super-resolution image generation unit requires the network to adopt a flexible and constrained loss function, and to use two losses, wavelet coefficient loss and image space pixel loss, to robustly calculate the similarity between the super-resolution image and the high-resolution image; the wavelet coefficient loss refers to the similarity comparison between all wavelet coefficient images of the obtained wavelet coefficient feature map of the same size as the original low-resolution wavelet coefficient feature map and the sub-band image obtained by wavelet transforming the input high-resolution image; the loss calculation in the image space refers to a balance between the texture and smoothness of the super-resolution image and the original high-resolution image.

Images