Remote sensing image colorization and super-resolution method and system based on neural network
Through the multi-scale residual block and attention mechanism based on neural network, combined with RGB color space, the problem of high and poor artificial intervention in colorization and super-resolution of remote sensing images is solved, and efficient image colorization and super-resolution is achieved Resolution processing improves the visual effects and detail reconstruction capabilities of images.
Patent Information
- Application Number
- CN202111670503.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-31
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-12-31
AI Technical Summary
The prior art has high artificial intervention and poor results in colorization and super-resolution of remote sensing images. Traditional methods are sensitive to image complexity and color changes, and the super-resolution method has low computational efficiency and is susceptible to amplification factors. .
Using a neural network-based method, the residual convolution branch network and improved U-net++ network are constructed, combined with RGB color space, and the end-to-end remote sensing image colorization and super-resolution are achieved, using multi-scale residual blocks and attention. The mechanism extracts high-dimensional feature information and performs image colorization and super-resolution processing.
With less information input, better image colorization and super-resolution performance are achieved. The resulting image is competitive in terms of visual effects and evaluation indicators, improving the readability and detail reconstruction capabilities of the image.
Smart Images

Figure CN114332625B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of computer vision, and in particular to a remote sensing image colorization and super-resolution method and system based on a neural network. Background Art
[0002] Remote sensing images are electromagnetic wave reflection photos of various objects such as jungles, ground fields, rectangular farmlands, houses, mountains, etc. The remote sensing information contained in them can not only truly and vividly display the current distribution of objects, but also discover the changes in the mutual influence between objects. Remote sensing images can effectively help people observe and understand the earth's information with their large amount of information, macroscopic, and dynamic monitoring characteristics. In recent years, remote sensing images have been combined with scene classification, target detection, target positioning and other technologies to provide people with many valuable practical applications.
[0003] Image colorization assigns a suitable color to each pixel of the target grayscale image, thereby increasing the color information of the image and improving the visual effect. In recent years, image colorization technology has been widely used in historical photo restoration, film and television production, and mural restoration. Therefore, applying colorization technology to remote sensing images can further improve their readability by enriching the visual information of remote sensing images.
[0004] At present, there are many solutions to the problem of image colorization, which can be roughly divided into three categories: graffiti-based colorization methods, reference image-based colorization methods, and learning-based colorization methods. (1) The graffiti-based colorization method requires manually adding color lines to the target image and then spreading the graffiti to the entire image through diffusion or variational methods to complete the image colorization. Due to the complexity and diversity of real images, it is difficult for humans to specify the appropriate color for each specific area through color lines. (2) The reference image-based colorization method transfers the color information of the reference image to the target grayscale image. However, this method requires manual setting of the reference image. In practical applications, it is very difficult to select a suitable color reference image. The above two methods not only require a lot of manual intervention, but also their performance is heavily dependent on the color of the manual graffiti and the color of the reference image. Incorrect graffiti information or color reference may cause unreasonable colorization effects. (3) With the introduction of deep learning, deep learning-based image colorization methods have emerged. This type of method uses models such as convolutional neural networks and generative adversarial networks to directly convert the input grayscale image into a color image. It overcomes the need for manual intervention in traditional colorization methods and saves a lot of resources.
[0005] Zhang et al. regard image colorization as a classification problem, input image brightness information into the CNN model to obtain feature information, and then obtain chromaticity information based on color distribution probability. The network structure of this method is relatively simple, the saturation of the colored image is low, and color overflow occurs. Iizuka et al. introduced a global feature extraction network and combined prior information to complete colorization, but this method fails on images with no obvious color changes. Yoo et al. proposed a novel enhanced memory network and used conditional GAN to achieve small sample colorization, but GAN is prone to model collapse during the learning process, such as: the generator only produces one result, which causes the discriminator to be unable to effectively supervise the generator to continue learning, and thus cannot continue training. Su et al. used the pre-trained Mask R-CNN to segment the target image and the background image, then colorized them separately, and finally merged them into a complete color image. This method is very dependent on the segmentation processing algorithm. For images that fail to segment, the colorization effect is average, and coloring the target image and the background image separately will also lead to an overall color transition that is not harmonious.
[0006] Image super-resolution is to convert LR images of the same scene into HR images using a specific algorithm. HR images have high pixel density and can obtain more detailed features of the image. Super-resolution technology has been widely used in image compression, medical imaging, remote sensing imaging, public security and other fields due to its flexibility and high practicality. It is currently a research hotspot in the field of image processing.
[0007] Image super-resolution methods can be roughly divided into three categories: (1) Interpolation-based super-resolution algorithms. This type of algorithm is relatively simple, but the images reconstructed using this type of algorithm have artifacts and ringing. (2) Modeling-based super-resolution algorithms. This type of algorithm has better reconstruction effect than the interpolation method, but the model operation efficiency is low and is greatly affected by the magnification factor. (3) As deep learning technology is applied to various artificial intelligence tasks, super-resolution technology based on deep learning has also been actively developed. At present, a variety of super-resolution methods based on deep learning have been proposed. These methods mainly have three strategies: a. Pre-sampling: First, the image is directly upsampled to a high resolution, and then fine-tuned using a deep neural network. Although the size of the input low-resolution image is not restricted, the pre-sampling method will produce side effects such as noise amplification and blurring when expanding the low-resolution image. b. Progressive upsampling super-resolution: The upsampling block is integrated in the middle or end of the network. Since this method only incurs a large computational cost in low-dimensional space, it has been widely used. c. Up-down sampling iterative super-resolution: Alternately connect the upsampling layer and the downsampling layer, and use all intermediate processes to reconstruct the super-resolution result.
[0008] Dong et al. proposed a convolutional neural network model SRCNN, which uses bicubic interpolation to enlarge the low-resolution image to the target size, then fits the nonlinear mapping through a three-layer convolutional network, and finally outputs the high-resolution image result. Inspired by ResNet, residual learning is also introduced into the VDSR super-resolution network. Kim et al. proposed a deep recursive convolutional network DRCN model. Lai et al. combined the traditional image algorithm Laplacian pyramid with deep learning and proposed a network LapSRN to achieve super-resolution. Shi et al. used sub-pixel up-convolution instead of deconvolution blocks in the sub-pixel convolution network ESPCN. Later, Lim et al. proposed an enhanced deep super-resolution network EDSR with good performance. The algorithm removes the batch normalization layer from the original residual block and improves the computational efficiency. The deep residual channel attention network RCAN was proposed by Zhang et al., which is the first time that the channel attention mechanism is applied to the super-resolution network.
[0009] From the above analysis, it can be seen that colorization and super-resolution of remote sensing images are important aspects of image processing. The CSRDNN proposed in the present invention adopts RGB color space, predicts RGB color images based on grayscale images, and constructs an end-to-end model, thereby avoiding color space transformation. Summary of the invention
[0010] The purpose of the present invention is to provide a remote sensing image colorization and super-resolution method and system based on neural network, which can improve the effect of remote sensing image colorization and super-resolution.
[0011] To achieve the above object, the present invention provides the following solutions:
[0012] A remote sensing image colorization and super-resolution method based on a neural network, comprising:
[0013] Acquire remote sensing images to be processed;
[0014] Preprocessing the remote sensing image to obtain a low-resolution remote sensing image;
[0015] Construct a residual convolution branch network;
[0016] Constructing an MRB network according to the residual convolution branch network;
[0017] Extracting high-dimensional feature information of the low-resolution remote sensing image according to the MRB network;
[0018] Get the improved U-net++ network;
[0019] Processing the high-dimensional feature information according to the improved U-net++ network to obtain a feature map;
[0020] The feature map is processed by using convolution calculation to obtain a high-resolution color image.
[0021] Optionally, the preprocessing includes downsampling and grayscaling.
[0022] Optionally, a BiCubic interpolation algorithm is used to downsample the remote sensing image.
[0023] Optionally, the MRB network organizes four residual convolution branch networks in parallel, and a skip connection is introduced in each of the residual convolution branch networks to further maintain the gradient.
[0024] Optionally, extracting high-dimensional feature information of the low-resolution remote sensing image according to the MRB network specifically includes:
[0025] Inputting the low-resolution remote sensing image into the MRB network to obtain first feature information;
[0026] The first feature information is input into the MRB network again to obtain the high-dimensional feature information.
[0027] Optionally, the improved U-net++ network includes an IDB subnetwork, an IMUB subnetwork and an ATB subnetwork, the IDB subnetwork is used for feature extraction, the IMUB subnetwork is used for feature reconstruction, and the ATB subnetwork is used for skip connection.
[0028] Optionally, the remote sensing image is grayscaled using a cvtColor function.
[0029] A remote sensing image colorization and super-resolution system based on a neural network, comprising:
[0030] An image acquisition module, used for acquiring remote sensing images to be processed;
[0031] A preprocessing module, used for preprocessing the remote sensing image to obtain a low-resolution remote sensing image;
[0032] Residual convolution branch network construction module, used to construct residual convolution branch network;
[0033] An MRB network construction module is used to construct an MRB network according to the residual convolution branch network;
[0034] A feature extraction module, used for extracting high-dimensional feature information of the low-resolution remote sensing image according to the MRB network;
[0035] Network acquisition module, used to obtain the improved U-net++ network;
[0036] A feature map module, used for processing the high-dimensional feature information according to the improved U-net++ network to obtain a feature map;
[0037] The convolution calculation module is used to process the feature map using convolution calculation to obtain a high-resolution color image.
[0038] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0039] The present invention discloses a remote sensing image colorization and super-resolution method and system based on a neural network. The remote sensing image can simultaneously achieve image colorization and super-resolution through an end-to-end colorization and super-resolution deep neural network. The whole process includes a feature extraction stage and a structure recovery stage. First, the low-resolution remote sensing image is passed through a feature extraction network composed of residual receptive domain blocks MRB to obtain extracted high-dimensional feature information. Secondly, the high-dimensional feature information is passed through a color information reconstruction network with U-net++ as the basic structure to achieve image colorization and super-resolution. The present invention can achieve better image colorization and super-resolution performance with less input information, and the obtained image has strong competitiveness in terms of visual effects and evaluation indicators. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0041] Figure 1 The present invention is a flowchart of a remote sensing image colorization and super-resolution method based on a neural network;
[0042] Figure 2 This is a schematic diagram of the MRB network structure of the present invention;
[0043] Figure 3 This is a schematic diagram of the improved U-net++ network structure of the present invention;
[0044] Figure 4 This is a schematic diagram of the IDB sub-network structure of the present invention;
[0045] Figure 5 This is a schematic diagram of the IMUB sub-network structure of the present invention;
[0046] Figure 6 ATB sub-network structure schematic diagram of the present invention;
[0047] Figure 7This is a comparison chart of the results of the present invention using 4 times super-resolution magnification on the NWPU-45 dataset;
[0048] Figure 8 This is a comparison chart of the super-resolution magnification 8 times result on the AID dataset of the present invention;
[0049] Fig. 9 This is a comparison chart of the colorization results of the present invention on the NWPU-45 dataset;
[0050] Fig.10 This is a comparison chart of the super-resolution magnification of the present invention on the NWPU-45 data set by 2 times and the colorization result;
[0051] Fig.11 This is a module diagram of the remote sensing image colorization and super-resolution system based on neural network of the present invention. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] The purpose of the present invention is to provide a remote sensing image colorization and super-resolution method and system based on neural network, which can improve the effect of remote sensing image colorization and super-resolution.
[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0055] Figure 1 The flowchart of the remote sensing image colorization and super-resolution method based on neural network of the present invention is as follows: Figure 1 As shown, a remote sensing image colorization and super-resolution method based on a neural network includes:
[0056] Step 101: Acquire a remote sensing image to be processed.
[0057] The remote sensing images are existing data sets, and the images in these data sets are used as labeled remote sensing images.
[0058] Step 102: preprocessing the remote sensing image to obtain a low-resolution remote sensing image.
[0059] Among them, preprocessing includes downsampling and grayscale processing:
[0060] S2-1 Downsampling operation: Use the BiCubic interpolation algorithm to downsample the high-resolution remote sensing image to obtain a low-resolution remote sensing image. It uses the grayscale values of 16 points around the sampling point for cubic interpolation, which not only needs to consider the grayscale values of the 4 directly adjacent points, but also needs to consider the grayscale value change rate between each adjacent point. Assume that the size of the source image A is m×n, and the size of the scaled target image B is M×N. Then according to the scaling ratio, we can get the corresponding coordinates of B(X, Y) on A as A(x, y) = A(X*(m / M), Y*(n / N)). At this time, the obtained x and y are small values. We can find the nearest 16 pixel points through this small value coordinate, and use the BiCubic basis function to find the corresponding weight of each pixel, and finally obtain B(X, Y).
[0061] S2-2 Grayscale processing: True color means that in each pixel value that makes up a color image, there are three primary color components, R, G, and B. Each primary color component directly determines the primary color intensity of the display device. The color generated in this way is called true color, which is the real original color. Therefore, the color image has 3 channels. Grayscale image is an image with only one grayscale value at the corresponding position, so it has only one channel. The image is composed of a pixel matrix, and the operation on the image is the operation on the pixel matrix. As long as the position of the pixel is found in this pixel matrix, such as the xth row and the yth column, the position of the pixel in this pixel matrix can be expressed as (x, y), because the color of a pixel is represented by three color variables of red, green, and blue (R, G, B). By assigning values to these three variables, the color of the pixel can be changed. We use the cvtColor function of the opencv library to make the three color variables of each pixel in the image equal, that is, R=G=B, and finally get the grayscale remote sensing image.
[0062] Step 103: Construct a residual convolution branch network.
[0063] Step 104: construct an MRB network according to the residual convolution branch network.
[0064] Step 105: extracting high-dimensional feature information of the low-resolution remote sensing image according to the MRB network.
[0065] Step 103 to step 105 specifically include:
[0066] The high-resolution remote sensing images are passed through two Multi-scale Residual Block (MRB) networks to extract multi-scale high-dimensional feature information from the input signal.
[0067] The MRB network organizes four residual convolution branches in parallel, and each feature map has a different receptive field. Skip-layer connections are introduced in each branch to further maintain the gradient. Compared with the initial InceptionV1 and V3, the MRB network removes the pooling layer and deepens the network depth. Its structure is as follows: Figure 2 shown.
[0068] We add a residual structure in each branch to add the reconstructed information to the original information.
[0069] S5-1: MRB uses a 1×1 convolution to determine the output dimension; the input of MRB includes three dimensions: width, height, and channel. The convolution kernel can be regarded as the weighted sum of a certain local area. It corresponds to local perception. It starts from the local area to understand the image, and then combines all local information, which corresponds to the convolution operation. The sizes of convolution kernels are generally 1×1, 3×3 and 5×5, and the number of convolution kernels corresponds to the number of desired output channels. When the number of 1×1 convolution kernels is less than the number of channels of the input feature, it plays a role in dimensionality reduction; when the number of 1×1 convolution kernels is greater than the number of channels of the input feature, it plays a role in dimensionality increase. For example, when the input is 6×6×32, the form of 1×1 convolution is 1×1×32. When there is only one 1×1 convolution kernel, the output is 6×6×1, which achieves the effect of dimensionality reduction. When there are 64 1×1 convolution kernels, the output is 6×6×64, which achieves the effect of dimensionality increase.
[0070] S5-2: Send the features to different branches. Each branch consists of a different number of 3×3 convolution blocks and residual structures to extract feature information of different scales. The first branch uses a 1×1 convolution, the second branch uses a 3×3 convolution, the third branch uses 2 3×3 convolutions, and the fourth branch uses 3 3×3 convolutions. The step size of all convolution operations is 1, and SAME padding is used. SAME means that the size of the output after convolution is consistent with the input size (step size is 1). The starting position of the convolution kernel is determined by aligning the center of the convolution kernel with the first point of the input, and then filling in the corresponding padding. In SAME mode, when the side length of the convolution kernel is an even number, the output size can be kept consistent with the input size by adding an extra row (column) of padding on one side. Therefore, four feature maps with the same size (width and height) can be output through four branches.
[0071] S5-3: Use Pytorch's concat() function to combine the feature information under the four receptive fields in the channel dimension. In PyTorch, tensors can be concatenated through the torch.concat(tensors, dim=1) function, where the parameter tensor stores all the sequences of tensors that need to be merged (any Python sequence object, such as lists, tuples, etc.), and the dim parameter specifies the dimension index that needs to be merged.
[0072] S5-4: According to the convolution operation principle, 1×1 convolution does not change the width and height of the output feature, and the number of channels of the output feature is equal to the number of convolution kernels. Therefore, a 1×1 convolution is used to change the number of channels of the feature information to restore the dimension of the feature and obtain high-dimensional feature information.
[0073] A large number of residual structures are introduced into the MRB network, and the residual structure is shown in formula (1). k represents a k×k convolution operation with an activation function, X represents the input, and Res k Represents a residual block.
[0074] Res k (X) = C k (X)+X (1)
[0075] Step 106: Obtain an improved U-net++ network.
[0076] Step 107: Process the high-dimensional feature information according to the improved U-net++ network to obtain a feature map.
[0077] Step 106-step 107 specifically include:
[0078] The high-dimensional feature information is input into the improved U-net++ network, which is recorded as IRA (Information Recovery Architecture) network for the convenience of description. Its structure is as follows Figure 3 As shown in the figure, the improved U-net++ network includes three sub-networks: IDB (Involution Based Down-sample Block), IMUB (Intensive Multi-scale Up-sample Block) and ATB (Attention Based Transfer Block). The IDB, IMUB and ATB sub-networks are used to further extract feature information and reconstruct high-resolution color information to obtain feature information with a size larger than the input image. The IDB sub-network is used for feature extraction, the IMUB sub-network is used for feature reconstruction, and the ATB sub-network is used for skip connections, aiming to transmit important information.
[0079] In the IRA network, the first layer of the network can stably transmit features, and the networks of other layers generate new information of different scales, and then continuously provide new information to the first layer. The IRA network does not use dense blocks, but uses attention mechanisms and residual structures to replace the original skip-layer structure. The root block and the end block are designed to face different computer vision tasks. The IRA network structure is shown in formula (2). Where X i,j represents the feature map, Represents an IDB block, Indicates ATB block, Represents an IMUB chunk.
[0080]
[0081] S7-1: IDB sub-network structure diagram as shown Figure 4 As shown in the figure, the overall structure of the module consists of three convolutional layers and corresponding activation layers. The first 1*1 convolutional layer is used to modify the channel dimension of the output feature, and the subsequent 3*3 convolutional layer is used to generate feature information that is downsampled by 2 times, and the last 3*3 convolution is used to reorganize the downsampled features and output the results.
[0082] S7-2: IMUB sub-network structure diagram as shown Figure 5 As shown in Figure 1, this module is built on the basis of the MRB structure and combined with the pyramid structure. The overall structure of IMUB consists of an extraction block that receives multi-scale features and a feature adjustment and combination module. The feature extraction block consists of several combined convolutional layers to extract features of different sizes and connect these features to output. The feature extraction block adopts the idea of the Laplacian pyramid network to obtain a set of multi-scale features by adjusting the size of the input features. They are then fed into the feature adjustment module and adjusted to the required dimension. These features are then combined and connected into the final result output.
[0083] S7-3: ATB sub-network structure diagram Figure 6 As shown in the figure, after the intermediate feature map is given, the ATB module uses the channel attention mechanism CA and the spatial attention mechanism SA to perform attention operations on different dimensions. Finally, the features output by the attention module are connected to the original input features to avoid gradient disappearance and output.
[0084] Step 108: Process the feature map using convolution calculation to obtain a high-resolution color image.
[0085] The feature size obtained in step 107 is larger than the size of the input image. In the RGB color space, a color image is represented by three channels, R, G, and B. Since the number of convolution kernels in the convolution operation can determine the number of channels of the output feature, step 108 uses a stack of convolution operations to change the number of channels of the feature obtained in step 107 to output a high-resolution color image.
[0086] In addition, in order to verify the effectiveness of each proposed module, the present invention conducts ablation experiments on the MRB network, IDB sub-network, IMRUB sub-network and ATB sub-network:
[0087] MRB network: In order to demonstrate the role of the MRB network, the original Inception structure is used instead of MRB for experiments.
[0088] IDB sub-network: To demonstrate the role of the IDB sub-network, experiments are performed using convolution operations instead of IDB.
[0089] IMRUB Subnetwork: To demonstrate the role of the IMRUB subnetwork, experiments are performed using convolution, sub-pixel upconvolution, and deconvolution operations instead of IMRUB.
[0090] ATB sub-network: In order to demonstrate the role of the ATB sub-network, experiments were conducted by removing the attention mechanism and replacing ATB with CBAM.
[0091] In order to verify the performance of the IRA network, the IRA network proposed in this paper is compared with advanced image super-resolution methods and image colorization methods respectively.
[0092] Figure 7 This is a comparison of the visual effects of different methods on the NWPU-45 dataset with 4 times super-resolution. The result generated by the present invention not only has the highest PSNR and SSIM, but also has accurate texture information and is closest to the original image.
[0093] Figure 8 This is a comparison of the visual effects of different methods on the AID dataset when the super-resolution is magnified 8 times. The result generated by the present invention not only has the highest PSNR and SSIM, but also can reconstruct reasonable details.
[0094] Fig. 9 This is the visual effect of different methods on the colorization of the NWPU-45 dataset. Fig. 9 As shown, the present invention can effectively reconstruct color information, and the experimental results are close to the real image.
[0095] Fig.10 This is the visual effect of the IRA network on the NWPU-45 dataset after super-resolution magnification by 2 times and colorization. Fig.10 As shown, reasonable details and color information can be reconstructed by adopting the present invention.
[0096] In order to evaluate the image quality from an objective perspective, the present invention uses some commonly used objective evaluation indicators to test the performance of the present invention, including: Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM). Table 1 shows the results of the ablation experiment, Table 2 shows the objective indicators of the super-resolution comparison experiment, and Table 3 shows the objective indicators of the colorization comparison experiment.
[0097] Table 1 Results of ablation experiments
[0098]
[0099] Table 2 Objective indicators of super-resolution comparison experiments
[0100]
[0101]
[0102] Table 3 Objective indicators of colorization comparison experiment
[0103] Methods PSNR SSIM Iizukas' 28.547 0.978 Isolas' 33.366 0.984 Sus' 32.972 0.988 Yoos' 31.068 0.979 Ours(depth=4) 34.597 0.989
[0104] As shown in Table 1, using the original Inception structure instead of MRB reduces the average PSNR. The results show that the introduction of MRB makes the perception field larger, thereby improving the performance of the model. Replacing the IDB module with a convolution operation will reduce the average PSNR value and the average SSIM value, which shows that the IDB module is more suitable for downsampling than the convolution operation. Replacing the IMRUB module with a convolution operation plus sub-pixel up-convolution or deconvolution reduces the average PSNR value and SSIM value, indicating that the introduction of the IMRUB module is useful. When the ATB module is replaced by a simple skip-layer structure or CBAM block, the average PSNR value and SSIM value are reduced, indicating that the ATB module transmits important feature information and improves model performance.
[0105] As shown in Table 2, compared with other super-resolution methods, the image processed by the present invention has the highest PSNR and SSIM. Compared with other super-resolution methods, IRA can reconstruct reasonable details and has more accurate texture information. Therefore, the IRA proposed in the present invention has good image super-resolution performance.
[0106] As shown in Table 3, compared with other colorization methods, the image processed by the present invention has the highest PSNR and SSIM. The IRA network can generate color images with higher colorization accuracy and better image quality. Therefore, the IRA network proposed in the present invention has good image colorization performance.
[0107] Based on the above method, the present invention also discloses a remote sensing image colorization and super-resolution system based on a neural network, such as Fig.11 As shown, including:
[0108] An image acquisition module 201 is used to acquire a remote sensing image to be processed;
[0109] A preprocessing module 202 is used to preprocess the remote sensing image to obtain a low-resolution remote sensing image;
[0110] A residual convolution branch network construction module 203, used to construct a residual convolution branch network;
[0111] An MRB network construction module 204 is used to construct an MRB network according to the residual convolution branch network;
[0112] A feature extraction module 205 is used to extract high-dimensional feature information of the low-resolution remote sensing image according to the MRB network;
[0113] A network acquisition module 206, used to acquire an improved U-net++ network;
[0114] A feature map module 207, used to process the high-dimensional feature information according to the improved U-net++ network to obtain a feature map;
[0115] The convolution calculation module 208 is used to process the feature map using convolution calculation to obtain a high-resolution color image.
[0116] In summary, the innovation of the present invention is:
[0117] 1. The present invention designs the same architecture SR and colorization for remote sensing images. Although SR and colorization are indeed different from each other, the proposed architecture can satisfy them. The effectiveness of the method is verified by analyzing satellite data.
[0118] 2. The present invention combines two visual problems, SR and colorization, into an integrated task, which means that the proposed architecture produces a color HR result given a gray LR image.
[0119] 3. Inspired by the Inception network, this paper proposes a multi-scale residual block (MRB) for feature extraction and reconstruction. Multi-scale paths and residual learning are introduced in MRB.
[0120] 4. Inspired by the attention mechanism module, this paper designs a new attention block (ATB) to transfer more important feature information by fusing the channel attention layer and the spatial attention layer.
[0121] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0122] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of the present invention. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A remote sensing image colorization and super-resolution method based on a neural network, characterized in that: include: Acquiring remote sensing images to be processed; Preprocessing the remote sensing image to obtain a low-resolution remote sensing image; Construct a residual convolution branch network; An MRB network is constructed according to the residual convolution branch network; the MRB network organizes four residual convolution branch networks in parallel, and a skip connection is introduced in each of the residual convolution branch networks to further maintain the gradient; Extracting high-dimensional feature information of the low-resolution remote sensing image according to the MRB network; Get the improved U-net++ network; The high-dimensional feature information is processed according to the improved U-net++ network to obtain a feature map; the improved U-net++ network includes an IDB subnetwork, an IMUB subnetwork and an ATB subnetwork, the IDB subnetwork is used for feature extraction, the IMUB subnetwork is used for feature reconstruction, and the ATB subnetwork is used for jump connection; specifically, the improved U-net++ network structure is shown in the following formula: Among them, X i,j represents the feature map, Represents an IDB block, Indicates ATB block, Represents an IMUB block; The overall structure of the IDB sub-network consists of three convolutional layers and corresponding activation layers. The first convolutional layer is a 1*1 convolutional layer, which is used to modify the channel dimension of the output feature; the second convolutional layer is a 3*3 convolutional layer, which is used to generate feature information that is downsampled by 2 times; the third convolutional layer is a 3*3 convolutional layer, which is used to reorganize the downsampled features and output the results; The IMUB sub-network structure is built on the basis of the MRB network structure and combined with the pyramid structure. Specifically, the overall structure of the IMUB sub-network consists of an extraction block that receives multi-scale features and a feature adjustment and combination module. The feature extraction block consists of multiple combined convolutional layers to extract features of different sizes and connect the extracted features of different sizes for output. Specifically, the feature extraction block adopts the idea of the Laplacian pyramid network to obtain a set of multi-scale features by adjusting the size of the input features. The multi-scale features are fed into the feature adjustment module, adjusted to the required dimensions, and then combined and connected into the final result output. After the intermediate feature map is given, the ATB sub-network uses the attention operations of the channel attention mechanism and the spatial attention mechanism in different dimensions to connect the features output by the attention module to the original input features to avoid gradient disappearance and output them; The feature map is processed by using convolution calculation to obtain a high-resolution color image.
2. The remote sensing image colorization and super-resolution method based on neural network according to claim 1, characterized in that: The preprocessing includes downsampling and gray-scaling.
3. The remote sensing image colorization and super-resolution method based on neural network according to claim 2, characterized in that: A BiCubic interpolation algorithm is used to downsample the remote sensing image.
4. The remote sensing image colorization and super-resolution method based on neural network according to claim 1, characterized in that: The step of extracting high-dimensional feature information of the low-resolution remote sensing image according to the MRB network specifically includes: Inputting the low-resolution remote sensing image into the MRB network to obtain first feature information; The first feature information is input into the MRB network again to obtain the high-dimensional feature information.
5. The remote sensing image colorization and super-resolution method based on neural network according to claim 1, characterized in that: The cvtColor function is used to grayscale the remote sensing image.
6. A remote sensing image colorization and super-resolution system based on neural network, characterized in that: include: An image acquisition module, used for acquiring remote sensing images to be processed; A preprocessing module, used for preprocessing the remote sensing image to obtain a low-resolution remote sensing image; Residual convolution branch network construction module, used to construct residual convolution branch network; An MRB network construction module is used to construct an MRB network according to the residual convolution branch network; the MRB network organizes four residual convolution branch networks in parallel, and a skip layer connection is introduced in each of the residual convolution branch networks to further maintain the gradient; A feature extraction module, used for extracting high-dimensional feature information of the low-resolution remote sensing image according to the MRB network; Network acquisition module, used to obtain the improved U-net++ network; The feature map module is used to process the high-dimensional feature information according to the improved U-net++ network to obtain a feature map; the improved U-net++ network includes an IDB subnetwork, an IMUB subnetwork and an ATB subnetwork, the IDB subnetwork is used for feature extraction, the IMUB subnetwork is used for feature reconstruction, and the ATB subnetwork is used for jump connection; specifically, the improved U-net++ network structure is shown in the following formula: Among them, X i,j represents the feature map, Represents an IDB block, Indicates ATB block, Represents an IMUB block; The overall structure of the IDB sub-network consists of three convolutional layers and corresponding activation layers. The first convolutional layer is a 1*1 convolutional layer, which is used to modify the channel dimension of the output feature; the second convolutional layer is a 3*3 convolutional layer, which is used to generate feature information that is downsampled by 2 times; the third convolutional layer is a 3*3 convolutional layer, which is used to reorganize the downsampled features and output the results; The IMUB sub-network structure is built on the basis of the MRB network structure and combined with the pyramid structure. Specifically, the overall structure of the IMUB sub-network consists of an extraction block that receives multi-scale features and a feature adjustment and combination module. The feature extraction block consists of multiple combined convolutional layers to extract features of different sizes and connect the extracted features of different sizes for output. Specifically, the feature extraction block adopts the idea of the Laplacian pyramid network to obtain a set of multi-scale features by adjusting the size of the input features. The multi-scale features are fed into the feature adjustment module, adjusted to the required dimensions, and then combined and connected into the final result output. After the intermediate feature map is given, the ATB sub-network uses the attention operations of the channel attention mechanism and the spatial attention mechanism in different dimensions to connect the features output by the attention module to the original input features to avoid gradient disappearance and output them; The convolution calculation module is used to process the feature map using convolution calculation to obtain a high-resolution color image.
