A method and device for enhancing remote sensing images of Mars

Abstract

The application discloses a Mars remote sensing image enhancement method and device, which comprises the following steps: acquiring a plurality of clear Mars remote sensing images shot by a Mars orbiter; using a sand dust rendering method to generate a plurality of corresponding images with sand dust for each clear Mars remote sensing image; training a deep neural network by using each plurality of clear sand dust images; inputting a real image with sand dust into the trained neural network, reducing the spatial resolution of the real image with sand dust and expanding the channel dimension through a convolutional neural network layer and two two-fold downsampling layers to obtain a feature map; sequentially passing the feature map through three dense deep convolution modules to extract features; and passing the extracted features through twice upsampling operations and once convolution operation. The application can be applied to a Mars orbiter computer vision system and Mars image ground analysis, and provides convenience for ground data analysis and Mars observation work.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and multimedia analysis, in particular to a remote sensing image enhancement method and device. BACKGROUND

[0002] For a remote sensing image affected by dust and covered by heavy dust, the goal of the enhancement method is to restore its clear state when it is not affected by dust. With the development of China's Tianwen-1 Mars exploration project, the Mars orbiter is responsible for observing the geological features of the Martian surface, but the periodic dust storms on the Martian surface have seriously affected the observation mission. The dust image restoration method proposed in the present application has an important auxiliary role in the observation mission of the Mars orbiter and the analysis of the Mars remote sensing image.

[0003] For the task of dust removal of remote sensing images, there is currently a lack of related research. The task in the image restoration field that is most relevant to it is image defogging. However, the defogging model cannot be directly applied to the task of dust removal of remote sensing images. The main reason is that the distribution of dust on the Martian surface is different from the distribution of fog and haze on Earth. The fog and haze on Earth tends to be uniformly distributed, while the distribution of dust on the Martian surface has a certain randomness. The intensity of dust at different positions of the Martian remote sensing image may change greatly on a global scale. In addition, due to the lack of paired data, the quality of image restoration is severely limited. Therefore, for the task of dust removal of Martian remote sensing images, it is necessary to improve the existing scattering model and design a data simulation method to carry out supervised training of deep neural networks. SUMMARY

[0004] According to the above actual needs and key problems, the purpose of the present application is to propose a Mars remote sensing image enhancement method and device, which restores the clear state from a single image given a real dust image.

[0005] The present application comprises the following 5 steps:

[0006] Step S100, a plurality of clear Martian remote sensing images photographed by the Mars orbiter are obtained, and a dust rendering method is used to generate a plurality of corresponding images with dust for each clear Martian remote sensing image, wherein the clear Martian remote sensing image refers to an image that is not blocked by dust;

[0007] Step S200, a deep neural network is trained using each plurality of clear dust image pairs generated, wherein the clear dust image pair in the plurality of clear dust image pairs is an image pair composed of a clear Martian remote sensing image and an image with dust corresponding to the clear Martian remote sensing image;

[0008] Step S300, input the real image with sand dust into the trained neural network, reduce the spatial resolution of the real image with sand dust and expand the channel dimension through a convolutional neural network layer and two two-fold downsampling layers to obtain a feature map;

[0009] Step S400, sequentially pass the feature map through three dense deep convolution modules to extract features;

[0010] Step S500, pass the extracted features through twice upsampling operations and once convolution operation to restore the spatial resolution and channel dimension to the same as those of the real image with sand dust, wherein the output of the trained neural network is a clear reconstruction result of the real image with sand dust, and the clear reconstruction result is a result of converting the real image with sand dust into an image not blocked by sand dust.

[0011] According to the method disclosed in the application, the method and device for enhancing Mars remote sensing images have three beneficial features compared with the traditional defogging model: 1) The method designs a simulation method for Mars sand dust images, which can render a clear remote sensing image with an effect similar to that of real sand dust. 2) By using the rendered paired data, the method learns the mapping relationship between the clear image and the sand dust image by using a supervised learning method, which greatly improves the image repair quality compared with the unsupervised method and the non-deep learning method. 3) The dense connection deep convolution module proposed in the method can greatly improve the feature extraction effect and can be directly applied to other neural network frameworks. 4) The deep neural network trained by the simulation data in the method can be directly applied to the real sand dust image for repair work, which can reduce the visual impact of sand dust on observation and analysis work. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and other features, advantages, and aspects of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0013] Figure 1 is a flowchart of the method and device for enhancing Mars remote sensing images according to the present application;

[0014] Figure 2 is an algorithm flowchart of the method and device for enhancing Mars remote sensing images according to the present application;

[0015] Figure 3 is a network module structure diagram of the method and device for enhancing Mars remote sensing images according to the present application. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0018] It should be noted that the modification of "one" and "multiple" mentioned in the present disclosure is illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0019] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0020] Reference Figure 1 The flowchart 100 of the embodiment of the present application comprises the following steps:

[0021] In step S100, a plurality of clear Mars remote sensing images photographed by a Mars orbiter are obtained, and a dust rendering method is used to generate a plurality of corresponding images with dust for each clear Mars remote sensing image.

[0022] In some embodiments, the execution subject of the Mars remote sensing image enhancement method can obtain a plurality of clear Mars remote sensing images photographed by a Mars orbiter, and use a dust rendering method to generate a plurality of corresponding images with dust for each clear Mars remote sensing image. Wherein, the clear Mars remote sensing image refers to an image that is not blocked by dust.

[0023] As an example, one clear Mars remote sensing image can obtain a plurality of images with dust through the dust rendering method. For example, 7 images with dust can be obtained.

[0024] In practice, in order to render a dust image with high realism, a general physical scattering model commonly used on earth can be used as a starting point, and the general physical scattering model can be gradually converted into a degradation model suitable for Mars remote sensing images. In step S100, for each clear Mars remote sensing image, a corresponding image with dust is generated by a physical scattering model:

[0025]

[0026] where x denotes a pixel position. λ denotes a wavelength of light. H denotes an intensity value tensor of the image with dust. H(x, λ) denotes an intensity value of light with wavelength λ at pixel position x in the image with dust. C denotes an intensity value tensor of the clear Mars remote sensing image. C(x, λ) denotes an intensity value of light with wavelength λ at pixel position x in the clear Mars remote sensing image. denotes a transmittance tensor. denotes a transmittance of light with wavelength λ at pixel position x. denotes a global illumination. denotes a global illumination of light with wavelength λ.

[0027] where, The generation process of is as follows:

[0028]

[0029] where α denotes a dust intensity parameter. M denotes a dust intensity tensor. M(x) denotes a dust intensity at pixel position x. M is generated by a 2-dimensional Perlin noise with fixed parameters.

[0030] where, The generation process of is as follows:

[0031]

[0032] where φ denotes a color correction coefficient. φ(λ) denotes a color correction coefficient of light with wavelength λ. λ0 denotes a wavelength of light at which an intensity value of the clear Mars remote sensing image takes a maximum value. C(x, λ0) denotes an intensity value of light with wavelength λ0 at pixel position x in the clear Mars remote sensing image. denotes a maximum intensity value among intensity values of all wavelengths in the clear Mars remote sensing image.

[0033] where the calculation process of φ(λ) is as follows:

[0034]

[0035] where, denotes a set of heavy dust regions in a plurality of real images with dust. denotes a number of heavy dust regions in a plurality of real images with dust. H' denotes a heavy dust region of a real image with dust. |H'| denotes a total number of pixels of the heavy dust region of the real image. H'(x, λ) denotes an intensity value of light with wavelength λ at pixel position x in H'. denotes a maximum intensity value of light with all wavelengths in H'.

[0036] Step S200, using the generated each multiple clear dust image pairs, training the deep neural network, obtaining the trained neural network. Wherein, the multiple real images with dust can be images taken by Mars orbiter.

[0037] In some embodiments, the above execution subject can use the generated each multiple clear dust image pairs to train the deep neural network, and obtain the trained neural network. Among them, the clear dust image pair in the multiple clear dust image pairs is an image pair composed of a clear Mars remote sensing image and a dust image corresponding to the clear Mars remote sensing image.

[0038] Wherein, the deep neural network has the following characteristics: the deep neural network adopts an encoder-decoder structure, which is composed of an encoder part, a feature extraction part and a decoder part. The encoder part is composed of one convolutional layer and two down-sampling layers. The feature extraction part is composed of three dense deep convolution modules. Each dense deep convolution module is composed of multiple feature attention modules and one deep separable convolution module at the beginning and the end. The decoder part is composed of two up-sampling layers and one convolutional layer.

[0039] As an example, one clear Mars remote sensing image uses the dust rendering method to obtain 7 images with dust, and the clear Mars remote sensing image can be made into a clear dust image pair with the 7 images with dust respectively, and 7 clear dust image pairs are obtained.

[0040] Step S300, input the real image with dust into the trained neural network, reduce the spatial resolution of the real image with dust and expand the channel dimension through a convolutional neural network layer and two two times down-sampling layers, and obtain a feature map.

[0041] In some embodiments, the above execution subject can input the real image with dust into the trained neural network, reduce the spatial resolution of the real image with dust and expand the channel dimension through a convolutional neural network layer and two two times down-sampling layers, and obtain a feature map. Wherein, the convolution operation does not change the spatial resolution and the channel number of the feature. The down-sampling operation will expand the channel number while reducing the spatial resolution of the feature map. The real image with dust can be an image taken by Mars orbiter.

[0042] Step S400, sequentially passing the feature map through three dense deep convolution modules to extract features.

[0043] In some embodiments, the above execution subject can sequentially pass the feature map through three dense deep convolution modules to extract features.

[0044] In some optional implementations of some embodiments, the above dense deep convolution module can have the following characteristics:

[0045] The feature map F is input into a depth separable convolution module. The depth separable convolution module includes a calculation process as follows:

[0046]

[0047] wherein F1 represents a first feature map. LR represents an activation function. BN represents a batch normalization operation. DSC represents a depth separable convolution operation. F represents a feature map. F2 represents a second feature map. [,,] represents a feature concatenation operation. [F, F1] represents a feature concatenation operation of F and F1. F3 represents a third feature map. [F, F1, F2] represents a feature concatenation operation of F, F1 and F2. F4 represents a fourth feature map. [F, F1, F2, F3] represents a feature concatenation operation of F, F1, F2 and F3. F5 represents a fifth feature map. [F, F1, F2, F3, F4] represents a feature concatenation operation of F, F1, F2, F3 and F4. O represents a final output feature map of the depth separable convolution module. The batch normalization operation can be a BatchNorm operation. BatchNorm is an algorithm frequently used in deep networks to accelerate neural network training, accelerate convergence speed and stability, and is an indispensable part of the current deep network. The activation function can be a LeakyReLU activation function. The LeakyReLU activation function can be a function running on the neurons of an artificial neural network, which can map the input of the neuron to the output end.

[0048] The obtained final output feature map is input into a plurality of feature attention modules to obtain an attention feature map. The feature attention module can be a module for extracting features from the input feature map by using a channel attention mechanism and a spatial attention mechanism.

[0049] The attention feature map is input into a depth separable convolution module to obtain a target output feature map. The target output feature map can be the final output of the dense depth convolution module.

[0050] Figure 2 is the algorithm flowchart of the Mars remote sensing image enhancement method and device of the application. Figure 2In the embodiment, the feature map F and the first feature map F1 can be subjected to a feature concatenation operation to obtain a second feature map F2. Then, the feature map F, the first feature map F1 and the second feature map F2 can be subjected to a feature concatenation operation to obtain a third feature map F3. After that, the feature map F, the first feature map F1, the second feature map F2 and the third feature map F3 can be subjected to a feature concatenation operation to obtain a fourth feature map F4. Next, the feature map F, the first feature map F1, the second feature map F2, the third feature map F3 and the fourth feature map F4 can be subjected to a feature concatenation operation to obtain a fifth feature map F5. Finally, the feature map F and the fifth feature map F5 can be subjected to an element-wise addition operation to obtain the final output feature map of the depth separable convolution module.

[0051] In step S500, the extracted features are subjected to twice upsampling operations and a convolution operation to restore the spatial resolution and the channel dimension to be the same as those of the real image with dust.

[0052] In some embodiments, the execution subject described above can subject the extracted features to twice upsampling operations and a convolution operation to restore the spatial resolution and the channel dimension to be the same as those of the real image with dust. The output of the trained neural network is a clear reconstruction result of the real image with dust, and the clear reconstruction result is a result of converting the real image with dust into an image without being blocked by dust.

[0053] The reconstruction process of the clear reconstruction result has the following characteristics: each upsampling operation reduces the channel number of the feature map and increases the spatial resolution. The final convolution operation does not change the spatial resolution and the channel number, and the shape of the output is the same as that of the input image. At this time, the final output is the dust removal result.

[0054] Figure 3 is a network module structure diagram of the Mars remote sensing image enhancement method and device. Figure 3 In the embodiment, the feature map is sequentially input into a convolution layer, two down-sampling layers, and the feature maps after twice down-sampling are subjected to a depth separable convolution module, a feature attention module and a depth separable convolution module. Then, the feature map subjected to the depth separable convolution module is input twice into a dense depth convolution module, and then subjected to twice up-sampling layers. Finally, the feature maps after twice up-sampling are input into a convolution layer to obtain the final feature map.

[0055] It can be understood that the units described in the Mars remote sensing image enhancement device are described in the method Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features and beneficial effects described above for the method also apply to the Mars remote sensing image enhancement device and the units contained therein, which will not be described here.

[0056] The above description is merely exemplary of some embodiments of the present disclosure and of the principles thereof. It is to be understood that the present disclosure is not limited to the specific technical features described above, and that the scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, but also covers other technical solutions formed by the combinations of the above technical features or equivalent features thereof without departing from the above inventive concept. For example, the above technical features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form technical solutions.

Claims

1. A method for enhancing Mars remote sensing images, comprising: Step S100: Acquire multiple clear Mars remote sensing images taken by the Mars orbiter, and use the dust rendering method to generate multiple corresponding dust-bearing images for each clear Mars remote sensing image. Here, a clear Mars remote sensing image refers to an image that is not obscured by dust. A clear Mars remote sensing image is used to generate multiple dust-bearing images from a single clear Mars remote sensing image through the dust rendering method. Step S200: Using the generated multiple clear dust image pairs, train a deep neural network to obtain a trained neural network. The clear dust image pair in the multiple clear dust image pairs is an image pair consisting of a clear Mars remote sensing image and an image with dust corresponding to the clear Mars remote sensing image. Step S300: Input the real image with sand and dust into the trained neural network. After passing through a convolutional neural network layer and two double downsampling layers, reduce the spatial resolution of the real image with sand and dust and expand the channel dimension to obtain the feature map. Step S400: The feature map is sequentially passed through three dense deep convolutional modules to extract features; Step S500: The extracted features are subjected to two upsampling operations and one convolution operation to restore the spatial resolution and channel dimension to the same as the spatial resolution and channel dimension of the real image with sand and dust. The output of the trained neural network is a clear reconstruction result of the real image with sand and dust. The clear reconstruction result is the result of transforming the real image with sand and dust into an image that is not covered by sand and dust. The sand and dust rendering method includes the following steps: For each clear Mars remote sensing image, a corresponding image with dust is generated using a physical scattering model: Where x represents the pixel location, λ represents the wavelength of light, H represents the intensity tensor of the image with dust, H(x, λ) represents the intensity value of light with wavelength λ at pixel location X in the image with dust, and C represents the intensity tensor of the clear Mars remote sensing image, C(x, λ) represents the intensity value of light with wavelength λ at pixel location X in the clear Mars remote sensing image. This represents the transmittance tensor. This represents the transmittance of light with wavelength λ at pixel location x. Indicates global illumination. Represents global illumination of light with wavelength λ; in, The generation process is as follows: Where α represents the dust intensity parameter, M represents the dust intensity tensor, M(x) represents the dust intensity at pixel position X, and M is generated by 2D Berlin noise with fixed parameters; in, The generation process is as follows: Where Φ represents the color correction coefficient, Φ(λ) represents the color correction coefficient for light with wavelength λ, λ0 represents the wavelength of light that maximizes the intensity value of a clear Mars remote sensing image, and C(x, λ0) represents the intensity value of light with wavelength λ0 at pixel position x in a clear Mars remote sensing image. This represents the maximum intensity value among all wavelengths in a clear Mars remote sensing image; The calculation process for Φ(λ) is as follows: in, This represents a collection of heavily dusty areas from multiple real images containing sand and dust. Let H′ represent the number of heavily dusty regions in multiple real images containing sand and dust, |H′| represent the total number of pixels in the heavily dusty regions of the real images, and H′(x, λ) represent the intensity value of light with wavelength λ at pixel position X in H′. This represents the maximum intensity value of light of all wavelengths in H′.

2. The method according to claim 1, wherein, Dense depthwise convolutional modules have the following characteristics: The feature map F is input into the depthwise separable convolution module, where the computation process of the depthwise separable convolution module is as follows: Where F1 represents the first feature map, LR represents the activation function, BN represents the batch normalization operation, DSC represents the depthwise separable convolution operation, F represents the feature map, F2 represents the second feature map, [,] represents the feature concatenation operation, [F,F1] represents the feature concatenation operation of F and F1, F3 represents the third feature map, [F,F1,F2] represents the feature concatenation operation of F, F1 and F2, F4 represents the fourth feature map, [F,F1,F2,F3] represents the feature concatenation operation of F, F1, F2 and F3, F5 represents the fifth feature map, [F,F1,F2,F3,F4] represents the feature concatenation operation of F, F1, F2, F3 and F4, and O represents the final output feature map of the depthwise separable convolution module; The final output feature map is passed through multiple feature attention modules to obtain the attention feature map; The attention feature map is passed through a depthwise separable convolutional module to obtain the target output feature map.

3. A Mars remote sensing image enhancement device, comprising: The acquisition unit is configured to acquire multiple clear Mars remote sensing images taken by the Mars orbiter, and to generate multiple corresponding images with dust for each clear Mars remote sensing image using a dust rendering method. Here, a clear Mars remote sensing image refers to an image that is not obscured by dust. The training unit is configured to train a deep neural network using the generated multiple pairs of clear dust images to obtain a trained neural network, wherein the multiple pairs of clear dust images are image pairs consisting of a clear Mars remote sensing image and an image with dust corresponding to the clear Mars remote sensing image. The input unit is configured to input a real image with sand and dust into the trained neural network. The image is then passed through a convolutional neural network layer and two downsampling layers to reduce the spatial resolution of the real image with sand and dust and expand the channel dimension to obtain a feature map. The extraction unit is configured to extract features by sequentially passing the feature map through three dense depthwise convolutional modules; The upsampling unit is configured to perform two upsampling operations and one convolution operation on the extracted features to restore the spatial resolution and channel dimension to the same as the spatial resolution and channel dimension of the real image with sand and dust. The output of the trained neural network is a clear reconstruction result of the real image with sand and dust, which is the result of transforming the real image with sand and dust into an image that is not occluded by sand and dust. The sand and dust rendering method includes the following steps: For each clear Mars remote sensing image, a corresponding image with dust is generated using a physical scattering model: Where X represents the pixel location, λ represents the wavelength of light, H represents the intensity tensor of the image with dust, H(x,λ) represents the intensity value of light with wavelength λ at pixel location X in the image with dust, and C represents the intensity tensor of the clear Mars remote sensing image, C(x,λ) represents the intensity value of light with wavelength λ at pixel location x in the clear Mars remote sensing image. This represents the transmittance tensor. This represents the transmittance of light with wavelength λ at pixel location X. Indicates global illumination. Represents global illumination of light with wavelength λ; in, The generation process is as follows: Where α represents the dust intensity parameter, M represents the dust intensity tensor, M(x) represents the dust intensity at pixel position X, and M is generated by 2D Berlin noise with fixed parameters; in, The generation process is as follows: Where Φ represents the color correction coefficient, Φ(λ) represents the color correction coefficient for light with wavelength λ, λ0 represents the wavelength of light that maximizes the intensity value of a clear Mars remote sensing image, and C(x,λ0) represents the intensity value of light with wavelength λ0 at pixel position X in a clear Mars remote sensing image. This represents the maximum intensity value among all wavelengths in a clear Mars remote sensing image; The calculation process for Φ(λ) is as follows: in, This represents a collection of heavily dusty areas from multiple real images containing sand and dust. Let H′ represent the number of heavily dusty regions in multiple real images containing sand and dust, |H′| represent the total number of pixels in the heavily dusty regions of the real images, and H′(x,λ) represent the intensity value of light with wavelength λ at pixel position X in H′. This represents the maximum intensity value of light of all wavelengths in H′.

Images