A Martian surface image enhancement method based on convolutional neural networks

Abstract

This invention discloses a method for enhancing Martian surface images based on convolutional neural networks, comprising: acquiring an image to be processed; performing an image enhancement operation on the image to be processed, the image enhancement operation including: inputting the image into a Martian surface image enhancement neural network model to obtain noise information; and removing the noise information to obtain an enhanced image. By modeling Martian surface images collected by the "Zhurong" Mars rover, the problem of Martian surface image enhancement is transformed into a problem of solving Martian surface image noise. Based on this, a convolutional neural network is designed to solve for the noise in the Martian surface images, and combined with the Martian surface image model, to achieve the enhancement of the Martian surface images. This invention can effectively improve the image acquisition quality of Mars rovers such as "Zhurong," effectively enhance the autonomous perception capabilities of the rovers, and thus contribute to the autonomous navigation and obstacle avoidance capabilities of Mars rovers such as "Zhurong."

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Mars rover autonomous intelligent perception, and in particular to a Mars surface image enhancement method based on a convolutional neural network. BACKGROUND

[0002] In the process of human exploration of Mars, images of the surface of Mars need to be continuously collected for analysis and exploration of the surface morphology and climate state of Mars. On the other hand, with the development of Mars autonomous rovers in recent years, especially with the start of the exploration of the Zhurong Mars rover, visual sensors are increasingly widely used in Mars rovers and can be used for autonomous perception of the environment in front of and behind the rover, thereby achieving autonomous navigation and obstacle avoidance. However, due to factors such as dust and atmosphere on the surface of Mars, the image clarity of the camera of a Mars explorer such as the Zhurong Mars rover is reduced during image shooting, thereby affecting subsequent image analysis and autonomous navigation and obstacle avoidance of the Zhurong Mars rover. Therefore, image enhancement is needed to improve the imaging quality of Mars surface images such as the Zhurong Mars rover.

[0003] Existing image enhancement methods mainly focus on image enhancement in weak light or low light environments, and there is little research on Mars surface image enhancement. On the other hand, current research on image enhancement based on deep learning or neural networks does not fully consider the visual imaging model and does not comprehensively consider the enhancement mechanism of images. SUMMARY

[0004] In view of the problem of Mars surface image enhancement of the Zhurong Mars rover and the like, the present application provides a Mars surface image enhancement method based on a convolutional neural network. The Mars surface image enhancement problem is converted into a Mars surface image noise solving problem by modeling the Mars surface images collected by the Zhurong Mars rover. On this basis, the noise of the Mars surface images is solved by designing a convolutional neural network, and the Mars surface image model is combined to achieve enhancement of the Mars surface images. The present application can effectively improve the image acquisition quality of the Zhurong Mars rover and the like, can effectively improve the autonomous perception capability of the Mars exploration vehicle, and can further help improve the autonomous navigation and obstacle avoidance capability of the Zhurong Mars rover and the like.

[0005] In a first aspect, a Mars surface image enhancement method is provided, comprising:

[0006] acquiring a to-be-processed image;

[0007] performing an image enhancement operation on the to-be-processed image, the image enhancement operation comprising: inputting an image into a Mars surface image enhancement neural network model to obtain noise information; removing the noise information to obtain an enhanced image.

[0008] In some implementations of the first aspect, the Mars surface image enhancement neural network model comprises 6 convolutional layers,

[0009] The first convolutional layer has a convolution kernel of 3*3, an expansion exponent of 1, contains a compression excitation SE block, and a receptive field of 3*3.

[0010] The second convolutional layer has a convolution kernel of 3*3, an expansion exponent of 2, contains a compression excitation SE block, and a receptive field of 5*5.

[0011] The third convolutional layer has a convolution kernel of 3*3, an expansion exponent of 4, contains a compression excitation SE block, and a receptive field of 9*9.

[0012] The fourth convolutional layer has a convolution kernel of 3*3, an expansion exponent of 8, contains a compression excitation SE block, and a receptive field of 17*17.

[0013] The fifth convolutional layer has a convolution kernel of 1*1, an expansion exponent of 1, does not contain a compression excitation SE block, and a receptive field of 17*17.

[0014] The fifth convolutional layer has a convolution kernel of 1*1, an expansion exponent of 1, does not contain a compression excitation SE block, and a receptive field of 17*17.

[0015] In some implementations of the first aspect, the Mars surface image enhancement neural network model further comprises a dimension reduction nonlinear mapping layer, and the dimension reduction nonlinear mapping layer satisfies:

[0016]

[0017] is a convolutional filter, the total number of convolutional filters is k*n1, and k is a parameter set for the dimension reduction nonlinear mapping, f1 is a filter size, 3 represents 3 color channels in the input image I(x), n1 represents the number of output feature maps; and * represents a convolution operation.

[0018] In some implementations of the first aspect, the Mars surface image enhancement neural network model realizes feature fusion by extracting features from a fully connected layer and extracting features from a convolutional layer.

[0019] In some implementations of the first aspect, the training loss function of the Mars surface image enhancement neural network model satisfies:

[0020]

[0021] O is a to-be-processed image, R is a noise image, and f(O) is a mapping based on O output by the Mars surface image enhancement neural network model.

[0022] With reference to the first aspect, in some implementations of the first aspect, the enhanced image B satisfies:

[0023]

[0024] O is a to-be-processed image, R j are noise of different layers, n is a number of noise layers, and a j correspond to weights of the respective noise layers.

[0025] With reference to the first aspect, in some implementations of the first aspect, the method further includes:

[0026] The image enhancement operation is performed in a loop, wherein an image obtained through a previous image enhancement operation is taken as an input of a next image enhancement operation.

[0027] I t+1 = f(I t )

[0028] wherein I t is a tth-time optimized image, and f is the image enhancement network proposed herein.

[0029] With reference to the first aspect, in some implementations of the first aspect, a mean value of (n-n+k)th-time optimization results is used as a final optimization result.

[0030] With reference to the first aspect, in some implementations of the first aspect, the image enhancement operation is performed in the loop, and the following is satisfied:

[0031]

[0032] is a jth noise layer decomposed in the ith-level loop, and b i correspond to weights of the respective noise layers.

[0033] With reference to the first aspect, in some implementations of the first aspect, the method further includes:

[0034] After each execution of the image enhancement operation, it is determined whether noise output by the image enhancement operation is greater than or equal to a preset threshold value.

[0035] If yes, the image enhancement operation is continuously executed.

[0036] If no, the loop is exited.

[0037] In a second aspect, an electronic device is provided for performing the method as described in any of the implementations of the first aspect above.

[0038] This invention provides a method for enhancing Martian surface images based on convolutional neural networks. This invention considers a visual imaging model and applies a convolutional neural network to achieve image enhancement of the Martian surface. Compared with existing technologies, the solution provided in this application has at least the following beneficial technical effects:

[0039] (1) This invention proposes an imaging model for Martian surface images. This invention models Martian surface images into clear images and noisy images, thereby transforming the problem of Martian surface image enhancement into a problem of solving Martian surface image noise. Compared with end-to-end neural networks for image enhancement, the image enhancement of this invention is more interpretable.

[0040] (2) This invention designs an image enhancement network based on a convolutional neural network. The network is designed with dilated convolution, multi-scale prediction and recurrent aggregation network. By fusing global and local features of the image and passing the image features, the noise of the image can be solved.

[0041] (3) This invention designs a method for generating a Mars surface dataset. In view of the limited availability of Mars surface images, this invention uses publicly available Mars surface images to generate blurred images of different degrees by setting the medium transmittance and noise ratio, in order to simulate Mars surface images. Based on this, a designed convolutional neural network is trained to enhance the Mars surface images. Attached Figure Description

[0042] Figure 1 A schematic diagram of the Martian surface imaging model provided for the invention.

[0043] Figure 2 A schematic diagram of the Mars surface enhancement convolutional neural network structure provided for the invention. Detailed Implementation

[0044] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0045] The present invention provides a method for enhancing Martian surface images based on convolutional neural networks, which mainly includes the following:

[0046] To address the issue of unclear Martian surface images due to dust, atmosphere, and other factors, this invention models images obtained by the "Zhurong" Mars rover's camera into both clear and noisy images, transforming the image enhancement problem into a "dual-signal separation problem" between clear and noisy images. Figure 1As shown, O is an image captured by the Mars rover, B is a clear image of the Martian surface, and R is noise in the image. That is, the Martian surface image O consists of a clear image B and a noisy image R.

[0047] O=B+R (1)

[0048] In real-world environments, the Martian surface noise images collected by the Zhurong Mars rover contain various noise data, such as those affected by dust and light scattering. To improve model accuracy, this invention further models the data augmentation model as multi-layered noise data, and the weight sparsity of each layer of noise data in the final image is not uniform; that is, different noise layers contain different weight sparsity α. Therefore, the further modeled Martian surface imaging model is as follows:

[0049]

[0050] In the above formula, R j For different layers of noise, n is the number of noise layers, and α j This corresponds to the weights of each noise layer. Therefore, the process of obtaining a clear image is as follows:

[0051]

[0052] To achieve clear image processing, this application provides a convolutional neural network model for enhancing Martian surface images, which enhances images of the Martian surface collected by the "Zhurong" Mars rover.

[0053] like Figure 1 As shown in the dashed box, the network depth in this embodiment can be set to 6 layers. The convolution method of some layers in the network adopts dilated convolution. The kernel size, dilation index, compression excitation (SE) block, and receptive field of each layer are shown in Table 1.

[0054] Table 1. Parameter information for each layer of the network

[0055] 0 1 2 3 4 5 Convolution kernel 3*3 3*3 3*3 3*3 3*3 1*1 Dilation exponent 1 2 4 8 1 1 SE block Comprise Comprise Comprise Comprise Comprise Do not comprise Receptive field 3*3 5*5 9*9 17*17 17*17 17*17

[0056] In this network, layer 0 transforms the noisy Martian surface image into a feature image; therefore, standard convolution is used in this layer with a dilation exponent of 1. In layers 1, 2, and 3, dilated convolution is used to increase the amount of information extracted from the image, with dilation exponents set to 2, 4, and 8 respectively, allowing the receptive field area to grow rapidly. Layers 4 and 5 perform inverse mapping; the image output from the front end is decoded in these two layers. Therefore, standard convolution is used in these two layers with a dilation exponent of 1. In layer 5, the three primary color channels of the original image are restored; therefore, linear operation is used in this layer, with a kernel size of 1*1, no SE block structure, and standard convolution.

[0057] In this embodiment, to further enhance the neural network's ability to extract features from Martian surface images, a nonlinear mapping operation for dimensionality reduction is performed in the network designed to extract the extreme values ​​in the color channels of noise features in the Martian surface images. The extreme value of the three primary color channels can be the maximum value among the three channels, meaning that the color proportion of a certain channel is the highest.

[0058] The hierarchical structure of the dimension reduction nonlinear mapping applied in this invention can act as an activation function of the network. By setting a parameter k in this layer, the dimension reduction nonlinear mapping will perform element-wise maximization operation according to the value of k to generate a new feature mapping.

[0059] Assuming the input is a 3D matrix of size X*Y*Z, the nonlinear mapping layer maps a 3D matrix with dimensions k*n in the Z direction to a vector of dimension n. For example, if the input is a 3D matrix of size 3*3*1000 and k is 5, the 3D matrix of size 3*3*1000 can be divided into 200 3D matrices of size 3*3*5. Element-wise maximization of each 3D matrix yields its maximum value. These 200 maximum values ​​can then be combined into a vector of dimension 200.

[0060] In the hidden layer of the dimension reduction nonlinear mapping, the calculation of each element (in the matrix) follows the formula:

[0061]

[0062] In the above formula, It is a convolutional filter, and the total number of convolutional filters is k*n1, and k is the parameter set for the dimension reduction nonlinear mapping, where f1 is the filter size and 3 represents the 3 color channels in the input image I(x). * represents the bias; n1 represents the number of output feature maps; * represents the convolution operation. W and B1 are the parameters that need to be learned.

[0063] The dimensionality reduction nonlinear mapping layer possesses extremely strong fitting capabilities, enabling it to fit any continuous function. This structure also exhibits strong learning capabilities, learning not only the relationships between hidden layers but also the activation functions of those hidden layers. This layer is not a traditional activation function; therefore, its generated representations are dense, but inherently possess sparse gradients. Furthermore, the dimensionality reduction nonlinear mapping has no upper or lower bounds, thus preventing saturation at any point. In the first layer of the denoising network in this embodiment, a structure combining a convolutional layer and a dimensionality reduction nonlinear mapping layer is employed.

[0064] Because the Martian surface images acquired by the Zhurong Mars rover exhibit diverse noise levels, a single neural network operation is insufficient to achieve ideal denoising results. Therefore, as... Figure 2 As shown, this invention performs multiple iterations on the network to filter out noise from different layers. This invention will... Figure 1 The neural network model shown divides each run into a unit module. Each unit module can eliminate noise in one layer and use the denoised image output from the previous unit module as the input image for the next stage. This allows for the successive elimination of noise, ultimately resulting in a completely denoised image.

[0065] To avoid the loss of feature information in the previous processing step due to simple cascading of single-layer networks, this invention, referencing the characteristics of RNN networks in handling time-series problems, can convert the image result sequence *O1,O2,…,O after each loop into O1,O2,…,O n The sequence can be viewed as occurring at time points 1, 2, ..., n. Image enhancement can be improved by using a multi-level network for iterative refinement. Noise reduction is corrected for each iteration.

[0066]

[0067] In the above formula, It is the j-th noise layer decomposed from the i-th level loop, β i This corresponds to the weights of each noise layer. Using the mean of the optimization results from the (n to n+k)th iterations as the final optimization result helps increase the robustness of the output image quality.

[0068] This application embodiment can also apply multi-scale prediction. Multi-scale features can be extracted and used for detection task prediction at different scales, thereby effectively improving the feature extraction capability of convolutional neural networks for Martian surface images collected by the "Zhurong" Mars rover. In this embodiment, multi-scale feature extraction adopts two methods to achieve feature extraction from fully connected layers and feature extraction from convolutional layers. Feature extraction from fully connected layers can be achieved by using spatial pyramid pooling layers instead of average pooling layers or max pooling layers at the front of the fully connected layers. Feature extraction from convolutional layers can be achieved by using the fully connected layers in the denoising convolutional neural network to perform multi-scale feature extraction. In this embodiment, information from fully connected layers and convolutional layers is extracted and merged to form multi-scale features.

[0069] Employing a multi-feature scale approach enables dense feature extraction from input Martian surface images at different spatial scales, achieving scale invariance and thus improving robustness of feature extraction at different resolutions. In the image enhancement network structure established in this embodiment, the second layer employs parallel convolution for computation. This layer designs three convolutional filters of different sizes, with the same number of filters of each size. The output of this layer is shown in the following equation:

[0070]

[0071] In the above formula, [] represents the operation of rounding up to the nearest integer; \ represents the operation of finding the remainder; n2 represents the dimension of the output of this layer. W2 and B2 contain 3 sets of parameters, with a total of n2 pairs of parameters. The index of the output feature map is i∈[1,n2].

[0072] After completing the above model design, the Mars surface image augmentation network model needs to be trained. The difference between the output of the Mars surface image augmentation network and the real output needs to be evaluated by optimizing the loss function to guide the model learning. The loss function is shown in the following formula.

[0073]

[0074] In the above equation, O represents the input image of the Martian surface, R is the actual noise image, and f(O) is the O-based mapping learned by the neural network. The loss function evaluates how close f(O) is to R. During training, the Adam algorithm is used to optimize the process. The Adam algorithm can update the weights in the neural network and then analyze the exponentially moving average of the gradient. This algorithm can replace the traditional stochastic gradient descent method and has better performance. This algorithm combines the advantages of root mean square propagation and adaptive gradient algorithms, which can calculate the learning rate of the adaptive parameters based on the first-order mean matrix and make full use of the biased variance of the gradient.

[0075] During model training, due to the limited number of Mars surface image samples, this invention further augmented the Mars surface dataset. A Mars surface dataset was created based on publicly available Mars surface data. By adding noise to the original Mars surface dataset, blurred Mars surface images were obtained. The added noise was then used as an optimization term to train the neural network model.

[0076] Because noise in real-world noisy images cannot be directly extracted, it is difficult to collect datasets of noisy images and corresponding noise-free scenes of the same location. On the other hand, since the content and noise in Martian surface images are independent, and the noise weighting coefficients are locally continuous, arbitrary noise weighting coefficients can be set for a single frame. This paper proposes a method to manually set the noise weight α based on noise-free Martian surface images. i This process generates noisy Martian surface images, which are then used to construct noise-free and noisy scene dataset pairs. In this embodiment, 2000 noise-free images are selected from the noise-free Martian surface image dataset without a network. For each noise-free image, five media transmission rates α are uniformly selected. i , and α i Between (0,1), a noiseless image frame is synthesized into a noisy image as described in formula (2), so that 5 noisy images are generated for each noiseless image frame. Therefore, there are a total of 10,000 noisy images in this paper.

[0077] Therefore, based on the trained neural network model parameters, the model parameters are saved, and the images of the "Zhurong" Mars rover are acquired in real time through camera drive. Based on the saved model parameters and the designed neural network model, the Martian surface is enhanced in real time to obtain a clear enhanced image of the Martian surface.

[0078] To improve the real-time performance of the model, this invention assesses the noise obtained after each loop, specifically by determining the value of R in formula (2). This invention sets a threshold T and determines whether the noise R after each loop is greater than or equal to the threshold T. If it is greater, the next denoising loop begins; if it is less, denoising is complete.

[0079] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope defined in the claims of the present invention.

Claims

1. A method of Mars surface image enhancement, characterized by, The method comprises: acquiring a to-be-processed image; performing an image enhancement operation on the to-be-processed image, the image enhancement operation comprising: inputting an image into a Mars surface image enhancement neural network model to obtain noise information; and removing the noise information to obtain an enhanced image; the Mars surface image enhancement neural network model comprises six convolutional layers, the first convolutional layer has a convolution kernel of 3*3, an expansion exponent of 1, contains a compression excitation SE block, and has a receptive field of 3*3; the second convolutional layer has a convolution kernel of 3*3, an expansion exponent of 2, contains a compression excitation SE block, and has a receptive field of 5*5; the third convolutional layer has a convolution kernel of 3*3, an expansion exponent of 4, contains a compression excitation SE block, and has a receptive field of 9*9; the fourth convolutional layer has a convolution kernel of 3*3, an expansion exponent of 8, contains a compression excitation SE block, and has a receptive field of 17*17; the fifth convolutional layer has a convolution kernel of 1*1, an expansion exponent of 1, does not contain a compression excitation SE block, and has a receptive field of 17*17; the Mars surface image enhancement neural network model further comprises a dimension reduction nonlinear mapping layer, and the dimension reduction nonlinear mapping layer satisfies: The method further comprises: W1 i,j is a convolution filter, the total number of convolution filters is k*n1, and k is a parameter set for the dimension reduction nonlinear mapping, f1 is the filter size, 3 represents 3 color channels in the input image I(x), represents the bias; n1 represents the number of output feature maps; * represents the convolution operation; cyclically performing the image enhancement operation, wherein an image obtained through a previous image enhancement operation is used as an input of a next image enhancement operation: using a mean value of optimization results of the (n~n+k)th to (n+k)th times as a final optimization result; I t+1 = φ(I t ) where I t is the t-th optimized image, and φ is the image enhancement network proposed in this paper. the cyclically performing the image enhancement operation satisfies: The Mars surface image enhancement neural network model realizes feature fusion by extracting features from a fully connected layer and extracting features from a convolutional layer. is the jth noise layer decomposed in the ith level of recursion, β i corresponds to the weight of each noise layer.

2. The method of claim 1, wherein, The training loss function of the Mars surface image enhancement neural network model satisfies:

3. The method of claim 1, wherein, O is the to-be-processed image, R is a noise image, and f(O) is a mapping based on O output by the Mars surface image enhancement neural network model. The enhanced image B satisfies:

4. The method of claim 1, wherein, The method further comprises: O is the image to be processed, R j is the noise of different layers, n is the number of noise layers, a j corresponds to the weight of each noise layer.

5. The method of claim 1, wherein, after each execution of the image enhancement operation, determining whether noise output by the image enhancement operation is greater than or equal to a preset threshold value; if yes, continuing to perform the image enhancement operation; if no, exiting the loop. The electronic device is configured to perform the method according to any one of claims 1 to 5.

6. An electronic device, comprising: ​

Images