A method and device for detecting anomalies in Mars images based on dense residual autoencoders

Abstract

This invention relates to a method and device for anomaly detection in Mars imagery based on a dense residual autoencoder. The invention proposes an innovative model that, while retaining the UNet architecture, introduces the concept of "dense residuals" on the encoding side. This involves using a series of densely connected convolutional networks to enhance the network's ability to learn deep semantic information from the image. The input masked Mars image serves as the reconstructed background. Convolutional and residual connections preserve the spatial details of the reconstructed background, and various loss functions are used to suppress the reconstruction of anomalies, achieving separation of background and anomalies in the network reconstruction result. This invention can solve the problem of deep learning-based anomaly detection in Mars imagery, achieving image-level anomaly detection results. This invention can effectively improve the applicability and accuracy of anomaly detection in Mars imagery.

Description

Technical Field

[0001] This invention is based on the field of remote sensing image processing technology, and specifically relates to a method for detecting anomalies in Mars images based on a dense residual autoencoder. Background Technology

[0002] Mars exploration is a multidisciplinary and highly integrated systems engineering project, of great significance for studying major scientific questions such as the origin and evolution of the solar system and the origin and evolution of life. With the continuous development of space technology and transmission technology, the number of images transmitted by Mars probes has increased dramatically, and visual observation alone cannot meet the actual needs of scientific research. Anomaly detection has become an effective method for detecting anomalous targets on Mars. By rapidly detecting valuable anomalous images on the Martian surface, the system can learn anomalous characteristics, which is of great significance for the study of Martian geography and the real-time decision-making and planning of Mars rovers, and has increasingly attracted the attention of the academic community.

[0003] Anomaly detection refers to modeling normal data to identify atypical samples. It plays a crucial role for researchers studying anomalous samples and analyzing anomalous distributions, and has promising prospects in various fields. In recent years, with the continuous development of deep learning, anomaly detection has gradually become a hot topic in planetary research.

[0004] However, past research on Mars image anomaly detection systems has been limited, primarily due to the lack of a Mars anomaly segmentation model. Traditional anomaly segmentation models, when faced with the rich surface information of Mars, suffer from reconstruction errors mainly at the pixel level rather than semantically, leading to the reconstruction of anomalous objects and failing to accurately provide anomaly information. Therefore, this research focuses on developing a rover anomaly detection system that simultaneously provides image-level anomaly structure, building upon existing anomaly segmentation models. This approach has significant research value and practical prospects. Summary of the Invention

[0005] The purpose of this invention is to propose a method for detecting anomalies in Mars images based on a dense residual autoencoder.

[0006] This invention uses image masking to mask Mars images and uses a convolutional model to repair the Mars images, thereby transforming the problem of Mars image anomaly detection into a problem of background and anomaly separation. The invention uses a network to infer the image, and the anomalies are manifested as network reconstruction errors, thus realizing the detection of anomalies in Mars images.

[0007] This invention proposes an innovative model, dUnet, which, while retaining the Unet architecture, introduces the concept of "dense residuals" on the encoding side. This involves using a series of densely connected convolutional networks to enhance its ability to learn deep semantic information from images. The input masked Mars image serves as the reconstructed background. Convolutional and residual connections preserve the spatial details of the reconstructed background, and a summation of various loss functions is used to suppress the reconstruction of anomalies, achieving separation between background and anomalies in the network's reconstruction result.

[0008] The innovative dUnet model is divided into two parts: the encoding side and the decoding side. The encoding side consists of five modules. The first module is a "convolutional layer-normalization layer-nonlinear activation layer" structure, used to extract semantic information from the surface of the image. Modules 2, 3, and 4 each consist of a dense residual module, a "convolutional layer-normalization layer-nonlinear activation layer" structure, and an average pooling layer. The dense residual module consists of multiple convolutional layers, which are interconnected, facilitating the extraction of deep semantics from the image; the average pooling layer extracts feature information from the image while simultaneously compressing the image scale. The fifth module is composed of a "convolutional layer-normalization layer-nonlinear activation layer" structure, aiming at deep semantic learning and image scale compression. The decoding side consists of four deconvolutional layers. Each deconvolutional layer outputs a reconstructed image based on the learned semantic information, resulting in a final output image with the exact same size as the original image. The first four modules of the encoding side are sequentially connected to the corresponding modules of the decoding side, forming the connection layers of the model. It has the following significant characteristics:

[0009] First, it enhances the ability to extract semantic information from images. Shallow networks are responsible for learning the shallow texture features of images, while as the network depth increases, the receptive field of the model expands, and deep networks are responsible for learning the semantic features of images. This structure facilitates the model's learning of semantic information at various levels of images.

[0010] Secondly, it enables the recovery of image edge features. Upsampling, while extracting image semantics, reduces the image size, resulting in the loss of image edge information. By using image stitching, the image edge information is recovered, avoiding the loss of edge information during anomaly detection.

[0011] Third, it mitigates the degradation effect of the model. As the number of layers in the network model increases, network degradation occurs, which affects the accuracy of anomaly detection and reduces its effectiveness. Residual networks effectively solve this problem by using short connections instead of ordinary connections, thus routing the input information outwards and preserving the integrity of image protection information as much as possible to reduce the loss of input information.

[0012] This invention provides a method for anomaly detection in Mars images based on a dense residual autoencoder, comprising a training step and a testing step, the implementation steps of which are as follows:

[0013] The following are the training steps:

[0014] Training Step 1: Preprocess Mars imagery. This step further includes: scaling the dataset to a central size of [size missing]. The Mars imagery is compressed to 64×64 pixels, and its pixel values ​​are normalized to within the range of 0 to 1, where H, W, and B represent the number of image rows, columns, and bands, respectively. The preprocessed dataset is output.

[0015] Training step 2: Select m different sizes of square grids to divide the Mars dataset, denoted as a1; a2; ...; a m Each partitioning result is then randomly divided into n disjoint sets. The pixel values ​​in each set are successively set to 0. The output result is denoted as a. 11 ,a 12 ,...,a 1n ;a 21 ,a 22 ,...,a 2n ;...;a m1 ,a m2 ,...,a mn .

[0016] Training step 3: For each partitioned dataset, the mask result a is... 11 ,a 12 ,...,a 1n ;a 21 ,a 22 ,...,a 2n ;...;a m1 ,a m2 ,...,a mn The input is fed into the encoder. For each image, the encoder learns the semantic information of the image and compresses the image scale to obtain five feature maps M1, M2, ..., M5 at different scales.

[0017] In training step 4, feature maps M1, M2, ..., M5 are input into the decoder for image inpainting. The decoder repairs the image based on semantic information, increasing the image scale. After repair, the lower-layer feature maps are stitched together with the upper-layer images of the same scale, and the repair process continues until the final feature map scale matches the original image. The repaired image is denoted as M'.

[0018] Training step 5 involves calculating the error between the restored image and the original image using multi-dimensional loss functions. This invention employs a weighted summation of multiple loss functions, simultaneously introducing three loss functions: mean squared error (MSE), multi-scale gradient magnitude similarity loss (MSGMS), and structural similarity loss (SSIM).

[0019] Assuming the original image is X, and H and W are the height and width of the image respectively, the restored image is X', X'∈R H×W N = H × W, where i and j represent the row and column of a pixel in the image, and the MSE loss is shown in Formula 1.

[0020]

[0021] The MSGMS loss fully considers the differences in image gradient and amplitude information at different scales, and effectively reflects the model's recovery effect on image gradient and amplitude information. First, the gradient amplitude similarity characteristics of the original and reconstructed images are calculated. The gradient amplitude map calculation method is shown in Equation 2.

[0022]

[0023] Where g(I) is the gradient magnitude map of image I, h x and h y It is a 3×3 Prewitt filter along the x and y directions of the image.

[0024] Original Image I and Restored Image I r The method for calculating the magnitude gradient similarity is shown in Formula 3.

[0025]

[0026] Where c is a normal number that provides numerical stability for the image.

[0027] In calculating the MSGMS loss, to extract multi-scale features from the image, the original image is downsampled four times with a sampling window size of 2×2 and a stride of 2. Then, the gradient magnitude loss between the reconstructed image and the original image is calculated based on the similarity of their gradient magnitudes. Here, the number of pixels at scale l (l=1,2,3,4) is denoted as N. l The original image is X l The image was repaired to X. l ',X l '∈R H×W N l =H×W, where H and W are the height and width of the image, respectively, and i and j represent the row and column of a pixel in the image. The MSGMS loss is shown in Formula 4.

[0028]

[0029] The SSIM loss fully considers relevant indicators such as brightness, contrast, and structure of the image to reflect the image details. Its range is [0,1], and the larger the value, the smaller the difference between the two images. Equation 5 shows the structure of the SSIM loss.

[0030]

[0031] As shown in Formula 6, the final total loss value is obtained by weighted summation of the above losses, where λ1 and λ2 are the weight values ​​of MSGMS loss and SSIM loss.

[0032] L = L MSE +λ1L MSGMS +λ2L SSIM (6)

[0033] Training step 6: Backpropagation loss values ​​update network model parameters. This step further includes:

[0034] Based on training step 5, the loss value of the masked pixel in the restored image and the original value of the original image before masking is calculated, and backpropagation is performed. The network parameters in training steps 2 to 4 are updated by the adaptive moment estimation algorithm to finally obtain the optimized network model.

[0035] The following are the testing steps:

[0036] Test Step 1: Anomaly detection using a trained model, Mars image anomaly detection heatmap ( Figure 4 This serves as the final anomaly detection result.

[0037] The test set of the MastCam-Noveltydet dataset A is input into the trained Mars anomaly segmentation model, and the output restored image a is generated. 11 ',a 12 ',...,a 1n ';a 21 ',a 22 ',...,a 2n ';...;a m1 ',a m2 ',...,a mn The restoration results obtained by the same segmentation method are stitched together to obtain the restored images a1'; a2'; ...; a m Gradient and amplitude information restoration of abnormal pixels is more difficult than restoration of normal pixels, resulting in larger error values ​​for abnormal pixels in the error map. Image restoration a1'; a2'; ...; a is calculated pixel by pixel. m The average value of the multi-scale gradient amplitude similarity error with the original image reflects the comprehensive ability of the model to detect anomalies under m scale divisions, and its output is used as a Mars anomaly detection heatmap.

[0038] Test step 2: Calculate the quantitative results of image-level Mars anomalies.

[0039] AUROC, calculated by the area under the ROC curve, can be used as an indicator to compare model accuracy. Inputting a Mars anomaly detection heatmap and a Mars anomaly annotation map, AUROC is calculated as the image-level Mars anomaly detection result. Since the maximum pixel value is representative of the distribution, it can effectively reflect the distribution characteristics of the entire image. The maximum values ​​of all original and restored images are obtained separately, and the AUROC index is calculated, serving as the image-level quantitative result of Mars anomalies in the MastCam-Noveltydet dataset A.

[0040] In a second aspect, the present invention provides an electronic device, including a processor and a memory;

[0041] Memory, used to store computer programs;

[0042] When the processor executes the program stored in the memory, it implements the above-described method for detecting anomalies in Mars images based on a dense residual autoencoder.

[0043] The method of the present invention has the following significant effects: (1) It introduces the idea of ​​dense residuals, which reduces the degradation effect of the model and enhances the network's ability to learn deep semantics in Mars images. (2) It adopts a weighted summation of multiple loss functions, which makes the network reconstruct the background with higher separation between anomalies and background. (3) As a result, the image reconstruction problem is transformed into an image restoration problem. The model no longer focuses on low-level semantic information in the image, but instead turns to the extraction of deep semantic information, which is conducive to improving the accuracy of image-level Mars anomaly detection. Attached Figure Description

[0044] Figure 1 This is an overall flowchart of an embodiment of the present invention.

[0045] Figure 2 This is a structural diagram of the Mars anomaly segmentation model used in the embodiments of the present invention.

[0046] Figure 3 This is a structural diagram of the dense residual module used in the embodiments of the present invention.

[0047] Figure 4 This is a Mars anomaly detection heatmap of the MastCam-Noveltydet dataset A, output by an embodiment of the present invention. Detailed Implementation

[0048] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0049] This invention provides a method for detecting anomalies in Mars images based on a dense residual autoencoder, which consists of a training step and a testing step.

[0050] The following are the training steps:

[0051] Training Step 1: Preprocess Mars imagery. This step further includes: scaling the Mars imagery in the MastCam-Noveltydet dataset A to a size of [size missing]. The Mars image is scaled to 64×64, and its pixel values ​​are normalized to within the range of 0 to 1, where H, W, and B are the number of image rows, columns, and bands, respectively. The preprocessed dataset A' is output.

[0052] Training step 2: Divide the Mars dataset A' into square grids of sizes 2×2, 4×4, and 8×8, denoted as a1, a2, and a3 respectively. Then, randomly divide each partition into three disjoint sets, and successively set the pixel values ​​in each set to 0. The output is denoted as a. 11 ,a 12 ,a 13 ;a 21 ,a 22 ,a 23 ;a 31 ,a 32 ,a 33 .

[0053] Training step 3: For each partitioned dataset, the mask result a is... 11 ,a 12 ,a 13 ;a 21 ,a 22 ,a 23 ;a 31 ,a 32 ,a 33 The input is fed into the encoder. For each image, the encoder learns the semantic information of the image and compresses the image scale to obtain five feature maps M1, M2, ..., M5 at different scales.

[0054] In training step 4, feature maps M1, M2, ..., M5 are input into the decoder for image inpainting. The decoder repairs the image based on semantic information, increasing the image scale. After repair, the lower-layer feature maps are stitched together with the upper-layer images of the same scale, and the repair process continues until the final feature map scale matches the original image. The repaired image is denoted as M'.

[0055] Training step 5 involves calculating the error between the restored image and the original image using multi-dimensional loss functions. This invention employs a weighted summation of multiple loss functions, simultaneously introducing three loss functions: mean squared error (MSE), multi-scale gradient magnitude similarity loss (MSGMS), and structural similarity loss (SSIM).

[0056] Assuming the original image is X, and H and W are the height and width of the image respectively, the restored image is X', X'∈RH×W N = H × W, where i and j represent the row and column of a pixel in the image, and the MSE loss is shown in Formula 1.

[0057]

[0058] The MSGMS loss fully considers the differences in image gradient and amplitude information at different scales, and effectively reflects the model's recovery effect on image gradient and amplitude information. First, the gradient amplitude similarity characteristics of the original and reconstructed images are calculated. The gradient amplitude map calculation method is shown in Equation 2.

[0059]

[0060] Where g(I) is the gradient magnitude map of image I, h x and h y It is a 3×3 Prewitt filter along the x and y directions of the image.

[0061] Original Image I and Restored Image I r The method for calculating the magnitude gradient similarity is shown in Formula 3.

[0062]

[0063] Where c is a normal number that provides numerical stability for the image.

[0064] In calculating the MSGMS loss, to extract multi-scale features from the image, the original image is downsampled four times with a sampling window size of 2×2 and a stride of 2. Then, the gradient magnitude loss between the reconstructed image and the original image is calculated based on the similarity of their gradient magnitudes. Here, the number of pixels at scale l (l=1,2,3,4) is denoted as N. l The original image is X l The image was repaired to X. l ',X l '∈R H×W N l =H×W, where H and W are the height and width of the image, respectively, and i and j represent the row and column of a pixel in the image. The MSGMS loss is shown in Formula 4.

[0065]

[0066] The SSIM loss fully considers relevant indicators such as brightness, contrast, and structure of the image to reflect the image details. Its range is [0,1], and the larger the value, the smaller the difference between the two images. Equation 5 shows the structure of the SSIM loss.

[0067]

[0068] As shown in Formula 6, the final total loss value is obtained by weighted summation of the above losses, where λ1 and λ2 are the weight values ​​of MSGMS loss and SSIM loss.

[0069] L = L MSE +λ1L MSGMS +λ2L SSIM (6)

[0070] Training step 6: Backpropagation loss values ​​update network model parameters. This step further includes:

[0071] Based on training step 5, the loss value of the masked pixel in the restored image and the original value of the original image before masking are calculated, and backpropagation is performed. The network parameters in training steps 3 to 5 are updated by the adaptive moment estimation algorithm to finally obtain the optimized network model.

[0072] The following are the testing steps:

[0073] Test Step 1: Anomaly detection using a trained model, Mars image anomaly detection heatmap ( Figure 4 This serves as the final anomaly detection result.

[0074] The test set of the MastCam-Noveltydet dataset A is input into the trained Mars anomaly segmentation model, and the output restored image a is generated. 11 ',a 12 ',a 13 ';a 21 ',a 22 ',a 23 ';a 31 ',a 32 ',a 33 The repair results obtained by the same segmentation method are stitched together to obtain repaired images a1', a2', and a3'. Repairing the gradient and amplitude information of anomalous pixels is more difficult than repairing normal pixels, therefore the error values ​​of anomalous pixels are larger in the error map. The multi-scale gradient amplitude similarity error between the repaired images a1', a2', and a3' and the original images is calculated pixel by pixel. The average value reflects the comprehensive anomaly detection capability of the model under the three scale segmentations, and its output is used as a Mars anomaly detection heatmap.

[0075] Test step 2: Calculate the quantitative results of image-level Mars anomalies.

[0076] AUROC, calculated by the area under the ROC curve, can be used as an indicator to compare model accuracy. Inputting a Mars anomaly detection heatmap and a Mars anomaly annotation map, AUROC is calculated as the image-level Mars anomaly detection result. Since the maximum pixel value is representative of the distribution, it can effectively reflect the distribution characteristics of the entire image. The maximum values ​​of all original and restored images are obtained separately, and the AUROC index is calculated, serving as the image-level quantitative result of Mars anomalies in the MastCam-Noveltydet dataset A.

[0077] Figure 4 This image shows the Mars anomaly detection distribution results of the MastCam-Noveltydet dataset A, output in test step 1 of the dUnet embodiment of this invention. From left to right, the image shows the original image, labeled image, restored image, and anomaly detection heatmap of the MastCam-Noveltydet dataset A. Observing the original and restored images, it can be found that the restoration effect of anomalous pixels is worse than that of normal pixels, which is basically consistent with the prediction of this invention; while the anomaly distribution of the labeled image and the anomaly detection heatmap is basically consistent, meeting the mission requirements for Mars anomaly target detection.

[0078] Table 1 shows the image-level Mars anomaly detection results of the MastCam-Noveltydet dataset A, output in test step 2 of the dUnet embodiment of this invention. The comparison shows that the image-level Mars anomaly detection results of the dUnet proposed in this invention are superior to the currently popular RIAD, Cae, and GANomaly algorithms, and can better meet the requirements of image-level Mars anomaly detection.

[0079] Table 1: Quantitative results of image-level anomalies in the MastCam-Noveltydet dataset A

[0080]

[0081] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for detecting anomalies in Mars images based on a dense residual autoencoder, characterized in that, The implementation steps include the following: Step 1: Input the Mars image set and scale and normalize the images in the dataset so that the pixel values ​​are within the range of 0 to 1. Step 2: Divide the Mars dataset into m different sized square grids, denoted as . Each partition result is randomly divided into n disjoint sets, and the pixel values ​​in each set are successively set to 0. The output result is denoted as... ; Step 3: Apply the masking results corresponding to each partitioned dataset. The input is fed into the encoder to obtain multiple feature maps at different scales; The encoder consists of five modules. The first module is a "convolutional layer-normalization layer-nonlinear activation layer" structure, used to extract semantic information from the surface of the image. The second, third, and fourth modules each consist of a dense residual module, a "convolutional layer-normalization layer-nonlinear activation layer" structure, and an average pooling layer. The dense residual module consists of multiple convolutional layers, which are interconnected. The average pooling layer is used to extract feature information from the image and simultaneously compress the image scale. The fifth module is composed of a "convolutional layer-normalization layer-nonlinear activation layer" structure and is used for deep semantic learning and image scale compression. Step 4: Input feature maps of different scales into the decoder for image inpainting; The decoder consists of four deconvolutional layers. Each deconvolutional layer outputs a restored image based on the learned semantic information, so that the decoder finally outputs a restored image with the same size as the original image. The four deconvolutional layers of the encoder are connected to the modules at the same level in the decoder in sequence, forming the connection layer of the model. Step 5: Calculate the error between the inferred image and the original image using a multi-dimensional loss function; Step 6: Update the encoder-decoder model parameters using the backpropagation loss function value. The encoder-decoder model is a Unet structure. Step 7: Use the trained model to perform anomaly detection, and use the Mars image anomaly detection heatmap as the final anomaly detection result. Record the output restored image as By stitching together the restoration results obtained using the same segmentation method, a restored image is obtained. Image restoration is calculated pixel by pixel. The average value of the multi-scale gradient magnitude similarity error with the original image reflects the comprehensive ability of the model to detect anomalies under m scale divisions, and its output is used as a Mars anomaly detection heatmap.

2. The Mars image anomaly detection method based on a dense residual autoencoder as described in claim 1, characterized in that: In step 2, the sizes are respectively... , , The Mars dataset is divided into square grids, denoted as . Each partition result is randomly divided into three disjoint sets. The pixel values ​​in each set are successively set to 0. The output result is denoted as... .

3. The Mars image anomaly detection method based on a dense residual autoencoder as described in claim 1, characterized in that: In step 5, the multi-dimensional loss function includes three loss functions: mean squared error (MSE) loss, multi-scale gradient magnitude similarity (MSGMS) loss, and structural similarity loss. The final total loss value is obtained by weighted summation of the three loss functions.

4. The Mars image anomaly detection method based on a dense residual autoencoder as described in claim 3, characterized in that: Set the original image as , and These are the height and width of the image, respectively. The restored image is... , , , and Let represent the row and column of a pixel in the image, and the MSE loss is shown in formula (1); （1）。 5. The Mars image anomaly detection method based on a dense residual autoencoder as described in claim 3, characterized in that: The MSGMS loss is calculated as follows: First, the gradient magnitude similarity characteristics between the original image and the reconstructed image are calculated. The gradient magnitude map is calculated as shown in formula (2): （2） in It is an image gradient magnitude plot, and It is along the image and Direction Prewitt filter; Original image With restored images The method for calculating the magnitude gradient similarity is shown in formula (3); （3） Where c is a normal constant that provides numerical stability for the image; In calculating the MSGMS loss, to extract multi-scale features from the image, the original image is downsampled four times, with a sampling window size of [size missing]. The stride is 2, and the gradient magnitude loss between the restored image and the original image is calculated based on the similarity of the image gradient magnitudes. The number of pixels at the scale is denoted as The original image is The image was restored as , , , and These are the height and width of the image, respectively. , and Let represent the row and column of a pixel in the image. The MSGMS loss is shown in formula (4): （4）。 6. The Mars image anomaly detection method based on a dense residual autoencoder as described in claim 3, characterized in that: The SSIM loss is shown in Equation (5): （5） The final total loss value is shown in formula (6): （6） in and MSGMS loss and SSIM loss The weight value, This is the MSE loss.

7. A Mars image anomaly detection device based on a dense residual autoencoder, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the program instructions in the memory to execute the Mars image anomaly detection method based on a dense residual autoencoder as described in any one of claims 1-6.

Images