A Martian impact crater identification method based on multimodal feature fusion

Abstract

This invention relates to the field of remote sensing image processing and intelligent target recognition technology, and discloses a Martian impact crater recognition method based on multimodal feature fusion. The method includes the following steps: constructing a multimodal dataset using three heterogeneous remote sensing data sets: Martian infrared imagery, color imagery, and digital elevation model (DEM); preprocessing the three heterogeneous remote sensing data sets and constructing a multi-scale image pyramid; inputting the three heterogeneous remote sensing data sets into three independent CNN-Transformer hybrid backbone networks respectively, and extracting local detail features through convolutional structures. This invention significantly improves the accuracy of Martian impact crater recognition, effectively reducing the false negative rate. By introducing a multimodal feature fusion mechanism, it significantly improves the overall recall rate of Martian impact crater recognition, especially for degraded and secondary impact craters that are difficult to accurately identify using traditional methods, effectively alleviating the serious false negative problem in existing technologies.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and intelligent target recognition technology, and in particular to a method for identifying Martian impact craters based on multimodal feature fusion. Background Technology

[0002] Martian impact craters, also known as meteorite craters or impact depressions, are ring-shaped depressions formed on the surface of planets, satellites, asteroids, or other terrestrial bodies due to meteorite impacts. As one of the most thoroughly studied terrestrial planets in the solar system, Mars has long been a crucial target for planetary science and the search for extraterrestrial life. The spatial distribution, morphological characteristics, and evolutionary patterns of Martian impact craters are of significant scientific importance for revealing Martian geological history, surface age, and tectonic activity. Therefore, high-precision, automated identification of Martian impact craters is a key technological requirement in planetary science research.

[0003] Currently, Mars impact crater identification technologies mainly fall into two categories: manual identification and computer vision-based automatic identification. Manual interpretation relies on expert experience to analyze remote sensing images, achieving high accuracy but low efficiency, making it difficult to meet the processing demands of massive amounts of Mars exploration data. Traditional automatic identification methods are mostly based on image processing algorithms or statistical machine learning models, such as Hough transform, template matching, or support vector machines. These methods typically rely on manually designed features, are sensitive to noise and complex backgrounds, have limited generalization ability, and are difficult to adapt to the complex and varied terrain of the Martian surface.

[0004] With the development of deep learning technology, object detection models based on convolutional neural networks have been gradually introduced into the field of Mars impact crater identification. For example, models such as YOLO and Faster R-CNN have been applied to impact crater detection in visible light or infrared imagery and have achieved certain results on labeled datasets. However, most of these methods rely on a single type of remote sensing data, such as using only visible light imagery or only digital elevation models (DEMs), making it difficult to fully overcome the inherent limitations of a single data source. Visible light imagery is susceptible to changes in lighting conditions and shadows, infrared imagery has limited ability to resolve terrain details, and DEMs are insufficient in their response to small-scale or degraded impact craters. These factors limit the accuracy and stability of existing methods under complex terrain conditions.

[0005] To improve the robustness of Martian impact crater identification, some studies have begun to explore the fusion of multi-source remote sensing data, such as combining imagery data with topographic data. However, existing multimodal fusion methods mostly remain at the decision-level fusion or simple feature stitching level, lacking effective modeling of the deep interaction relationships between multimodal features, failing to fully explore the complementarity between spectral, textural, and topographic information, and are difficult to adapt to the complex and varied terrain of the Martian surface.

[0006] Furthermore, existing impact crater identification models generally suffer from limitations in scale perception. The identification capabilities of these models vary across different scales and morphological types (such as simple craters, complex craters, degraded craters, and secondary impact craters), particularly in complex terrain regions like volcanic areas and transitional zones between the Northern and Southern Hemispheres, where the models exhibit poor generalization ability and robustness. This leads to significant differences in the number and accuracy of publicly available Mars impact crater databases constructed using different research methods, especially in matching accuracy within key scale ranges, thus limiting the effective application of impact crater data in planetary science research.

[0007] In recent years, the Transformer architecture has demonstrated excellent performance in computer vision tasks due to its global context modeling capabilities, and is gradually forming a hybrid architecture with convolutional neural networks. However, in the field of planetary remote sensing, especially in Mars impact crater identification tasks, there is still a lack of mature and systematic technical solutions for combining Transformer with multimodal feature deep fusion technology to simultaneously process heterogeneous remote sensing data such as infrared imagery, color imagery, and digital elevation models. Existing methods have not yet achieved an end-to-end multimodal deep fusion framework for complex terrain environments, making it difficult to achieve high-precision and high-recall automatic identification of various types of Mars impact craters across the entire scale.

[0008] In summary, existing Mars impact crater identification technologies still have shortcomings in terms of data modality utilization, feature fusion depth, scale perception capability, and robustness under complex terrain conditions. There is an urgent need for an automatic Mars impact crater identification technology that can fully integrate multimodal remote sensing data, has multi-scale perception capability, and adapts to complex terrain environments. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for identifying Mars impact craters based on multimodal feature fusion.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A method for identifying Martian impact craters based on multimodal feature fusion includes the following steps:

[0012] A multimodal dataset was constructed using three heterogeneous remote sensing data: Mars infrared imagery, color imagery, and digital elevation model. The three heterogeneous remote sensing data were preprocessed, and a multi-scale image pyramid was constructed.

[0013] The three types of heterogeneous remote sensing data are respectively input into three independent CNN-Transformer hybrid backbone networks. Local detail features are extracted through convolutional structures, and global context features are extracted through self-attention modules to obtain multi-scale features.

[0014] The obtained multi-scale features are progressively fused. In the shallow feature stage, the homogeneous modal features of infrared and color images are channel fusion. In the deep feature stage, the deep features of infrared and color images are spectral feature interaction through a cross-attention module with shared weights. Then, the spectral features after spectral feature interaction are fused with the terrain features of the digital elevation model through another cross-attention module.

[0015] The fused multi-scale features are input into a Transformer-based visual encoder, and the fused multi-scale features are encoded through a multi-scale deformable attention mechanism to output an encoded feature sequence.

[0016] The encoded feature sequence is input into a Transformer-based visual decoder. The encoded feature sequence interacts with a learnable target query vector, outputting a fixed number of predicted bounding boxes and corresponding classification confidence scores. A one-to-one set matching mechanism based on the Hungarian algorithm is used to achieve the optimal matching between the prediction results and the ground truth labels.

[0017] The model is trained based on a joint loss function, which includes the main detection head loss function and multiple auxiliary detection head loss functions. Dynamic denoising training is introduced during the training process to improve the robustness of the model.

[0018] Preferably, the steps of constructing a multimodal dataset using three heterogeneous remote sensing data—Mars infrared imagery, color imagery, and digital elevation model (DEM)—and preprocessing the three heterogeneous remote sensing data to construct a multi-scale image pyramid include:

[0019] Preprocessing is performed on three types of heterogeneous remote sensing data: Mars infrared imagery, color imagery, and digital elevation model. The preprocessing includes spatial alignment and georegistration of the three types of heterogeneous remote sensing data, and unification to the same spatial resolution, so that the different modal data correspond one-to-one in spatial location.

[0020] Based on three types of preprocessed heterogeneous remote sensing data, a multimodal dataset containing information on multiple modalities is constructed.

[0021] The aligned remote sensing data is cut into image patches of fixed size, which serve as the basic input units for model training and inference.

[0022] A multi-scale image pyramid is constructed based on the image patches, and images of different scales are downsampled before being input into the network to achieve the identification of Martian impact craters of different scales.

[0023] Preferably, the fixed-size image block is a 640×640 pixel image block.

[0024] Preferably, the scale input of the multi-scale image pyramid includes 1280×1280 and 2560×2560.

[0025] Preferably, the cross-attention module generates query vectors, key vectors, and value vectors through depthwise separable convolution, obtains attention weights by normalization using the Softmax function, and performs weighted fusion on the features corresponding to the value vectors.

[0026] Preferably, in the cross-attention module, features from two different modalities are used as inputs, wherein the query vector comes from the input features of the first modality, and the key vector and value vector come from the input features of the second modality. The output is obtained by weighted summation of the features corresponding to the value vector and adding it to the input features of the first modality.

[0027] Preferably, the steps of inputting the three types of heterogeneous remote sensing data into three independent CNN-Transformer hybrid backbone networks, extracting local detail features through convolutional structures, and extracting global contextual features through self-attention modules to obtain multi-scale features include:

[0028] In the shallow stages of the CNN-Transformer hybrid backbone network, convolutional structures are used to process the input remote sensing data to extract local detail features.

[0029] In the deep stages of the CNN-Transformer hybrid backbone network, a self-attention module is introduced to process the features extracted in the shallow stages in order to extract global contextual features.

[0030] The shallow and deep stages output features of different scales respectively, forming multi-scale features.

[0031] Preferably, the joint loss function includes a quality focus loss function for optimizing classification confidence and localization quality, a bounding box loss function for regressing position and size, and an intersection-union loss function for directly optimizing the overlap between the predicted box and the ground truth box.

[0032] A Mars impact crater identification system based on multimodal feature fusion, the system comprising:

[0033] The multimodal data construction and preprocessing module is used to construct a multimodal dataset using three heterogeneous remote sensing data: Mars infrared imagery, color imagery, and digital elevation model. It also preprocesses the three heterogeneous remote sensing data and constructs a multi-scale image pyramid.

[0034] The multi-branch feature extraction module is used to input the three types of heterogeneous remote sensing data into three independent CNN-Transformer hybrid backbone networks, extract local detail features through convolutional structures, and extract global context features through self-attention modules to obtain multi-scale features.

[0035] The multimodal feature fusion module is used to progressively fuse the obtained multi-scale features. In the shallow feature stage, channel fusion is performed on the homogeneous modal features of infrared and color images. In the deep feature stage, the deep features of infrared and color images are interacted through a cross-attention module with shared weights. Then, the spectral features after spectral feature interaction are fused with the terrain features of the digital elevation model through another cross-attention module.

[0036] The feature encoding module is used to input the fused multi-scale features into the Transformer-based visual encoder, encode the fused multi-scale features through a multi-scale deformable attention mechanism, and output the encoded feature sequence.

[0037] The target decoding and detection module is used to input the encoded feature sequence into a Transformer-based visual decoder, interact with the encoded feature sequence through a learnable target query vector, output a fixed number of predicted bounding boxes and corresponding classification confidence scores, and use a one-to-one set matching mechanism based on the Hungarian algorithm to achieve the optimal matching between the prediction results and the real annotations.

[0038] The model training module is used to train the model based on a joint loss function, which includes the main detection head loss function and multiple auxiliary detection head loss functions. Dynamic denoising training is introduced during the training process to improve the robustness of the model.

[0039] Compared with existing technologies, the present invention demonstrates significant and verifiable technical effects in Mars impact crater identification tasks, specifically in the following aspects.

[0040] This invention significantly improves the accuracy of Mars impact crater identification, effectively reducing the false negative rate. By introducing a multimodal feature fusion mechanism, this invention significantly improves the overall recall rate of Mars impact craters, especially for degraded and secondary impact craters that are difficult to identify accurately using traditional methods, effectively alleviating the serious false negative problem in existing technologies. Validation on a standard test set shows that the overall recall rate of this invention reaches 0.940, with the recall rate for degraded impact craters increasing to 0.887 and the recall rate for secondary impact craters increasing to 0.861. These effects are mainly attributed to the multimodal feature fusion mechanism, which uses a cross-attention module to jointly model the spectral information of Mars infrared images, the texture details of color images, and the topographic features of digital elevation models. This allows the model to obtain a more comprehensive and robust feature representation, thereby enhancing the ability to identify impact craters with blurred or partially incomplete shapes.

[0041] This invention possesses full-scale perception capabilities, enabling high-precision identification of Martian impact craters of varying scales. Addressing the issue of uneven perception capabilities in existing models when crater scales vary significantly, this invention combines multi-scale feature extraction with a multi-scale image pyramid to achieve effective identification of Martian impact craters ranging from sub-kilometer to hundreds of kilometers in scale. Experimental verification shows that the model maintains high recall rates in both extremely small-scale craters (diameter less than 2 km) and extremely large-scale craters (diameter greater than 80 km). This is primarily attributed to the multi-scale feature extraction backbone network's ability to output multi-scale features at different levels, while the image pyramid preprocessing strategy ensures that impact craters of different scales are input into the network at appropriate scales, thereby enhancing the model's overall perception capability for targets of different scales.

[0042] This invention exhibits strong robustness in complex terrain environments, maintaining stable recognition performance under various complex background conditions. Through a multimodal feature fusion strategy and Transformer-based global context modeling capabilities, it achieves stable recognition with high accuracy even in complex Martian terrain regions. In complex terrain areas such as volcanic activity zones and the transition zone between the northern and southern hemispheres, where traditional algorithms are susceptible to background interference, this invention effectively reduces false positives and false negatives. This robustness primarily stems from the Transformer encoder's ability to model global context information and the deep fusion of spectral and terrain information, enabling the model to better distinguish between impact crater structures and surrounding complex terrain features.

[0043] Furthermore, this invention demonstrates significant scientific value and practical benefits in real-world applications. By applying this invention to regional detection within 65° north and south latitude on Mars, over 630,000 Martian impact craters were automatically identified, including approximately 280,000 newly discovered craters not yet included in existing mainstream databases (including the Robbins and Hynek databases), while maintaining a low false positive rate of 9.4%. These results indicate that this invention effectively suppresses false positives while maintaining a high recall rate, providing richer and more reliable impact crater data support for Martian geological evolution research. This achievement is closely related to the one-to-one set matching mechanism employed, which avoids biases introduced during post-processing of prediction results. Simultaneously, cross-validation between multimodal information fundamentally reduces the possibility of misclassifying non-crater terrain as impact craters. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the overall structure of a Mars impact crater identification method based on multimodal feature fusion proposed in an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram of the progressive fusion process and module structure of a Mars impact crater identification method based on multimodal feature fusion proposed in an embodiment of the present invention.

[0046] Figure 3 This is a schematic diagram of the cross-attention module structure in a Mars impact crater identification method based on multimodal feature fusion proposed in an embodiment of the present invention. Detailed Implementation

[0047] To provide a clearer understanding of the technical features, objectives, and beneficial effects of this invention, the technical solution of this invention is described in detail below, but this should not be construed as limiting the scope of implementation of this invention. Unless otherwise specified, the methods used in this invention are conventional methods in this technical field. In this invention, materials, reagents, or instruments whose manufacturers are not specified are all conventional products that can be purchased commercially.

[0048] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0049] Reference Figures 1 to 3 This invention proposes a method for identifying Martian impact craters based on multimodal feature fusion, specifically including the following steps:

[0050] Step S100: Construct a multimodal dataset using three heterogeneous remote sensing data: Mars infrared imagery, color imagery, and digital elevation model. Preprocess the three heterogeneous remote sensing data and construct a multi-scale image pyramid.

[0051] Step S100 specifically includes the following steps:

[0052] Step S101: Preprocess the three heterogeneous remote sensing data, namely, Mars infrared image, color image and digital elevation model. The preprocessing includes spatial alignment and georegistration of the three heterogeneous remote sensing data and unifying them to the same spatial resolution so that the different modal data correspond one-to-one in spatial location.

[0053] Step S102: Based on the three types of heterogeneous remote sensing data that have completed preprocessing, construct a multimodal dataset containing information on multiple modalities;

[0054] Step S103: Cut the aligned remote sensing data into image patches of fixed size, which serve as the basic input units for model training and inference;

[0055] Step S104: Construct a multi-scale image pyramid based on the image patch, and perform downsampling processing on images of different scales before inputting them into the network to achieve the identification of Martian impact craters of different scales.

[0056] In this embodiment of the invention, three heterogeneous remote sensing data—Mars infrared imagery, color imagery, and digital elevation model (DEM)—are simultaneously used as input data sources. The Mars infrared imagery is THEMIS daytime infrared imagery, the color imagery is the Tianwen-1 MoRIC color imagery, and the DEM is the HRSC-MOLA digital elevation model. Because these three heterogeneous remote sensing data sources differ in their resolution and projection methods, preprocessing is required before constructing the multimodal dataset to ensure a one-to-one spatial correspondence between the different modalities.

[0057] The preprocessing includes spatial alignment and georegistration of Mars infrared imagery, color imagery, and digital elevation model. By unifying the three heterogeneous remote sensing data to the same spatial resolution and performing the same projection transformation, the same geographical location corresponds to the same pixel position in different modal data. In this embodiment, the three heterogeneous remote sensing data are unified to a spatial resolution of 75 meters per pixel, thereby ensuring the accurate spatial correspondence between the different modal data.

[0058] After spatial alignment and georegistration, a multimodal dataset containing various modal information is constructed based on preprocessed Mars infrared and color images and a digital elevation model. Subsequently, the aligned remote sensing data is segmented into fixed-size image patches, which serve as the basic input units for model training and inference. In this embodiment, the fixed-size image patch is a 640×640 pixel image patch.

[0059] Based on this, a multi-scale image pyramid is constructed using the image patches to accommodate the identification needs of Martian impact craters at different scales. When constructing the multi-scale image pyramid, image formats with different scales, such as 1280×1280 pixels and 2560×2560 pixels, are used. Before inputting the images into the network, downsampling is performed on the images at different scales to ensure they meet the network's input size requirements. This ensures consistency in network input while effectively preserving and utilizing the overall morphological features of large-scale impact craters.

[0060] Furthermore, the accurate annotations of impact craters are sourced from authoritative manual databases, including the Robbins-Hynek database and impact crater annotation data established by Lagain, among others. To ensure the accuracy of the monitoring information, special structural features such as splash zones in some layered impact craters were manually corrected and supplemented with additional annotations, thereby improving the consistency between the accurate annotations and the actual terrain features.

[0061] Furthermore, the Mars impact crater identification method based on multimodal feature fusion also includes the following steps:

[0062] Step S200: Input the three types of heterogeneous remote sensing data into three independent CNN-Transformer hybrid backbone networks respectively, extract local detail features through convolutional structure, and extract global context features through self-attention module to obtain multi-scale features.

[0063] Step S200 specifically includes the following steps:

[0064] Step S201: In the shallow stage of the CNN-Transformer hybrid backbone network, the input remote sensing data is processed using a convolutional structure to extract local detail features.

[0065] Step S202: In the deep stage of the CNN-Transformer hybrid backbone network, a self-attention module is introduced to process the features extracted in the shallow stage in order to extract global context features.

[0066] Step S203: Features of different scales are output from the shallow stage and the deep stage respectively to form multi-scale features.

[0067] In this embodiment of the invention, a three-branch input architecture is adopted, in which three heterogeneous remote sensing data—Mars infrared imagery, color imagery, and digital elevation model—are respectively input into three independent CNN-Transformer hybrid backbone networks for feature extraction. Each backbone network has the same structure but independent parameters to accommodate the differences in feature distribution among the different modalities of remote sensing data.

[0068] The CNN-Transformer hybrid backbone network achieves multi-scale feature extraction from input remote sensing data through the synergistic effect of convolutional structures and self-attention modules. In the feature extraction process, the network first processes the input remote sensing data through convolutional structures in the shallow stage to extract local detail features; in the deep stage, the self-attention module further processes the features extracted by the convolutional structures to extract global contextual features. This staged processing approach, combining shallow and deep layers, enables the network to effectively acquire global contextual information while preserving local detail information.

[0069] In the CNN-Transformer hybrid backbone network, the convolutional structure and the self-attention module output feature representations at different scales in the shallow and deep stages, respectively, thereby forming multi-scale features for subsequent multimodal feature fusion and Mars impact crater identification processing.

[0070] In this embodiment, a lightweight CNN-Transformer hybrid architecture is preferably used as the backbone network of each branch to reduce computational complexity while ensuring feature representation capability, thereby achieving efficient identification of Martian impact craters.

[0071] Furthermore, the Mars impact crater identification method based on multimodal feature fusion also includes the following steps:

[0072] Step S300: Progressive fusion of the obtained multi-scale features is performed. In the shallow feature stage, channel fusion is performed on the homogeneous modal features of infrared and color images. In the deep feature stage, spectral feature interaction is performed on the deep features of infrared and color images through a cross-attention module with shared weights. Then, cross-modal fusion is performed on the spectral features after spectral feature interaction and the terrain features of the digital elevation model through another cross-attention module.

[0073] In this embodiment of the invention, multi-scale features extracted by the CNN-Transformer hybrid backbone network are fed into a multimodal feature fusion module for fusion processing. The multimodal feature fusion module adopts a progressive fusion strategy, performing different forms of fusion operations in the shallow feature stage and the deep feature stage according to the different layers of the features.

[0074] In the shallow feature stage ( Figure 2 In the middle (S1, S2), corresponding to the shallow features in the multi-scale features, channel fusion processing is performed on the homogeneous modal features of infrared and color images. The features of the two spectral modes are merged by channel fusion to fuse the differences between spectral information, and local details such as texture and edges of small-scale Martian impact craters are preserved during the fusion process.

[0075] In the deep feature stage ( Figure 2 In section S3 and S4, corresponding to the deep features in the multi-scale features, spectral feature interaction and cross-modal fusion of spectral features and terrain features are performed sequentially. First, a cross-attention module with shared weights performs spectral feature interaction on the deep features of infrared and color images to achieve information fusion between the two spectral modalities. Subsequently, another cross-attention module performs cross-modal fusion of the spectral features after spectral feature interaction with the terrain features extracted by the digital elevation model, thereby combining terrain undulation information with spectral texture information to enhance the expressive ability of Martian impact crater structural features.

[0076] The Cross-Attention module (Cross-Attn) takes features from two different modalities as input, with features from one modality used to generate the query vector. Another modality's features are used to generate key vectors. Sum value vector Specifically, the cross-attention module generates query vectors through depthwise separable convolutions. Key vector Sum value vector and through query vectors With key vector The correlation between features is calculated using matrix multiplication, and then normalized using the Softmax function to obtain the attention weights. These attention weights reflect the degree of correlation between different modal features in spatial location. The normalization calculation process is shown in the following equation:

[0077] ;

[0078] in, Scaling factor For the number of channels, The number of attention heads controls the amount of attention energy, and the Softmax function converts the attention energy into a probability distribution.

[0079] After obtaining the attention weights, the features corresponding to the value vectors are weighted and summed using these weights to obtain an enhanced feature representation. This enhanced feature representation is then added to the input features of the modality corresponding to the query vector, serving as the output of the cross-attention module. This processing method enables features from one modality to adaptively acquire complementary information from features from another modality.

[0080] By performing the above fusion operations in the shallow feature stage and the deep feature stage respectively, the progressive fusion of multi-scale features is achieved, providing a fused multimodal feature representation for subsequent feature encoding and Mars impact crater identification.

[0081] Furthermore, the Mars impact crater identification method based on multimodal feature fusion also includes the following steps:

[0082] Step S400: Input the fused multi-scale features into a Transformer-based visual encoder, encode the fused multi-scale features through a multi-scale deformable attention mechanism, and output the encoded feature sequence.

[0083] In this embodiment of the invention, the multi-scale features processed by the multi-modal feature fusion module are input into a Transformer-based visual encoder for encoding. The visual encoder employs a multi-scale deformable attention mechanism, which can effectively model long-distance dependencies with low computational cost and encode the fused multi-scale features to obtain an encoded feature sequence.

[0084] The described multi-scale deformable attention mechanism enables joint modeling of features at different scales during the encoding process, allowing the visual encoder to model the contextual relationships between features while maintaining computational efficiency. By performing the above encoding operation on the fused multi-scale features, an encoded feature sequence containing contextual information is obtained, which is used for subsequent feature decoding and Mars impact crater identification processing. This is crucial for locating and identifying impact craters in complex backgrounds.

[0085] Furthermore, the Mars impact crater identification method based on multimodal feature fusion also includes the following steps:

[0086] Step S500: Input the encoded feature sequence into the Transformer-based visual decoder, interact with the encoded feature sequence through a learnable target query vector, output a fixed number of predicted bounding boxes and corresponding classification confidence scores, and use a one-to-one set matching mechanism based on the Hungarian algorithm to achieve the optimal matching between the prediction results and the ground truth annotations.

[0087] In this embodiment of the invention, a Transformer-based visual decoder receives an encoded feature sequence output by a visual encoder and a set of learnable target query vectors. The decoder decodes the encoded features through the interaction between the target query vectors and the encoded feature sequence. The visual decoder uses a multi-layer decoding structure to progressively interact the target query vectors with the encoded feature sequence, thereby generating a prediction result for Mars impact crater identification.

[0088] The visual decoder outputs a fixed number of predicted bounding boxes (including category confidence, center point coordinates, and size). These predicted bounding boxes represent the spatial location range of a Martian impact crater, and the category confidence represents the probability that the corresponding prediction belongs to a Martian impact crater.

[0089] After the prediction results are generated, a one-to-one set matching mechanism is used, based on the Hungarian algorithm, to match the prediction results with the ground truth labels, in order to achieve the optimal correspondence between the prediction results and the ground truth labels. During model training, the model parameters are optimized based on the matching results.

[0090] In addition, during the training phase, multiple auxiliary detection heads can be set up to assist in the prediction of the encoded feature sequences and provide additional supervision information for model training, thereby working together with the main detection head to complete the model training process.

[0091] Furthermore, the Mars impact crater identification method based on multimodal feature fusion also includes the following steps:

[0092] Step S600: Train the model based on the joint loss function, which includes the main detection head loss function and multiple auxiliary detection head loss functions. Dynamic denoising training is introduced during the training process to improve the robustness of the model.

[0093] In this embodiment of the invention, the model training employs a joint loss function to optimize the model parameters. This joint loss function includes a main detector head loss function and multiple auxiliary detector head loss functions. The main detector head loss function is used to optimize the classification confidence and localization results of the prediction results, ensuring that each real target corresponds to a unique predicted bounding box, thereby completing the task of detecting Martian impact craters.

[0094] The joint loss function also includes multiple auxiliary detection head loss functions to provide auxiliary supervision signals for the model training process. These auxiliary detection head loss functions are applied to different detection branches to provide denser supervision information to the encoder output, thereby enhancing the discriminability of the encoded features.

[0095] In this embodiment, the joint loss function can be expressed as a weighted sum of multiple loss function terms, wherein the main detection head loss function corresponds to the loss term of the main detection task, and the multiple auxiliary detection head loss functions correspond to the loss terms of the auxiliary detection branches. Each loss term is weighted and combined by a preset weight coefficient to form the total loss function used in the model training process.

[0096] In this embodiment, the joint loss function includes a quality focus loss function (used to optimize classification confidence and localization quality), a bounding box loss function (used to regress position and size), and an intersection-union loss function (used to directly optimize the overlap between the predicted box and the ground truth box).

[0097] The joint loss function is defined as follows:

[0098]

[0099] in, For the total loss function, The loss function of the main detection head is used to maintain end-to-end detection capability, ensuring that each real target has a unique predicted bounding box, thereby avoiding the use of non-maximum suppression for post-processing.

[0100] , , These are the loss functions corresponding to the RPN auxiliary detection head, ATSS auxiliary detection head, and RoI auxiliary detection head, respectively, used to provide denser one-to-many supervision signals for the encoder output, thereby enhancing the discriminability of the encoded features. , , These are the hyperparameters used to adjust the weights of the loss function for each auxiliary detection head.

[0101] By introducing the aforementioned joint loss function, the decoder can obtain a large number of stable positive sample queries during training, thereby improving the learning efficiency and training stability of the cross-attention module.

[0102] By using the aforementioned joint loss function for training, the model can simultaneously consider detection accuracy and feature representation ability during training, thereby improving the training stability and detection performance of the model in the Mars impact crater identification task.

[0103] Meanwhile, dynamic denoising training is introduced during the model training process. By dynamically perturbing the input samples during the training phase, the model can maintain stable feature learning ability even in the presence of noise interference, thereby enhancing the robustness of the model in complex scenarios.

[0104] To verify the effectiveness of this invention, the method described herein was validated on a constructed test set. The experimental data comprehensively utilized THEMIS infrared imagery, Tianwen-1 MoRIC color imagery, and the HRSC-MOLA digital elevation model, and Martian impact craters were labeled with reference to the Robbins-Hynek artificial database and artificial databases established by Lagain et al.

[0105] In the experimental setup, the training set covered 22 typical regions of Mars, containing a total of 24,125 impact craters; the test set covered 30 regions, containing a total of 24,721 impact craters. These datasets were used to validate the model's ability to identify Martian impact craters under different regional conditions.

[0106] During the experiment, the MC-DETR model proposed in this invention (which uses a multimodal fusion method combining infrared imagery, color imagery, and digital elevation model) was compared with various single-modal and dual-modal fusion models. The focus was on evaluating its recognition performance on four typical Martian impact craters. The experimental results are shown in Table 1. The MC-DETR model proposed in this invention outperforms the comparative methods in terms of overall impact crater recognition recall.

[0107] Table 1. Comparative Experimental Results of MC-DETR

[0108]

[0109] As shown in Table 1, the experimental results demonstrate that the MC-DETR model proposed in this invention achieves superior recognition performance for most Martian impact crater types, outperforming or approaching the best comparative method overall. Particularly noteworthy is the highest recall rate of 0.887 for the most difficult-to-identify degraded impact crater type. This indicates that the recall rate of the method described in this invention on this test set is higher than the control method, fully demonstrating the significant advantages of multimodal feature fusion in characterizing blurred and degraded morphologies.

[0110] For standard impact crater types, the MC-DETR model also achieved good recognition results, with a recall rate of 0.968, demonstrating stable performance in typical impact crater recognition tasks. For layered impact craters and secondary / secondary impact craters, although the recall rate of the MC-DETR model is slightly lower than that of some bimodal combinations, its overall recognition performance is more balanced, without any obvious weaknesses.

[0111] The experimental results above demonstrate that this invention, by deeply fusing information from three modalities—Mars infrared images, color images, and digital elevation models—can effectively address the challenge of identifying various complex types of impact craters on the Martian surface, thereby enhancing the model's comprehensive detection capabilities and robustness.

[0112] The core of this invention lies in providing a complete intelligent identification scheme for Mars impact craters based on deep learning. Its innovation is mainly reflected in the collaborative utilization of multimodal data, the architecture design of multimodal feature fusion, and the end-to-end Mars impact crater identification framework. The key technical points and the technical solutions to be protected by this invention are systematically described below.

[0113] This invention seeks protection for a method for identifying Mars impact craters based on multimodal feature fusion, the overall method architecture of which constitutes an important part of the protected content. The method employs an end-to-end processing flow, starting with the input of multimodal remote sensing data, sequentially completing feature extraction, multimodal feature fusion, and the identification and localization of Mars impact craters, forming a complete and trainable deep learning model. Through this end-to-end processing flow, joint modeling of multimodal information and automatic identification of Mars impact craters are achieved.

[0114] In the overall methodological architecture described above, three types of heterogeneous remote sensing data are explicitly and simultaneously used as model inputs: Martian infrared imagery, color imagery, and digital elevation model (DEM). By simultaneously introducing these three types of heterogeneous remote sensing data, the model can comprehensively utilize spectral and topographic information, providing richer and more complementary data support for the identification of Martian impact craters.

[0115] This invention also claims protection for the multi-branch feature extraction network structure used in the method. The feature extraction network employs a three-branch CNN-Transformer hybrid backbone network structure, with each branch operating independently and processing remote sensing data of different modalities. Each backbone network adopts a staged design, extracting local detail features through convolutional structures in the shallow stage and introducing a self-attention mechanism to capture global contextual information in the deep stage, thereby achieving multi-scale feature extraction from the input data.

[0116] Furthermore, the multi-branch CNN-Transformer hybrid backbone network outputs feature representations at multiple scales, forming multi-scale features, which provide basic support for subsequent multimodal feature fusion and Mars impact crater identification.

[0117] The multimodal feature fusion module is the core component for achieving the technical effects of this invention and is also one of its key protected aspects. This module employs a progressive fusion strategy, following the progressive fusion principle of "first fusing homogeneous modal features, then interacting with heterogeneous modal features." In the shallow feature stage, features from similar Martian infrared and color images are preferentially fused. For example, channel fusion is used to merge the two spectral modal features to preserve detailed information about small-scale Martian impact craters.

[0118] In the deep feature stage, multimodal features are further fused. First, a cross-attention module is used to achieve spectral feature interaction between deep features of Martian infrared and color images. Then, the spectral features after spectral feature interaction are fused with the topographic features of the digital elevation model across modes, thereby achieving a joint expression of spectral and topographic information to enhance the model's ability to identify different types of Martian impact craters.

[0119] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A Mars impact crater identification method based on multi-modal feature fusion, characterized in that, The method comprises the following steps: a multi-modal data set is constructed using three heterogeneous remote sensing data of Mars infrared images, color images and digital elevation models, the three heterogeneous remote sensing data are preprocessed, and a multi-scale image pyramid is constructed; the three heterogeneous remote sensing data are respectively input into three independent CNN-Transformer hybrid backbone networks, local detail features are extracted through convolution structure, and global context features are extracted through self-attention modules, to obtain multi-scale features; the obtained multi-scale features are progressively fused, wherein, in the shallow feature stage, the channel fusion of the homogeneous modal features of the infrared image and the color image is performed, in the deep feature stage, the spectral feature interaction of the deep features of the infrared image and the color image is performed through the cross-attention module with shared weights, and another cross-attention module is used to perform cross-modal fusion of the spectral features after the spectral feature interaction and the terrain features of the digital elevation model; the fused multi-scale features are input into a Transformer-based visual encoder, the fused multi-scale features are encoded through a multi-scale deformable attention mechanism, and an encoded feature sequence is output; the encoded feature sequence is input into a Transformer-based visual decoder, the encoded feature sequence is interacted with a learnable target query vector, a fixed number of predicted bounding boxes and corresponding classification confidence are output, and a one-to-one set matching mechanism is used to realize the optimal matching between the predicted results and the real labels based on the Hungarian algorithm; the model is trained based on a joint loss function, the joint loss function comprises a main detection head loss function and a plurality of auxiliary detection head loss functions, and dynamic denoising training is introduced in the training process.

2. The method of claim 1, wherein the method is based on multi-modal feature fusion. The step of constructing a multi-modal data set using three heterogeneous remote sensing data of Mars infrared images, color images and digital elevation models, preprocessing the three heterogeneous remote sensing data, and constructing a multi-scale image pyramid comprises: the three heterogeneous remote sensing data of Mars infrared images, color images and digital elevation models are preprocessed, the preprocessing comprises spatial alignment and geographic registration of the three heterogeneous remote sensing data, and unification to the same spatial resolution, so that different modal data correspond one by one in spatial position; based on the three heterogeneous remote sensing data after preprocessing, a multi-modal data set containing multiple modal information is constructed; the aligned remote sensing data is cut into fixed-size image blocks as the basic input unit of model training and inference; based on the image blocks, a multi-scale image pyramid is constructed, and different scale images are down-sampled before inputting into the network, to realize the identification of different scale Mars impact craters.

3. The method of claim 2, wherein the method is based on multi-modal feature fusion. The fixed-size image block is a 640x640 pixel image block.

4. The method of claim 1, wherein the method is based on multi-modal feature fusion. The scale input of the multi-scale image pyramid comprises 1280x1280 and 2560x2560.

5. The method of claim 1, wherein the method is based on multi-modal feature fusion. The cross-attention module generates query vectors, key vectors and value vectors through depth separable convolution, obtains attention weights through Softmax function normalization, and performs weighted fusion on the features corresponding to the value vectors.

6. The method of claim 5, wherein the method is based on multi-modal feature fusion. In the cross attention module, the features from two different modalities are taken as input, wherein the query vector is from the input features of the first modality, the key vector and the value vector are from the input features of the second modality, and the output is obtained by weighting and summing the corresponding features of the value vector and adding the input features of the first modality.

7. The method of claim 1, wherein the method is based on multi-modal feature fusion. The three kinds of heterogeneous remote sensing data are respectively input into three independent CNN-Transformer hybrid backbone networks, local detail features are extracted through convolution structure, and global context features are extracted through self-attention module, and the steps of obtaining multi-scale features include: In the shallow stage of the CNN-Transformer hybrid backbone network, the input remote sensing data is processed by using the convolution structure to extract local detail features; In the deep stage of the CNN-Transformer hybrid backbone network, the features extracted in the shallow stage are processed by introducing the self-attention module to extract global context features; Different scale features are output by the shallow stage and the deep stage respectively to form multi-scale features.

8. The method of claim 1, wherein the method is based on multi-modal feature fusion. The joint loss function includes quality focus loss function, bounding box loss function and intersection over union loss function. 9.A Mars impact crater identification system based on multi-modal feature fusion, characterized in that, The system comprises: A multi-modal data construction and preprocessing module is used to construct a multi-modal data set by using three kinds of heterogeneous remote sensing data of Mars infrared image, color image and digital elevation model, to preprocess the three kinds of heterogeneous remote sensing data, and to construct a multi-scale image pyramid; A multi-branch feature extraction module is used to input the three kinds of heterogeneous remote sensing data into three independent CNN-Transformer hybrid backbone networks respectively, to extract local detail features through convolution structure, and to extract global context features through self-attention module, and to obtain multi-scale features; A multi-modal feature fusion module is used to gradually fuse the obtained multi-scale features, wherein the channel fusion of the homogeneous modal features of the infrared image and the color image is performed in the shallow feature stage, the spectral feature interaction of the deep features of the infrared image and the color image is performed through the cross attention module with shared weights in the deep feature stage, and the cross-modal fusion of the spectral features after the spectral feature interaction and the terrain features of the digital elevation model is performed through another cross attention module; A feature encoding module is used to input the fused multi-scale features into a Transformer-based visual encoder, to encode the fused multi-scale features through a multi-scale deformable attention mechanism, and to output an encoded feature sequence; A target decoding and detection module is used to input the encoded feature sequence into a Transformer-based visual decoder, to interact the encoded feature sequence with a learnable target query vector, to output a fixed number of predicted bounding boxes and corresponding classification confidence, and to realize the optimal matching between the predicted results and the real labels based on the Hungarian algorithm through a one-to-one set matching mechanism. The model training module is configured to train the model based on a joint loss function, the joint loss function comprising a main detection head loss function and a plurality of auxiliary detection head loss functions, and introducing dynamic denoising training in the training process.

Images