Multi-modal image fusion method based on neural kernel method and incremental deep network

Abstract

The invention belongs to the technical field of space target perception and detection, and particularly relates to a multi-modal image fusion method based on a neural kernel method and an incremental deep network, and the method comprises the steps: 1, collecting the paired visible light VIS and near infrared NIR multi-modal perception time sequence image data of a spacecraft target in a near-earth orbit environment; step 2, constructing and pre-training a teacher fusion network model of bimodal visible light VIS + near infrared NIR input; 3, building a lightweight student network model which only receives single-mode visible light VIS input; 4, building a multi-level knowledge distillation composite loss function; and 5, freezing teacher model parameters. The trained student model can generate a high-fidelity fusion feature image for a subsequent target detection task only through visible light input, and the method combines a neural kernel function with cosine similarity, obtains better maintenance feature representation for a feature extraction layer, and more intuitively reflects whether the feature spaces of the two models are opposite or not.

Description

Technical Field

[0001] This invention belongs to the field of space target perception and detection technology, and particularly relates to a multimodal image fusion method based on neural kernel method and incremental deep network. Background Technology

[0002] In recent years, the influence of various countries in space has been continuously expanding, and the demand for human exploration, development, and utilization of space has been increasing. The technological level of spacecraft has been continuously improving, and the number of launches has surged. As orbital space becomes increasingly crowded, the risk of collisions has increased dramatically, posing a severe challenge to the safe operation of spacecraft. On the one hand, space debris and space junk created by spacecraft failures occupy a large amount of orbital space resources, posing a significant threat to the safe and stable operation of other spacecraft. On the other hand, space competition is becoming increasingly fierce, with the possibility of non-cooperative targets carrying offensive payloads harassing our spacecraft. Therefore, improving on-orbit servicing technology against non-cooperative targets is crucial. To achieve autonomous threat avoidance against space debris, failed spacecraft, and hostile spacecraft, spacecraft need to be able to measure their own state and environmental scenarios, identify threat characteristics, predict threat behaviors, and determine threat levels.

[0003] However, spacecraft orbits have a wide range, and the measurement distance of threatening targets varies from meters to kilometers. Long-range imaging resolution is low, sometimes only line-of-sight measurements are possible, or a large number of images need to be clustered for feature analysis. The behavioral characteristics of orbital threats such as collisions, close encounters, and fly-arounds are not obvious, and the threat's impact is long-lasting, requiring a comprehensive analysis of morphology, motion, and even historical behavior to make an accurate judgment. Due to the large spatiotemporal scale and complex space environment, accurate perception of orbital threats is difficult. Perception involves measuring the spacecraft's own state and the environmental scene, identifying threat characteristics, predicting threat behavior, and determining the threat level. The large spatial span of orbits and the measurement distance of threatening targets ranging from meters to kilometers result in very low long-range imaging resolution, sometimes only line-of-sight measurements. Complex lighting conditions lead to uneven target reflection, resulting in overexposed and noisy images with limited effective information, posing challenges to the extraction of target morphological features and the identification of abnormal behavior; information on the target's trajectory and morphology is incomplete. The behavioral characteristics of orbital threats such as collisions, close approaches, and fly-arounds are not obvious, and the duration of their impact is relatively long. Accurately determining the type, intent, and level of a threat requires a comprehensive analysis of the spacecraft's morphology, trajectory, historical behavior, and prior knowledge. Therefore, under conditions of limited spaceborne resources, the rapid and accurate assessment of threats from unknown non-cooperative targets in space faces significant challenges.

[0004] Current detection methods primarily rely on optical sensors to perceive the pose and shape information of targets, mainly using methods based on visible light cameras, depth cameras, infrared cameras, and lidar. However, commonly used spaceborne monocular visible light cameras cannot address the lack of usable semantic information in low-exposure, noisy environments, and also lack usable near-infrared and visible light data pairs due to equipment limitations. Therefore, it is necessary not only to combine the advantages of different modal data but also to use technical means to simulate the feature relationships of different modalities under the premise of data scarcity, and enhance the spacecraft's perception capabilities through single-modal simulation. To this end, this invention provides a multimodal image fusion method based on neural kernel methods and incremental depth networks. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal image fusion method based on neural kernel method and incremental deep network, which uses only a single modality to achieve the fusion effect of VIS & NIR images in low exposure and high noise environment, thereby improving the target perception effect under complex spatial lighting conditions.

[0006] The specific technical solution adopted by this invention is as follows: A multimodal image fusion method based on neural kernel methods and incremental deep networks includes the following steps: Step 1: Collect paired visible light (VIS) and near-infrared (NIR) multimodal sensing time-series image data of spacecraft targets in the near-Earth orbit environment, as well as paired image data in near-Earth orbit.

[0007] Step 2: Construct and pre-train a teacher fusion network model with dual-modal visible light (VIS) + near-infrared (NIR) inputs, and use cosine similarity-improved NTK incremental learning loss to enhance the modality fusion effect; Step 3: Build a lightweight student network model that only accepts single-modal visible light (VIS) input. The network consists of the following modules in sequence: visible light feature extractor, NIR feature simulator, multi-scale feature fusion module, and image reconstruction module. Each module is designed and a two-stage simulation strategy is used: an initial feature transformation module and a feature refinement module based on channel and spatial attention. This enables the network to infer and generate high-quality pseudo-NIR features from multi-scale visible light features.

[0008] Step 4: Construct a multi-level knowledge distillation composite loss function, which consists of three weighted parts: Feature distillation loss: used to minimize the L2 distance between pseudo-NIR features and true NIR features; Attention distillation loss: used to align the attention graphs of the teacher-student network in the fusion module; Output distillation loss: used to ensure pixel-level consistency in the final fused image; Step 5: Freeze the teacher model parameters and train the student model using only the visible light image and the multi-level distillation loss function. The trained student model only requires visible light input to generate a high-fidelity fused feature image for subsequent object detection tasks.

[0009] The technical effects achieved by this invention are as follows: This invention combines neural kernel functions with cosine similarity, without directly calculating the kernel matrix or considering the effects of scale, to obtain better feature representation for the feature extraction layer, and more intuitively reflects whether the feature spaces of the two models are aligned.

[0010] This invention proposes a NIR feature simulator incorporating a two-stage simulation strategy. This module, through initial feature transformation and attention-guided feature refinement, combined with channel attention and spatial attention, enables the student network to accurately infer and reconstruct missing pseudo-NIR feature information from single-modal visible light features.

[0011] This invention innovatively designs a multi-level knowledge distillation loss function L. feature and L attention By forcing the student network to mimic the teacher network at the level of feature representation and decision logic, a high-fidelity fusion effect is ensured, which is significantly better than traditional single-output distillation. Attached Figure Description

[0012] Figure 1 This is a complete flowchart of the multimodal image fusion method based on neural kernel method and incremental deep network of the present invention; Figure 2 This is a structural diagram of the teacher network of the present invention; Figure 3 This is a structural diagram of the single-modal student fusion network of the present invention; Figure 4 This is a flowchart of the NIR feature simulator of the present invention; Figure 5 This is a flowchart of the multi-scale feature fusion module of the present invention. Detailed Implementation

[0013] To make the objectives and advantages of this invention clearer, the invention will be specifically described below with reference to embodiments. It should be understood that the following text is merely used to describe one or more specific embodiments of the invention and does not strictly limit the scope of protection specifically claimed by the invention.

[0014] like Figures 1-5As shown, a multimodal image fusion method based on neural kernel methods and incremental deep networks is used in a near-Earth orbit environment. Based on knowledge distillation, a multimodal feature simulation and fusion method based on single-modal visible light input is used to achieve efficient target detection of non-cooperative / cooperative spacecraft in the absence of near-infrared (NIR) sensing data. The method includes the following steps.

[0015] Step 1: Collect paired visible light (VIS) and near-infrared (NIR) multimodal sensing time-series image data of spacecraft targets in the near-Earth orbit environment, as well as paired image data in near-Earth orbit.

[0016] In step one, specifically, a publicly available multispectral dataset is selected as the training data; a multispectral dataset such as the KAIST dataset contains 95,328 paired RGB / VIS and near-infrared (NIR) images; visible light data is obtained by integrating 939 visible light VIS image data from speedplusv2 or spaceX datasets.

[0017] Step 2: Construct and pre-train a teacher fusion network model with dual-modal visible light (VIS) + near-infrared (NIR) inputs, and use cosine similarity-improved NTK incremental learning loss to enhance the modality fusion effect; In step two, the teacher fusion network model with dual-modal visible light (VIS) and near-infrared (NIR) inputs is trained using a composite total loss function. This total loss function includes at least: a pixel loss term based on structural similarity measurement, used to preserve the structural integrity and edge details of the input image; a feature loss term based on feature contrast learning, which promotes the learning of shared semantic structures by maximizing the lower bound of mutual information between the fused features and the features of each source modality; and an incremental learning loss term based on gradient correlation analysis, which ensures that the fusion process learns unbiased knowledge representation by calculating and minimizing the gradient cosine difference between the current model and the previous model at the feature extraction layer, while introducing weight pruning based on Zipf's law to reduce model redundancy.

[0018] Step 2 involves constructing and pre-training a teacher fusion network model with dual-modal (VIS+NIR) inputs, and using cosine similarity-enhanced NTK incremental learning loss to improve the modality fusion effect.

[0019] The teacher model T employs a dual-branch input, multi-scale fusion network architecture. During training, paired training methods are used. and As input, by minimizing a composite total loss function To conduct end-to-end training.

[0020] The total loss function is defined as:

[0021] in: For the pixel term: the pixel differences between the input image pairs and the fusion result are calculated, and the weighting coefficients obtained by measuring complementary information control the degree of information preservation. Although L1 loss ensures the overall quality of the generated image, it often ignores edge regions, which are crucial but only account for a small portion of the image, leading to blurring. To address this issue, this invention replaces L1 loss with the structural similarity index metric SSIM and its enhanced version, multi-scale SSIM (MS-SSIM). These metrics are perceptually more consistent with human vision, preserving the structural integrity and edge details of the input image.

[0022] Given two registered images and For a pixel p located in one of the two images, SSIM can be calculated as:

[0023] in , , These are the mean of X, the variance of X, and the covariance of X and Y, calculated using a Gaussian kernel. (Constant) and It's set based on grayscale levels. Essentially, and This represents an estimate of the brightness and contrast of X, while This measures the structural similarity between X and Y.

[0024]

[0025] For feature terms: Calculate the feature differences between fused blocks. Ensure that each DualAttentionFusionBlock in the network can be trained effectively to guarantee the effectiveness of multi-scale information interaction. The goal is to maximize the intermediate fused features. Its corresponding infrared features and visible light characteristics Mutual information between features is important, but directly computing the mutual information (MI) between high-dimensional continuous feature maps is extremely difficult. Therefore, a computable lower bound for MI is used instead. Lfeature is defined as the sum of the InfoNCE losses between the fused features and the source features across five fusion scales:

[0026] Define the projection head as two 1x1 convolutional layers, denoted as H. ir and H visThese projectors map the NIR and RGB feature maps respectively. The two projectors share parameters across all five scales (t=1…5). At any scale t, there are four embedded feature maps ( From the embedding map of fused features. In the process, a feature vector is extracted at spatial position (i,j). Embedded map of infrared source features In the process, the corresponding feature vector is extracted at the same position (i,j). .from Extract the feature vectors of all other positions (k, l) ≠ (i, j). For a given value q, the loss function is:

[0027] Where s is the similarity function, usually cosine similarity. This is the temperature hyperparameter, used to adjust the smoothness of the loss; it is set to 0.2. Then, it is calculated at all positions (i,j) across the entire feature map. The average value is used to obtain the total scale loss. :

[0028] Force the fusion module through representation learning It's not just about simple pixel blending, but about learning... and The shared semantic structure. To achieve a low score in the contrastive loss, The representation at position (i, j) must simultaneously be with ,and Their characteristics are similar.

[0029] L GTK Based on high-level features extracted from pre-trained networks, perceptual loss measures image similarity more robustly than pixel loss during training. To achieve better fusion results compared to the original network, this invention interprets the fusion task as cross-class incremental information learning. It innovatively introduces a Neural Tangent Kernel (NTK) to perform correlation analysis on the gradients of the tail layer weights of the current and unimodal networks, ensuring that the fused network can learn unbiased knowledge representations.

[0030] NTK is used to describe the convergence behavior of neural networks during training. A kernel matrix is ​​defined to represent the influence of the neural network parameters on the output.

[0031] in It is input The network output, and It is the network output of parameters The partial derivatives of the network are used. Furthermore, to address the limitation of only being applicable to infinitely wide neural networks, Gradient Tangent Kernel (GTK) is employed. Cosine similarity is used to measure the gradient difference between the two models, avoiding dependence on network width and providing a direct reflection of whether the feature spaces of the two models are aligned.

[0032] This invention evaluates the gradient differences only based on the feature extraction layers of the VGG-16 network, avoiding interference from shallow network features. During training, only the gradient of the previous model needs to be calculated once, eliminating the need for repeated calculations and thus reducing computational cost. The kernel function objective equation is defined as follows:

[0033] To further reduce network computational redundancy and enable the model to better meet the stringent real-time requirements of the on-board closed-loop system, this invention introduces Ziff's law to prune the tail layer weights. Ziff's law is widely used in information theory; specifically, it applies to the frequency of a word. f And its ranking r The following relationships exist:

[0034] in, k It is a constant. Its application here helps the present invention reduce unimportant feature redundancy while maintaining the overall expressiveness of the model.

[0035] Step 3: Build a lightweight student network model that only accepts single-modal visible light (VIS) input. This lightweight student network model includes, in sequence: a visible light feature extractor, an NIR feature simulator, a multi-scale feature fusion module, and an image reconstruction module. Design each module and use a two-stage simulation strategy: an initial feature transformation module and a feature refinement module based on channel and spatial attention, to enable it to infer and generate high-quality pseudo-NIR features from multi-scale visible light features.

[0036] In step three, the lightweight student network model uses a NIR feature simulator that infers pseudo-NIR features from visible light features through three stages: initial feature transformation, cross-modal cross-attention refinement, and a learnable NIR prototype feature library. The multi-scale feature fusion module employs a bidirectional cross-attention mechanism, enabling visible light features and pseudo-NIR features to query complementary information and perform weighted fusion by dynamically predicting spatially adaptive fusion weights through a lightweight convolutional network. The entire student network is trained using a multi-level knowledge distillation framework, and is guided by the bimodal teacher network at the feature, attention mechanism, and output levels.

[0037] Step 3: In actual operation, a lightweight student network model that only accepts single-modal visible light (VIS) input is built. The network includes, in sequence: visible light feature extractor, NIR feature simulator, multi-scale feature fusion module and image reconstruction module.

[0038] Specifically, this invention proposes an innovative Knowledge Distillation (KD) framework. This framework aims to transfer knowledge from a powerful, pre-trained bimodal (RGB-NIR) teacher fusion network to a lightweight 8-bit student network that only receives unimodal (RGB / VIS) input. The invention designs a core NIR feature simulator, enabling the student network to infer and generate high-quality pseudo-NIR features from VIS features. To this end, the invention proposes a multi-level distillation strategy, performing refined knowledge transfer at three levels: feature generation, attention mechanism, and final output. This method allows the student network to generate fused images comparable to those of the bimodal teacher network, even with only VIS input. This provides an efficient and low-cost solution for achieving augmented perception on devices without NIR hardware. The specific structure is shown in the diagram below. Figure 3 As shown: The core idea is to use a pre-trained bimodal teacher model T to guide the training of a unimodal student model S. The teacher model T receives pairs of VIS images X. VIS and NIR image X NIR As input, the student model S only receives X VIS To compensate for the lack of NIR information, the architecture of the student network S was carefully designed, consisting of four core components connected in series: VIS feature extractor F... VIS NIR Feature Simulator F SIM、 Multi-scale feature fusion module F FUS and image reconstruction module F REC .

[0039] Visible light feature extractor (F VIS The forward propagation process of a network begins with F. VIS The Visible Light Feature Extractor (VISFeatureExtractor) employs a multi-scale encoder structure, extracting features from the VIS image hierarchically through a series of residual down-sampling blocks. This process can be formally represented as:

[0040] in, Representing different feature map scales; relative to the input image size, This represents the VIS feature map extracted at scale l. This hierarchical structure helps the network capture rich information from low-level texture to high-level semantics.

[0041] NIR Feature Simulator (F vis This paper proposes to replace the traditional CBAM (Spatial Attention & Channel Attention Paradigm) with a Cross-Modal Cross-Attention (CMCA) mechanism, enabling the student network to explicitly learn how to query the representation form that NIR features should have from VIS features. Simultaneously, this invention introduces a Learnable NIR Prototype Bank, allowing the model to generate high-quality pseudo-NIR features based on the cross-modal mapping knowledge learned during training, even without the guidance of real NIR features from the teacher network during the inference phase.

[0042] The core of feature learning is the NIR feature simulator (F SIM This module is designed to utilize VIS features. The corresponding pseudo-NIR features were inferred from the data. The present invention redesigns its process into three stages, such as Figure 4 As shown: First, an initial feature transformation is performed: two convolutional layers are used to perform a preliminary nonlinear mapping on the VIS features to obtain a rough NIR feature estimate.

[0043]

[0044] Where W and b represent the learnable convolutional kernel weights and bias terms, respectively, and σ is the PReLU activation function.

[0045] Then, cross-modal cross-attention refinement: a cross-modal cross-attention mechanism is introduced to enable VIS features to actively query the representation of the NIR feature space.

[0046] The preliminary estimated NIR characteristics The query is obtained through linear projection:

[0047] Where H and W are the height and width of the feature map, and d is the feature dimension.

[0048] During the pre-training phase, the keys and values ​​are derived from the real NIR features extracted by the teacher network. ,

[0049] During the inference phase, due to the lack of real NIR inputs, this invention uses a learnable NIR prototype feature library. ;in Number of prototypes:

[0050] Calculate the attention weight between the query and the key, and apply it to the value:

[0051] To preserve information from the initial feature transformation and stabilize training, this invention employs residual connections:

[0052] Finally, NIR prototype feature library learning: prototype feature library P NIR This prototype library is used as a model parameter and is jointly trained with other parts of the network. Its learning objective is to capture common representations of NIR modalities, such as edge intensity distribution and texture patterns, by aligning with the teacher's NIR features during the training phase. During the inference phase, this prototype library acts as a "memory," enabling the student network to dynamically retrieve the most relevant NIR prototype combinations based on VIS features. The number of prototypes, Np, needs to be set to balance expressive power and computational cost; in the experiments, this invention sets Np... p =16.

[0053] Multi-scale feature fusion module (F FUS In the teacher network, the multi-scale feature fusion module employs a strategy of independently generating channel and spatial attention maps, and then weighting and summing the VIS and NIR features separately. This method has two limitations: Unidirectional enhancement: VIS and NIR features generate attention weights independently, failing to capture the complementarity and synergy between the two.

[0054] Fixed blending strategy: Simple element summation assumes that the importance of the two modes is symmetrical in all spatial locations, ignoring that different regions may need different modal preferences (e.g., shadow regions are more dependent on NIR, and textured regions are more dependent on VIS).

[0055] This invention proposes a bidirectional cross-attention mechanism that enables VIS and NIR features to query each other's complementary information and dynamically determines which modality each spatial location should rely on more through an adaptive fusion weight network.

[0056] For scale l VIS features and simulated NIR characteristics This invention performs cross-attention in two directions:

[0057] The cross-attention function is defined as follows:

[0058] Unlike simple addition or fixed weights, this invention designs a lightweight convolutional network that adaptively fuses weights based on the enhanced features in the dynamic prediction space. :

[0059] Among them, W f It is a 1×1 convolution kernel, with softmax normalization along the channel dimension to ensure α l +β l =1.

[0060] The tail layer achieves dot product fusion through a linear layer:

[0061] in, This indicates element-wise multiplication.

[0062] Overall advantages: Complementary modeling: Bidirectional cross-attention explicitly models the complementary relationship between the two modalities, enabling the fused features to simultaneously contain the semantic information of VIS and the robustness of NIR.

[0063] Spatial Adaptation: The fusion weights α and β are adaptively adjusted in different spatial locations, such as automatically increasing the NIR feature weights in low-light regions.

[0064] Step 4: Construct a multi-level knowledge distillation composite loss function, which consists of three weighted parts: Feature distillation loss: used to minimize the L2 distance between pseudo-NIR features and true NIR features; Attention distillation loss: used to align the attention graphs of the teacher-student network in the fusion module; Output distillation loss: used to ensure pixel-level consistency in the final fused image; The total loss function in step four includes at least: feature distillation loss, used to minimize the L2 distance between the pseudo-NIR features generated by the student network and the real NIR features extracted by the teacher network; attention distillation loss, used to align the distribution consistency of the student network and the teacher network on the prototype activation modes; and output distillation loss, used to minimize the pixel-level L2 difference of the final fused output of the student network and the teacher network. Furthermore, the loss function also includes a prototype alignment loss, which drives the learnable NIR prototype feature library to capture common representations of NIR modalities by calculating the maximum mean difference between the distribution of pseudo-NIR features generated by the student network and the distribution of real NIR features extracted by the teacher network.

[0065] Step four involves constructing a multi-level knowledge distillation composite loss function in actual operation. This function consists of three weighted components: Feature distillation loss: used to minimize the L2 distance between pseudo-NIR features and true NIR features; Attention distillation loss is used to align the attention graph of the teacher-student network in the fusion module; The output distillation loss is used to ensure pixel-level consistency in the final fused image.

[0066] To ensure that the student model can fully learn the knowledge of the teacher model, this invention designs a composite optimization objective that includes task loss and multi-level distillation loss. The total loss function L... total Defined as:

[0067] Where λ is the weighting coefficient of the prototype alignment loss; it is set to 0.1.

[0068] To ensure the NIR prototype feature library P NIR This invention effectively captures typical representations of NIR modes and adds a prototype alignment loss L. proto : Among them, the maximum mean discrepancy is used to measure the difference between two distributions. This represents the operation of converting the feature map into a distribution; it is achieved through global average pooling.

[0069]

[0070] Where λ is the weighting coefficient of the prototype alignment loss; it is set to 0.1.

[0071] Distillation loss :

[0072] Where β1, β2, and β3 are the weights of each term. First, characteristic distillation loss. The learning of the NIR feature simulator is directly supervised. It minimizes the pseudo-NIR features generated by the student model. Features extracted from real NIR images by the teacher model The L2 distance between them is used to force student networks to learn accurate cross-modal mappings:

[0073] The concept of prototype activation consistency loss is proposed to constrain a "prototype selection" pattern where student online learning and teacher online learning are identical.

[0074] Among them, A l It is the Prototype Activation Vector, defined as: Prototype activation of the student network: The student network obtains attention weights through cross-modal cross-attention computation. Global prototype activation is obtained by aggregating along spatial dimensions:

[0075] Prototype activation of teacher networks: Although teacher networks do not use cross attention, this invention can achieve this by calculating their true NIR features. With prototype library P NIR Using similarity to construct the ideal prototype activation:

[0076] in, The feature vector is obtained through global average pooling:

[0077] Finally, the attention loss from the VIS and NIR branches was merged, and the output distillation loss (L) was reduced. output The final output y is obtained by minimizing the teacher-student model. T and y SThe L2 norm between these parameters ensures pixel-level result consistency. It is worth noting that this loss is related to the task loss L of this invention. task They are formally equivalent; this loss is formally equivalent to the task loss of this invention, and together they enhance the supervision of the quality of the final fused image.

[0078] Step 5: Freeze the teacher model parameters and train the student model using only the visible light image and the multi-level distillation loss function. The trained student model only requires visible light input to generate a high-fidelity fused feature image for subsequent object detection tasks.

[0079] This invention employs a clear two-stage training process: Phase 1: Teacher Model Pre-training: The teacher model is fully trained using paired VIS-NIR image datasets until convergence. Then freeze all its network parameters.

[0080] Phase Two: First: The first 10 epochs: Freeze the NIR prototype library and train only the Q / K / V projection matrix of the cross-attention module, so that the network learns the basic cross-modal mapping first.

[0081] Then: Subsequent epochs: The temperature is dynamically adjusted according to the training progress, the prototype library is gradually unfrozen, and all parameters are trained end-to-end.

[0082] Finally: Learning rate scheduling: For each sample, the temperature is dynamically adjusted based on the difference between teacher and student predictions. When teacher and student predictions are consistent (small KL divergence) → low temperature, exact matching; when teacher and student predictions differ greatly (large KL divergence) → high temperature, softened guidance.

[0083] t is the current epoch, and T is the total number of epochs; , Let λ be the predicted distribution of the sample by teachers and students, and λ be the scaling factor. This is the initial temperature value.

[0084] Based on the experimental results, determine the values ​​of the parameters α and λ for the total training loss function: .

[0085] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained in this invention are implemented according to conventional methods in the art unless otherwise specified or limited.

Claims

1. A multimodal image fusion method based on neural kernel methods and incremental deep networks, characterized in that: Includes the following steps: Step 1: Collect paired visible light (VIS) and near-infrared (NIR) multimodal sensing time-series image data of spacecraft targets in the near-Earth orbit environment, as well as paired image data in near-Earth orbit. Step 2: Construct and pre-train a teacher fusion network model with dual-modal visible light (VIS) + near-infrared (NIR) inputs, and use cosine similarity-improved NTK incremental learning loss to enhance the modality fusion effect; Step 3: Build a lightweight student network model that only accepts single-modal visible light (VIS) input. This lightweight student network model includes, in sequence: visible light feature extractor, NIR feature simulator, multi-scale feature fusion module, and image reconstruction module. Design each module and use a two-stage simulation strategy: an initial feature transformation module and a feature refinement module based on channel and spatial attention, to enable it to infer and generate high-quality pseudo-NIR features from multi-scale visible light features. Step 4: Construct a multi-level knowledge distillation composite loss function, which consists of three weighted parts: Feature distillation loss: used to minimize the L2 distance between pseudo-NIR features and true NIR features; Attention distillation loss: used to align the attention graphs of the teacher-student network in the fusion module; Output distillation loss: used to ensure pixel-level consistency in the final fused image; Step 5: Freeze the teacher model parameters and train the student model using only the visible light image and the multi-level distillation loss function; the trained student model only requires visible light input to generate a high-fidelity fused feature image for subsequent target detection tasks.

2. The multimodal image fusion method based on neural kernel method and incremental deep network according to claim 1, characterized in that: In step one, a publicly available multispectral dataset is selected as the training data; 939 visible light VIS images are obtained by integrating data from the speedplusv2 or spaceX datasets.

3. The multimodal image fusion method based on neural kernel method and incremental deep network according to claim 1, characterized in that: In step two, the teacher fusion network model with dual-modal visible light (VIS) and near-infrared (NIR) inputs is trained using a composite total loss function. This total loss function includes at least: a pixel loss term based on structural similarity measurement, used to preserve the structural integrity and edge details of the input image; a feature loss term based on feature contrast learning, which promotes the learning of shared semantic structures by maximizing the lower bound of mutual information between the fused features and the features of each source modality; and an incremental learning loss term based on gradient correlation analysis, which ensures that the fusion process learns unbiased knowledge representation by calculating and minimizing the gradient cosine difference between the current model and the previous model at the feature extraction layer, while introducing weight pruning based on Zipf's law to reduce model redundancy.

4. The multimodal image fusion method based on neural kernel method and incremental deep network according to claim 1, characterized in that: In step three, the lightweight student network model uses a NIR feature simulator that infers pseudo-NIR features from visible light features through three stages: initial feature transformation, cross-modal cross-attention refinement, and a learnable NIR prototype feature library. The multi-scale feature fusion module employs a bidirectional cross-attention mechanism, enabling visible light features and pseudo-NIR features to query complementary information and perform weighted fusion by dynamically predicting spatially adaptive fusion weights through a lightweight convolutional network. The entire student network is trained using a multi-level knowledge distillation framework, and is guided by the bimodal teacher network at the feature, attention mechanism, and output levels.

5. The multimodal image fusion method based on neural kernel method and incremental deep network according to claim 1, characterized in that: The total loss function in step four includes at least: feature distillation loss, used to minimize the L2 distance between the pseudo-NIR features generated by the student network and the real NIR features extracted by the teacher network; attention distillation loss, used to align the distribution consistency of the student network and the teacher network on the prototype activation modes; and output distillation loss, used to minimize the pixel-level L2 difference of the final fused output of the student network and the teacher network. In addition, the loss function also includes a prototype alignment loss, which drives the learnable NIR prototype feature library to capture the common representation of NIR modes by calculating the maximum mean difference between the distribution of pseudo-NIR features generated by the student network and the distribution of real NIR features extracted by the teacher network.

Images