Cross-scene multi-domain fusion small sample remote sensing target robust identification method

Through meta-learning and improved mask autoencoder, combined with deep neural network and multi-level feature fusion, the problems of small sample learning and multi-modal data fusion in remote sensing image processing are solved, and efficient and accurate goals are achieved in complex scenarios Identification improves the robustness and adaptability of the model.

CN118918476AActive Publication Date: 2024-11-08BEIJING UNIV OF POSTS & TELECOMM

Patent Information

Application Number
CN202410997142.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-08
Estimated Expiration
2044-07-24

AI Technical Summary

Technical Problem

In a complex and changeable battlefield environment, remote sensing image processing and target detection face problems such as small sample learning, domain offset, low multimodal data fusion accuracy, insufficient feature extraction and high detection model complexity, resulting in target recognition accuracy and resilience Not good enough.

Method used

Using meta-learning strategies and improved mask autoencoder for cross-scene generalization small sample learning, combining deep neural networks to build a lightweight real-time object recognition model, and improving feature extraction and fusion through multi-level feature fusion and cross-modal attention mechanism Ability to use multi-domain heterogeneous image registration technology based on feature matching and deep residual network for high-precision feature extraction and detection.

Benefits of technology

It significantly improves the accuracy and robustness of remote sensing object detection, reduces the dependence on large amounts of sample data, realizes efficient and accurate object recognition in complex scenarios, and enhances the adaptability and real-timeness of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918476A_ABST
    Figure CN118918476A_ABST
Patent Text Reader

Abstract

The invention provides a cross-scene multi-domain fusion small sample remote sensing target robust recognition method, and belongs to the technical field of target recognition. According to the method, cross-scene multi-domain multi-modal remote sensing images are fused, a meta-learning strategy and a mask auto-encoder are adopted for representation learning, and the efficient learning and accurate generalization ability of a recognition model in a complex and changeable environment with limited annotation data is remarkably improved; through an improved multi-modal image registration technology and an improved cross-modal multi-head self-attention mechanism, in combination with an improved multi-level feature fusion method, registration precision and feature expression ability are improved; on the basis of a lightweight real-time target recognition model of an improved deep neural network, an anchor frame mechanism and a non-maximum suppression algorithm are combined, and the real-time performance and accuracy of target detection are achieved. According to the method, a cross-scene generalization small sample learning and representation learning method is innovated, a multi-domain heterogeneous image registration, fusion and lightweight real-time target identification technology is innovated, and the cross-scene target identification precision and robustness are remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of target recognition technology, and in particular to a cross-scene multi-domain fusion small sample remote sensing target robust recognition method, which improves the accuracy and robustness of target detection and recognition by fusing cross-scene multi-domain multi-modal remote sensing image data. Background Art

[0002] Remote sensing image processing and target detection technologies play an important role in environmental monitoring, disaster assessment, urban planning and other fields. By acquiring and analyzing remote sensing images, accurate monitoring and identification of surface objects and phenomena can be achieved, thereby providing critical decision support. The application of these technologies can not only improve the efficiency of resource management and environmental protection, but also enhance the reconnaissance and monitoring capabilities in the security field.

[0003] However, remote sensing image processing and target detection and recognition in adversarial environments face many challenges and problems. In adversarial environments, advanced combat equipment is often highly concealed and technologically novel, which makes it extremely difficult to obtain sufficient target data through conventional means before the war. In addition, the ever-changing war environment makes it very difficult to accumulate enough sample data for model training in a short period of time. The lack of sample data directly limits the effective perception and recognition capabilities of new or rare targets. Due to the short intelligence collection window, key information data such as the enemy's new weapons and equipment, tactical layout, etc. are difficult to be widely collected, resulting in extremely limited available data. The target information data successfully obtained is often concentrated in specific scenarios or conditions, which is quite different from the diverse environment in actual combat. The target recognition model built based on these limited and scenario-restricted data often shows poor generalization performance when facing new environments or new targets, and cannot accurately identify or adapt to the changes in the characteristics of the target in different environments.

[0004] When dealing with such small sample learning problems, traditional intelligent algorithms have limited feature representation capabilities and are unable to extract sufficient and representative features from a small number of samples. In addition, they have insufficient generalization capabilities and cannot accurately predict or classify unseen samples even if they perform well on training data. Especially in cross-scenario situations, the training set and test set come from different fields and have different data distributions, while the training set and test set in traditional intelligent algorithms come from the same field. The resulting domain shift problem will have a negative impact on the performance of small sample learning, seriously hindering the rapid and accurate identification of targets in complex and changing battlefield environments, thereby affecting the timeliness and accuracy of decision-making.

[0005] In addition, it is difficult for single-mode data to fully reflect the target features in complex scenes, and thus it is difficult to ensure high-precision target detection. Remote sensing images come from a wide range of sources, including visible light, infrared, synthetic aperture radar (SAR) and other modalities. The data of each modality has different imaging principles and characteristics, and can provide different information. For example, visible light images can provide rich color and texture information, infrared images can obtain data at night or in bad weather conditions, and SAR images can penetrate clouds and vegetation to provide surface structure information. In remote sensing image processing, images of different modalities each have their own unique advantages, but they also have their own limitations: visible light images can provide rich color and texture information, have strong detail expression capabilities, and are intuitive and easy to understand, but they are sensitive to lighting conditions. At night or in bad weather, the image quality is significantly reduced, and they are affected by occlusion and cannot penetrate obstructions such as clouds, smoke or vegetation, which limits their application in complex environments. Infrared images can obtain data at night or in bad weather conditions. They are sensitive to temperature differences and are suitable for detecting heat sources, but they have low resolution. Usually, the resolution of infrared images is lower than that of visible light images, and the detail expression ability is poor. At the same time, the color information is missing, resulting in poor visual intuitiveness of the image. Thermal interference, heat sources and temperature changes will affect the imaging effect, and there is a possibility of false alarms. Synthetic aperture radar (SAR) images can penetrate clouds and vegetation, provide surface structure information, are not affected by lighting conditions, and are suitable for all-weather imaging, but the noise is high. Speckle noise often exists in SAR images, which affects image quality and detail expression, making interpretation difficult. The image is poorly intuitive and it is difficult to directly extract useful information from the image. Professional processing and analysis are required. Geometric distortion. Due to different imaging principles, SAR images may have geometric distortion and need geometric correction. By combining the advantages of these modalities and overcoming their respective shortcomings, targets in complex scenes can be detected and identified more comprehensively and accurately, and the overall performance of remote sensing image processing can be improved.

[0006] The registration of multimodal data is the premise of data fusion. However, due to the geometric distortion, resolution difference and changes in imaging conditions between different modal images, traditional registration methods (such as grayscale or feature point based methods) are difficult to achieve high-precision registration in complex scenes. This directly affects the accuracy of subsequent feature extraction and fusion. Second, feature extraction is insufficient. Feature extraction is a key step in target detection. Existing methods mostly rely on single-modality feature extraction or simple multimodal feature fusion, and fail to fully explore and utilize the complementary information of multimodal data. For example, traditional manual feature extraction methods (such as SIFT and SURF) are not robust enough in complex backgrounds and between different modalities. Although some deep learning methods can extract relatively rich features, they often fail to effectively combine the advantages of each modality when processing multimodal data. Third, the detection model is highly complex. In order to improve the accuracy of target detection, some existing methods design very complex detection models. These models usually contain a large number of parameters and calculations, which are difficult to meet the constraints of real-time and computing resources in practical applications. For example, although the detection method based on deep neural network has high accuracy, the computational complexity in the training and reasoning process is high, which limits its promotion in real-time applications. Finally, the multimodal data fusion method is insufficient: in the existing technology, although there are some multimodal data fusion methods, the existing methods often have problems of information loss and redundancy when dealing with the fusion of different modal features, and because the splicing weighted fusion strategy used cannot give full play to the complementarity of different modal data, the expression ability of the fusion feature is insufficient.

[0007] To address the above problems, the present invention proposes a cross-scenario generalized small sample learning based on improved domain mapping and a lightweight real-time target recognition model based on deep neural network, which has the following innovations and advantages:

[0008] The meta-learning strategy and representation learning based on improved masked autoencoders are used to improve the accuracy and generalization ability of remote sensing target recognition. By using the meta-learning strategy to build a K-way-N-shot learning scenario, only a small number of labeled samples are needed to drive the rapid iteration and upgrade of the target recognition model. When encountering new tasks, it can quickly adapt and accurately perform classification tasks, significantly reducing the dependence on a large amount of sample data.

[0009] Using masked autoencoders for pre-training, the self-supervised learning of image features does not rely on labeled data. Masked autoencoders randomly mask parts of the input image and use the Vision Transformer (ViT) efficient encoder to generate low-dimensional latent variables containing global context and local details. The encoder part is used as a feature extractor to effectively decouple domain features from target features, thereby establishing a common feature space between the source domain and the target domain, which helps to narrow the data distribution difference.

[0010] The multi-domain heterogeneous image registration technology based on feature matching is adopted. By extracting significant boundary features and feature points based on grayscale, the scale-invariant feature transform (SIFT) threshold is dynamically adjusted, and weight values ​​are calculated to strengthen important feature points to ensure high-precision registration in complex scenes. Secondly, the registration results based on feature points and boundary features are integrated, and the registration accuracy is improved using incremental optimization methods. New weights are designed according to the accuracy of target detection, and the registration algorithm is iteratively optimized to improve its adaptability in different scenes.

[0011] A deep residual network (ResNet-101) is used to extract features from the registered visible light, infrared and SAR images, and the high-level and low-level features are enhanced through the cross-modal attention mechanism. A multi-level feature fusion strategy is adopted to concatenate and reduce the dimension of high-level and low-level features to form the output feature map of the encoder part, which fully retains important information.

[0012] Construct a lightweight target detection model, reduce the number of network layers and optimize the network structure, use lightweight modules (such as depthwise separable convolution) to reduce the number of parameters and computational complexity, so as to ensure the efficient operation of the model in a resource-constrained environment. Use the anchor box mechanism to delineate the bounding box, preset multi-scale and multi-ratio anchor boxes on each grid cell of the input image, and adjust the position and size of the anchor box through model learning to achieve accurate positioning of the target object. Use the maximization of the IoU value as the loss function. During the training process, optimize the model parameters through back propagation to improve the accuracy of the prediction, and post-process through the non-maximum suppression (NMS) algorithm to sort the detection results by confidence. Starting from the bounding box with the highest confidence, compare the IoU values ​​of other bounding boxes one by one, remove redundant detections, retain the optimal detection results, and ensure the accuracy and effectiveness of the final detection results.

[0013] Through these innovations, the present invention can solve the problems of low multimodal remote sensing image registration accuracy, insufficient feature extraction, high complexity of the detection model and poor multimodal data fusion effect, thereby achieving high-precision and high-efficiency target detection in complex scenarios and significantly improving the reliability and practicability of remote sensing image processing. Summary of the invention

[0014] In view of this, the present invention provides a cross-scene multi-domain fusion small-sample remote sensing target robust recognition method, which reduces or improves the problem of domain offset caused by large differences in feature data distribution between source domain datasets and target domain datasets in small sample learning in the prior art, solves the problems of low multimodal data fusion registration accuracy, insufficient feature extraction and high detection model complexity, thereby improving the accuracy and robustness of target detection.

[0015] The present invention provides a cross-scenario generalized small sample learning based on improved domain mapping, characterized in that a meta-learning strategy is adopted to construct a K-way-N-shot learning framework, and the model is quickly iterated and updated through a small number of labeled samples to improve the adaptability of the target recognition model to new tasks. It is based on a certain degree of similarity between the training set category (old category) and the category to be tested (new category), and the model that performs well on the training set can also be applied to the new category. Meta-learning constructs the training set into a series of small sample tasks with the same settings as the test task, and then the training model has good performance on a large number of small sample tasks to ensure that it can also be quickly classified when adapted to new tasks. A small sample classifier and a domain classifier are used in the learning process, and the method includes the following steps:

[0016] Pre-training stage: Use the improved mask autoencoder to train on the source domain dataset and learn image features in a self-supervised manner. Use the encoder module of the trained mask autoencoder as a feature extractor, which decouples the image features of the source domain and the target domain to obtain domain-independent features and domain features. Map the features of the target domain to the source domain feature space, thereby helping to narrow the difference in data distribution between the source domain and the target domain.

[0017] Meta-learning stage: In order to weaken the impact of domain shift on the performance of small-sample learning and reduce distribution differences, the core idea is to map the features of the target domain to the source domain, and guide the learning of the model by obtaining a very small amount of labeled data belonging to the target domain and adding it to the training dataset. That is, using the very small amount of labeled data in the target domain as auxiliary data to improve the ability of cross-domain small-sample learning to cope with domain shift. A very small number of samples in the target domain are set as an auxiliary dataset through the dataset mixer, and fused with the dataset of the source domain to form a mixed dataset, which is then redivided into a source domain support set, an auxiliary support set, and a mixed query set, which effectively promotes the fusion and migration of features between the source domain and the target domain and reduces the phenomenon of domain shift. The source domain support set, auxiliary support set, and mixed query set are sent to the feature extractor to obtain their respective domain-independent features and domain features. H1 and H2 are used to represent the domain-independent features and domain features, respectively. The domain-independent features and domain features of the source domain support set are represented as and The domain-independent features and domain features of the auxiliary support set are expressed as and The domain-independent features and domain features of the mixed query set are expressed as and Feed the domain-independent features into the small-shot classifier g fsl Classify and compare with the label to get the loss function L FSL ; Send domain-independent features and domain features to the domain classifier g dom Perform domain classification in and compare with domain labels to obtain the loss function and Overall loss The Adam gradient descent optimizer is used to optimize L to minimize L.

[0018] The small sample classifier is characterized in that: and Input small sample classifier and get classification loss and Input small sample classifier and get classification loss Since the mixed query set is a mixture of the source domain query set and the auxiliary query set with a ratio of λ, the loss function of the small sample classifier is

[0019] The domain classifier is characterized in that, for domain-dependent features, the label of the source domain is set to 1, the label of the target domain is set to 0, and for domain-independent features, the domain label vector is set to y1=[0.5,0.5]; is a domain-independent feature, and is input into the domain classifier g dom The loss is calculated by using the relative entropy KL divergence (Kullback-Leibler divergence, KL) The input is compared with the respective domain labels, and the loss is calculated using cross-entropy (CE).

[0020] The present invention provides a representation learning based on an improved masked autoencoder, which is characterized in that a random partial area of ​​an input image is masked based on a random mask mechanism, and the original complete image is predicted through a reconstruction operation process to achieve effective encoding of image data; in the masked autoencoder, the encoder is composed of a VisionTransformer (ViT) module combination, specifically ViT-L / 16, which is composed of 24 layers of ViT modules stacked together, and each image block is mapped to a 1024-dimensional feature vector; the decoder can adopt a lightweight Transformer module, which is composed of 8 layers of ViT modules stacked together, and each image block is mapped to a 512-dimensional feature vector, and the calculation amount of the decoder is only 10% of that of the encoder, thereby improving training and execution efficiency; since it is necessary to infer complete information from incomplete input, the masked autoencoder learns how to capture global context information and local details, thereby generating a more robust and generalized feature representation, so that the masked autoencoder can also have a good performance under cross-scene conditions.

[0021] Among them, ViT is a model based on the Transformer architecture. By transforming the encoder part of the traditional Transformer to adapt to computer vision tasks, it marks the successful migration of the Transformer architecture from the field of natural language processing (NLP) to the field of computer vision. It mainly includes the following key modules:

[0022] Input processing: First, assume that the dimensions of the input image are H×W×C, representing height, width, and number of channels, respectively, and are divided into N fixed-size patches. The dimensions of each patch block are P×P×C, then N=HW / P 2 Flatten each patch into a dimension of P 2 The one-dimensional vector of C is then mapped to a vector space of dimension D through a linear layer to obtain an embedded vector. This process helps the ViT model learn higher-level feature representations.

[0023] Position encoding: Since the Transformer architecture itself does not contain position information, a position encoding vector needs to be added to the embedding vector corresponding to each patch to tell the model the position of each patch in the original image. These position encoding vectors are added to the embedding vector results of the patch and used as the input of the Transformer encoder.

[0024] Transformer encoder: After the input sequence is normalized by the layer, the multi-head self-attention layer is used to focus on different positions of the input sequence in parallel, and the ViT model understands the dependencies between different patches by calculating the query, key, and value matrices. It then passes through the feedforward neural network layer, including two linear layers and a nonlinear activation function (such as the ReLU function), to further transform and enrich the feature representation. Residual connections are used after the self-attention layer and the feedforward neural network layer to alleviate the gradient vanishing and gradient exploding problems, promote feature extraction, and accelerate the training process.

[0025] The present invention provides an improved multi-domain heterogeneous image registration technology based on feature matching, characterized in that the method extracts significant boundary features and feature points based on a grayscale method for registration, thereby ensuring high-precision registration in complex scenes. The method comprises the following steps:

[0026] Multimodal remote sensing image registration based on local boundary features: According to the method of maximizing the internal points and controlling redundant points, feature points are extracted by extracting significant boundary features and grayscale-based methods, and the scale-invariant feature transform (SIFT) threshold is dynamically adjusted to extract feature points. The weight value is calculated to strengthen important feature points to ensure high-precision registration in complex scenes. The method of dynamically adjusting the SIFT threshold is performed through the following formula:

[0027]

[0028] Among them, α is the adjustment coefficient, the feature point strength i is the intensity of the i-th feature point, and n is the total number of feature points.

[0029] Based on the improved method of extracting significant boundary feature points, multimodal remote sensing image registration of local boundary features is realized. The specific process includes:

[0030] After inputting the reference image R and the image to be registered S, the reference image R is used as the standard image for reference, and the image to be registered S is used as the image aligned with the reference image. By extracting significant boundary features and feature points based on grayscale, dynamically adjusting the scale-invariant feature transform (SIFT) threshold, extracting feature points and calculating weight values ​​to strengthen important feature points, high-precision registration is achieved in complex scenes. The specific implementation process includes: for the general case of the input image, that is, the image I of one of the reference image R or the image to be registered S, calculate its significant boundary feature point set F = {f i |i=1,2,…,n}, weight value of feature point Among them, g i is the feature point f i Gray value, α and β are adjustment parameters;

[0031] The improved incremental optimization method detects the boundary and boundary feature points, calculates the local boundary features and boundary descriptors, and performs feature matching. The specific implementation process includes: at each iteration step t, calculating the loss L of the current registration result total =L match +L boundary , update the parameters based on the gradient of the loss function

[0032] Based on the incremental optimization method, the registration results based on feature points and boundary features are integrated, and the accuracy indicators of the respective registration results are evaluated and compared. New weights are designed according to the target recognition accuracy, and the registration algorithm is iteratively optimized to improve the adaptability in different scenarios. The specific implementation process includes: in each incremental optimization step t, the feature point weights are updated Among them, TA represents the target recognition accuracy, and CA represents the current recognition accuracy;

[0033] Based on the improved image registration and exchange method, the registered image is transformed to ensure the accuracy and robustness of target detection. The image transformation matrix is ​​calculated as:

[0034] Based on the target detection evaluation index analysis method, the detection results are analyzed to evaluate the effects of registration and detection, and the mean average precision (mAP) is obtained to ensure the applicability and reliability of the system in complex scenarios. The calculation of the mean average precision is:

[0035] Based on the incremental optimization method, the registration results of feature points and target boundary features are integrated to evaluate and compare the accuracy indicators of each registration result. New weights are designed according to the target recognition accuracy, and the registration algorithm is iteratively optimized to improve the adaptability in different scenarios. Specifically, the evaluation of the registration results is performed using the following formula:

[0036] RA stands for registration accuracy, CRP stands for the number of correctly registered points, and TRP stands for the total number of registered points.

[0037] Design a new weight w and iteratively optimize the registration algorithm to improve adaptability in different scenarios. The new weight design is calculated by the following formula:

[0038] At each incremental optimization step t, update the feature point weights Among them, TA represents the target recognition accuracy, and CA represents the current recognition accuracy;

[0039] Based on the improved multi-level feature fusion method, the cross-modal attention mechanism and adaptive weight adjustment are introduced to fuse features of different levels to improve the detection effect. The general multi-level feature fusion method is: Among them, α i Represents the fusion weight of the i-th layer feature, which is optimized through training. The specific implementation process includes the following steps: extracting multi-level features from the reference image and the image to be registered, performing cross-modal attention enhancement on the extracted features, calculating the attention weight, assigning adaptive weights to features at different levels, and dynamically adjusting the weight parameters through training. The improved fusion formula is: Among them A i is the attention weight matrix of the i-th layer feature, α i The fusion weights are dynamically optimized through training. The specific implementation process includes: extracting multi-level features {f i |1,2,…,n}, calculate the attention weight A i =Softmax(Sim(f i ,fi )),in, Dynamically optimize the fusion weight α through training i , so that the fused features can better adapt to different scenarios and task requirements. Through the above improved method, more robust and efficient feature fusion can be achieved in different scenarios, improving the target detection accuracy and robustness of multimodal remote sensing images;

[0040] Based on the deep residual network (ResNet-101) backbone network feature extractor, feature extraction is performed on the registered visible light, infrared and SAR images. The cross-modal attention mechanism is used to enhance the high-level and low-level features to improve the expressiveness of the features. Multi-level feature fusion concatenates and reduces the dimensions of high-level and low-level features to form the output feature map of the encoder part to retain important information. The enhancement effect of the cross-modal attention mechanism is quantified by the following formula: F ResNet-101 ={f1,f2,…,f n}, where F ResNet-101 Represents a multi-layer feature set extracted by ResNet-101;

[0041] Among them, the attention weight is calculated by the following formula:

[0042] AW=Softmax(SM)

[0043] The calculation formula of the enhanced feature is:

[0044] EF=OF×AW

[0045] EF stands for Enhanced Features (EF), OF stands for Original Features (OF), AW stands for Attention Weights (AW), and SM stands for Similarity Matrix (SM). The similarity is calculated by cosine similarity and other methods. The relevant formula is as follows:

[0046] Vector dot product calculation formula:

[0047] The calculation formula of the vector norm is:

[0048] Cosine similarity calculation formula:

[0049] Through these technologies, the present invention can solve the problems of low multimodal remote sensing image registration accuracy, insufficient feature extraction, high complexity of the detection model and poor multimodal data fusion effect, thereby achieving high-precision and high-efficiency target detection in complex scenes and significantly improving the reliability and practicability of remote sensing image processing.

[0050] An improved cross-modal multi-head self-attention mechanism is introduced. By introducing a cross-modal attention mechanism between multi-modal features including visible light, infrared and synthetic aperture radar (SAR) images, the features of different modalities are multi-head segmented and processed in parallel to capture more diverse feature relationships and semantic information, ultimately enhancing the correlation between features and improving the accuracy of target detection. The attention weight is obtained by calculating the similarity through the Softmax function. The relevant formula is: attention =softmax(S)·V. Where S represents the similarity matrix, which is calculated by cosine similarity, and V represents the value matrix. The calculation formula of cosine similarity is: Among them, Q i and K j denote the query vector and key vector respectively, · denotes the dot product operation, ‖·‖ denotes the norm of the vector Among them, H represents the number of heads, W O is the output weight matrix;

[0051] The present invention also provides an improved method for designing and constructing a lightweight real-time target recognition model based on a deep neural network, characterized in that a lightweight target detection model is constructed based on a deep neural network, and an improved anchor box mechanism is used to delineate a bounding box. The deep learning model is optimized by maximizing the Intersection over Union (IoU) value as the loss function to ensure the accuracy and real-time performance of the detection. The detection results are post-processed using a non-maximum suppression (NMS) algorithm to retain the best detection results and remove redundant results. The method specifically includes the following steps:

[0052] First, the bounding box is defined based on an improved anchor box mechanism. The improved anchor box mechanism is characterized in that the position and size of the anchor box are adjusted through model learning to achieve accurate positioning of the target object. The specific process includes:

[0053] By presetting multi-scale and multi-ratio anchor boxes on each grid unit of the input image, a similar frame is used to match the desired prediction target. The frame size is set as follows: Set L as the loss function, L conf represents the confidence loss, L loc represents the position loss, L cls represents the classification loss, λ1 and λ2 are weight coefficients, then L = Lconf +λ1L loc +λ2L cls ; A is defined to represent the set of all preset anchor boxes. Assuming the grid size is S×S, and the number of preset anchor boxes on each grid unit is k, the anchor box on each grid unit (i, j) can be expressed as:

[0054]

[0055] in, and is the center coordinate of the frame, w k and h k The width and height of the frame.

[0056] The predicted frame adjustment amount Δ = (Δx, Δy, Δw, Δh) is learned by the model, and the position and size of the frame are adjusted according to the adjustment amount:

[0057]

[0058]

[0059]

[0060]

[0061] Generate multiple preset bounding boxes at different scales and ratios. Let the scale set be {s1,s2,…,s n}, the ratio set is {r1,r2,…,r m}, the width and height of the anchor box are:

[0062]

[0063] In order to reduce the number of network layers of the model and optimize the structure of the network, a lightweight deep neural network model is constructed. The depthwise separable convolution lightweight module is used to reduce the number of parameters and reduce the computational complexity so that the model can run efficiently in a resource-constrained environment. The depthwise separable convolution decomposes the standard convolution into depthwise convolution and pointwise convolution to reduce the computational complexity. Assume that the size of the input feature map is H×W×C, the size of the convolution kernel is K×K, and the number of output channels is C′.

[0064] The calculation formula for the standard convolution computational complexity is:

[0065] Standard Convolution FLOPs=H×W×C×K 2 ×C′

[0066] The relevant formula for the computational complexity of depth-separable convolution is:

[0067] Depthwise Convolution FLOPs=H×W×C×K 2

[0068] Pointwise Convolution FLOPs=H×W×C×C′

[0069] Total FLOPs = H × W × C × K 2 +H×W×C×C′

[0070] Then, based on IoU optimization and non-maximum suppression (NMS) post-processing methods, redundant detections are removed, the optimal detection results are retained, and the accuracy and effectiveness of the final detection results are ensured. The specific process includes:

[0071] Taking the maximum IoU value as the loss function, during the training process, the maximum IoU value between the predicted bounding box and the true bounding box is taken as the goal, and the model parameters are optimized through back propagation to improve the accuracy of the prediction. The specific loss function can be expressed as:

[0072] IoU loss = 1-IoU(B p ,B t )

[0073] Among them, B p and B t Represent the predicted bounding box and the real bounding box respectively, and the IoU calculation formula is:

[0074]

[0075] Use the non-maximum suppression (NMS) algorithm for post-processing. This algorithm can retain the best target detection results by removing bounding boxes with high overlap. Sort all predicted bounding boxes by confidence, starting with the bounding box with the highest confidence, and compare the IoU values ​​of other bounding boxes one by one. The specific steps are as follows:

[0076] 1. Sort the predicted bounding boxes by confidence from high to low.

[0077] 2. Select the bounding box with the highest confidence from the sorted list as the current bounding box.

[0078] 3. Calculate the IoU value between the current bounding box and the remaining bounding boxes. If the IoU value is greater than the set threshold (such as 0.5), remove these bounding boxes.

[0079] 4. Repeat steps 2 and 3 until all bounding boxes have been processed.

[0080] The formula of the NMS algorithm is as follows:

[0081] NMS(B, scores, IoU th ) = {Bi | IoU(Bi, Bj) < IoU threshold}

[0082] In this formula, B represents the set of predicted bounding boxes, scores represents the corresponding confidence scores, and IoU th represents the set threshold.

[0083] Based on the improved dynamic setting of thresholds and weights, a dynamic adjustment mechanism based on incremental optimization is implemented. The relevant parameters will be updated in real time according to the performance metrics of the current model during each training cycle. The specific process includes:

[0084] During the model training process, the thresholds and weights of various parameters are dynamically adjusted through cross-validation to ensure the best detection effect. Different from traditional methods, the specific implementation process includes: θ * = argmin θ E[L(θ)], where θ * represents the optimization objective, θ represents the model parameters, and L(θ) represents the loss function.

[0085] Perform model training and optimization, data preparation and preprocessing: Collect multi-modal remote sensing image datasets, and perform annotation and preprocessing (such as image enhancement and denoising) to ensure data quality. Model training: Use the preprocessed dataset for training, and gradually optimize the model performance through batch training and hyperparameter tuning. Performance evaluation and adjustment: Evaluate the detection performance of the model on the test set, including metrics such as accuracy, recall rate, and F1-score, and adjust the model structure and parameters according to the evaluation results to ensure the best performance of the model.

[0086] Actual application and deployment, deploy the optimized model into the target detection system, and optimize the model running efficiency in combination with hardware devices (such as GPUs or embedded devices) to ensure real-time detection capabilities. Real-time detection: Input the remotely sensed images obtained in real time, and the model performs target detection and outputs the bounding boxes and class information. Result verification and correction: Manually verify and make necessary corrections to the detection results to ensure the reliability and accuracy of the detection results.

[0087] The multi-modal remote sensing image registration method realizes more robust and accurate image registration through the extraction of significant boundary feature points and the construction of local boundary descriptors. The incremental optimization method can design new weights according to the target detection accuracy, significantly improving the adaptability and stability of the registration method in different scenarios.

[0088] The feature extraction and fusion method further enhances the accuracy and robustness of target detection through a cross-modal attention mechanism and a multi-level feature fusion strategy. The target detection model reduces the computational complexity through lightweight design and optimization, meeting the real-time requirements in practical applications.

[0089] On the other hand, the present invention provides a system for implementing the above method, characterized in that the system adopts the above steps and methods to improve the accuracy and robustness of target detection by fusing multimodal remote sensing image data, solves the problems existing in the prior art, and provides an effective solution for remote sensing image processing and target detection.

[0090] The beneficial effects of the present invention are at least:

[0091] The present invention provides a cross-scene multi-domain fusion small sample remote sensing target robust recognition method, which belongs to the field of remote sensing image processing and target detection and recognition. The method is based on cross-scene generalized small sample learning of improved domain mapping. By mapping the features of the target domain to the source domain feature space, the data distribution difference between the source domain and the target domain is effectively reduced, and the robustness and generalization ability of the target recognition model in different scenarios are improved. In particular, in the small sample learning scenario, the recognition ability of the target recognition model for new tasks and unseen categories is significantly enhanced, and a meta-learning strategy is adopted so that the target recognition model can be quickly adjusted and updated using a small number of labeled samples. Based on the representation learning of the improved mask autoencoder, a self-supervised learning method is used, and ViT is used as the core module. The learning efficiency of image features is improved by using data parallel processing, and the feature extractor is strengthened to understand the global context and capture local details, so as to generate a more robust and generalized feature representation. Multimodal remote sensing image registration based on local boundary features, by extracting significant boundary features of image pairs and extracting feature points in combination with grayscale-based methods, achieves high-precision registration in complex scenes; extracts feature points based on dynamically adjusted SIFT thresholds, and calculates weight values ​​to strengthen important feature points to ensure the robustness and accuracy of registration; uses a deep residual network (ResNet-101) backbone network feature extractor to extract features from registered visible light, infrared and SAR images, and enhances high-level and low-level features through a cross-modal attention mechanism to further optimize the expressiveness of features. Based on the multi-level feature fusion strategy provided by the present invention, important information can be retained to improve the accuracy and robustness of target detection; the registration results based on feature points and boundary features are fused through an incremental optimization method, and a new weighted iterative optimization registration algorithm is designed, which significantly improves the adaptability and stability of the registration method in different scenarios; a lightweight target detection model is constructed based on a deep neural network, and the bounding box is delineated using an anchor box mechanism to maximize the IoU value as a loss function to ensure the accuracy and real-time performance of detection. The detection results are post-processed using a non-maximum suppression algorithm to effectively retain the best detection results and remove redundant results. The innovative method and system provided by the present invention can significantly improve the registration accuracy, feature extraction and fusion effects of multimodal remote sensing images, and the accuracy and real-time performance of target detection, which not only meets the real-time requirements in practical applications, but also effectively solves the problems of low registration accuracy, insufficient feature extraction, and high complexity of detection models in the prior art, providing an effective solution for remote sensing image processing and target detection.

[0092] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention are achieved and obtained through the structures specifically pointed out in the specification and the drawings.

[0093] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0095] Figure 1 The figure is a schematic diagram of steps for cross-scenario generalized small sample learning based on improved domain mapping in one embodiment of the present invention.

[0096] Figure 2 The figure is a flowchart of cross-scenario generalized small sample learning based on improved domain mapping in one embodiment of the present invention.

[0097] Figure 3 The figure is a schematic diagram of the steps of a target detection method enhanced by multi-domain heterogeneous data fusion in one embodiment of the present invention.

[0098] Figure 4 The present invention is a flowchart of a method for target detection enhanced by fusion of multi-domain heterogeneous data in one embodiment of the present invention.

[0099] Figure 5 The present invention is a flowchart of a multi-domain heterogeneous image registration technology based on feature matching in one embodiment of the present invention.

[0100] Figure 6 The present invention is a framework diagram of a multimodal remote sensing image semantic segmentation solution in one embodiment of the present invention. DETAILED DESCRIPTION

[0101] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0102] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0103] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0104] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection with an intermediate.

[0105] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0106] like Figure 1 As shown, the cross-scenario generalized small sample learning based on improved domain mapping and the representation learning based on improved masked autoencoder of the present invention include the following steps S101 to S105:

[0107] Step S101: Use the source domain dataset as a sample to train the improved masked autoencoder, mask a random part of the input image, input it into the encoder part of the autoencoder to obtain the feature representation of the image, and then restore it to the original complete image through the lightweight decoder. After the training is completed, the encoder part is used as a feature extractor.

[0108] Step S102: Set a small number of samples from the target domain as an auxiliary dataset, and mix them with the source domain dataset through a dataset mixer to obtain a mixed dataset, and re-divide the mixed dataset into a source domain support set, an auxiliary support set, and a mixed query set. The source domain support set maintains representative samples of the source domain, helping the target recognition model to maintain and consolidate the learning results in the source domain. The auxiliary support set contains a small number of samples from the target domain, which serves as auxiliary information to guide the model to gradually adapt to the feature distribution of the target domain and alleviate the domain shift problem. The mixed query set is composed of a mixture of the source domain and target domain datasets, which is used to test the generalization ability of the model when facing unknown or cross-domain samples.

[0109] Step S103: Input the image data of the source domain support set, the auxiliary support set and the mixed query set into the feature extractor to obtain domain-independent features and domain features. Domain-independent features refer to features that are not affected by a specific domain and are commonly present in various domains, such as some basic visual patterns or the general outline of an object; domain features refer to features related to a specific domain, which reflect the specificity of different scenes or environments.

[0110] Step S104: The domain-independent features are sent to the small sample classifier to perform the classification task and compared with the corresponding category labels to obtain the target classification loss function. The domain-independent features of the three-category set are input to the domain classifier for domain classification and compared with the domain labels to evaluate and obtain the domain classification loss function. The domain features of the three-category set are also processed in the same way to obtain the domain classification loss value. All loss functions are added together to obtain the overall loss function.

[0111] Step S105: Adopting the Adam optimization algorithm, an efficient gradient descent optimization process is performed on the overall loss function to iteratively optimize the parameters of the target recognition model to ensure the continuous improvement of the model performance and the gradual convergence of the overall error.

[0112] like Figure 2 As shown, it is a flowchart of cross-scene generalized small sample learning based on improved domain mapping in a cross-scene multi-domain fusion small sample remote sensing target robust recognition method.

[0113] In step S101, the feature representation of the source domain data set is first learned through a masked autoencoder, and a random mask strategy is used to partially mask the image so that the autoencoder can learn the image feature representation in the absence of information. The encoder then encodes the processed image to generate a feature vector containing information such as local texture, contextual relationship, and semantic information. The lightweight decoder receives these feature vectors and strives to reconstruct the original image. This process not only ensures the validity of the features, but also enhances the generalization ability of the model. After training, the encoder part is selected as the feature extractor. This process belongs to the pre-training stage.

[0114] In step S102, in order to cope with the challenges of cross-domain learning, a small number of samples from the target domain are introduced as auxiliary datasets, which are fused with the source domain dataset through a dataset mixer to create a hybrid dataset that contains both source domain characteristics and target domain characteristics. The hybrid dataset is then reorganized into a source domain support set, an auxiliary support set, and a hybrid query set, each of which carries a specific training mission: the source domain support set ensures the performance of the model in familiar domains, the auxiliary support set guides the model to gradually explore and adapt to the characteristics of the target domain, and the hybrid query set tests the generalization ability of the model under unknown or cross-domain conditions.

[0115] In step S103, the images in the source domain support set, the auxiliary support set, and the mixed query set are deeply analyzed by the feature extractor to extract domain-independent features and domain features. Domain-independent features are common features across multiple domains, which form the general basis of recognition tasks; while domain features specifically correspond to the unique properties of a specific domain, which helps the model recognize and distinguish the feature distribution of different scenes.

[0116] In step S104, an overall loss function is designed to ensure the simultaneous optimization of the model in the classification task and the domain discrimination task. The domain-independent features are fed into the small sample classifier for target category prediction and compared with the actual label to calculate the classification loss. At the same time, the domain-independent features and domain features are fed into the domain classifier to calculate the domain classification loss by relative entropy and cross entropy, respectively. The sum of these loss functions forms the overall optimization goal of the model, reflecting the comprehensive pursuit of the model in recognition accuracy and domain adaptability.

[0117] In step S105, in order to efficiently adjust the model parameters, the Adam optimization algorithm is used. This is an adaptive learning rate method that can dynamically adjust the learning rate based on historical gradient information, thereby more accurately guiding the model to approach the global minimum loss point during the gradient descent process. Through continuous iterative optimization, the performance of the model is continuously improved, while ensuring the gradual reduction of the overall error, and ultimately achieving high accuracy and strong robustness in cross-scenario small sample target recognition.

[0118] like Figure 3 As shown, the multi-domain heterogeneous image registration technology based on feature matching and the lightweight real-time target recognition model based on deep neural network of the present invention include the following steps S301 to S307:

[0119] Step S301: A multimodal remote sensing image registration method based on local boundary features is used to extract significant boundary features of the image pair and extract feature points for registration in combination with a grayscale-based method. In order to improve the accuracy of registration, the SIFT threshold is dynamically adjusted to extract feature points, and weight values ​​are calculated to strengthen important feature points. Multimodal remote sensing image registration based on boundary features extracts significant boundary feature points and constructs local boundary descriptors for feature matching to ensure high-precision registration in complex scenes.

[0120] Step S302: The registration results based on feature points and boundary features are integrated through an incremental optimization method, and the accuracy indicators of the respective registration results are evaluated and compared. New weights are designed according to the target detection accuracy, and the registration algorithm is iteratively optimized to improve the adaptability in different scenarios.

[0121] Step S303: Use the deep residual network (ResNet-101) backbone network feature extractor to extract features from the registered visible light, infrared and SAR images. Use the cross-modal attention mechanism to enhance the high-level and low-level features to improve the expressiveness of the features. Concatenate the high-level and low-level features and reduce their dimensionality to form the output feature map of the encoder part, and realize multi-level feature fusion to retain important information.

[0122] Step S304: A lightweight target detection model is constructed based on a deep neural network to ensure the accuracy and real-time performance of the detection. In this model, the bounding box is delineated using the anchor box mechanism. By predefining a set of anchor boxes of different scales and aspect ratios and adjusting the positions and sizes of these anchor boxes during the training process, the target object is accurately enclosed. The model generates multiple anchor boxes on the grid cells of each input image and predicts the offset and category probability of each anchor box to achieve bounding box delineation for target detection. Then, the deep learning model is optimized by maximizing the Intersection over Union (IoU) value as the loss function to improve the accuracy of the target position. IoU is used to measure the overlap between the predicted bounding box and the true bounding box. In model training, the performance of the model is effectively improved by maximizing the IoU value as the loss function. Finally, the detection results are post-processed using the non-maximum suppression (NMS) algorithm to remove redundant detection results and retain the best detection results, further improving the accuracy and real-time performance of target detection.

[0123] Step S305: In the target detection method, a deep residual network (ResNet-101) backbone network feature extractor is used to extract features from the registered visible light, infrared and SAR images. This step inputs images of different modalities into the ResNet-101 network and gradually extracts high-level and low-level features of the image through the hierarchical structure of the network.

[0124] Step S306: In object detection, in order to improve the expressiveness of features, a cross-modal attention mechanism is introduced to enhance the high-level and low-level features extracted by ResNet-101. The attention mechanism can adaptively adjust the weights between different modal features, so that more important features receive more attention, thereby improving the performance of object detection.

[0125] Step S307: Based on the cross-modal attention mechanism, a multi-level feature fusion strategy is used in target detection to fuse high-level and low-level features. Specifically, for the feature map of each modality, it is concatenated with the feature map of other modalities, and the dimension of the feature is reduced through dimensionality reduction operations, and finally the output feature map of the encoder part is formed. This multi-level feature fusion strategy can retain important information and improve the accuracy and robustness of target detection.

[0126] like Figure 4 The figure shows a flowchart of a multi-domain heterogeneous image registration technology based on feature matching in a cross-scene multi-domain fusion small sample remote sensing target robust recognition method and a lightweight real-time target recognition model based on deep neural network.

[0127] In step S301, firstly, significant boundary features are extracted from the image by using an edge detection algorithm (such as Canny). The edge detection algorithm can effectively identify the boundaries and contours in the image, thereby extracting significant boundary feature points. These boundary feature points usually have high stability and reliability, providing strong support for subsequent image registration.

[0128] Next, feature points are extracted based on the grayscale method. By analyzing the image using the grayscale co-occurrence matrix (GLCM), feature points that reflect the texture characteristics of the image can be extracted. The grayscale co-occurrence matrix is ​​a statistical method used to describe the grayscale linkage information of an image. It can capture the spatial relationship of pixel grayscale in an image, thereby extracting representative feature points.

[0129] In order to further improve the registration accuracy, the scale-invariant feature transform (SIFT) threshold is dynamically adjusted to extract feature points. Depending on the image features, the threshold of the SIFT algorithm needs to be dynamically adjusted to extract more stable and representative feature points. The SIFT algorithm extracts and matches feature points by finding key points in the image and describing them.

[0130] After extracting feature points, the importance weight value of each feature point is calculated to strengthen the significant feature points. The importance weight value of feature points is calculated by various methods, such as comprehensive evaluation based on factors such as the repeatability and stability of feature points. By giving significant feature points higher weight values, the role of these feature points can be more prominent in the registration process, improving the registration accuracy.

[0131] Finally, the local boundary descriptor is constructed using the salient boundary feature points and feature matching is performed to achieve image registration. The local boundary descriptor is a descriptor used to describe the local boundary features of an image, which can effectively characterize the local structure and shape information of the image. By using these local boundary descriptors for feature matching, high-precision image registration can be achieved in complex scenes. The entire registration process ensures high-precision registration in complex scenes by extracting salient boundary feature points and constructing local boundary descriptors.

[0132] In step S302, the registration results based on feature points and boundary features are first fused. In the multimodal remote sensing image registration process, the registration results of different feature points and boundary features have their own advantages and disadvantages. By fusing these two registration results, the information of both can be comprehensively utilized to improve the overall registration accuracy. Specifically, the registration results based on feature points are integrated with the registration results based on boundary features to generate a comprehensive registration result.

[0133] Then evaluate the accuracy of each registration result. During the fusion process, the accuracy of different registration results needs to be evaluated to select the best registration result. Common evaluation indicators include mean square error, number and distribution of matching points, etc. By comprehensively analyzing these indicators, the performance of each registration method in different scenarios can be determined, so as to select the best registration result.

[0134] Design new weights based on the accuracy of target detection. In order to further optimize the registration results, it is necessary to design a new weight allocation strategy based on the accuracy of target detection. Specifically, by analyzing the accuracy of target detection, it is possible to determine which feature points and boundary features contribute more to target detection. Based on this information, the weights of different feature points and boundary features in the registration results are adjusted so that the final registration result can maximize the accuracy of target detection.

[0135] Finally, the registration algorithm is optimized iteratively. The incremental optimization method optimizes the registration results through continuous iteration. In each iteration, the registration algorithm is adjusted and optimized according to the new weights designed in the previous step. Through repeated iterations, the adaptability and stability of the registration algorithm in different scenarios are gradually improved. The registration results after each iteration are further used for evaluation and weight adjustment until the expected accuracy and robustness goals are achieved. The whole process ensures that the registration algorithm can maintain efficient and reliable performance in complex and changing remote sensing image scenes.

[0136] In step S303, the registered visible light, infrared and SAR images are first fused. Through the previous registration step, the correspondence between the images of different modalities in the same coordinate system has been obtained. Next, these registered image data need to be fused. Specifically, the visible light image, infrared image and SAR image are superimposed in a spatially aligned manner so that their pixels are aligned in space. During the fusion process, weighted averaging, maximum value selection and other methods can be used to ensure that the image information of different modalities can be effectively integrated, thereby generating a fused image containing rich information.

[0137] After that, the initial fused image data is constructed. The fused image data contains rich information from three different modalities: visible light, infrared, and SAR, with a higher signal-to-noise ratio and more comprehensive feature representation. The goal of this step is to generate an initial fused image to provide a high-quality data foundation for subsequent feature extraction and target detection. When constructing the initial fused image data, a variety of image processing techniques, such as image filtering, denoising, and enhancement, can be used to further improve the quality and clarity of the fused image. Through these preprocessing steps, it can be ensured that the fused image has high visual effects and information fidelity, thus laying a solid foundation for subsequent deep feature extraction and target detection.

[0138] In step S304, a lightweight target detection model based on a deep neural network is first designed. This model is optimized for the real-time requirements of target detection tasks, and the computational complexity of the model is reduced by reducing the number of network layers, optimizing the network structure and the number of parameters. Such a lightweight design ensures efficient operation in a resource-constrained environment and also has significant performance in target detection.

[0139] Next, the bounding box is delineated using the anchor box mechanism. The anchor box mechanism is one of the commonly used techniques in target detection models. It predefines a set of anchor boxes of different scales and aspect ratios, and enables the model to learn to adjust the position and size of these anchor boxes during the training process to accurately surround the target object. Specifically, on each grid cell of the input image, the model generates multiple anchor boxes and predicts the offset and category probability of each anchor box to delineate the bounding box for target detection. This mechanism can significantly improve the model's ability to detect target objects of different scales and shapes.

[0140] Then, the maximum IoU value is used as the loss function. IoU is an indicator that measures the degree of overlap between the predicted bounding box and the true bounding box. During the model training process, by maximizing the IoU value as the loss function, the model can be effectively optimized to more accurately predict the location of the target object. Specifically, the loss function calculates the IoU value between the predicted box and the true box, and adjusts the model parameters through back propagation to improve the IoU value, thereby improving the accuracy of target detection.

[0141] Finally, the non-maximum suppression (NMS) algorithm is used to post-process the detection results. Since the target detection model may predict multiple overlapping bounding boxes for the same target object, the NMS algorithm is introduced to remove these redundant detection results. In the NMS process, the confidence of all predicted bounding boxes is first sorted, and then the IoU values ​​of other bounding boxes are compared one by one, starting from the bounding box with the highest confidence. If the IoU value exceeds the set threshold, these bounding boxes are considered redundant and are removed. In this way, the NMS algorithm can retain the optimal detection results and remove redundant results, thereby improving the accuracy and efficiency of target detection.

[0142] In step S305, images of different modalities are first input into the ResNet-101 network. In order to make full use of the information of multimodal remote sensing images, the registered visible light, infrared and SAR images are input into the pre-trained ResNet-101 network. ResNet-101 is a deep residual network with 101 layers. By introducing residual connections, the gradient vanishing problem in deep network training can be effectively alleviated to ensure the stability and effectiveness of feature extraction.

[0143] Then, the high-level and low-level features of the image are extracted through the ResNet-101 network. In the ResNet-101 network, the first few layers are mainly responsible for extracting low-level features of the image, which usually contain local information such as edges and textures; while the latter layers extract high-level features of the image, which can capture global information such as the shape and category of the object. By extracting multi-level features of the image layer by layer, it ensures that image information of different modalities can be fully mined and expressed, providing rich feature representation for subsequent feature fusion and target detection.

[0144] In step S306, an improved cross-modal multi-head self-attention module is first introduced to adaptively adjust feature weights. In the process of multimodal feature fusion, the contributions of image features of different modalities to target detection may be different. To this end, an improved cross-modal multi-head self-attention module is introduced to adaptively adjust these weights by calculating the importance weights of each modal feature so that more important features receive more attention during the fusion process. This adaptive weight adjustment mechanism can effectively improve the effect of multimodal feature fusion.

[0145] Then, the attention mechanism is used to enhance the expressive power of high-level and low-level features. The improved cross-modal multi-head self-attention module can not only adjust the weights between features of different modalities, but also enhance features of different levels of the same modality. In the specific implementation process, the attention mechanism amplifies the feature response of high-weight areas by calculating the attention weights of the feature map, thereby enhancing the expressive power of high-level and low-level features. The feature map enhanced by the attention mechanism can more accurately represent the target object in the image, providing more precise feature support for the final target detection.

[0146] In step S307, a multi-level feature fusion strategy is first used to fuse high-level and low-level features. In order to make full use of the information of features at different levels, a multi-level feature fusion strategy is adopted, that is, high-level features extracted from a deep residual network (such as ResNet-101) and low-level features are fused. High-level features usually contain more semantic information, while low-level features contain more detailed information. By fusing these two types of features, various characteristics of the target object can be better captured, improving the effect of target detection.

[0147] Next, the feature maps of different modalities are spliced ​​together and the feature dimensions are reduced through dimensionality reduction operations. During the fusion process, feature maps of different modalities (such as feature maps of visible light, infrared, and SAR images) are spliced ​​together to form a multimodal feature map. In order to reduce the computational complexity and avoid the feature map dimension being too high, the spliced ​​feature map is processed by dimensionality reduction operations to retain the main information while reducing redundant features. This step ensures the compactness and efficiency of the fused features.

[0148] like Figure 5 As shown in FIG. 1 , a multi-domain heterogeneous image registration technology based on feature matching is shown in the figure. As shown in the figure, the technology is executed in the image registration system and includes the following steps:

[0149] Step S501: Multimodal remote sensing image registration based on local boundary features. Input the reference image R and the image to be registered S, extract the feature points by extracting significant boundary features and grayscale-based methods, dynamically adjust the scale-invariant feature transform (SIFT) threshold, extract feature points and calculate weight values ​​to strengthen important feature points, and ensure high-precision registration in complex scenes.

[0150]

[0151] Among them, w i is the feature point p i The weight of score(p i ) is the score of the feature point.

[0152] Step S502: Multi-domain heterogeneous image registration based on feature points. The extracted SIFT feature points are used for feature matching, and weight calculation and feature matching strategies are adopted to achieve high-precision image registration.

[0153]

[0154] Among them, dist(p i .p j ) is the feature point p i and p j The distance between them, ∈ is the distance threshold.

[0155] Step S503: Multi-domain heterogeneous image registration based on boundary features: By detecting boundaries and boundary feature points, local boundary features and boundary descriptors are calculated, and feature matching is performed.

[0156]

[0157] Among them, desc(p i ) is the feature point p i The boundary descriptor of and are the gradients of the image in the x and y directions, respectively.

[0158] Step S504: Incremental optimization registration. The incremental optimization method is integrated with the registration results based on feature points and boundary features, and the accuracy indicators of the respective registration results are evaluated and compared. New weights are designed according to the target detection accuracy, and the registration algorithm is iteratively optimized to improve the adaptability in different scenarios.

[0159]

[0160] Among them, loss is the loss function, w i is the weight of the feature point, is the feature point p i Its corresponding point The distance between.

[0161] Step S505: Image registration and transformation: Transform the registered image to ensure the accuracy and robustness of target detection.

[0162]

[0163] Where T is the registration transformation matrix.

[0164] Step S506: Target detection evaluation index analysis: Analyze the detection results, evaluate the effects of registration and detection, and ensure the applicability and reliability of the system in complex scenarios.

[0165]

[0166] Among them, IoU is Intersection over Union.

[0167] The figure shows the specific implementation process of each step, including inputting the reference image R and the image to be registered S, extracting SIFT feature points, calculating weights and performing feature matching, using incremental optimization registration, and finally realizing image registration and transformation, as well as evaluation and analysis of target detection results.

[0168] The present invention also provides a scheme framework for semantic segmentation of multimodal remote sensing images, such as Figure 6 As shown, the technology is implemented in an image segmentation system and includes the following steps:

[0169] Step S501: Input the reference image and the image to be registered, including the visible light image, infrared image and SAR image. For the input visible light, infrared and SAR images, extract low-level and high-level features through the backbone network. Let the input image be I, and the low-level features and high-level features extracted by the backbone network be represented as F low and F high

[0170] F low =Backbone low (I)

[0171] F high =Backbone high (I)

[0172] Step S502: Process the low-level features through the cross-modal attention module to achieve feature splicing and obtain low-level splicing features. Let the low-level feature splicing result be A low-dw

[0173] A low-dw =Attention low (F low,OPT ,F low,SAR )

[0174] Step S503: Process the high-level features through the cross-modal attention module to achieve feature splicing and obtain high-level splicing features. high-ms

[0175] A high-ms =Attention high (F high,OPT ,F high,SAR )

[0176] Step S504: Comprehensively stitch the low-level and high-level stitching features to generate the final stitching feature A low-high :

[0177] A low-high =Concatenate(A low-dw ,A high-ms )

[0178] Step S505: Input the concatenated features into a regression predictor based on a deep neural network to perform target detection and semantic segmentation, and output a prediction result P.

[0179] P = RegressionNetwork(A low-high )

[0180] Step S506: Post-process and optimize the detection results, use the non-maximum suppression (NMS) algorithm to remove redundant detection frames, set the confidence threshold to t, retain the best detection result, and output the final target detection result. The processing formula of NMS is:

[0181]

[0182] The figure shows the specific implementation process of each step, including inputting the reference image and the image to be registered, feature extraction, cross-modal attention module processing, feature splicing, regression prediction based on deep neural network, and post-processing optimization process. Finally, accurate semantic segmentation of multimodal remote sensing images is achieved.

[0183] Then the output feature map of the encoder part is formed to retain important information. The feature map after multi-level fusion and dimensionality reduction processing is used as the output feature map of the encoder part. These feature maps integrate feature information of different modalities and different levels, and can better represent various information in the image, providing a solid foundation for subsequent target detection.

[0184] Finally, based on the fused feature map, target detection and recognition are performed to improve detection accuracy and robustness. Using an optimized target detection model (such as a lightweight deep neural network model), the fused feature map is input into the detection model to identify and locate the target object. The above multi-layer feature fusion strategy can effectively improve the accuracy and robustness of target detection, ensuring excellent detection performance in complex scenarios. The non-maximum suppression (NMS) algorithm further removes redundant results in the post-processing of the detection results to ensure the accuracy of the detection results.

[0185] In summary, the present invention provides a target detection and image registration method based on multimodal remote sensing images, which belongs to the field of image processing and remote sensing. The method is based on edge detection algorithm (such as Canny), gray level co-occurrence matrix (GLCM) and dynamically adjusted scale invariant feature transform (SIFT), and realizes accurate image registration by extracting significant boundary features, texture features and stable feature points; and by calculating the importance weights of feature points and constructing local boundary descriptors, the role of significant feature points is strengthened to ensure high-precision image registration. The method also improves the accuracy and robustness of registration by fusing the registration results based on feature points and boundary features, evaluating the accuracy index of the registration results and designing new weights. Based on the registered multimodal images, a lightweight deep neural network target detection model is designed, and the anchor frame mechanism, the maximum IoU value loss function and the non-maximum suppression algorithm are used to achieve efficient and accurate target detection. Finally, by introducing an improved cross-modal multi-head self-attention module, a multi-level feature fusion strategy and a dimensionality reduction operation, feature information of different modalities and different levels are effectively fused, thereby improving the accuracy and robustness of target detection. The whole method ensures efficient and accurate target detection and image registration in complex and changeable remote sensing image scenes, providing an effective solution for remote sensing image processing.

[0186] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0187] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0188] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0189] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A robust recognition method for small sample remote sensing targets with cross-scene multi-domain fusion, characterized in that: The method comprises the following steps: Step 1: Based on the cross-scenario generalized small-sample learning of improved domain mapping, a meta-learning strategy is used to build a K-way-N-shot learning scenario. A small amount of labeled samples is used to drive the iterative upgrade of the model, so that it can quickly adapt to and accurately perform classification tasks when faced with new tasks. Step 2: Based on the representation learning of the improved masked autoencoder, a random mask mechanism is used to extract high-level semantic feature representations from large-scale image data without relying on labeled data. Step 3: Based on the improved feature matching multi-domain heterogeneous image registration technology, the significant boundary features of the multimodal remote sensing images are extracted, and the images are registered with the feature points extracted based on the grayscale method. The visible light, infrared and SAR images after image registration and exchange are subjected to feature extraction again, and the improved incremental optimization method is combined with the improved anchor frame mechanism to delineate the boundary frame. Step 4: Based on the lightweight real-time target recognition model of the improved deep neural network, the ResNet-101 backbone network feature extractor is used to extract features of the registered multimodal remote sensing images; and target detection and recognition are performed through the improved cross-modal self-attention mechanism and multi-level feature fusion method.

2. The cross-scene multi-domain fusion small sample remote sensing target robust recognition method according to claim 1 is characterized in that: The cross-scenario generalized small sample learning based on improved domain mapping includes the following steps: Through the dataset mixer, a very small number of samples in the target domain are set as auxiliary datasets, and merged with the dataset of the source domain, and re-divided into source domain support set, auxiliary support set and mixed query set, so as to comprehensively utilize the sample data of the source domain and target domain, and improve the ability of cross-domain small sample learning to cope with domain shift; In the pre-training stage, the masked autoencoder is used for training on the dataset of the source domain to learn image features in a self-supervised manner. The encoder part of the trained masked autoencoder is used as a feature extractor. The feature extractor decouples the image features of the source domain and the target domain to obtain domain-independent features and domain features, and maps the features of the target domain to the source domain, thereby helping to narrow the gap in data distribution between the source domain and the target domain, and converting the data of the source domain and the target domain into a common feature space; the learning of the model is guided by obtaining a very small amount of labeled data belonging to the target domain and adding it to the training dataset, that is, using the very small amount of labeled data of the target domain as auxiliary data to improve the ability of cross-domain small sample learning to cope with domain shift, and enable the meta-learning-based object recognition model to have good generalization ability on the target domain dataset; In the meta-learning stage, the source domain support set, auxiliary support set, and mixed query set are sent to the feature extractor to obtain their respective domain-independent features and domain features. H1 and H2 are used to represent the domain-independent features and domain features, respectively. The domain-independent features and domain features of the source domain support set are expressed as and The domain-independent features and domain features of the auxiliary support set are expressed as and The domain-independent features and domain features of the mixed query set are expressed as and Feed the domain-independent features into the small-shot classifier g fsl Classify and compare with the label to get the loss function L FSL ; Send domain-independent features and domain features to the domain classifier g dom Perform domain classification in and compare with domain labels to obtain the loss function and Overall loss Use the Adam gradient descent optimizer to optimize L to minimize L; The small sample classifier will and Input small sample classifier and get classification loss and Input small sample classifier and get classification loss Since the mixed query set is a mixture of the source domain query set and the auxiliary query set with a ratio of λ, the loss function of the small sample classifier is The domain classifier, for domain-dependent features, sets the source domain label to 1 and the target domain label to 0, and for domain-independent features, sets the domain label vector to y1 = [0.5, 0.5]; is a domain-independent feature, and is input into the domain classifier g dom It is classified and compared with the label y1, and the loss is calculated using the relative entropy KL divergence The input is compared with the respective domain labels, and the loss is calculated using cross entropy 3. The cross-scene multi-domain fusion small sample remote sensing target robust recognition method according to claim 1 is characterized in that: The representation learning based on the improved masked autoencoder includes the following steps: The random mask mechanism is used to mask random areas of the high-dimensional input image, and the reconstruction operation of the autoencoder is used to predict the masked area to achieve effective encoding of the data. Without relying on labeled training data, a variety of general feature representations such as local texture, contextual relationship, semantic information, etc. are learned from massive unlabeled data. In the masked autoencoder, the encoder is composed of a combination of ViT modules, specifically ViT-L / 16; the decoder can use a lightweight Transformer module to improve training and execution efficiency; since it is necessary to infer complete information from incomplete input, the masked autoencoder learns how to capture global context information and local details, thereby generating a more robust and generalizable feature representation, so that the masked autoencoder can also have good feature representation capabilities under cross-scenario conditions; The ViT module transforms the encoder part of the traditional Transformer and uses the attention mechanism to parallelize the training of the mask autoencoder, speed up the training and inference of the mask autoencoder, and handle problems with long-term dependencies. The image is divided into blocks of the same size, and then these image blocks are linearly mapped into embedded vectors through a fully connected layer. Since the attention mechanism lacks position information representation, the vector converted from the image block is coupled with the vector representing the position information as the input of ViT. With the help of position encoding, the model can capture more refined position features. The lightweight Transformer module has a depth of 8, that is, it is composed of 8 layers of ViT modules stacked together, and a width of 512, that is, each image block is processed into a 512-dimensional feature vector; the computational complexity of the decoder composed of it is only 10% of that of the encoder.

4. The cross-scene multi-domain fusion small sample remote sensing target robust recognition method according to claim 1 is characterized in that: The multi-domain heterogeneous image registration technology based on improved feature matching includes the following steps: The improved method for extracting significant boundary feature points is used to extract significant boundary feature points and construct a description of the boundaries of each target to be detected and identified in the image, so as to achieve more stable and accurate image registration. After executing the image registration and exchange method, the improved incremental optimization method and the improved anchor frame mechanism are combined to improve the adaptability and stability in different scenarios. The cross-scenario refers to the scene with a long span, large terrain undulations, large sea conditions, and inconsistent meteorological conditions in some areas. The above method and steps can effectively improve the robustness and accuracy of multi-domain heterogeneous image registration, and ensure the robust recognition performance in the above application scenarios according to the target detection evaluation index analysis method; Using an improved method for extracting significant boundary feature points, multimodal remote sensing image registration based on local boundary features is performed. After inputting a reference image R and an image to be registered S, the reference image R is used as a standard image as a reference, and the image to be registered S is used as an image aligned with the reference image; by extracting significant boundary features and extracting feature points based on a grayscale method, dynamically adjusting the scale-invariant feature transformation threshold, extracting feature points and calculating weight values ​​to strengthen important feature points, and ensuring high-precision registration in complex scenes. The specific implementation process includes: for the general case of an input image, that is, an image I of one of the reference image R or the image to be registered S, calculating its significant boundary feature point set F = {f i |i=1,2,…,n}, weight value of feature point where g i is the feature point f i Gray value, α and β are adjustment parameters; The image registration and exchange method transforms the registered image to ensure the accuracy and robustness of target detection. The image transformation matrix is ​​calculated as follows: Using the improved incremental optimization method, by detecting the boundary and boundary feature points, calculating the local boundary features and boundary descriptors, and performing feature matching, this method not only considers the registration of feature points, but also integrates the boundary features to update the optimization parameters. The specific implementation process includes: at each iteration step t, calculating the loss L of the current registration result total =L match +L boundary , update the parameters based on the gradient of the loss function The improved incremental optimization method is used to fuse the registration results based on feature points and boundary features, evaluate and compare the accuracy indicators of each registration result, and then design new weights according to the target recognition accuracy, iteratively optimize the registration algorithm, and finally improve the generalization ability of the data for different scenarios. The specific implementation process includes: in each incremental optimization step t, update the feature point weight Among them, TA represents the target recognition accuracy, and CA represents the current recognition accuracy; The improved anchor frame mechanism presets a multi-scale and multi-ratio anchor frame on each grid unit of the input image, adjusts the position and size of the anchor frame through model learning, and uses the adjusted anchor frame mechanism to delineate the boundary frame. The specific implementation method includes: L = L conf +λ1L loc +λ2L cls , where L is the loss function, L conf represents the confidence loss, L loc represents the position loss, L cls represents classification loss, λ1 and λ2 are weight coefficients; The target detection evaluation index analysis method analyzes the detection results, evaluates the effects of registration and detection, and obtains the average precision mean to ensure the applicability and reliability of the system in complex scenarios. The calculation of the average precision mean is:

5. The cross-scene multi-domain fusion small sample remote sensing target robust recognition method according to claim 1 is characterized in that: The lightweight real-time target recognition model based on deep neural network includes the following steps: By using the ResNet-101 backbone network feature extractor and the improved cross-modal multi-head self-attention mechanism and the improved multi-level feature fusion method, the accuracy and robustness of target detection and recognition are enhanced, and the computational complexity is reduced through lightweight design and optimization methods combined with improved threshold and weight dynamic settings to meet the real-time requirements in practical applications; The ResNet-101 backbone network feature extractor extracts multi-layer features of the image through the ResNet-101 backbone network feature extractor to ensure that the extracted features have high expressiveness and robustness. The specific implementation process includes: ResNet-101 ={f1,f2,…,f n }, where F ResNet-101 Represents a multi-layer feature set extracted by ResNet-101; The improved cross-modal multi-head self-attention mechanism introduces a cross-modal attention mechanism between multi-modal features including visible light, infrared and synthetic aperture radar images, and captures more diverse feature relationships and semantic information by multi-head segmentation and parallel processing of features of different modalities, so as to enhance the correlation between features and improve the accuracy of target detection. The attention weight is obtained by calculating the similarity through the Softmax function. The specific implementation process includes: A attention =softmax(S)·V, where S represents the similarity matrix, which is calculated by cosine similarity, and V represents the value matrix. The calculation formula of cosine similarity is: Where Q i and K j denote the query vector and key vector respectively, · denotes the dot product operation, ‖·‖ denotes the norm of the vector Where H represents the number of heads, W O is the output weight matrix; The improved multi-level feature fusion method introduces a cross-modal attention mechanism and adaptive weight adjustment to fuse features at different levels to improve the detection effect. Among them, α i The fusion weight of the i-th layer feature is optimized through training. The specific implementation process of the fusion method before the improvement includes the following steps: extracting multi-level features from the reference image and the image to be registered, performing cross-modal attention enhancement on the extracted features, calculating the attention weight, assigning adaptive weights to features at different levels, and dynamically adjusting the weight parameters through training. The improved fusion formula of the present invention is: Among them A i represents the attention weight matrix of the i-th layer feature, α i represents the fusion weight dynamically optimized through training. The specific implementation process of the improved fusion method includes: extracting multi-level features {f i |1,2,…,n}, calculate the attention weight A i =Softmax(Sim(f i ,f i )),in Dynamically optimize the fusion weight α through training i , so that the fused features can better adapt to different scenarios and task requirements. Through the above improved method, more robust and efficient feature fusion can be achieved in different scenarios, improving the target detection accuracy and robustness of multimodal remote sensing images; The lightweight design and optimization method described above reduces the computational complexity of the model through depthwise separable convolution, ensuring efficient operation in resource-constrained environments. The calculation method of depthwise separable convolution is: D = D depthwise (F fusion )·D pointwise (F fusion ), where D depthwise Denotes depth convolution, D pointwise Represents point-wise convolution; The improved threshold and weight dynamic setting, in the process of model training, dynamically adjusts the threshold and weight of each parameter through cross-validation to ensure the best detection effect. Different from the traditional method, the present invention adopts a dynamic adjustment mechanism based on incremental optimization. In each training cycle, the parameters are updated in real time according to the performance indicators of the current model. The specific implementation process includes: optimizing the target θ * = arg min θ E[L(θ)], where θ represents the model parameters and L(θ) represents the loss function.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Citation Information

Patent Citations

  • Image foreground segmentation method combining subject determination and edge precision

    CN113592893A

  • General target detection method for adaptive attention guidance mechanism

    WO2021139069A1

Cited By

  • Multi-view feature fusion road vehicle detection method and system

    CN119495076A

  • Multi-modal image registration method and system based on feature decoupling and computer equipment

    CN119625039A

  • Dynamic map modal semantic segmentation method and system for vehicle-mounted mining video monitoring

    CN119625323A

  • Object-oriented change target detection method and system based on improved YOLO

    CN119832293A

  • An object-oriented changing target detection method and system based on improved YOLO

    CN119832293B