Remote sensing image surface anomaly detection method and device based on image-text collaborative processing

Abstract

The present application provides a method and device for detecting surface anomalies in remote sensing images based on collaborative image and text processing. The method comprises: obtaining text features of each surface anomaly type label; performing band extraction and fusion on the multispectral remote sensing image to be detected, obtaining the RGB image and SWIR image of the multispectral remote sensing image, inputting them into an image encoder module, obtaining initialization RGB features and initialization SWIR features, inputting them into a multiscale spatial adapter module, obtaining multiscale enhanced spatially adapted RGB features and multiscale enhanced spatially adapted SWIR features, inputting them into a cross-band alignment module, obtaining RGB image features and SWIR image features, inputting them into a dynamic spectral weighted fusion module, and generating fused image features; performing cosine similarity analysis on the fused image features and the text features of each surface anomaly type label to determine the surface anomaly type of the multispectral remote sensing image. This improves the accuracy and generalization of surface anomaly detection.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a method and device for detecting surface anomalies in remote sensing images based on collaborative image and text processing. Background Art

[0002] Earth Surface Anomaly Detection (ESAD) from remote sensing images plays a vital role in emergency response, disaster assessment, and environmental monitoring. In recent years, with global climate change and rapid economic growth, various surface anomalies, both natural and human, have become increasingly frequent. Their rapid evolution poses a threat to public safety and sustainable development, placing higher demands on surface anomaly detection technology.

[0003] Surface anomalies are numerous and their remote sensing response characteristics vary. The key to surface anomaly detection lies in extracting effective anomaly features, including spectral and spatial features. For example, surface anomalies such as earthquakes, landslides, and debris flows exhibit quantitative relationships with visible light (spectral, geometric, and texture) and radar (backscatter and texture) response characteristics. Vegetation surface anomalies caused by forest and grassland fires, for example, are primarily related to thermal infrared radiation temperature. Traditional methods utilize prior knowledge to design targeted artificial features, such as various remote sensing indices, and then combine them with manual interpretation to achieve surface anomaly detection. Advances in sensor technology, in particular, have continuously broadened and enriched the detection range and spectral information of high-resolution satellite remote sensing, reducing the cost of large-scale surface anomaly detection. However, these prior knowledge-based methods have a long development cycle and, due to the complex and diverse data distribution, are difficult to generalize to surface anomaly events across diverse temporal and spatial scales.

[0004] The rise of deep learning technology has changed this landscape. Data-driven deep learning models have gradually replaced expert-driven feature engineering. In the field of remote sensing image processing, deep learning methods are primarily based on convolutional neural networks (CNNs) and visual transformers (ViTs). CNNs automatically extract local features from images, such as edges and textures, through convolutional layers, and reduce feature dimensionality through pooling layers to improve computational efficiency. ViTs, through a self-attention mechanism, can capture long-range dependencies between arbitrary locations in an input image sequence, thereby enhancing understanding of global context. By deeply mining high-level semantic information in remote sensing imagery, deep learning methods have surpassed most methods based on hand-crafted features in terms of accuracy and transferability. However, cross-scenario and cross-category surface anomaly detection remains challenging. Furthermore, deep learning models typically require a large amount of training data to achieve good generalization, which presents a challenge for remote sensing imagery with limited availability and high annotation costs for surface anomalies.

[0005] Existing surface anomaly detection methods often rely on limited band combinations, such as using only visible light remote sensing imagery. This underutilizes spectral information, leading to missed or false surface anomalies. To more effectively capture surface anomaly features, many improved methods incorporate additional spectral band data as auxiliary branches, such as near-infrared imagery and synthetic aperture radar (SAR) imagery, and then achieve surface anomaly detection through feature fusion. However, while this design can capture key anomaly features to a certain extent, it has significant limitations. First, the additional spectral band data increases acquisition and processing costs, especially when registering heterogeneous SAR images. Second, the spectral response characteristics of different surface anomalies often vary. Existing improved methods are often limited to a specific type of surface anomaly and lack a general paradigm applicable to most surface anomalies. Furthermore, modal semantic conflict limits the ability of deep learning models to represent anomaly features, while linear fusion methods cannot exploit deep band correlations, making it difficult to effectively fuse multimodal features.

[0006] In recent years, large pre-trained models have gained widespread application in computer vision due to their strong generalization capabilities. For example, the Contrastive Language-Image Pre-training (CLIP) model is a powerful classification model that achieves cross-modal semantic understanding and association by aligning the semantic spaces of images and text. However, directly applying the CLIP model to the task of surface anomaly detection faces two major generalization bottlenecks. First, surface anomaly events themselves are characterized by low frequency and uneven distribution. Remote sensing images are often affected by cloud occlusion, and sample annotation is expensive, resulting in a scarcity of high-quality training samples and a tendency for the model to overfit. Second, existing large pre-trained models, such as the CLIP model, primarily utilize natural image datasets for weight training. However, remote sensing images differ significantly from natural images, including differences in spectral distribution and spatial scale. Developing a cross-domain generalization mechanism that leverages the spectral and spatial characteristics of remote sensing images to learn highly generalizable, cross-scenario surface anomaly features using a small number of samples remains an open and ongoing challenge.

[0007] Existing surface anomaly detection methods still have many shortcomings in large-scale practical applications, especially in open-ended detection. Traditional methods are mostly closed-set detection, meaning that certain anomaly categories must be predefined during the training phase. For new, unseen types, retraining is required based on the existing model. However, in real-world scenarios, the connotation and extension of surface anomalies continue to deepen and expand, and various surface anomalies intertwine and overlap, presenting complex and diverse evolutionary trends. Therefore, from the perspective of the effectiveness of surface anomaly emergency response and the cost of deep model training, the limited types assumed by the closed-set model are seriously out of sync with the unlimited demands of actual detection. Furthermore, due to the intra-class variability and inter-class similarity of remote sensing objects, the manifestation of surface anomalies in remote sensing imagery changes with the scene. Existing models lack the ability to model the essential characteristics of surface anomalies, resulting in poor performance in the detection of unknown anomalies.

[0008] In summary, the current surface anomaly detection methods based on remote sensing images have low accuracy and generalization for surface anomaly detection. Summary of the Invention

[0009] Based on this, it is necessary to provide a remote sensing image surface anomaly detection method, device, electronic device and storage medium based on image-text collaborative processing that can improve the accuracy and generalization of surface anomaly detection in response to the above technical problems.

[0010] This application provides a method for detecting surface anomalies in remote sensing images based on collaborative image and text processing, comprising the following steps:

[0011] Step 1: Obtain text features corresponding to each surface anomaly type label. The text features corresponding to the surface anomaly type label are pre-converted into corresponding natural sentences based on the surface anomaly type label and a customized text prompt template, and feature extraction is performed on the natural sentences using a text encoder to obtain text features represented by vectors.

[0012] Step 2: performing band extraction and fusion on the multispectral remote sensing image to be detected to obtain an RGB image and a SWIR image of the multispectral remote sensing image;

[0013] Step 3: Input the RGB image and the SWIR image into an image encoder module for feature extraction to obtain initialized RGB features and initialized SWIR features;

[0014] Step 4: Input the initialized RGB features and the initialized SWIR features into a multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted RGB features and multi-scale enhanced spatially adapted SWIR features;

[0015] Step 5: Input the multi-scale enhanced spatially adapted RGB features and the multi-scale enhanced spatially adapted SWIR features into a cross-band alignment module for spectral adjustment to obtain RGB image features and SWIR image features;

[0016] Step 6: Input the RGB image features and the SWIR image features into a dynamic spectral weighted fusion device for feature fusion to generate fused image features;

[0017] Step 7: Perform cosine similarity analysis on the fused image features and the text features corresponding to each surface anomaly type label to determine the surface anomaly type of the multispectral remote sensing image.

[0018] In one embodiment, the multi-scale spatial adapter module includes two multi-scale spatial adapters with the same network structure;

[0019] Inputting the initialized RGB features and the initialized SWIR features into a multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted RGB features and multi-scale enhanced spatially adapted SWIR features, including:

[0020] Inputting the initialized RGB features into one of the multi-scale spatial adapters of the multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted RGB features;

[0021] The initialized SWIR features are input into another multi-scale spatial adapter of the multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted SWIR features.

[0022] In one embodiment, the processing process of the multi-scale spatial adapter is:

[0023] Separating the initialization features input into the multi-scale spatial adapter into a CLS sequence and an image block sequence;

[0024] The image block sequence is reconstructed into a 2D feature map that retains spatial information:

[0025] The 2D feature map is input into the convolution enhancement unit for multi-scale convolution processing to obtain multi-scale fusion features. The convolution enhancement unit is a four-branch multi-convolution structure;

[0026] The multi-scale fusion features are input into the global feature extraction unit for feature extraction to obtain global features;

[0027] Adding the global feature to the CLS sequence to obtain an enhanced CLS sequence;

[0028] The multi-scale fusion features are restored to a sequence format and recombined with the enhanced CLS sequence to obtain multi-scale enhanced spatial adaptation features.

[0029] In one embodiment, the cross-band alignment module includes two linear layers with the same network structure and two cross-band alignment units with the same network structure;

[0030] The step of inputting the multi-scale enhanced spatially adapted RGB features and the multi-scale enhanced spatially adapted SWIR features into a cross-band alignment module for spectral adjustment to obtain RGB image features and SWIR image features includes:

[0031] Inputting the multi-scale enhanced spatially adapted RGB features into one of the linear layers of the cross-band alignment module for processing to obtain RGB query features, RGB key features, and RGB numerical features;

[0032] Inputting the multi-scale enhanced spatially adapted SWIR features into another linear layer of the cross-band alignment module for processing to obtain SWIR query features, SWIR key features, and SWIR numerical features;

[0033] Inputting the SWIR query feature, the RGB key feature, and the RGB numerical feature into one of the cross-band alignment units of the cross-band alignment module for spectral adjustment to obtain RGB image features;

[0034] The RGB query feature, the SWIR key feature, and the SWIR numerical feature are input into another cross-band alignment unit of the cross-band alignment module for spectral adjustment to obtain a SWIR image feature.

[0035] In one embodiment, inputting the SWIR query feature, the RGB key feature, and the RGB numerical feature into one of the cross-band alignment units of the cross-band alignment module for spectral adjustment to obtain the RGB image feature includes:

[0036] The SWIR query feature, the RGB key feature, and the RGB numerical feature are subjected to similarity measurement and weighted summation to obtain a multispectral attention weighted RGB feature that focuses on similar features of the RGB image and the SWIR image. The expression is:

[0037] ;

[0038] in, represents the multi-spectral attention weighted RGB features, represents the Softmax activation function, is the attention weight coefficient of one of the cross-band alignment units, Represents RGB numerical features, R represents real number space, N represents the length of the feature, and C represents the number of channels of the feature;

[0039] Among them, spectral information divergence is used for SWIR query features and RGB key features The similarity measure of , obtains the attention weight coefficient of one of the cross-band alignment units , the expression is:

[0040] ;

[0041] in, Refers to the similarity measurement method, is the natural exponential function, represents the spectral information divergence matrix between the SWIR query feature and the RGB key feature. The first Spectral information divergence of the column The expression is:

[0042] ;

[0043] in, represents the probability normalized indivual sequence, , is the logarithmic function with base 10, represents the probability normalized indivual sequence, , superscript represents transpose;

[0044] The RGB numerical features With the multispectral attention weighted RGB features After matrix subtraction and linear mapping, the SWIR query feature is obtained as a residual structure Perform residual connection to obtain RGB difference features;

[0045] The RGB difference features are processed by layer normalization and multi-layer perceptron, and then connected with the RGB difference feature residual to obtain RGB image features;

[0046] The step of inputting the RGB query feature, the SWIR key feature, and the SWIR numerical feature into another cross-band alignment unit of the cross-band alignment module for spectral adjustment to obtain the SWIR image feature comprises:

[0047] The RGB query feature, the SWIR key feature, and the SWIR numerical feature are subjected to similarity measurement and weighted summation to obtain a multispectral attention weighted SWIR feature that focuses on similar features of the SWIR image and the RGB image. The expression is:

[0048] ;

[0049] in, represents the multispectral attention-weighted SWIR features, is the attention weight coefficient of another cross-band alignment unit, Indicates SWIR numerical characteristics;

[0050] Among them, spectral information divergence is used for RGB query features and SWIR key characteristics Similarity measure, obtain the attention weight coefficient of another cross-band alignment unit , the expression is:

[0051] ;

[0052] in, represents the spectral information divergence matrix between RGB query features and SWIR key features. The first Spectral information divergence of the column The calculation formula is:

[0053] ;

[0054] in, represents the probability normalized indivual sequence, , represents the probability normalized indivual sequence, ;

[0055] The SWIR numerical characteristics With the multispectral attention weighted SWIR features After matrix subtraction and linear mapping, the RGB query feature is used as a residual structure Perform residual connection to obtain SWIR difference features;

[0056] The SWIR difference features are processed by layer normalization and a multi-layer perceptron, and then connected with the SWIR difference feature residual to obtain the SWIR image features.

[0057] In one embodiment, inputting the RGB image features and the SWIR image features into a dynamic spectral weighted fusion device for feature fusion to generate fused image features includes:

[0058] The RGB image features are input into a dynamic spectral weighted fusion device, and then subjected to a layer normalization operation and feature separation to obtain a CLS sequence of the RGB image features and an image block sequence of the RGB image features;

[0059] The SWIR image features are input into a dynamic spectral weighted fusion device and subjected to a layer normalization operation to perform feature separation into a CLS sequence of the SWIR image features and an image block sequence of the SWIR image features;

[0060] Reconstructing the image block sequence of the RGB image feature and the image block sequence of the SWIR image feature into a 2D feature map of the RGB image feature and a 2D feature map of the SWIR image feature;

[0061] After splicing the 2D feature map of the RGB image features and the 2D feature map of the SWIR image features along the channel dimension, the 2D feature map is input into a gating mechanism to generate an element-by-element gating signal;

[0062] According to the element-by-element gating signal, a multimodal feature adaptive fusion is performed on the 2D feature map of the RGB image feature and the 2D feature map of the SWIR image feature to obtain a multimodal fusion feature, which is expressed as:

[0063] ;

[0064] in, Reshape the array. Represents the element-by-element gating signal, H and W represent the length and width of the 2D feature map respectively, represents a matrix whose elements are all 1, Represents element-wise multiplication; represents multimodal fusion features, A sequence of image patches representing RGB image features, A sequence of image patches representing SWIR image features;

[0065] After performing global average pooling on the multimodal fusion feature, the multimodal fusion feature is spliced and flattened with the CLS sequence of the RGB image feature and the CLS sequence of the SWIR image feature to obtain a flattened image feature;

[0066] The flattened image features are subjected to a channel reweighting operation using a three-branch parallel structure to obtain fused image features. The expression of the channel reweighting operation using the three-branch parallel structure is:

[0067] ;

[0068] in, represents the Softmax activation function, represents a 1D convolution operation, represents the fused image features, represents the flattened image features, represents a fully connected layer.

[0069] In one embodiment, performing cosine similarity analysis on the fused image features and text features corresponding to each surface anomaly type label to determine the surface anomaly type of the multispectral remote sensing image includes:

[0070] Mapping the fused image features and the text features corresponding to each surface anomaly type label to a shared semantic space of the same dimension, and calculating the cosine similarity between the fused image features and the text features corresponding to each surface anomaly type;

[0071] The cosine similarity between the fused image features and the text features corresponding to each surface anomaly type is processed through the Softmax activation function to obtain the probability that the fused image features belong to each surface anomaly type label, and the surface anomaly type label with the greatest probability is used as the surface anomaly type of the multispectral remote sensing image.

[0072] A remote sensing image surface anomaly detection device based on image-text collaborative processing, comprising:

[0073] A text feature acquisition module is used to obtain text features corresponding to each surface anomaly type label. The text features corresponding to the surface anomaly type label are pre-converted into corresponding natural sentences based on the surface anomaly type label and a customized text prompt template, and the text encoder is used to extract features from the natural sentences to obtain text features represented by vectors;

[0074] A band extraction module is used to extract and fuse the bands of the multispectral remote sensing image to be detected to obtain an RGB image and a SWIR image of the multispectral remote sensing image;

[0075] An image encoder module is used to extract features from the RGB image and the SWIR image to obtain initialized RGB features and initialized SWIR features;

[0076] A multi-scale spatial adapter module is used to spatially adjust the initialized RGB features and the initialized SWIR features to obtain multi-scale enhanced spatially adapted RGB features and multi-scale enhanced spatially adapted SWIR features;

[0077] a cross-band alignment module, configured to perform spectral adjustment on the multi-scale enhanced spatially adapted RGB features and the multi-scale enhanced spatially adapted SWIR features to obtain RGB image features and SWIR image features;

[0078] A dynamic spectral weighted fusion device, used for fusing the RGB image features and the SWIR image features to generate fused image features;

[0079] The similarity analysis module is used to perform cosine similarity analysis on the fused image features and the text features corresponding to the surface anomaly type labels to determine the surface anomaly type of the multispectral remote sensing image.

[0080] A computer device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the remote sensing image surface anomaly detection method based on collaborative image and text processing are implemented.

[0081] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the remote sensing image surface anomaly detection method based on collaborative image and text processing are implemented.

[0082] The above-mentioned remote sensing image surface anomaly detection method based on collaborative processing of images and text obtains the text features corresponding to each surface anomaly type label. The text features corresponding to the surface anomaly type label are converted into corresponding natural sentences in advance based on the surface anomaly type label and the customized text prompt template, and the natural sentences are subjected to feature extraction by the text encoder to obtain text features represented by vectors; the multispectral remote sensing image to be detected is subjected to band extraction and fusion to obtain the RGB image and SWIR image of the multispectral remote sensing image; the RGB image and the SWIR image are input into the image encoder module for feature extraction to obtain the initialized RGB features and the initialized SWIR features; the initialized RGB features are extracted The initialized SWIR features are input into the multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted RGB features and multi-scale enhanced spatially adapted SWIR features. These multi-scale enhanced spatially adapted RGB features and multi-scale enhanced spatially adapted SWIR features are then input into the cross-band alignment module for spectral adjustment to obtain RGB image features and SWIR image features. The RGB image features and SWIR image features are then input into the dynamic spectral weighted fusion module for feature fusion to generate fused image features. Cosine similarity analysis is performed on the fused image features and the text features corresponding to each surface anomaly type label to determine the surface anomaly type in the multispectral remote sensing image. Thus, by introducing the multi-scale spatial adapter and cross-band alignment module, fine-tuning the spatial and spectral adaptability of multispectral remote sensing images is achieved, significantly improving the accuracy and generalization of surface anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 1 is a flow chart of a method for detecting surface anomalies in remote sensing images based on collaborative processing of images and text in one embodiment;

[0084] Figure 2 A schematic diagram of a method for detecting surface anomalies in remote sensing images based on collaborative image and text processing in one embodiment;

[0085] Figure 3 A schematic diagram of the structure of a multi-scale spatial adapter in one embodiment;

[0086] Figure 4 2. It is a schematic structural diagram of a cross-band alignment unit in one embodiment;

[0087] Figure 5 Schematic diagram of the structure of a dynamic spectral weighted fusion device in one embodiment. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0089] It should be noted that the illustrations provided in the embodiments of the present application are only schematic illustrations of the basic concept of the present application. Therefore, the illustrations only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complex.

[0090] It should also be noted that in this application, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" and the like are used to indicate positions or locations based on those shown in the accompanying drawings. These terms are intended solely to facilitate the description of this application and simplify the description, and are not intended to indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting this application. Furthermore, the terms "first" and "second" are used solely for descriptive and distinguishing purposes and should not be construed as indicating or implying relative importance.

[0091] In addition, it should be noted that the features of the various embodiments of the present application can be combined or integrated in part or in whole, and as will be understood by those skilled in the art, can interact and operate in different ways. Each embodiment can be implemented independently of each other or in an associated relationship.

[0092] In one embodiment, Figure 1 and Figure 2 As shown, a method for detecting surface anomalies in remote sensing images based on collaborative processing of images and text is provided, which is characterized by comprising the following steps:

[0093] Step S1, obtain the text features corresponding to each surface anomaly type label. The text features corresponding to the surface anomaly type label are pre-converted into corresponding natural sentences based on the surface anomaly type label and a customized text prompt template, and the natural sentences are subjected to feature extraction through a text encoder to obtain text features represented by vectors.

[0094] Among them, the surface anomaly type label is determined based on the possible surface types, such as abnormal surface types such as forest fires and earthquakes, and normal surface types.

[0095] Among them, the text encoder is used for text feature encoding. The encoder of the pre-trained Transformer model is used to process the surface anomaly type labels in the open set of surface anomaly type labels. Each surface anomaly type label is combined with the corresponding text prompt template to form a natural sentence to extract a set of text features required for contrastive learning. The open set of surface anomaly type labels ensures the openness of the detection system to different anomaly types.

[0096] Among them, such as Figure 2 As shown, an open set of textual surface anomaly type labels can be created based on common surface types (such as forest fire, earthquake, and normal). Using textual hint templates, each surface anomaly type label can be converted into a natural sentence with richer semantic information. For example, a surface anomaly type label of "forest fire" can be converted into "This is an anomalous remote sensing image of a forest fire." Textual hint templates can also be designed for multispectral remote sensing images of surface anomalies to supplement relevant information, such as the irregular spatial characteristics and complex spectral characteristics of the surface anomaly coverage area. Using the encoder of a pre-trained Transformer model as the text encoder, each natural sentence is input into the text encoder and converted into a series of text features represented as vectors.

[0097] Among them, the surface anomaly type label open set Contains an indefinite number of surface types, which are converted into natural sentences through text prompt templates; the text encoder converts natural sentences, such as "This is an abnormal remote sensing image of a forest fire," into a series of text features represented by vectors , where R represents the real number space, n represents the number of sequences of text feature t (i.e., the number of categories in the open set of surface anomaly type labels), and D represents the feature dimension of text feature t (i.e., the data point in each feature sequence is represented by a vector composed of D numerical values).

[0098] This example uses the text encoder of the Contrastive Language-Image Pre-training (CLIP) model to convert each natural sentence into a series of text features represented by vectors. Specifically, the Transformer-based text encoder in the CLIP model and the encoder in the pre-trained Transformer model have the same structure but different parameters. The CLIP model's text encoder is pre-trained on the WebImage Text (WIT) dataset, a dataset developed by OpenAI that contains 400 million image and text pairs, covering a diverse range of natural images and natural language descriptions on the internet, specifically designed for multimodal contrastive learning.

[0099] Among them, the text encoder of the CLIP model keeps all parameters in a frozen state during use, avoiding the risk of overfitting in training with a small number of surface anomaly samples and performance degradation caused by fine-tuning, and utilizing its powerful semantic prior to improve the generalization ability of the model.

[0100] Step S2: performing band extraction and fusion on the multispectral remote sensing image to be detected to obtain an RGB image and a SWIR image of the multispectral remote sensing image.

[0101] In one example, the Sentinel-2 L2A multispectral data product, that is, the multispectral remote sensing image to be detected, is obtained. After preprocessing such as band extraction and fusion, the red, green, and blue bands of the multispectral remote sensing image are visualized as an RGB image, that is, a true color image, and the short-wavelength infrared 2 (SWIR2), short-wavelength infrared 1 (SWIR1), and near-infrared (NIR) bands of the multispectral remote sensing image are visualized as SWIR images, that is, pseudo-color images.

[0102] Among them, the central wavelength of the shortwave infrared 2 band of the multispectral remote sensing image is 2.190μm.

[0103] Among them, the central wavelength of the shortwave infrared 1 band of the multispectral remote sensing image is 1.610μm.

[0104] Step S3: Input the RGB image and the SWIR image into the image encoder module for feature extraction to obtain initialized RGB features and initialized SWIR features.

[0105] Among them, the image encoder module includes two image encoders with the same network structure, one image encoder is used to process RGB images, and the other image encoder is used to process SWIR images.

[0106] The image encoder is used to divide the RGB image and SWIR image into U parts along the height and V parts along the width to obtain image blocks and linearly embed the image blocks into a vector with a fixed number of channels C to form an image block sequence, where , the values of U and V are set in the image encoder; an additional learnable classification CLS (Classification, CLS) sequence is added for the classification task; the image block sequence and CLS sequence are composed of a length of N and a number of feature channels The full embedding of is forward processed, keeping the scale unchanged, .

[0107] In an example, an RGB image of size 512×512×3 is processed in the image encoder (the image encoder is set to be split into 16 parts along the height and 16 parts along the width): first divided into 16*16=256 image blocks, each image block is 32×32×3, and then each image block is encoded into a vector with a channel number C=768 (feature sequence, size is 1×768), and then an additional learnable classification CLS (Classification, CLS) sequence is added for the classification task. The CLS sequence and all image blocks are added together, and finally the encoder outputs the initialized RGB features with feature length N=256+1 and channel number C=768.

[0108] Among them, the image encoder is used for the preliminary extraction of image features of multispectral remote sensing images. The encoder of the pre-trained ViT model is used to process RGB images and SWIR images respectively to extract initialized image features in order to enhance feature generalization in the spatial and spectral fine-tuning stages.

[0109] This example uses the image encoder of the Contrastive Language-Image Pre-training (CLIP) model. Specifically, the ViT-based image encoder in the CLIP model shares the same structure as the encoder in the pre-trained ViT model, differing only in parameters. The CLIP model's image encoder is pre-trained on the WebImage Text (WIT) dataset, a dataset developed by OpenAI that contains 400 million image and text pairs, covering a diverse range of natural images and natural language descriptions found on the internet. It is specifically designed for multimodal contrastive learning.

[0110] Among them, the encoder of the pre-trained Vision Transformer (ViT) model is used as the image encoder.

[0111] Among them, the image encoder of the CLIP model keeps all parameters in a frozen state during use, avoiding the risk of overfitting in training with a small number of surface anomaly samples and performance degradation caused by fine-tuning, and utilizing its powerful semantic prior to improve the generalization ability of the model.

[0112] The image encoder of the CLIP model divides the image (i.e., RGB image or SWIR image) into image blocks, and linearly embed these image blocks into a vector with a fixed number of channels C, and then add an additional learnable classification (CLS) sequence for the classification task. The image block and CLS sequence form a network with a length of , a series of forward processing is performed on the complete embedding with the number of feature channels C, keeping the scale unchanged, and obtaining the initialized image features (i.e., initialized RGB features or initialized SWIR features).

[0113] It should be understood that multispectral remote sensing images differ significantly from natural images, including differences in spectral distribution and spatial scale. The CLIP model is pre-trained only on datasets containing natural images. Therefore, a targeted cross-domain generalization mechanism is needed for the initialized image features to bridge the distribution gap. This is done in two stages: spatial fine-tuning and spectral fine-tuning.

[0114] Among them, the two image encoders receive and process RGB images and SWIR images respectively, extract the initialized RGB features and initialized SWIR features, and then enter the spatial fine-tuning stage and the spectral fine-tuning stage in sequence.

[0115] In step 4, the initialized RGB features and the initialized SWIR features are input into the multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted RGB features and multi-scale enhanced spatially adapted SWIR features.

[0116] Among them, the multi-scale spatial adapter module is used to perform spatial fine-tuning on the initialized RGB features and initialized SWIR features.

[0117] The multi-scale spatial adapter module includes two multi-scale spatial adapters with the same network structure. The initialized RGB features are processed by a multi-scale spatial adapter (MSSA) to improve the ability to extract local details such as the morphology and texture of the surface anomaly coverage area in the RGB image, thereby obtaining multi-scale enhanced spatially adapted RGB features. At the same time, the initialized SWIR image features are processed by another multi-scale spatial adapter to improve the ability to extract local details such as the morphology and texture of the surface anomaly coverage area in the SWIR image, thereby obtaining multi-scale enhanced spatially adapted SWIR features.

[0118] Among them, the spatial fine-tuning stage of the cross-domain generalization mechanism is reflected in step 4 of this embodiment, and a multi-scale spatial adapter is established after the image encoder in the RGB path (i.e., the path composed of an image encoder for processing RGB images, a multi-scale spatial adapter, a linear layer, and a cross-band alignment unit) and the SWIR path (i.e., the path composed of an image encoder for processing SWIR images, a multi-scale spatial adapter, a linear layer, and a cross-band alignment unit) to improve the model's local detail modeling and global semantic understanding capabilities.

[0119] Among them, the initialized image features (including initialized RGB features and initialized SWIR features) are taken as input and separated into CLS sequence and image block sequence to process global and local information respectively; after the image block sequence is reconstructed into a 2D feature map, multi-scale features are extracted and fused through the residual structure and dilated convolution with different expansion rates; then, the fused features guide the update of the CLS sequence through global average pooling, and finally output multi-scale enhanced spatial adaptation features (including multi-scale enhanced spatial adaptation RGB features and multi-scale enhanced spatial adaptation SWIR features).

[0120] In one embodiment, the multi-scale spatial adapter module includes two multi-scale spatial adapters with the same network structure;

[0121] Inputting the initialized RGB features and the initialized SWIR features into a multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted RGB features and multi-scale enhanced spatially adapted SWIR features, including: inputting the initialized RGB features into one of the multi-scale spatial adapters of the multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted RGB features; inputting the initialized SWIR features into another multi-scale spatial adapter of the multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted SWIR features.

[0122] In one embodiment, the processing of the multi-scale spatial adapter is as follows:

[0123] The initialization features of the input multi-scale spatial adapter are separated into CLS sequences and image block sequences; the image block sequence is reconstructed into a 2D feature map that retains spatial information: the 2D feature map is input into the convolution enhancement unit for multi-scale convolution processing to obtain multi-scale fusion features. The convolution enhancement unit is a four-branch multi-convolution structure; the multi-scale fusion features are input into the global feature extraction unit for feature extraction to obtain global features; the global features are added to the CLS sequence to obtain an enhanced CLS sequence; the multi-scale fusion features are restored to a sequence format and recombined with the enhanced CLS sequence to obtain multi-scale enhanced spatial adaptation features.

[0124] It should be understood that if the multi-scale spatial adapter is used to process the initialization RGB features, accordingly, the initialization features of the input multi-scale spatial adapter are the initialization RGB features, and the corresponding output multi-scale enhanced spatial adaptation features are the multi-scale enhanced spatial adaptation RGB features; if the multi-scale spatial adapter is used to process the initialization SWIR features, accordingly, the initialization features of the input multi-scale spatial adapter are the initialization SWIR features, and the corresponding output multi-scale enhanced spatial adaptation features are the multi-scale enhanced spatial adaptation SWIR features.

[0125] Among them, the multi-scale spatial adapter MSSA is used for spatial fine-tuning of the cross-domain generalization mechanism of features, enhances the ability to extract local details through multi-branch convolution, and integrates multi-scale receptive field differences to improve the system's expression ability and detection accuracy.

[0126] In one embodiment, taking the multi-scale spatial adapter MSSA in the RGB path as an example, Figure 3 As shown, the initialized image features are initialized RGB features, and the multi-scale enhanced spatial adaptation features are multi-scale enhanced spatial adaptation RGB features. The specific operations are as follows:

[0127] Initialize the RGB features Feature separation into CLS sequences and image block sequence , respectively enhance the modeling capabilities of global background and local details;

[0128] In order to enhance the ability to capture spatial details at different scales, a convolutional enhancement unit is used to achieve multi-scale enhancement: specifically, the image block sequence Reconstructed into a 2D feature map that preserves spatial information , the process can be expressed as: , Indicates reshaping the array shape to facilitate subsequent convolution processing. H and W represent the length and width of the 2D feature map respectively. The value of H is equal to the value of U, and the value of W is equal to the value of V.

[0129] For 2D feature maps A convolution enhancement unit is used, which is a four-branch multi-convolution structure: first, a 1×1 convolution operation is performed in each branch to adjust the number of feature channels to 1 / 4; branch 1 does not perform the remaining convolution processing and acts as a residual structure, thereby generating The equivalent feature map of is used to retain the original local details; the other three branches perform dilated convolution operations respectively. The convolution kernel size of the three branches is 3×3, and the expansion rates of the three branches are 1, 2, and 3 respectively, generating multi-scale features with different local receptive fields, which can be expressed as:

[0130] ,

[0131] ,

[0132] ,

[0133] ;

[0134] in, 、 、 、 Represent the convolution results of the first branch, the second branch, the third branch, and the fourth branch, respectively. , , , ; 、 、 、 The receptive field sizes are 1, 3, 5, and 8 respectively; Represents standard convolution, i.e. 1×1 convolution, which is used to adjust the channel dimension; 、 、 Represent the expansion rate in the second branch The dilated convolution with a dilation rate of 1 and the expansion rate in the third branch The dilated convolution with a dilation rate of 2 and the expansion rate in the fourth branch The dilated convolution with a size of 3 directly fuses the multi-scale features of the four branches to obtain:

[0135] ;

[0136] in, represents multi-scale fusion features, Indicates splicing along the channel dimension;

[0137] In order to enhance the model's ability to model global context information, multi-scale fusion features are processed , to update the CLS sequence representing global semantics : A global feature extraction unit is introduced, and the global features are extracted through the global average pooling GAP of the global feature extraction unit. The global features are added to the CLS sequence to obtain the enhanced CLS sequence, which is expressed as:

[0138] ;

[0139] Among them, GAP represents global average pooling, represents the fully connected layer, which is used to integrate global context information. express activation function, Represents the enhanced CLS sequence, encoding both local anomaly details and global scene semantics;

[0140] Finally, the multi-scale fusion features Restore to sequence format, and the enhanced CLS sequence Recombined to finally form multi-scale enhanced spatial adaptation RGB features , expressed as:

[0141] .

[0142] Similarly, the multi-scale spatial adapter MSSA in the SWIR path has the same structure and steps, and finally forms a multi-scale enhanced spatial adaptation SWIR feature , I will not go into details here.

[0143] The Multi-Scale Spatial Adapter (MSSA) effectively integrates multi-scale spatial perception with global semantic modeling capabilities, improving the model's perception of fine-grained structures while also enhancing its cross-scale contextual understanding. In the task of surface anomaly detection, feature representations that are both scale-invariant and semantically consistent are crucial for complex scene recognition and significantly enhance the model's generalization and adaptability in processing heterogeneous multispectral imagery.

[0144] In step 5, the multi-scale enhanced spatially adapted RGB features and the multi-scale enhanced spatially adapted SWIR features are input into the cross-band alignment module for spectral adjustment to obtain RGB image features and SWIR image features.

[0145] Among them, the cross-band alignment module is used to fine-tune the spectrum of the cross-domain generalization mechanism of features, achieve semantic alignment between visible light, near-infrared, and short-wave infrared bands, suppress redundant features with high spectral correlation, and improve the accuracy and interpretability of detection results.

[0146] Among them, the cross-band alignment module includes two linear layers with the same network structure and two cross-band alignment units with the same network structure; the multi-scale enhanced spatially adapted RGB features are processed by a linear layer and a cross-band alignment unit to supplement the response characteristics of surface anomalies in the near-infrared, short-wave infrared 1, and short-wave infrared 2 bands, and obtain RGB image features that have been spatially and spectrally fine-tuned; at the same time, the multi-scale enhanced spatially adapted SWIR features are processed by another linear layer and another cross-band alignment unit to supplement the response characteristics of surface anomalies in the red, green, and blue bands, and obtain SWIR image features that have been spatially and spectrally fine-tuned.

[0147] In one embodiment, the cross-band alignment module includes two linear layers with the same network structure and two cross-band alignment units with the same network structure;

[0148] The multi-scale enhanced spatially adapted RGB features and the multi-scale enhanced spatially adapted SWIR features are input into the cross-band alignment module for spectral adjustment to obtain RGB image features and SWIR image features, including:

[0149] The multi-scale enhanced spatially adapted RGB features are input into one of the linear layers of the cross-band alignment module for processing to obtain RGB query features, RGB key features and RGB numerical features; the multi-scale enhanced spatially adapted SWIR features are input into another linear layer of the cross-band alignment module for processing to obtain SWIR query features, SWIR key features and SWIR numerical features; the SWIR query features, RGB key features and RGB numerical features are input into one of the cross-band alignment units of the cross-band alignment module for spectral adjustment to obtain RGB image features; the RGB query features, SWIR key features and SWIR numerical features are input into another cross-band alignment unit of the cross-band alignment module for spectral adjustment to obtain SWIR image features.

[0150] The spectral fine-tuning stage of the cross-domain generalization mechanism is reflected in step 5 of this embodiment. A cross-band alignment module (CBAM) is established after the multi-scale spatial adaptation module to achieve feature complementarity through difference modeling and residual fusion, thereby improving the ability to perceive abnormal changes and the consistency of cross-band expression. Figure 4 As shown in the figure, the multi-scale enhanced spatial adaptation features (multi-scale enhanced spatial adaptation RGB features or multi-scale enhanced spatial adaptation SWIR features) are linearly projected through the linear layer to obtain the intermediate variables of the multi-spectral attention mechanism (i.e. query features, key features and numerical features), and then the similarity measurement is realized based on the spectral information divergence (SID) through the cross-band alignment unit, and a series of forward processing is implemented to finally obtain the image features (i.e. RGB image features or SWIR image features) that have been fine-tuned in space and spectrum. The linear layer in the RGB path projects the multi-scale enhanced spatial adaptation RGB features. After projection through a learnable linear layer, we generate RGB query features , RGB key features , RGB numerical features , the calculation formula is:

[0151] ;

[0152] in, represents a linear layer;

[0153] The linear layer in the SWIR path adapts the multi-scale enhanced space to the SWIR features After projection through a learnable linear layer, we generate SWIR query characteristics SWIR key features , SWIR numerical characteristics , the calculation formula is:

[0154] ;

[0155] Subsequently, the query features, key features, and numerical features are used as inputs of the cross-band alignment unit to start the multispectral attention mechanism MSAttention.

[0156] In one embodiment, a series of forward processing is performed: taking the RGB path as an example, the SWIR query feature, RGB key feature, and RGB numerical feature are input into one of the cross-band alignment units of the cross-band alignment module for spectral adjustment to obtain RGB image features, including:

[0157] The SWIR query features, RGB key features, and RGB numerical features are similarly measured and weighted summed to obtain the multispectral attention weighted RGB features that focus on the similar features of the RGB image and the SWIR image. The expression is:

[0158] ;

[0159] in, represents the multi-spectral attention weighted RGB features, represents the Softmax activation function, is the attention weight coefficient of one of the cross-band alignment units, calculated by and Similarity is obtained; Represents RGB numerical features, R represents real number space, N represents the length of the feature, and C represents the number of channels of the feature;

[0160] Among them, the attention weight coefficient Generally, the matrix dot product is used to obtain: , the superscript T represents transposition; however, this similarity measurement method is difficult to effectively characterize the complex relationship between the bands and regions of the multispectral remote sensing image of surface anomalies. Therefore, in the multispectral attention mechanism MSAttention, Spectral Information Divergence (SID) is used for SWIR query features. and RGB key features The similarity measure of is used to obtain the attention weight coefficient of one of the cross-band alignment units (i.e., the cross-band alignment unit on the RGB path). , the expression is:

[0161] ;

[0162] in, Refers to the similarity measurement method, is the natural exponential function, Represents the spectral information divergence matrix between the SWIR query feature and the RGB key feature. Spectral information divergence SID is a spectral classification method based on information theory to measure the difference between two spectra. It takes the spectral vector as a random variable and analyzes the similarity of two random vectors based on probability statistics theory. It is more consistent with the actual physical process and is not sensitive to noise and illumination changes. It is more suitable for complex nonlinear spectral analysis of multispectral remote sensing images. The first The first Spectral information divergence of the column The expression is:

[0163] ;

[0164] in, represents the probability normalized indivual sequence, , is the logarithmic function with base 10, represents the probability normalized indivual sequence, , superscript represents the transpose; the first part of the spectral information divergence matrix between the SWIR query feature and the RGB key feature The first The smaller the spectral information divergence value of the column, the more similar the two are.

[0165] In order to more effectively characterize the surface anomaly characteristics, the RGB numerical features and multi-spectral attention weighted RGB features After matrix subtraction and linear mapping, the SWIR query feature is used as a residual structure Perform residual connection to obtain RGB difference features , the process is expressed mathematically as:

[0166] ;

[0167] In order to further enhance the complementarity of cross-band features, the RGB difference features are processed by layer normalization and multi-layer perceptron, and then connected with the RGB difference feature residual to obtain the RGB image features, which can be expressed mathematically as follows: ,in, Representation layer normalization is used to eliminate feature distribution offset and accelerate convergence; Represents a multi-layer perceptron that refines features through local nonlinear transformations, supplementing fine-grained information that may be ignored by the attention mechanism; Represents the RGB image feature, which is the RGB feature after spatial and spectral adaptation.

[0168] In one embodiment, the RGB query features, the SWIR key features, and the SWIR numerical features are input into another cross-band alignment unit of the cross-band alignment module for spectral adjustment to obtain SWIR image features, including:

[0169] The RGB query feature, SWIR key feature, and SWIR numerical feature are similarly measured and weighted summed to obtain the multispectral attention weighted SWIR feature that focuses on the similar features of the SWIR image and the RGB image. The expression is:

[0170] ;

[0171] in, represents the multispectral attention-weighted SWIR features, is the attention weight coefficient of another cross-band alignment unit, Indicates SWIR numerical characteristics;

[0172] Among them, spectral information divergence is used for RGB query features and SWIR key characteristics Similarity measure, obtain the attention weight coefficient of another cross-band alignment unit , the expression is:

[0173] ;

[0174] in, represents the spectral information divergence matrix between RGB query features and SWIR key features. The first Spectral information divergence of the column The calculation formula is:

[0175] ;

[0176] in, represents the probability normalized indivual sequence, , represents the probability normalized indivual sequence, ;

[0177] SWIR numerical features and multispectral attention weighted SWIR features After matrix subtraction and linear mapping, the RGB query feature is used as a residual structure Residual connection is performed to obtain SWIR difference features; after layer normalization and multi-layer perceptron processing, the SWIR difference features are residually connected with the SWIR difference features to obtain SWIR image features.

[0178] Similarly, the cross-band alignment unit in the SWIR path has the same structure and steps, with Query characteristics , from Key features and numerical features As the input of the cross-band alignment unit in the SWIR path, the multispectral attention mechanism MSAttention is started, and a series of forward processing is implemented to finally form the SWIR image features. , that is, the SWIR features that have been fine-tuned spatially and spectrally, realize the two-way information interaction between RGB images and SWIR images.

[0179] The multispectral attention mechanism, combined with the spectral information divergence (SID) metric, effectively enhances the collaborative modeling capabilities between the multispectral bands of RGB and SWIR images. Its advantages lie in its better alignment with the physical distribution characteristics of multispectral data, robustness, and precise representation of details. Through bidirectional feature fusion and difference enhancement, the model demonstrates greater sensitivity and expressiveness to ground feature changes and anomalies, providing a more adaptable feature modeling solution for semantic understanding of remote sensing scenes.

[0180] In step 6, the RGB image features and the SWIR image features are input into the dynamic spectral weighted fusion device for feature fusion to generate fused image features.

[0181] Among them, the dynamic spectral weighted fusion (DSWF) is used for feature fusion after the cross-domain generalization mechanism. It realizes adaptive fusion of multi-spectral features based on the dynamic weights generated by the gating mechanism, and uses the fused features as the image features required for contrastive learning, thereby improving the overall performance of the system.

[0182] Among them, the RGB image features and SWIR image features are simultaneously input into the Dynamic Spectrally Weighted Fuser (DSWF) to highlight the key abnormal responses of the multispectral remote sensing image and obtain the adaptive fused multispectral features after spatial and spectral fine-tuning, that is, the fused image features.

[0183] The feature fusion after the spatial and spectral fine-tuning phase of the cross-domain generalization mechanism is reflected in step 6 of this embodiment. In order to further improve the fusion expression ability of RGB image features and SWIR image features in remote sensing anomaly detection, a dynamic spectral weighted fusion DSWF is established after the cross-band alignment module CBAM. Figure 5 As shown in the figure, the spatially and spectrally fine-tuned RGB image features and the spatially and spectrally fine-tuned SWIR image features are used as input, and dynamic weights are generated through a gating mechanism to adjust the contributions of different modalities element by element, and the key abnormal response features are highlighted based on global and local channel interactions, thereby achieving more accurate cross-band feature fusion, and finally outputting the spatially and spectrally fine-tuned adaptive fused multispectral features (i.e., the fused image features).

[0184] In one embodiment, the RGB image features and the SWIR image features are input into a dynamic spectral weighted fusion device for feature fusion to generate fused image features, including:

[0185] The RGB image features are input into the dynamic spectral weighted fusion device and subjected to layer normalization operation for feature separation to obtain the CLS sequence of RGB image features and the image block sequence of RGB image features; the SWIR image features are input into the dynamic spectral weighted fusion device and subjected to layer normalization operation for feature separation to obtain the CLS sequence of SWIR image features and the image block sequence of SWIR image features; the image block sequence of RGB image features and the image block sequence of SWIR image features are reconstructed into 2D feature maps of RGB image features and 2D feature maps of SWIR image features; the 2D feature maps of RGB image features and the 2D feature maps of SWIR image features are spliced along the channel dimension and input into the gating mechanism to generate an element-by-element gating signal; the 2D feature maps of RGB image features and the 2D feature maps of SWIR image features are adaptively fused according to the element-by-element gating signal to obtain the multimodal fusion feature, which is expressed as:

[0186] ;

[0187] in, Reshape the array. Represents the element-by-element gating signal, H and W represent the length and width of the 2D feature map respectively, represents a matrix whose elements are all 1, Represents element-wise multiplication; represents multimodal fusion features, A sequence of image patches representing RGB image features, A sequence of image patches representing SWIR image features;

[0188] After performing global average pooling on the multimodal fusion features, they are concatenated and flattened with the CLS sequence of RGB image features and the CLS sequence of SWIR image features to obtain the flattened image features. The channel reweighting operation of the three-branch parallel structure is performed on the flattened image features to obtain the fused image features. The expression of the channel reweighting operation using the three-branch parallel structure is:

[0189] ;

[0190] in, represents the Softmax activation function, represents a 1D convolution operation, represents the fused image features, represents the flattened image features, represents a fully connected layer.

[0191] Specifically: RGB image features after spatial and spectral fine-tuning Separated into CLS sequences after layer normalization operation and image block sequence At the same time, the SWIR image features are fine-tuned spatially and spectrally Separated into CLS sequences after layer normalization operation and image block sequence ;

[0192] The image block sequence Reconstructed into a 2D feature map, and then spliced along the channel dimension, to generate element-by-element dynamic weights (i.e., element-by-element gating signals) based on the gating mechanism to achieve adaptive fusion of multimodal features, expressed as:

[0193] ,

[0194] ;

[0195] in, The number of input channels is 2C, the number of output channels is C, is the Sigmoid activation function, represents the element-wise gating signal, Represents a matrix whose elements are all 1, by element-by-element multiplication Realize adaptive fusion of RGB and SWIR multimodal features; Represents multimodal fusion features;

[0196] In order to integrate the global anomaly information of multispectral remote sensing images for surface anomaly detection, the multimodal fusion features Perform global average pooling , and is concatenated and flattened with the CLS sequence that aggregates all image block information, expressed as:

[0197] ;

[0198] in, represents the flattening operation, Represents the flattened image features, integrating multimodal information of spatial and spectral multi-scale interactions;

[0199] In order to further enhance the accuracy of anomaly detection, A three-branch parallel structure with channel reweighting is used to achieve global and local cross-channel information interaction: one branch is not weighted and retains important information as a residual structure; the other two branches, one using a fully connected layer to model global channel correlations, and the other using a 1D convolution operation to model local channel correlations, reweight important channels that are sensitive to surface anomalies from different perspectives, as expressed as:

[0200] ;

[0201] in, represents a 1D convolution operation, It represents the adaptive fused multispectral features (i.e., fused image features) that have been fine-tuned spatially and spectrally, that is, the image features that are finally passed to the contrastive learning stage, which can highlight the key abnormal responses.

[0202] The Dynamic Spectral Weighted Fusion (DSWF) effectively integrates global and local cross-channel response characteristics through channel-dimensional splicing and a parallel information interaction structure, enhancing the sensitivity of surface anomaly detection. The resulting fused features have strong generalization capabilities for anomaly representation and can be accurately aligned with text features across modalities, providing reliable support and semantic discrimination for anomaly detection tasks.

[0203] Step 7: Perform cosine similarity analysis on the fused image features and the text features corresponding to each surface anomaly type label to determine the surface anomaly type of the multispectral remote sensing image.

[0204] In one embodiment, a cosine similarity analysis is performed on the fused image features and the text features corresponding to each surface anomaly type label to determine the surface anomaly type of the multispectral remote sensing image, including:

[0205] The fused image features and the text features corresponding to each surface anomaly type label are mapped to a shared semantic space of the same dimension, and the cosine similarity between the fused image features and the text features corresponding to each surface anomaly type is calculated. The cosine similarity between the fused image features and the text features corresponding to each surface anomaly type is processed through the Softmax activation function to obtain the probability that the fused image features belong to each surface anomaly type label. The surface anomaly type label with the highest probability is used as the surface anomaly type of the multispectral remote sensing image.

[0206] The determination of the type of surface anomaly in the multispectral remote sensing image based on the acquired fused image features is embodied in step 7 of this embodiment. Figure 2 As shown in the figure, contrastive learning is used to establish cross-modal associations and generate surface anomaly type labels corresponding to images. The specific steps include:

[0207] The fused image features With text features As input, both are mapped to a shared semantic space of the same dimension, represented as:

[0208] ,

[0209] ;

[0210] in, is the image feature after mapping, is the text feature after mapping, Indicates the dimension size of the shared semantic space, which is generally set to 512.

[0211] Calculate the mapped image features and mapped text features The cosine similarity matrix is obtained by the Softmax activation function to obtain the normalized probability:

[0212] ;

[0213] in, is to calculate the cosine similarity, Indicates the probability that the fused image features belong to each surface anomaly type label, which is a 1-line For example, the label with the highest probability is “forest fire”, and the label with the highest probability “forest fire” is output as the surface anomaly type of the multispectral remote sensing image.

[0214] In summary, this application is different from the traditional closed detection paradigm that is difficult to generalize. This application proposes an architecture based on multimodal contrastive learning of visual language combined with efficient fine-tuning of lightweight adapters, which effectively bridges the domain differences between natural images and multispectral remote sensing images without significantly increasing the computational overhead. By introducing the multi-scale spatial adapter MSSA and the cross-band alignment module, the spatial and spectral adaptive fine-tuning of multispectral remote sensing images is achieved, which significantly improves the accuracy and generalization of surface anomaly detection. In addition, the dynamic spectral weighted fusion device DSWF is used to achieve adaptive fusion of multispectral features in order to establish cross-modal associations and achieve accurate identification and precise classification of surface anomalies. This method not only provides a solution to open detection, but also significantly improves the accuracy and generalization of surface anomaly detection in remote sensing images under complex scenes.

[0215] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0216] In one embodiment, a device for detecting surface anomalies in remote sensing images based on collaborative image and text processing is provided, which is characterized by comprising:

[0217] The text feature acquisition module is used to obtain the text features corresponding to each surface anomaly type label. The text features corresponding to the surface anomaly type label are pre-converted into corresponding natural sentences based on the surface anomaly type label and a customized text prompt template. The text encoder is used to extract features from the natural sentences to obtain text features represented by vectors.

[0218] The band extraction module is used to extract and fuse the bands of the multispectral remote sensing image to be detected to obtain the RGB image and SWIR image of the multispectral remote sensing image;

[0219] The image encoder module is used to extract features from RGB images and SWIR images to obtain initialized RGB features and initialized SWIR features;

[0220] The multi-scale spatial adapter module is used to spatially adjust the initialized RGB features and the initialized SWIR features to obtain multi-scale enhanced spatially adapted RGB features and multi-scale enhanced spatially adapted SWIR features;

[0221] The cross-band alignment module is used to spectrally adjust the multi-scale enhanced spatially adapted RGB features and the multi-scale enhanced spatially adapted SWIR features to obtain RGB image features and SWIR image features;

[0222] Dynamic spectral weighted fusion, used to fuse RGB image features and SWIR image features to generate fused image features;

[0223] The similarity analysis module is used to perform cosine similarity analysis on the fused image features and the text features corresponding to the surface anomaly type labels to determine the surface anomaly type of the multispectral remote sensing image.

[0224] Regarding the specific definition of the remote sensing image surface anomaly detection device based on image-text collaborative processing, please refer to the definition of the remote sensing image surface anomaly detection method based on image-text collaborative processing above, which will not be repeated here. The various modules in the above-mentioned remote sensing image surface anomaly detection device based on image-text collaborative processing can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0225] A computer device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the above-mentioned remote sensing image surface anomaly detection method based on collaborative image and text processing are implemented.

[0226] A computer-readable storage medium stores a computer program thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned remote sensing image surface anomaly detection method based on collaborative image and text processing are implemented.

[0227] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0228] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0229] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements are all within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for detecting surface anomalies in remote sensing images based on collaborative processing of images and text, characterized in that: The following steps are involved: Step 1: Obtain text features corresponding to each surface anomaly type label. The text features corresponding to the surface anomaly type label are pre-converted into corresponding natural sentences based on the surface anomaly type label and a customized text prompt template, and feature extraction is performed on the natural sentences using a text encoder to obtain text features represented by vectors. Step 2: performing band extraction and fusion on the multispectral remote sensing image to be detected to obtain an RGB image and a SWIR image of the multispectral remote sensing image; Step 3: Input the RGB image and the SWIR image into an image encoder module for feature extraction to obtain initialized RGB features and initialized SWIR features; Step 4: Input the initialized RGB features and the initialized SWIR features into a multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted RGB features and multi-scale enhanced spatially adapted SWIR features; Step 5: Input the multi-scale enhanced spatially adapted RGB features and the multi-scale enhanced spatially adapted SWIR features into a cross-band alignment module for spectral adjustment to obtain RGB image features and SWIR image features; Step 6: Input the RGB image features and the SWIR image features into a dynamic spectral weighted fusion device for feature fusion to generate fused image features; Step 7, performing cosine similarity analysis on the fused image features and the text features corresponding to the surface anomaly type labels to determine the surface anomaly type of the multispectral remote sensing image; The cross-band alignment module includes two linear layers with the same network structure and two cross-band alignment units with the same network structure; The step of inputting the multi-scale enhanced spatially adapted RGB features and the multi-scale enhanced spatially adapted SWIR features into a cross-band alignment module for spectral adjustment to obtain RGB image features and SWIR image features includes: Inputting the multi-scale enhanced spatially adapted RGB features into one of the linear layers of the cross-band alignment module for processing to obtain RGB query features, RGB key features, and RGB numerical features; Inputting the multi-scale enhanced spatially adapted SWIR features into another linear layer of the cross-band alignment module for processing to obtain SWIR query features, SWIR key features, and SWIR numerical features; Inputting the SWIR query feature, the RGB key feature, and the RGB numerical feature into one of the cross-band alignment units of the cross-band alignment module for spectral adjustment to obtain RGB image features; The RGB query feature, the SWIR key feature, and the SWIR numerical feature are input into another cross-band alignment unit of the cross-band alignment module for spectral adjustment to obtain a SWIR image feature.

2. The method for detecting surface anomalies in remote sensing images based on collaborative image and text processing according to claim 1, characterized in that: The multi-scale space adapter module includes two multi-scale space adapters with the same network structure; Inputting the initialized RGB features and the initialized SWIR features into a multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted RGB features and multi-scale enhanced spatially adapted SWIR features, including: Inputting the initialized RGB features into one of the multi-scale spatial adapters of the multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted RGB features; The initialized SWIR features are input into another multi-scale spatial adapter of the multi-scale spatial adapter module for spatial adjustment to obtain multi-scale enhanced spatially adapted SWIR features.

3. The method for detecting surface anomalies in remote sensing images based on collaborative image and text processing according to claim 2, characterized in that: The processing process of the multi-scale space adapter is as follows: Separating the initialization features input into the multi-scale spatial adapter into a CLS sequence and an image block sequence; The image block sequence is reconstructed into a 2D feature map that retains spatial information: The 2D feature map is input into the convolution enhancement unit for multi-scale convolution processing to obtain multi-scale fusion features. The convolution enhancement unit is a four-branch multi-convolution structure; The multi-scale fusion features are input into the global feature extraction unit for feature extraction to obtain global features; Adding the global feature to the CLS sequence to obtain an enhanced CLS sequence; The multi-scale fusion features are restored to a sequence format and recombined with the enhanced CLS sequence to obtain multi-scale enhanced spatial adaptation features.

4. The method for detecting surface anomalies in remote sensing images based on collaborative image and text processing according to claim 1, characterized in that: The step of inputting the SWIR query feature, the RGB key feature, and the RGB numerical feature into one of the cross-band alignment units of the cross-band alignment module for spectral adjustment to obtain the RGB image feature includes: The SWIR query feature, the RGB key feature, and the RGB numerical feature are subjected to similarity measurement and weighted summation to obtain a multispectral attention weighted RGB feature that focuses on similar features of the RGB image and the SWIR image. The expression is: ； in, represents the multi-spectral attention weighted RGB features, represents the Softmax activation function, is the attention weight coefficient of one of the cross-band alignment units, Represents RGB numerical features, R represents real number space, N represents the length of the feature, and C represents the number of channels of the feature; Among them, spectral information divergence is used for SWIR query features and RGB key features The similarity measure of , obtains the attention weight coefficient of one of the cross-band alignment units , the expression is: ； in, Refers to the similarity measurement method, is the natural exponential function, represents the spectral information divergence matrix between the SWIR query feature and the RGB key feature. The first Spectral information divergence of the column The expression is: ； in, represents the probability normalized indivual sequence, , is the logarithmic function with base 10, represents the probability normalized indivual sequence, , superscript represents transpose; The RGB numerical features With the multispectral attention weighted RGB features After matrix subtraction and linear mapping, the SWIR query feature is obtained as a residual structure Perform residual connection to obtain RGB difference features; The RGB difference features are processed by layer normalization and multi-layer perceptron, and then connected with the RGB difference feature residual to obtain RGB image features; The step of inputting the RGB query feature, the SWIR key feature, and the SWIR numerical feature into another cross-band alignment unit of the cross-band alignment module for spectral adjustment to obtain the SWIR image feature comprises: The RGB query feature, the SWIR key feature, and the SWIR numerical feature are subjected to similarity measurement and weighted summation to obtain a multispectral attention weighted SWIR feature that focuses on similar features of the SWIR image and the RGB image. The expression is: ； in, represents the multispectral attention-weighted SWIR features, is the attention weight coefficient of another cross-band alignment unit, Indicates SWIR numerical characteristics; Among them, spectral information divergence is used for RGB query features and SWIR key characteristics Similarity measure, obtain the attention weight coefficient of another cross-band alignment unit , the expression is: ； in, represents the spectral information divergence matrix between RGB query features and SWIR key features. The first Spectral information divergence of the column The calculation formula is: ； in, represents the probability normalized indivual sequence, , represents the probability normalized indivual sequence, ; The SWIR numerical characteristics With the multispectral attention weighted SWIR features After matrix subtraction and linear mapping, the RGB query feature is used as a residual structure Perform residual connection to obtain SWIR difference features; The SWIR difference features are processed by layer normalization and a multi-layer perceptron, and then connected with the SWIR difference feature residual to obtain the SWIR image features.

5. The method for detecting surface anomalies in remote sensing images based on collaborative image and text processing according to claim 4, characterized in that: The step of inputting the RGB image features and the SWIR image features into a dynamic spectral weighted fusion device for feature fusion to generate fused image features includes: The RGB image features are input into a dynamic spectral weighted fusion device, and then subjected to a layer normalization operation and feature separation to obtain a CLS sequence of the RGB image features and an image block sequence of the RGB image features; The SWIR image features are input into a dynamic spectral weighted fusion device and subjected to a layer normalization operation to perform feature separation into a CLS sequence of the SWIR image features and an image block sequence of the SWIR image features; Reconstructing the image block sequence of the RGB image feature and the image block sequence of the SWIR image feature into a 2D feature map of the RGB image feature and a 2D feature map of the SWIR image feature; After splicing the 2D feature map of the RGB image features and the 2D feature map of the SWIR image features along the channel dimension, the 2D feature map is input into a gating mechanism to generate an element-by-element gating signal; According to the element-by-element gating signal, a multimodal feature adaptive fusion is performed on the 2D feature map of the RGB image feature and the 2D feature map of the SWIR image feature to obtain a multimodal fusion feature, which is expressed as: ； in, Reshape the array. Represents the element-by-element gating signal, H and W represent the length and width of the 2D feature map respectively, represents a matrix whose elements are all 1, Represents element-wise multiplication; represents multimodal fusion features, A sequence of image patches representing RGB image features, A sequence of image patches representing SWIR image features; After performing global average pooling on the multimodal fusion feature, the multimodal fusion feature is spliced and flattened with the CLS sequence of the RGB image feature and the CLS sequence of the SWIR image feature to obtain a flattened image feature; The flattened image features are subjected to a channel reweighting operation using a three-branch parallel structure to obtain fused image features. The expression of the channel reweighting operation using the three-branch parallel structure is: ； in, represents the Softmax activation function, represents a 1D convolution operation, represents the fused image features, represents the flattened image features, represents a fully connected layer.

6. The method for detecting surface anomalies in remote sensing images based on collaborative image and text processing according to claim 1, characterized in that: The performing cosine similarity analysis on the fused image features and the text features corresponding to the surface anomaly type labels to determine the surface anomaly type of the multispectral remote sensing image includes: Mapping the fused image features and the text features corresponding to each surface anomaly type label to a shared semantic space of the same dimension, and calculating the cosine similarity between the fused image features and the text features corresponding to each surface anomaly type; The cosine similarity between the fused image features and the text features corresponding to each surface anomaly type is processed through the Softmax activation function to obtain the probability that the fused image features belong to each surface anomaly type label, and the surface anomaly type label with the greatest probability is used as the surface anomaly type of the multispectral remote sensing image.

7. A remote sensing image surface anomaly detection device based on image and text collaborative processing, characterized in that: include: A text feature acquisition module is used to obtain text features corresponding to each surface anomaly type label. The text features corresponding to the surface anomaly type label are pre-converted into corresponding natural sentences based on the surface anomaly type label and a customized text prompt template, and the text encoder is used to extract features from the natural sentences to obtain text features represented by vectors; A band extraction module is used to extract and fuse the bands of the multispectral remote sensing image to be detected to obtain an RGB image and a SWIR image of the multispectral remote sensing image; An image encoder module is used to extract features from the RGB image and the SWIR image to obtain initialized RGB features and initialized SWIR features; A multi-scale spatial adapter module is used to spatially adjust the initialized RGB features and the initialized SWIR features to obtain multi-scale enhanced spatially adapted RGB features and multi-scale enhanced spatially adapted SWIR features; a cross-band alignment module, configured to perform spectral adjustment on the multi-scale enhanced spatially adapted RGB features and the multi-scale enhanced spatially adapted SWIR features to obtain RGB image features and SWIR image features; A dynamic spectral weighted fusion device, used for fusing the RGB image features and the SWIR image features to generate fused image features; A similarity analysis module is used to perform cosine similarity analysis on the fused image features and the text features corresponding to the surface anomaly type labels to determine the surface anomaly type of the multispectral remote sensing image; The cross-band alignment module includes two linear layers with the same network structure and two cross-band alignment units with the same network structure; The step of inputting the multi-scale enhanced spatially adapted RGB features and the multi-scale enhanced spatially adapted SWIR features into a cross-band alignment module for spectral adjustment to obtain RGB image features and SWIR image features includes: Inputting the multi-scale enhanced spatially adapted RGB features into one of the linear layers of the cross-band alignment module for processing to obtain RGB query features, RGB key features, and RGB numerical features; Inputting the multi-scale enhanced spatially adapted SWIR features into another linear layer of the cross-band alignment module for processing to obtain SWIR query features, SWIR key features, and SWIR numerical features; Inputting the SWIR query feature, the RGB key feature, and the RGB numerical feature into one of the cross-band alignment units of the cross-band alignment module for spectral adjustment to obtain RGB image features; The RGB query feature, the SWIR key feature, and the SWIR numerical feature are input into another cross-band alignment unit of the cross-band alignment module for spectral adjustment to obtain a SWIR image feature.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the remote sensing image surface anomaly detection method based on image-text collaborative processing according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the remote sensing image surface anomaly detection method based on image-text collaborative processing according to any one of claims 1 to 6 are implemented.

Images