Method and device for constructing visual large model based on multi-source remote sensing image, equipment and medium

Abstract

The application relates to a multi-source remote sensing image-based visual large model construction method and device, equipment and medium. The method comprises the following steps: in view of the problems that the features of optical remote sensing images, SAR remote sensing images and infrared remote sensing images are quite different, and the model has weak cross-source generalization, a multi-source remote sensing image-based visual large model is researched, complementary features of multi-modal data are fused, a cross-source target unified representation space is established, the bottleneck of insufficient adaptability of a single model to different image source features is solved, the correlation of target texture, geometry and radiation characteristics in multi-source images is focused on, a multi-level feature interaction and adaptive fusion mechanism is designed, the semantic consistency understanding of the model to the cross-source target is enhanced, finally, a visual large model with strong generalization capability is constructed, high-precision supervision signals are provided for on-board lightweight small models, and the stability of cross-source remote sensing image target detection performance is ensured. The application can reduce the dependence on target domain labeled data, and make the model maintain good generalization capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a method for constructing a visual large model based on multi-source remote sensing images, a corresponding device, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] Current remote sensing image interpretation mainly adopts special models for single sensor data, such as convolutional neural networks for optical images, polarization decomposition algorithms for SAR images, and temperature feature classifiers for infrared images. Existing technologies usually independently design model architectures for specific sensor types (such as optical, SAR, or infrared), and train target recognition or feature classification models in a single scene through a large amount of labeled data. Some improved technologies attempt to improve the cross-source adaptability of models through transfer learning or data enhancement, but the core method still takes single sensor data as input, without breaking through the limitations of multi-source data collaborative modeling. Under the existing technical system, parameters are not shared between different sensor models, and feature expressions are not compatible, resulting in the need to retrain or fine-tune models when interpreting cross-source images.

[0003] The workflow of the existing technology is divided into three stages: data preprocessing, feature extraction, and classification recognition. For optical images, a convolutional neural network based on RGB channels is used to extract spatial and spectral features; SAR image processing relies on a coherent speckle noise suppression module and a polarization scattering feature extractor to model target characteristics through backscattering intensity; and infrared images are detected through temperature threshold segmentation and thermal radiation feature analysis. On the model structure, existing technologies usually adopt a phased processing architecture: sensor data is processed through independent input channels, and then the results are output through fully connected layers or decision fusion modules. Transfer learning methods achieve cross-source adaptation by freezing some pre-trained layers and replacing the top classifier, but they do not fundamentally solve the problem of joint expression of multi-source features, and existing technologies have the following technical defects:

[0004] First, the feature expression of single sensor models is limited to specific physical characteristics (such as reflectivity for optical images and scattering characteristics for SAR), and cannot fuse the complementary information of multi-source data, resulting in insufficient diversity of target features in cross-source scenarios;

[0005] Second, the model architecture lacks a unified multi-source feature fusion mechanism, and different sensor data needs to be processed independently before shallow fusion, making it difficult to mine deep correlations of cross-modal features;

[0006] Third, existing transfer learning methods rely on target domain labeled data, and when dealing with new sensors or cross-source combined data, they need to be re-labeled and trained, resulting in high model generalization costs. In addition, traditional models have limited parameters, and are not good at learning high-dimensional and heterogeneous features of multi-source data, making it difficult to support large-scale cross-domain generalization requirements.

[0007] In summary, the feature expression of the single sensor model in the prior art is limited to specific physical characteristics, cannot fuse the complementary information of multi-source data, leads to insufficient diversity of target features in cross-source scenes, and the model architecture lacks a unified multi-source feature fusion mechanism, etc. The applicant makes corresponding exploration to solve the problem. SUMMARY

[0008] The purpose of the present application is to solve the above problems and provide a visual large model construction method based on multi-source remote sensing images, a corresponding device, an electronic equipment and a computer readable storage medium.

[0009] To meet the various purposes of the present application, the present application adopts the following technical solutions:

[0010] A visual large model construction method based on multi-source remote sensing images is proposed to adapt to one of the purposes of the present application, comprising:

[0011] Obtain radar remote sensing images containing each target object and its corresponding infrared remote sensing images and optical remote sensing images, generate a random mask matrix using a preset multi-scale masking strategy to dynamically mask specific regions of the radar remote sensing images, the infrared remote sensing images and the optical remote sensing images, to determine the random masked images of the radar remote sensing images, the infrared remote sensing images and the optical remote sensing images;

[0012] In the feature extraction network of the encoder, based on the Swin Transformer model, the local window self-attention mechanism is used to capture the local detail features of each target object in the optical remote sensing images according to the encoding data corresponding to the random masked images of the optical remote sensing images, and based on the Vision Transformer model, the global attention mechanism is used to extract the global semantic information of each target object in the radar remote sensing images and the infrared remote sensing images according to the encoding data corresponding to the random masked images of the radar remote sensing images and the infrared remote sensing images;

[0013] In the cross-modal fusion layer of the encoder, the local detail features of each target object and the global semantic information of the target object are fused by using the gated cross-attention mechanism to determine the multi-source fusion features corresponding to each target object;

[0014] The multi-source fusion features are reconstructed in the decoder to determine the reconstruction loss between the reconstructed images and the original images, and the training is performed to a certain number of iterations to make the reconstruction loss as small as possible, and the construction of the visual large model based on multi-source remote sensing images and its pre-training are completed;

[0015] The pre-trained encoder is added with a specific task head for fine-tuning and then applied to a downstream interpretation task, which includes but is not limited to an object detection task, an object recognition task or a change detection task, wherein the task head includes but is not limited to an object recognition head, an object detection head or a change detection head.

[0016] Optionally, the step of obtaining radar remote sensing images containing each target object and corresponding infrared remote sensing images and optical remote sensing images comprises:

[0017] In response to a data preprocessing instruction, for the optical remote sensing image, geometric transformation is used to eliminate illumination and viewing angle interference, and image denoising technology is used to suppress clouds and noise;

[0018] For the radar remote sensing image, spot noise suppression and terrain radiation correction are used to extract backscattering coefficient features, and a vertex suppression algorithm is used to eliminate interference from overlapping and shadow areas;

[0019] For the infrared remote sensing image, based on thermal radiation calibration and dynamic range compression, the temperature difference contrast between the target object and the background is enhanced.

[0020] Optionally, the step of capturing local detail features of each target object in the optical remote sensing image based on the local window self-attention mechanism of the Swin Transformer model according to the encoding data corresponding to the randomly masked image of the optical remote sensing image comprises:

[0021] The encoding data corresponding to the randomly masked image of the optical remote sensing image is input into the Swin Transformer model, and the optical remote sensing image is divided into multiple local windows according to a preset size, wherein each local window represents a local area;

[0022] In each local window, the Swin Transformer model uses a self-attention mechanism to perform local self-attention calculation on the unmasked area features according to the encoding data corresponding to the randomly masked image of the optical remote sensing image to determine the relationship between pixels within the window. Each pixel is assigned a different weight based on the features of other pixels to capture the spatial relationship and detail features between each target object within the local area.

[0023] Optionally, the step of determining the multi-source fusion features corresponding to each target object by fusing the local detail features corresponding to each target object with the global semantic information corresponding to the target object using a gated cross-attention mechanism comprises:

[0024] acquire local detail features corresponding to each target object in the optical remote sensing image and global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image, to generate a query matrix corresponding to the local detail features and a key matrix corresponding to the global semantic information;

[0025] adopt a preset Sigmoid gating function to weight according to the matching degree of the query matrix and the key matrix, so that the features of different modalities are weighted and fused in different regions;

[0026] concatenate the features after gating weighting to integrate the information of each modality into the same space to retain their local detail features and global semantic information;

[0027] After feature concatenation, the concatenated feature map is re-labeled through channel re-labeling to determine a unified cross-modal representation, wherein the cross-modal representation contains the local detail features and global semantic information of each target object.

[0028] Optionally, the pre-trained encoder is added with a specific task head for fine-tuning and then applied to a downstream interpretation task, which includes but is not limited to the steps of a target detection task, a target recognition task or a change detection task, including:

[0029] For the target detection task, the target position and class probability are generated through an anchor box adaptive mechanism;

[0030] For the change detection task, a classification network is used to process the input data, output the probability distribution of each pixel belonging to different change categories, and then determine the change category of the pixel according to the maximum probability;

[0031] For the target recognition task, a fully connected layer classification head is designed to determine the category of the target with the maximum probability according to the probability distribution.

[0032] Optionally, the multi-scale masking strategy includes one or any combination of small size random masking, large size random masking, small size channel random masking, and large size channel random masking.

[0033] Optionally, the radar remote sensing image includes a SAR image;

[0034] The reconstruction loss represents the pixel-by-pixel mean square error between the reconstructed image output by the decoder and the original image corresponding to the radar remote sensing image, the infrared remote sensing image and the optical remote sensing image.

[0035] The reconstructed image represents a remote sensing image that integrates the local detail features and the global semantic information corresponding to each target object.

[0036] The application discloses a multi-source remote sensing image-based visual large model construction device.

[0037] The shielding image determination module is configured to acquire radar remote sensing images containing each target object and corresponding infrared remote sensing images and optical remote sensing images, generate a random mask matrix to dynamically shield specific regions of the radar remote sensing images, the infrared remote sensing images and the optical remote sensing images by adopting a preset multi-scale shielding strategy, and determine random shielding images of the radar remote sensing images, the infrared remote sensing images and the optical remote sensing images.

[0038] The semantic information extraction module is configured to capture local detail features of each target object in the optical remote sensing images according to encoding data corresponding to the random shielding images of the optical remote sensing images by adopting a local window self-attention mechanism based on a Swin Transformer model in a feature extraction network of an encoder, and extract global semantic information of each target object in the radar remote sensing images and the infrared remote sensing images according to encoding data corresponding to the random shielding images of the radar remote sensing images and the infrared remote sensing images by adopting a global attention mechanism based on a Vision Transformer model.

[0039] The fusion feature determination module is configured to perform feature fusion of the local detail features of each target object and the global semantic information of the target object by adopting a gated cross-attention mechanism in a cross-modal fusion layer of the encoder, so as to determine multi-source fusion features corresponding to each target object.

[0040] The encoder pre-training module is configured to reconstruct the multi-source fusion features in a decoder to determine a reconstruction loss between reconstructed images and original images, train until a certain number of iterations to make the reconstruction loss as small as possible, and complete construction and pre-training of the multi-source remote sensing image-based visual large model.

[0041] The downstream task execution module is configured to add a specific task head to the pre-trained encoder for fine-tuning, and then apply the fine-tuned encoder to downstream interpretation tasks, including but not limited to a target detection task, a target recognition task or a change detection task, wherein the task head includes but is not limited to a target recognition head, a target detection head and a change detection head.

[0042] The application also provides an electronic device, which comprises a central processor and a memory, and the central processor is configured to call and run a computer program stored in the memory to perform the steps of the multi-source remote sensing image-based visual large model construction method.

[0043] Another object adapted to the present application is to provide a computer-readable storage medium storing a computer program implemented according to the method for constructing a visual large model based on multi-source remote sensing images in the form of computer-readable instructions. When the computer program is invoked and run by a computer, the steps included in the corresponding method are executed.

[0044] Compared with the prior art, the present application is directed to the problems in the prior art that the feature expression of a single sensor model is limited to specific physical characteristics and cannot fuse complementary information of multi-source data, resulting in insufficient diversity of target features in cross-source scenes and lack of unified multi-source feature fusion mechanism in the model architecture, etc. The present application includes but is not limited to the following beneficial effects:

[0045] Firstly, the present application breaks through the limitations of single sensor feature expression. By cooperative modeling of multi-source remote sensing images, the present application uses data of different types of sensors (optical, SAR and infrared) for deep feature mining, which can fully fuse the complementary information in various types of sensor data. Through such cooperative modeling of cross-source data, the model can capture the multi-dimensional features of target objects, and is no longer limited to the physical characteristics of a single sensor, effectively improving the accuracy of target recognition and ground feature classification.

[0046] Secondly, the prior art usually adopts a phased processing architecture, and different sensor data is processed independently and then shallowly fused, which often leads to information loss and cannot fully mine the deep correlations of cross-modal features. The architecture based on Swin Transformer and Vision Transformer proposed by the present application can process the local detail features of optical images and the global semantic information of radar and infrared images in the encoder, respectively, and realize deep feature fusion through a gated cross-attention mechanism in the cross-modal fusion layer. The unified large model architecture enables multi-source data to be jointly learned end-to-end, eliminating information loss in traditional techniques and greatly improving the performance of cross-source remote sensing image interpretation.

[0047] Thirdly, existing transfer learning methods usually rely on target domain labeled data, and when new sensors or cross-source combined data appear, the model needs to be re-labeled and trained, which limits the generalization ability of the model. Especially when facing multi-source heterogeneous data, the adaptability of existing models is poor. The present application designs a self-supervised pre-training strategy, which can reduce the dependence on target domain labeled data, so that the model can maintain good generalization ability in cross-source and cross-domain scenarios. In particular, in the case of unseen sensors or cross-source combined data, the zero-shot capability of the model is significantly improved. This makes the present application have high adaptability and flexibility, and can cope with changing remote sensing image data and scenes.

[0048] Fourthly, the existing traditional model usually has the problems of limited parameter quantity and insufficient ability to learn high-dimensional features when processing multi-source remote sensing data, which makes it difficult to meet the demand of large-scale data processing. The present application can effectively learn in a large-scale parameter space by introducing powerful network architectures such as SwinTransformer and Vision Transformer. These models have efficient feature extraction capabilities, especially when processing multi-source heterogeneous data, they can capture rich contextual information and cross-modal features, meeting the demand of large-scale remote sensing data processing and cross-domain generalization.

[0049] Fifthly, the traditional training method usually relies on a large amount of labeled data for supervised learning, however, in the case of cross-source data, it is difficult to obtain labeled data and the cost is high, and the performance of the existing model is also limited in this respect. The present application can automatically learn deep features of multi-source remote sensing images without labeled data through a self-supervised pre-training mechanism. Self-supervised learning not only reduces the dependence on labeled data, but also improves the model's ability to abstract potential patterns in remote sensing data, further enhancing its feature expression and recognition ability. BRIEF DESCRIPTION OF DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings, in which:

[0051] Figure 1 A flowchart of a method for constructing a visual large model based on multi-source remote sensing images in an embodiment of the present application;

[0052] Figure 2 An exemplary network architecture of a system for constructing a visual large model based on multi-source remote sensing images in an embodiment of the present application;

[0053] Figure 3 A principle block diagram of an apparatus for constructing a visual large model based on multi-source remote sensing images in an embodiment of the present application;

[0054] Figure 4 A structural schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation on the present application.

[0056] It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, "connected," "coupled," and / or "coupling," can include both direct connections and / or indirect connections (i.e., via one or more other elements). As used herein, "connection" or "coupling" can include a wireless connection or a wireless coupling. As used herein, the term "and / or" comprises all of the associated listed items, one or more of the associated listed items, and all combinations of the associated listed items.

[0057] It will be further understood that, as used herein, the terms "comprises" and / or "comprising," while they can be used in the context of compositions of matter, preferably, are not used in a restrictive sense, for example, to denote the presence of stated features, integers, steps or components thereof, but to denote the presence at least the stated features, integers, steps or components thereof. As used herein, unless otherwise stated, the term "or" includes both exclusive and inclusive or. As used herein, unless otherwise stated, the term "and / or" includes all of the associated listed items, one or more of the associated listed items, and all combinations of the associated listed items.

[0058] Those skilled in the art will understand that, as used herein, the terms "client," "terminal," and "terminal device" include both devices that are solely wireless signal receivers and devices that have both receiving and transmitting hardware that can communicate bi-directionally over a bi-directional communication link. Such devices can include cellular or other communication devices with single-line or multiple-line displays, or no display, Personal Communications Service (PCS) devices that can combine a voice and / or data processor, a PDA that can include a radio frequency receiver and a pager, Internet and / or Intranet access, a Web browser, a calendar, and / or a GPS receiver, a conventional laptop and / or palmtop computer and / or other devices that have a radio frequency receiver. As used herein, the terms "client," "terminal," and "terminal device" can be portable, transportable, mounted in a vehicle (aeronautical, maritime, and / or land), or adapted and / or configured for local and / or distributed operation on Earth and / or any other location in space. As used herein, the terms "client," "terminal," and "terminal device" can also be a communication terminal, an Internet terminal, a music / video playing terminal, such as a PDA, a Mobile Internet Device (MID), and / or a mobile phone with music / video playing function, a smart television, a set-top box, and / or the like.

[0059] As used herein, the terms "server," "client," "service node," and the like refer to hardware that has the equivalent capability of a personal computer, i.e., an electronic device having a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device, and the like necessary components disclosed by the Von Neumann principle. A computer program is stored in the memory, the central processing unit loads the program stored in the external memory into the memory and runs it, executes the instructions in the program, and interacts with the input and output devices, thereby completing a specific function.

[0060] It should be noted that the concept of "server" in the present application can also be extended to the case of a server cluster. According to the principle of network deployment understood by those skilled in the art, the servers should be logically divided, and in physical space, these servers can be independent of each other but can be called through an interface, or can be integrated into a physical computer or a computer cluster. Those skilled in the art should understand this variation and should not be restricted by the implementation of the network deployment of the present application.

[0061] One or more technical features of the present application, unless explicitly specified, can be deployed on a server for implementation and accessed by a client remotely calling an online service interface provided by the server, or can be directly deployed and run on a client for implementation and access.

[0062] The neural network model referred to or possibly referred to in the present application, unless explicitly specified, can be deployed on a remote server and remotely called at the client, or can be deployed on a client with sufficient device capability for direct calling. In some embodiments, when it runs on a client, its corresponding intelligence can be obtained through transfer learning to reduce the requirement for client hardware running resources and avoid excessive occupation of client hardware running resources.

[0063] The various data involved in the present application, unless explicitly specified, can be remotely stored on a server or stored on a local terminal device, as long as it is suitable for being called by the technical solutions of the present application.

[0064] Those skilled in the art should know that the various methods of the present application, although based on the same concept and described to present commonality among them, are independently executable unless otherwise specified. Similarly, for each embodiment disclosed in the present application, it is based on the same inventive concept, so the same concept is understood to be equivalent, and although the concept is expressed differently, it is only for convenience and is appropriately transformed.

[0065] Unless it is explicitly stated that the embodiments disclosed in the present application are mutually exclusive, the technical features involved in each embodiment can be combined flexibly to construct new embodiments, as long as such combination does not deviate from the spirit of the present application and can meet the needs of the prior art or solve some aspects of the prior art. For this variation, those skilled in the art should know.

[0066] Please refer to Figure 1 and Figure 2 The visual large model construction method based on multi-source remote sensing images of the present application includes, in one embodiment thereof:

[0067] Step S10, obtain radar remote sensing images containing each target object and its corresponding infrared remote sensing images, optical remote sensing images, and generate a random mask matrix to dynamically mask the specific area of the radar remote sensing images, the infrared remote sensing images and the optical remote sensing images using a preset multi-scale masking strategy, to determine the random masking images of the radar remote sensing images, the infrared remote sensing images and the optical remote sensing images.

[0068] The visual large model construction system based on multi-source remote sensing images in the terminal device can obtain radar remote sensing images containing each target object and its corresponding infrared remote sensing images, optical remote sensing images, and generate a random mask matrix to dynamically mask the specific area of the radar remote sensing images, the infrared remote sensing images and the optical remote sensing images using a preset multi-scale masking strategy, to determine the random masking images of the radar remote sensing images, the infrared remote sensing images and the optical remote sensing images. Wherein, the random masking image represents the remote sensing image after the radar remote sensing image, the infrared remote sensing image and the optical remote sensing image are masked by the random mask matrix; the multi-scale masking strategy includes one or any combination of small size random masking, large size random masking, small size channel random masking, and large size channel random masking; the target object includes buildings, roads, vegetation, etc., and the radar remote sensing image includes SAR (Synthetic Aperture Radar) image. Specifically, the SAR image uses synthetic aperture radar technology to obtain information of ground objects by transmitting microwave signals and receiving reflected echo signals.

[0069] Specifically, after obtaining radar remote sensing images containing each target object and its corresponding infrared remote sensing images, optical remote sensing images, a multi-scale masking strategy such as small size random masking, large size random masking, small size channel random masking, and large size channel random masking can be used to generate a random mask matrix to dynamically mask the specific area of the radar, infrared, and optical remote sensing images. In this way, a random masking image can be generated, which specifies which areas in the radar remote sensing image, infrared remote sensing image, and optical remote sensing image will be masked to simulate data missing or noise conditions. The mask matrix is usually a binary matrix that marks which pixels need to be masked, usually 0, indicating that the area is masked, while other pixels remain unchanged. The random masking image of the radar remote sensing image, the infrared remote sensing image and the optical remote sensing image will be used for processing of input images to help the model learn how to effectively extract features in the case of missing part of the information.

[0070] In some embodiments, the small size random masking refers to randomly selecting small areas in the image for masking (occlusion), usually with small size. The masked area is set to zero or a certain constant value, simulating the missing part of the information in the image. This method is often used for data enhancement or training of deep learning models, so that the model has stronger robustness to local missing; the large size random masking is similar to the small size random masking, but the size of the masked area is larger, which may cover part of the image or one or more target areas. The size of the mask is randomly selected; the small size channel random masking means random masking in the channel dimension of the image. That is, some channels are randomly selected, and then small range masking is performed on these channels. The masked channel may randomly set to zero or change the pixel value at a certain position; the large size channel random masking is similar to the small size channel random masking, but here the range of the masked channel is larger, which may be some important channels or multiple channels of the entire image. The above multi-scale masking strategy is mainly used to enhance the robustness of the deep learning model, especially in image processing and remote sensing image analysis, to simulate different types of data missing or occlusion. By introducing these masking strategies in the training process, the model can learn how to handle the case of partial information missing, and improve its generalization ability in practical application.

[0071] In some embodiments, the step of acquiring radar remote sensing images containing each target object and its corresponding infrared remote sensing images and optical remote sensing images comprises:

[0072] Step S101, in response to a data preprocessing instruction, for the optical remote sensing image, geometric transformation is used to eliminate the interference of light and viewing angle, and image denoising technology is used to suppress cloud and noise;

[0073] Step S102, for the radar remote sensing image, spot noise suppression and terrain radiation correction are used to extract the backscattering coefficient feature, and a vertex suppression algorithm is used to eliminate the interference of the shadow area and the shadow area;

[0074] Step S103, for the infrared remote sensing image, based on thermal radiation calibration and dynamic range compression to enhance the temperature difference contrast of target objects and background.

[0075] Specifically, the geometric transformation includes geometric correction, radiometric correction, etc., and the image denoising technology includes mean filtering, etc. Optical remote sensing images are easily affected by lighting conditions (such as the angle of the sun) and viewing angles (such as the shooting angle), resulting in some areas in the image being too bright or too dark, thereby affecting the identification and detection of the target. Geometric transformation eliminates these effects by geometric correction of the image (such as radiometric correction, orthorectification, etc.). Geometric correction adjusts the geometric shape of the image to conform to the geographic coordinate system, eliminating image distortion caused by sensor viewing angle, terrain undulation, etc.; radiometric correction adjusts the lighting conditions of the image to enable effective comparison of partial information missing under different time, place, and weather conditions, thereby improving its generalization ability in practical applications. Clouds often block or interfere with the observation of target objects in optical remote sensing images. Cloud detection and removal based on multispectral images can effectively suppress the effects of clouds. Optical remote sensing images may be affected by sensor noise or atmospheric effects (such as atmospheric scattering) to produce noise points. Denoising techniques (such as median filtering, wavelet denoising, etc.) are used to remove these noises to improve image quality.

[0076] Further, radar remote sensing images often have speckle noise, which is a kind of random noise caused by scattering characteristics, affecting the quality of the image. Speckle noise suppression techniques (such as filtering techniques, mean filtering, band transformation, etc.) help reduce noise interference and extract clearer target information; radar images are affected by terrain, especially in mountainous or undulating areas, where terrain changes can cause differences in radiation intensity at different locations. Terrain radiometric correction techniques (such as using digital elevation models) can correct this terrain effect, thereby more accurately reflecting the backscatter characteristics of the ground surface.

[0077] Radar remote sensing images acquire ground object information by analyzing the reflection (backscatter) of radar waves by ground objects. By correcting the backscatter characteristics, physical features of target objects such as surface roughness, moisture, and vegetation cover can be extracted. For example, by processing SAR images, the backscatter coefficients of different ground objects can be obtained, which helps to distinguish different types of ground objects. In radar remote sensing images, due to the shooting angle of the sensor or the influence of the terrain, ground object overlap or shadow areas may occur, which may not provide effective ground object information. Vertex suppression algorithms are a technique that suppresses the effects of these areas by analyzing the geometric shapes of objects in the image. Vertex suppression algorithms improve the usability and quality of the image by identifying and removing these areas.

[0078] Further, infrared remote sensing images acquire information by receiving thermal radiation of ground objects, but due to the interference of atmosphere and sensor, the original infrared image may have deviation, and the thermal radiation calibration technology is used to correct the image according to the known radiation standard, to ensure the accuracy of the thermal radiation in the image; The calibration process usually includes corresponding the digital value of the image with the actual radiation value (or temperature) of the ground object, so as to obtain accurate temperature information; The temperature range of infrared image may be very extensive, resulting in difficulty in distinguishing the target with small temperature difference from the background, and dynamic range compression adjusts the brightness and contrast of the image, so that the temperature contrast is more prominent, thereby enhancing the difference between the target object and the background. This technology helps to improve the ability to identify low temperature targets (such as human temperature, vegetation, etc.) in the background.

[0079] In step S20, in the feature extraction network of the encoder, based on the Swin Transformer model, the local window self-attention mechanism is used to capture the local detail features of each target object in the optical remote sensing image according to the encoding data corresponding to the random masked image of the optical remote sensing image, and based on the Vision Transformer model, the global attention mechanism is used to extract the global semantic information of each target object in the radar remote sensing image and the infrared remote sensing image according to the encoding data corresponding to the random masked image of the radar remote sensing image and the infrared remote sensing image.

[0080] The radar remote sensing image containing each target object and its corresponding infrared remote sensing image and optical remote sensing image are obtained, a random mask matrix is generated by using a preset multi-scale masking strategy to dynamically mask a specific area of the radar remote sensing image, the infrared remote sensing image and the optical remote sensing image, and after determining the random masked image of the radar remote sensing image, the infrared remote sensing image and the optical remote sensing image, in the feature extraction network of the encoder, based on the Swin Transformer model, the local window self-attention mechanism is used to capture the local detail features of each target object in the optical remote sensing image according to the encoding data corresponding to the random masked image of the optical remote sensing image, and based on the Vision Transformer model, the global attention mechanism is used to extract the global semantic information of each target object in the radar remote sensing image and the infrared remote sensing image according to the encoding data corresponding to the random masked image of the radar remote sensing image and the infrared remote sensing image.

[0081] In some embodiments, the step of capturing the local detail features of each target object in the optical remote sensing image according to the encoding data corresponding to the random masked image of the optical remote sensing image based on the Swin Transformer model using the local window self-attention mechanism includes:

[0082] Step S201, input the encoding data corresponding to the random masking image of the optical remote sensing image into the SwinTransformer model, divide the optical remote sensing image into a plurality of local windows according to a preset size, wherein each local window represents a local area;

[0083] Step S202, in each local window, the Swin Transformer model uses a self-attention mechanism to determine the relationship between pixels within the window based on the encoding data corresponding to the random masking image of the optical remote sensing image, and each pixel is assigned a different weight based on the features of other pixels to capture the spatial relationship and detailed features between target objects within the local area.

[0084] Specifically, Swin Transformer is a deep learning model based on a self-attention mechanism, which uses a "windowed" self-attention mechanism, especially suitable for tasks with locality and large-scale data. The core advantage of SwinTransformer is that it processes images through local window division, allowing the model to efficiently learn local features.

[0085] When inputting the encoding data corresponding to the random masking image of the optical remote sensing image into the SwinTransformer, the optical remote sensing image will be divided into a plurality of local windows according to a preset size (e.g. 7x7 or 8x8), and each window represents a local area.

[0086] In each window, Swin Transformer calculates the relationship between pixels within the window through a self-attention mechanism, and each pixel is assigned a different weight based on the features of other pixels, thereby capturing the spatial relationship and features within the local area. According to the random masking image of the optical remote sensing image, the corresponding input feature data is masked. During training, the features of the masked area are "ignored" and only rely on the local self-attention calculation of the unmasked area features. This strategy can help the model enhance the robustness of the local missing or occluded area.

[0087] In each local window, the Swin Transformer model learns features based on a self-attention mechanism. Through self-attention calculation, the model can focus on key information within the local area, effectively capturing the detailed features of target objects, even if some areas may lose information due to masking. Through multi-level local window attention mechanism, the model not only learns the detailed features of each local area, but also integrates these local features at a deeper level to capture the context information across local areas, thereby more comprehensively understanding the target objects.

[0088] By using the local window self-attention mechanism based on the Swin Transformer model, the local detail features of the target objects in the optical remote sensing image can be effectively captured. Through the masking strategy and the self-attention mechanism, the model can still learn useful local features from the unmasked areas in the case of partial area missing, and can effectively perform cross-modal feature fusion, thereby improving the performance of the remote sensing image analysis task.

[0089] In some embodiments, the Vision Transformer model adopts a global attention mechanism to extract global semantic information of each target object in the radar remote sensing image and the infrared remote sensing image according to the encoded data corresponding to the randomly masked image of the radar remote sensing image and the infrared remote sensing image. Specifically, first, the radar remote sensing image and the infrared remote sensing image are divided into small blocks (patches), and each radar remote sensing image or infrared remote sensing image is divided into fixed-size blocks (for example, 16x16 or 32x32 small blocks), which makes each small block become an input unit that can be processed by the Vision Transformer model. Each small block will be mapped to an embedding vector through a linear transformation. In this way, each small block (for example, a 16x16 image block) of the image will be converted into a one-dimensional vector representation as the input of the subsequent Vision Transformer model.

[0090] In the Vision Transformer model, the self-attention mechanism calculates the relationship between image blocks globally, and each image block updates its representation through interaction with other image blocks. That is, each image block dynamically adjusts its weight according to global information and is weighted according to the features of other image blocks. For each image block (patch), the Vision Transformer model calculates an attention weight with all other image blocks. These weights are calculated based on the content and position of the image blocks through inner product operation and softmax function. The representation of each image block will be aggregated with the weighted information of other image blocks, thereby obtaining a globally context-aware representation.

[0091] Based on the encoded data corresponding to the randomly masked image of the radar remote sensing image and the infrared remote sensing image, the masked parts can be ignored or given lower attention weights when calculating self-attention. In this way, the model can avoid over-reliance on the masked areas and focus on the unmasked areas. Through the multi-layer structure of the Vision Transformer, the network updates and adjusts the image blocks at each layer to gradually extract high-level semantic features of the image.

[0092] Through the global attention mechanism of multiple layers, the Vision Transformer model gradually fuses the information from different image blocks and learns the dependency between the blocks. In particular, for radar and infrared remote sensing images, the Vision Transformer model can learn different types of features, such as the shape, size, motion characteristics (for radar remote sensing images), and thermal radiation information (for infrared remote sensing images) of the target objects.

[0093] Finally, after processing by multiple layers of self-attention mechanisms, the model obtains a set of high-dimensional global feature vectors representing the global semantic information of each target object in the image. These information can reflect the specific location, category, structure, etc. of the target object.

[0094] In step S30, the local detail features corresponding to each target object and the global semantic information corresponding to the target object are fused by using a gated cross-attention mechanism in the cross-modal fusion layer of the encoder to determine the multi-source fusion features corresponding to each target object.

[0095] After the Vision Transformer model extracts the global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image based on the encoded data corresponding to the randomly masked radar remote sensing image and the randomly masked infrared remote sensing image, in the cross-modal fusion layer of the encoder, the local detail features corresponding to each target object and the global semantic information corresponding to the target object are fused by using a gated cross-attention mechanism to determine the multi-source fusion features corresponding to each target object.

[0096] In some embodiments, the step of fusing the local detail features corresponding to each target object and the global semantic information corresponding to the target object by using a gated cross-attention mechanism to determine the multi-source fusion features corresponding to each target object includes:

[0097] In step S301, the local detail features corresponding to each target object in the optical remote sensing image and the global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image are obtained to generate a query matrix corresponding to the local detail features and a key matrix corresponding to the global semantic information.

[0098] In step S302, a pre-set Sigmoid gating function is used to weight according to the matching degree of the query matrix and the key matrix, so that the features of different modalities are weighted and fused in different regions.

[0099] Step S303, the gated weighted features are spliced together to integrate the information of each modality into the same space to retain their local detail features and global semantic information;

[0100] Step S304, after feature splicing, the spliced feature map is re-calibrated by channel re-calibration to determine a unified cross-modal representation, wherein the cross-modal representation contains the local detail features and global semantic information of each target object.

[0101] Specifically, in the cross-modal fusion layer of the encoder, the main purpose is to fuse the local detail features and global semantic information extracted from radar remote sensing images, infrared remote sensing images and optical remote sensing images, and then obtain a multi-source fusion feature. The gated cross-attention mechanism is a cross-modal attention mechanism that introduces a gating mechanism. It optimizes the fusion of multi-source data by effectively controlling the interaction between local detail features and global semantic information. Its workflow includes:

[0102] The local detail features and global semantic information are respectively mapped to the same dimensional space, which is to ensure that the local detail features and global semantic information can be fused in the same feature space. The core of the gating mechanism is to dynamically adjust the fusion degree of local detail features and semantic information by learning a gating variable (such as attention weight). Specifically, the gating mechanism generates a "gating coefficient" for each target object, which controls the fusion ratio of the local detail features extracted from the optical remote sensing image and the semantic information extracted from the radar and infrared image. The gating coefficient can be obtained by calculating the similarity of the input features (such as through self-attention mechanism or other metrics).

[0103] Then, the gating coefficient is used to weight the local detail features and semantic information, so that they can be selectively fused according to the actual situation. This step is dynamically changed for different target objects to ensure that the feature fusion method of each target object is targeted.

[0104] On the basis of the gating mechanism, the cross-attention mechanism further enhances the interaction between different modalities. Under this mechanism, the local detail features from the optical image will be weighted and calculated with the semantic information from the radar and infrared images, and vice versa. The goal of cross-attention is to focus on the features of different modalities alternately, so as to achieve more efficient information fusion.

[0105] Finally, the local detail features and global semantic information are fused through cross-attention and gating mechanisms to generate the final multi-source fusion features. These fusion features contain both detailed information and semantic context information about the target objects, providing more comprehensive and accurate feature representations for downstream tasks such as object detection and change detection.

[0106] Furthermore, local detail features can help capture details in the image, while global semantic information can provide an understanding of the overall structure and relationships. Through the gating cross-attention mechanism, these two types of information can be combined flexibly to improve the quality of the fusion features. After feature fusion, each target object can utilize both local details and global semantic information, resulting in more accurate feature representations.

[0107] The gating cross-attention mechanism dynamically adjusts the fusion of local detail features and global semantic information, controls the weighting of cross-attention using a gating mechanism, and ensures that different modalities of information can be effectively combined to produce high-quality multi-source fusion features. These features provide more comprehensive and accurate feature representations for subsequent remote sensing image analysis tasks.

[0108] Step S40, reconstructing the multi-source fusion features in the decoder to determine the reconstruction loss between the reconstructed image and the original image, training to a certain number of iterations to make the reconstruction loss as small as possible, completing the construction and pre-training of the visual large model based on multi-source remote sensing images;

[0109] In the cross-modal fusion layer of the encoder, the local detail features corresponding to each target object and the global semantic information corresponding to the target object are fused using the gating cross-attention mechanism to determine the multi-source fusion features corresponding to each target object. Then, in the decoder, the multi-source fusion features are reconstructed to determine the reconstruction loss between the reconstructed image and the original image. The model is trained to a certain number of iterations to make the reconstruction loss as small as possible, completing the construction and pre-training of the visual large model based on multi-source remote sensing images. The reconstruction loss represents the pixel-by-pixel mean square error between the reconstructed image output by the decoder and the original image corresponding to the radar remote sensing image, infrared remote sensing image, and optical remote sensing image. The reconstructed image represents the remote sensing image that combines the local detail features and global semantic information corresponding to each target object.

[0110] Specifically, in the decoder, the multi-source fusion features are reconstructed, and the purpose of this process is to convert the fused features back to the original image form to evaluate the performance of the model. By calculating the difference between the reconstructed image and the original image (i.e., the reconstruction loss), the model can learn how to better extract and fuse information from different modalities. Through iterative training, the model gradually optimizes until the reconstruction loss reaches as small as possible, completing the construction of the visual large model based on multi-source remote sensing images and its pre-training. The training in this stage is to pre-train the encoder so that it can perform better generalization in the following downstream interpretation tasks.

[0111] After the pre-trained encoder is added with a specific task head for fine-tuning and applied to downstream interpretation tasks, including but not limited to object detection tasks, object recognition tasks, or change detection tasks, where the task head includes but is not limited to an object recognition head, an object detection head, and a change detection head.

[0112] After the pre-trained encoder is added with a specific task head for fine-tuning and applied to downstream interpretation tasks, including but not limited to object detection tasks, object recognition tasks, or change detection tasks, where the task head includes but is not limited to an object recognition head, an object detection head, and a change detection head. Specifically, the pre-trained encoder can be used for a series of downstream remote sensing image interpretation tasks, such as object detection, object recognition, change detection, etc. These tasks require the model to be able to understand and recognize various targets in remote sensing images, such as buildings, roads, vegetation, etc.

[0113] In some embodiments, based on the task requirements, a special task output head is designed for the execution of specific downstream tasks. After the pre-trained encoder is added with a specific task head for fine-tuning and applied to downstream interpretation tasks, including but not limited to object detection tasks, object recognition tasks, or change detection tasks, the steps include:

[0114] For object detection tasks, an anchor box adaptive mechanism is used to generate target position and class probability;

[0115] For change detection tasks, a classification network is used to process the input data, output the probability distribution of each pixel belonging to different change categories, and then determine the change category of the pixel according to the maximum probability;

[0116] For object recognition tasks, a fully connected layer classification head is designed to determine the class of the target with the maximum probability according to the probability distribution.

[0117] In some embodiments, the multi-source remote sensing image set is represented as:

[0118]

[0119] wherein X (k) represents the original input image data of the kth modality, K represents the number of modalities, k = 1, 2, 3,... K, for example, k = 1 represents a SAR image, k = 2 can represent an infrared remote sensing image, and k = 3 represents an optical remote sensing image.

[0120] The expression for generating a random masked image is:

[0121]

[0122] wherein, represents the image after the mask operation, and M represents a random mask matrix.

[0123] Based on the Swin Transformer model, the local window self-attention mechanism is used to capture the local detail features of each target object in the optical remote sensing image according to the encoding data corresponding to the random masked image of the optical remote sensing image. The local window self-attention calculation is:

[0124]

[0125] wherein Q represents a query matrix, K represents a key matrix, V represents a value matrix, Attention(Q, W) represents the local window self-attention calculation result, the Softmax function is used to calculate the similarity of Q and K, and the attention output is finally obtained by weighting the similarity by V, QW Q (KW K ) T represents the matrix multiplication of Q and K, which is used to calculate their similarity; VW V represents the result of the V matrix after weighting.

[0126] Based on the Vision Transformer model, the global attention mechanism is used to extract the global semantic information of each target object in the radar remote sensing image and the infrared remote sensing image according to the encoding data corresponding to the random masked image of the radar remote sensing image and the infrared remote sensing image. The global attention output is:

[0127] Z (k) = LayerNorm(MultiHead(E (k) )+E (k) ),

[0128] wherein LayerNorm represents layer normalization, MultiHead represents a multi-head attention mechanism for enhancing the model's attention to different information sources, and E (k) represents the embedding representation of the k-th modality, obtained by embedding operation on

[0129] In the cross-modal fusion layer of the encoder, a gated cross-attention mechanism is used to fuse the local detailed features corresponding to each target object with the global semantic information corresponding to the target object to determine the multi-source fusion features corresponding to each target object, and the gated cross-attention mechanism is defined as:

[0130]

[0131] wherein σ represents a Sigmoid gating function for weighting the interaction between different modalities of cross-attention. The output of σ is a value between 0 and 1, which is used to control the influence degree of each modality. represents a feature concatenation operation for fusing information from different modalities into a unified representation. The concatenated feature map can contain information from multiple modalities. represents the feature map after gating.

[0132] The expression of channel reparameterization is:

[0133]

[0134] wherein, represents the feature map after channel reparameterization; γ represents a scaling parameter in the channel reparameterization operation; and β represents a translation parameter in the channel reparameterization operation.

[0135] The reconstruction loss between the reconstructed image output by the decoder and each original image is represented as:

[0136]

[0137] As can be seen from the above embodiments, compared with the prior art, the present application is aimed at the problems in the prior art that the feature expression of a single sensor model is limited to specific physical characteristics, cannot fuse complementary information of multi-source data, leads to insufficient diversity of target features in cross-source scenes, and the model architecture lacks a unified multi-source feature fusion mechanism. The present application includes but is not limited to the following beneficial effects:

[0138] ​Firstly, the application breaks through the limitations of single sensor feature expression. Through the collaborative modeling of multi-source remote sensing images, the application uses data from different types of sensors (optical, SAR, and infrared) to conduct deep feature mining, which can fully fuse the complementary information in various sensor data. Through this cross-source data collaborative modeling, the model can capture the multi-dimensional features of the target object, no longer limited to the physical characteristics of a single sensor, effectively improving the accuracy of target recognition and feature classification.

[0139] Secondly, existing technologies usually use a phased processing architecture, with different sensor data being processed independently and then shallowly fused, which often leads to information loss and cannot fully mine the deep correlations of cross-modal features. The architecture based on Swin Transformer and Vision Transformer proposed by the application can process the local detail features of optical images and the global semantic information of radar and infrared images in the encoder, respectively, and achieve deep feature fusion through the gated cross-attention mechanism in the cross-modal fusion layer. The unified large model architecture enables multi-source data to be learned end-to-end, eliminating the information loss in traditional technologies and greatly improving the performance of cross-source remote sensing image interpretation.

[0140] Thirdly, existing transfer learning methods usually rely on target domain labeled data, and when new sensors or cross-source combined data appear, the model needs to be re-labeled and trained, which limits the model's generalization ability. Especially when facing multi-source heterogeneous data, the adaptability of existing models is poor. The application designs a self-supervised pre-training strategy to reduce the dependence on target domain labeled data, enabling the model to maintain good generalization ability in cross-source and cross-domain scenarios. Especially in the case of unseen sensors or cross-source combined data, the model's zero-shot ability has been significantly improved. This makes the application highly adaptable and flexible, capable of coping with changing remote sensing image data and scenarios.

[0141] Fourthly, existing traditional models often have limited parameter quantity and insufficient ability to learn high-dimensional features when processing multi-source remote sensing data, making it difficult to meet the needs of large-scale data processing. The application introduces powerful network architectures such as SwinTransformer and Vision Transformer, which can effectively learn in large-scale parameter space. These models have high feature extraction capabilities, especially when dealing with multi-source heterogeneous data, they can capture rich contextual information and cross-modal features, meeting the needs of large-scale remote sensing data processing and cross-domain generalization.

[0142] Fifthly, the traditional training method usually relies on a large amount of labeled data for supervised learning. However, in the case of cross-source data, it is difficult to obtain labeled data and the cost is high. The performance of the existing model is also limited in this respect. Through the self-supervised pre-training mechanism, the model can automatically learn the deep features of multi-source remote sensing images without labeled data. Self-supervised learning not only reduces the dependence on labeled data, but also improves the model's ability to abstract potential patterns in remote sensing data, further enhancing its feature expression and recognition capabilities.

[0143] Please refer to Figure 3, provided by one of the purposes of the application, a visual large model construction device based on multi-source remote sensing images, comprising a masking image determination module 1100, a semantic information extraction module 1200, a fusion feature determination module 1300, an encoder pre-training module 1400 and a downstream task execution module 1500. Among them, the masking image determination module 1100 is set to obtain radar remote sensing images containing each target object and its corresponding infrared remote sensing images, optical remote sensing images, generate a random mask matrix to dynamically mask a specific area of the radar remote sensing image, the infrared remote sensing image and the optical remote sensing image by adopting a preset multi-scale masking strategy, to determine the random masking image of the radar remote sensing image, the infrared remote sensing image and the optical remote sensing image; the semantic information extraction module 1200 is set to capture the local detail features of each target object in the optical remote sensing image according to the encoding data corresponding to the random masking image of the optical remote sensing image based on the Swin Transformer model using the local window self-attention mechanism in the feature extraction network of the encoder, and extract the global semantic information of each target object in the radar remote sensing image and the infrared remote sensing image according to the encoding data corresponding to the random masking image of the radar remote sensing image and the infrared remote sensing image based on the Vision Transformer model using the global attention mechanism; the fusion feature determination module 1300 is set to perform feature fusion on the local detail features of each target object and the global semantic information of the target object by adopting the gated cross-attention mechanism in the cross-modal fusion layer of the encoder to determine the multi-source fusion features corresponding to each target object; the encoder pre-training module 1400 is set to reconstruct the multi-source fusion features in the decoder to determine the reconstruction loss between the reconstructed image and the original image, and train to a certain number of iterations to make the reconstruction loss as small as possible, complete the construction and pre-training of the visual large model based on multi-source remote sensing images; the downstream task execution module 1500 is set to add a specific task head to the pre-trained encoder for fine-tuning, and then applied to downstream interpretation tasks, including but not limited to target detection tasks, target recognition tasks or change detection tasks, wherein the task head includes but is not limited to a target recognition head, a target detection head, and a change detection head.

[0144] On the basis of any embodiment of the present application, please refer to Figure 4 Another embodiment of the present application also provides an electronic device, which can be realized by a computer device, such as Figure 4As shown, the internal structure diagram of the computer device is shown. The computer device includes a processor, a computer readable storage medium, a memory and a network interface connected by a system bus. Among them, the computer readable storage medium of the computer device stores an operating system, a database and computer readable instructions, the database can store control information sequence, and the computer readable instructions are executed by the processor to enable the processor to implement a visual large model construction method based on multi-source remote sensing images. The processor of the computer device is used to provide computing and control capability to support the operation of the entire computer device. The memory of the computer device can store computer readable instructions, and the computer readable instructions are executed by the processor to enable the processor to execute the visual large model construction method based on multi-source remote sensing images of the present application. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art can understand, Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0145] The processor in the embodiment is used to execute the specific functions of each module in Figure 3 The memory stores the program codes and various data required to execute the above-mentioned modules. The network interface is used for data transmission between the user terminal or the server. The memory in the embodiment stores the program codes and data required to execute all modules in the visual large model construction device based on multi-source remote sensing images of the present application, and the server can call the program codes and data of the server to execute the functions of all modules.

[0146] The present application also provides a storage medium storing computer readable instructions, which are executed by one or more processors to enable one or more processors to execute the steps of the visual large model construction method based on multi-source remote sensing images described in any embodiment of the present application.

[0147] The present application also provides a computer program product including computer programs / instructions, which are executed by one or more processors to implement the steps of the visual large model construction method based on multi-source remote sensing images described in any embodiment of the present application.

[0148] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the application can be completed by a computer program instructing relevant hardware, and the computer program can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiments of the methods can be included. The storage medium can be a computer readable storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0149] The above only describes some embodiments of the application. It should be pointed out that those skilled in the art can make some improvements and refinements without departing from the principles of the application, and these improvements and refinements should also be considered as the protection scope of the application.

Claims

1. A method for constructing a visual large model based on multi-source remote sensing images, characterized in that, The method comprises the steps of: acquiring radar remote sensing images containing target objects and corresponding infrared remote sensing images and optical remote sensing images, and generating random mask matrices to dynamically mask specific regions of the radar remote sensing images, the infrared remote sensing images and the optical remote sensing images by using a preset multi-scale masking strategy, so as to determine random masked images of the radar remote sensing images, the infrared remote sensing images and the optical remote sensing images; in the feature extraction network of the encoder, based on the Swin Transformer model, local window self-attention mechanism is used to capture local detail features of each target object in the optical remote sensing images according to the encoding data corresponding to the random masked images of the optical remote sensing images, and based on the Vision Transformer model, global attention mechanism is used to extract global semantic information of each target object in the radar remote sensing images and the infrared remote sensing images according to the encoding data corresponding to the random masked images of the radar remote sensing images and the infrared remote sensing images; in the cross-modal fusion layer of the encoder, the local detail features of each target object and the global semantic information of the target object are fused by using the gated cross-attention mechanism to determine the multi-source fusion features corresponding to each target object, which comprises: acquiring the local detail features of each target object in the optical remote sensing images and the global semantic information of each target object in the radar remote sensing images and the infrared remote sensing images to generate a query matrix corresponding to the local detail features and a key matrix corresponding to the global semantic information; using a preset Sigmoid gating function to weight according to the matching degree of the query matrix and the key matrix, so that the features of different modalities are weighted and fused in different regions; the features after gating and weighting are spliced together to integrate the information of each modality into the same space to retain their local detail features and global semantic information; after feature splicing, the spliced feature map is re-labeled by channel re-labeling to determine a unified cross-modal representation, wherein the cross-modal representation contains the local detail features and the global semantic information of each target object; reconstructing the multi-source fusion features in the decoder to determine the reconstruction loss between the reconstructed image and the original image, training to a certain number of iterations to make the reconstruction loss as small as possible, and completing the construction and pre-training of the visual large model based on the multi-source remote sensing images; after adding a specific task head to the pre-trained encoder for fine-tuning, the encoder is applied to downstream interpretation tasks, which include target detection tasks, target recognition tasks or change detection tasks, wherein the task head includes a target recognition head, a target detection head and a change detection head.

2. The method of claim 1, wherein the method further comprises: The step of acquiring radar remote sensing images containing target objects and corresponding infrared remote sensing images and optical remote sensing images comprises: in response to a data preprocessing instruction, for the optical remote sensing images, geometric transformation is used to eliminate the interference of light and viewing angle, and image denoising technology is used to suppress clouds and noise; For radar remote sensing images, speckle noise suppression and topographic radiometric correction are used to extract backscattering coefficient features, and vertex suppression algorithm is used to eliminate overlapping and shadow area interference. For infrared remote sensing images, thermal radiation calibration and dynamic range compression are used to enhance the temperature difference contrast between the target object and the background.

3. The method of claim 1, wherein the method further comprises: Based on the Swin Transformer model, a local window self-attention mechanism is used to capture the local detail features corresponding to each target object in the optical remote sensing image according to the encoded data corresponding to the randomly occluded image. The steps include: The encoded data corresponding to the randomly occluded images of the optical remote sensing image are input into the Swing Transformer model, and the optical remote sensing image is divided into multiple local windows according to a preset size, wherein each local window represents a local region. Within each local window, the Swin Transformer model employs a self-attention mechanism to perform local self-attention calculations on the unmasked region features based on the encoded data corresponding to the randomly occluded images of the optical remote sensing image, in order to determine the relationship between pixels within the window. Each pixel is assigned a different weight based on the features of other pixels, in order to capture the spatial relationships and detailed features between various target objects within the local region.

4. The method of claim 1, wherein the method further comprises: After fine-tuning the pre-trained encoder by adding specific task heads, it is applied to downstream interpretation tasks, including object detection, object recognition, or change detection tasks, and includes the following steps: For object detection tasks, an anchor box adaptive mechanism is used to generate object location and class probability. For the change detection task, the input data is processed by a classification network to output the probability distribution of each pixel belonging to different change categories, and then the change category of the pixel is determined based on the maximum probability. For target recognition tasks, a fully connected layer classification head is designed to determine the category of the target with the highest probability based on the probability distribution.

5. The method of claim 1 to 4, wherein, The multi-scale masking strategy includes one or more of the following: small-size random masking, large-size random masking, small-size channel random masking, and large-size channel random masking.

6. The method of claim 1 to 4, wherein, The radar remote sensing images include SAR images; The reconstruction loss characterizes the pixel-wise mean square error between the reconstructed image output by the decoder and the original images of the radar remote sensing image, infrared remote sensing image, and optical remote sensing image corresponding to the reconstructed image. The reconstructed image representation is a remote sensing image that integrates local detail features and global semantic information corresponding to each target object.

7. A device for constructing a visual large model based on multi-source remote sensing images, characterized in that, include: The masking image determination module is configured to acquire radar remote sensing images containing each target object and its corresponding infrared remote sensing images and optical remote sensing images, and use a preset multi-scale masking strategy to generate a random mask matrix to dynamically mask specific areas of the radar remote sensing images, the infrared remote sensing images and the optical remote sensing images, so as to determine the random masking images of the radar remote sensing images, the infrared remote sensing images and the optical remote sensing images. The semantic information extraction module is configured to, in the feature extraction network of the encoder, use a local window self-attention mechanism based on the Swin Transformer model to capture the local detail features corresponding to each target object in the optical remote sensing image according to the encoded data corresponding to the randomly occluded image of the optical remote sensing image, and use a global attention mechanism based on the Vision Transformer model to extract the global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image according to the encoded data corresponding to the randomly occluded image of the infrared remote sensing image. The feature fusion determination module is configured to use a gated cross-attention mechanism in the cross-modal fusion layer of the encoder to fuse the local detail features corresponding to each target object with the global semantic information corresponding to the target object, so as to determine the multi-source fusion features corresponding to each target object, which includes: Local detail features corresponding to each target object in the optical remote sensing image and global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image are obtained to generate a query matrix corresponding to the local detail features and a key matrix corresponding to the global semantic information. A preset sigmoid gating function is used to weight the query matrix and the key matrix according to their matching degree, so that features of different modalities are weighted and fused in different regions. The gated and weighted features are concatenated to integrate the information of each modality into the same space to preserve their local detail features and global semantic information. After feature concatenation, the concatenated feature map is recalibrated through channel recalibration to determine a unified cross-modal representation, wherein the cross-modal representation includes the local detail features and global semantic information of each target object. The encoder pre-training module is configured to reconstruct the multi-source fusion features in the decoder to determine the reconstruction loss between the reconstructed image and the original image, and train it to a certain number of iterations to minimize the reconstruction loss, thereby completing the construction and pre-training of a large visual model based on multi-source remote sensing images. The downstream task execution module is configured to add specific task heads to the pre-trained encoder for fine-tuning before applying it to downstream interpretation tasks, including target detection tasks, target recognition tasks, or change detection tasks. The task heads include target recognition heads, target detection heads, and change detection heads.

8. An electronic device comprising a central processing unit and a memory, characterized in that The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 6, which, when invoked by a computer, executes the steps included in the corresponding method.

Images