Visual large model construction method and device based on multi-source remote sensing image, equipment and medium

Abstract

The invention relates to a multi-source remote sensing image-based visual large model construction method, device and equipment and a medium, and the method comprises the steps: researching a multi-source remote sensing image-based visual large model for solving the problems that the image feature difference of an optical remote sensing image, an SAR remote sensing image and an infrared remote sensing image is large, and the cross-source generalization of the model is weak; by fusing multi-modal data complementary features and establishing a cross-source target unified representation space, the bottleneck that a single model is insufficient in adaptability to different image source features is solved, the relevance of target texture, geometry and radiation characteristics in a multi-source image is emphatically mined, a multi-level feature interaction and adaptive fusion mechanism is designed, and the multi-source image feature fusion is realized. The semantic consistency understanding of the model on the cross-source target is enhanced, a visual large model with strong generalization ability is finally constructed, a high-precision supervision signal is provided for an on-satellite lightweight small model, and the stability of the cross-source remote sensing image target detection performance is ensured. According to the method, the dependence on the target domain annotation data can be reduced, so that the model keeps good generalization ability.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a method for constructing a large visual model based on multi-source remote sensing images, corresponding devices, electronic equipment and computer-readable storage media. Background Art

[0002] The current field of remote sensing image interpretation mainly uses dedicated models for single sensor data, such as convolutional neural networks for optical images, polarization decomposition algorithms for SAR images, and temperature feature classifiers for infrared images. Existing technologies usually independently design model architectures for specific sensor types (such as optical, SAR, or infrared), and use large amounts of labeled data to train target recognition or ground object classification models in a single scenario. Some improved technologies attempt to enhance the cross-source adaptability of models through transfer learning or data augmentation, but the core methods still use single sensor data as input and have not broken through the limitations of collaborative modeling of multi-source data. Under the existing technology system, parameters are not shared and feature expressions are incompatible between different sensor models, resulting in the need to retrain or fine-tune the model for cross-source image interpretation.

[0003] The workflow of existing technologies is divided into three stages: data preprocessing, feature extraction, and classification and recognition. For optical images, a convolutional neural network based on RGB channels is used to extract spatial spectral features; SAR image processing relies on a coherent speckle noise suppression module and a polarization scattering feature extractor to model target characteristics through backscattering intensity; infrared images achieve target detection through temperature threshold segmentation and thermal radiation feature analysis. In terms of model structure, existing technologies usually adopt a staged processing architecture: after the sensor data is processed through independent input channels, the results are output through a fully connected layer or a decision fusion module. Transfer learning methods achieve cross-source adaptation by freezing some pre-trained layers and replacing the top-level classifier, but do not fundamentally solve the problem of joint expression of multi-source features. The existing technologies have the following technical defects:

[0004] First, the feature expression of a single sensor model is limited to specific physical properties (such as the reflectivity of optical images and the scattering characteristics of SAR), and it is unable to integrate the complementary information of multi-source data, resulting in insufficient target feature diversity in cross-source scenarios.

[0005] Second, the model architecture lacks a unified multi-source feature fusion mechanism. Different sensor data must be processed independently before shallow fusion, making it difficult to mine deep correlations between cross-modal features.

[0006] Third, existing transfer learning methods rely on labeled data in the target domain. When processing new sensors or cross-source combined data, re-labeling and training are required, resulting in high model generalization costs. In addition, traditional models have a limited number of parameters and insufficient ability to learn the high-dimensional and heterogeneous features of multi-source data, making it difficult to support large-scale cross-domain generalization needs.

[0007] To sum up, the feature expression of a single sensor model in the existing technology is limited to specific physical characteristics and cannot fuse the complementary information of multi-source data, resulting in insufficient target feature diversity in cross-source scenarios and a lack of a unified multi-source feature fusion mechanism in the model architecture. In order to solve this problem, the applicant has made corresponding explorations. Summary of the Invention

[0008] The purpose of this application is to solve the above problems and provide a method for constructing a large visual model based on multi-source remote sensing images, corresponding devices, electronic equipment and computer-readable storage medium.

[0009] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0010] A method for constructing a large visual model based on multi-source remote sensing images is proposed to meet one of the purposes of this application, including:

[0011] Acquire a radar remote sensing image containing each target object and its corresponding infrared remote sensing image and optical remote sensing image, and use a preset multi-scale masking strategy to generate a random mask matrix to dynamically mask specific areas of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image to determine a randomly masked image of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image;

[0012] In the feature extraction network of the encoder, a local window self-attention mechanism is adopted based on the Swin Transformer model according to the encoded data corresponding to the randomly masked image of the optical remote sensing image to capture the local detail features corresponding to each target object in the optical remote sensing image. A global attention mechanism is adopted based on the Vision Transformer model according to the encoded data corresponding to the randomly masked images of the radar remote sensing image and the infrared remote sensing image to extract the global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image.

[0013] In the cross-modal fusion layer of the encoder, a gated cross-attention mechanism is used to fuse the local detail features corresponding to each target object with the global semantic information corresponding to the target object to determine the multi-source fusion features corresponding to each target object;

[0014] The multi-source fusion features are reconstructed in the decoder to determine the reconstruction loss between the reconstructed image and the original image, and training is performed to a certain number of iterations to minimize the reconstruction loss, thereby completing the construction and pre-training of a large visual model based on multi-source remote sensing images;

[0015] The pre-trained encoder is fine-tuned by adding a specific task head and then applied to downstream interpretation tasks, including but not limited to target detection tasks, target recognition tasks or change detection tasks, where the task head includes but is not limited to target recognition head, target detection head, and change detection head.

[0016] Optionally, the step of obtaining a radar remote sensing image containing each target object and its corresponding infrared remote sensing image and optical remote sensing image includes:

[0017] In response to the data preprocessing instruction, for the optical remote sensing image, geometric transformation is used to eliminate the interference of illumination and viewing angle, and image denoising technology is combined to suppress cloud and noise;

[0018] For radar remote sensing images, speckle noise suppression and terrain radiation correction are used to extract backscatter coefficient features, and vertex suppression algorithm is used to eliminate overlap and shadow area interference;

[0019] For infrared remote sensing images, the temperature difference contrast between the target object and the background is enhanced based on thermal radiation calibration and dynamic range compression.

[0020] Optionally, the step of capturing local detail features corresponding to each target object in the optical remote sensing image based on the encoded data corresponding to the randomly masked image of the optical remote sensing image using a local window self-attention mechanism based on the Swin Transformer model includes:

[0021] Inputting the encoded data corresponding to the random masked image of the optical remote sensing image into the SwinTransformer model, dividing the optical remote sensing image into a plurality of local windows according to a preset size, wherein each local window represents a local area;

[0022] Within each local window, the Swin Transformer model uses a self-attention mechanism to perform local self-attention calculations on the unmasked area features based on the encoded data corresponding to the randomly masked image of the optical remote sensing image to determine the relationship between pixels within the window. Each pixel is assigned different weights based on the characteristics of other pixels to capture the spatial relationship and detailed features between the target objects within the local area.

[0023] Optionally, the step of using a gated cross-attention mechanism to perform feature fusion on the local detail features corresponding to each target object and the global semantic information corresponding to the target object to determine the multi-source fusion features corresponding to each target object includes:

[0024] Acquire local detail features corresponding to each target object in the optical remote sensing image and global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image, so as to generate a query matrix corresponding to the local detail features and a key matrix corresponding to the global semantic information;

[0025] A preset Sigmoid gating function is used to perform weighting according to the matching degree of the query matrix and the key matrix, so that the features of different modalities are weighted and fused in different areas;

[0026] The gated weighted features are concatenated to integrate the information of each modality into the same space to preserve their local detail features and global semantic information;

[0027] After feature splicing, the spliced ​​feature maps are recalibrated through channel recalibration to determine a unified cross-modal representation, where the cross-modal representation contains local detail features and global semantic information of each target object.

[0028] Optionally, the pre-trained encoder is fine-tuned by adding a specific task head and then applied to downstream interpretation tasks, including but not limited to object detection tasks, object recognition tasks, or change detection tasks, including:

[0029] For target detection tasks, the target location and category probability are generated through the anchor box adaptation mechanism;

[0030] For the change detection task, the input data is processed by the classification network, and the probability distribution of each pixel belonging to different change categories is output. Then, the change category of the pixel is determined based on the maximum probability.

[0031] For the target recognition task, a fully connected layer classification head is designed to determine the category of the target with the highest probability based on the probability distribution.

[0032] Optionally, the multi-scale masking strategy includes one or any multiple of small-size random masking, large-size random masking, small-size channel random masking, and large-size channel random masking.

[0033] Optionally, the radar remote sensing image includes a SAR image;

[0034] The reconstruction loss represents the pixel-by-pixel mean square error between the reconstructed image output by the decoder and the original image of the radar remote sensing image, infrared remote sensing image and optical remote sensing image corresponding to the reconstructed image;

[0035] The reconstructed image represents a remote sensing image that fuses local detail features corresponding to each target object and global semantic information.

[0036] A device for constructing a large visual model based on multi-source remote sensing images is provided to meet another purpose of the present application, comprising:

[0037] a masked image determination module configured to obtain a radar remote sensing image containing each target object and its corresponding infrared remote sensing image and optical remote sensing image, and to use a preset multi-scale masking strategy to generate a random mask matrix to dynamically mask specific areas of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image to determine a random masked image of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image;

[0038] The semantic information extraction module is configured to, in a feature extraction network of an encoder, capture local detail features corresponding to each target object in the optical remote sensing image based on the encoded data corresponding to the randomly masked image of the optical remote sensing image using a local window self-attention mechanism based on a Swin Transformer model, and extract global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image based on the encoded data corresponding to the randomly masked image of the radar remote sensing image and the infrared remote sensing image using a global attention mechanism based on a Vision Transformer model;

[0039] a fusion feature determination module configured to use a gated cross-attention mechanism in a cross-modal fusion layer of the encoder to perform feature fusion on the local detail features corresponding to each target object and the global semantic information corresponding to the target object to determine a multi-source fusion feature corresponding to each target object;

[0040] An encoder pre-training module is configured to reconstruct the multi-source fusion features in the decoder to determine the reconstruction loss between the reconstructed image and the original image, train to a certain number of iterations to minimize the reconstruction loss, and complete the construction and pre-training of a large visual model based on multi-source remote sensing images;

[0041] The downstream task execution module is configured to add a specific task head to the pre-trained encoder for fine-tuning before applying it to downstream interpretation tasks, which include but are not limited to target detection tasks, target recognition tasks or change detection tasks, wherein the task head includes but is not limited to target recognition head, target detection head, and change detection head.

[0042] An electronic device provided to meet another purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method for constructing a large visual model based on multi-source remote sensing images described in the present application.

[0043] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the method for constructing a large visual model based on multi-source remote sensing images in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0044] Compared with the existing technology, this application addresses the problems in which the feature expression of a single sensor model in the existing technology is limited to specific physical characteristics, cannot fuse the complementary information of multi-source data, resulting in insufficient target feature diversity in cross-source scenarios, and lacks a unified multi-source feature fusion mechanism in the model architecture. This application includes but is not limited to the following beneficial effects:

[0045] First, this application breaks through the limitations of single-sensor feature representation. Through collaborative modeling of multi-source remote sensing imagery, this application uses data from different sensor types (optical, SAR, and infrared) for deep feature mining, fully integrating the complementary information from various sensor data types. Through this cross-source collaborative modeling, the model can capture the multidimensional characteristics of the target object, no longer limited to the physical characteristics of a single sensor, effectively improving the accuracy of target recognition and object classification.

[0046] Secondly, existing technologies usually adopt a staged processing architecture, in which different sensor data are processed independently and then shallowly fused. This approach often leads to information loss and cannot fully explore the deep correlation of cross-modal features. The architecture based on Swin Transformer and Vision Transformer proposed in this application can process the local detail features of optical images and the global semantic information of radar and infrared images respectively in the encoder, and realize deep feature fusion through the gated cross-attention mechanism in the cross-modal fusion layer. The unified large model architecture enables end-to-end joint learning of multi-source data, eliminates the information loss in traditional technologies, and greatly improves the performance of cross-source remote sensing image interpretation.

[0047] Third, existing transfer learning methods usually rely on labeled data in the target domain. When new sensors or cross-source combination data appear, the model needs to be re-labeled and trained, which limits the generalization ability of the model. Especially when faced with multi-source heterogeneous data, the adaptability of existing models is poor. This application can reduce the dependence on labeled data in the target domain by designing a self-supervised pre-training strategy, so that the model can maintain good generalization capabilities in cross-source and cross-domain scenarios. Especially in the absence of sensor or cross-source combination data, the zero-sample capability of the model has been significantly improved. This makes this application highly adaptable and flexible, and can cope with ever-changing remote sensing image data and scenes.

[0048] Fourthly, existing traditional models often have limited parameters and insufficient ability to learn high-dimensional features when processing multi-source remote sensing data, which makes it difficult to meet the needs of large-scale data processing. This application introduces powerful network architectures such as SwinTransformer and Vision Transformer, which can effectively learn in large-scale parameter spaces. These models have efficient feature extraction capabilities, especially when processing multi-source heterogeneous data, and can capture rich contextual information and cross-modal features to meet the needs of large-scale remote sensing data processing and cross-domain generalization.

[0049] Fifth, traditional training methods usually rely on large amounts of labeled data for supervised learning. However, in the case of cross-source data, labeled data is difficult to obtain and expensive, and the performance of existing models in this regard is also limited. This application uses a self-supervised pre-training mechanism to enable the model to automatically learn the deep features of multi-source remote sensing images without labeled data. Self-supervised learning not only reduces the dependence on labeled data, but also improves the model's ability to abstract potential patterns in remote sensing data, further enhancing its feature expression and recognition capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0051] Figure 1 Schematic diagram of the process of constructing a large visual model based on multi-source remote sensing images in an embodiment of the present application;

[0052] Figure 2 This is an exemplary network architecture of a system for building a large visual model based on multi-source remote sensing images in an embodiment of the present application;

[0053] Figure 3 This is a functional block diagram of a device for constructing a large visual model based on multi-source remote sensing images in an embodiment of the present application;

[0054] Figure 4 Schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0055] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0056] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0058] It will be understood by those skilled in the art that the terms "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, i.e., devices that only have wireless signal receivers without transmission capabilities, and devices that have receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers and tablet computers, which have single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service), which may combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices, which have and / or include a radio frequency receiver. As used herein, the terms "client," "terminal," or "terminal device" may be portable, transportable, or installed in a vehicle (air, sea, and / or land), or may be adapted and / or configured to operate locally and / or in a distributed manner at any other location on Earth and / or in space. As used herein, the terms "client," "terminal," or "terminal device" may also refer to a communication terminal, an Internet terminal, or a music / video playback terminal, such as a PDA, an MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or may include a smart TV, a set-top box, or other device.

[0059] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit loads the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0060] It should be noted that the concept of "server" referred to in this application can also be extended to server clusters. Based on the network deployment principles understood by those skilled in the art, the servers described should be logically divided. In physical space, these servers can be independent of each other but callable through interfaces, or integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method of this application.

[0061] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for implementation.

[0062] Unless expressly specified otherwise, the neural network models referenced or may be referenced in this application may be deployed on a remote server and remotely called on the client, or may be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence may be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0063] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as they are suitable for being called by the technical solution of this application.

[0064] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus exhibit commonality, unless otherwise specified, these methods can be independently executed. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept. Therefore, concepts with the same expression, as well as concepts that are appropriately transformed for convenience despite different expression, should be understood as equivalent.

[0065] Unless expressly stated to be mutually exclusive, the various embodiments disclosed in this application may be cross-combined with the relevant technical features of the various embodiments to flexibly construct new embodiments, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or resolve certain deficiencies in the prior art. Those skilled in the art should be aware of such flexibility.

[0066] See also Figure 1 and Figure 2 In one embodiment, the method for constructing a large visual model based on multi-source remote sensing images of the present application includes:

[0067] Step S10: obtaining a radar remote sensing image containing each target object and its corresponding infrared remote sensing image and optical remote sensing image, and using a preset multi-scale masking strategy to generate a random mask matrix to dynamically mask specific areas of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image to determine a randomly masked image of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image;

[0068] The visual large model construction system based on multi-source remote sensing images in the terminal device can obtain radar remote sensing images containing various target objects and their corresponding infrared remote sensing images and optical remote sensing images, and use a preset multi-scale masking strategy to generate a random mask matrix to dynamically mask specific areas of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image to determine the randomly masked image of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image; wherein the randomly masked image represents the remote sensing image after the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image are masked using a random masking matrix; the multi-scale masking strategy includes one or any multiple of small-size random masking, large-size random masking, small-size channel random masking, and large-size channel random masking; the target objects include buildings, roads, vegetation, etc., and the radar remote sensing images include SAR (synthetic aperture radar) images. Specifically, SAR images use synthetic aperture radar technology to obtain information about ground objects by transmitting microwave signals and receiving reflected echo signals.

[0069] Specifically, after obtaining radar remote sensing images containing each target object and their corresponding infrared remote sensing images and optical remote sensing images, multi-scale masking strategies such as small-scale random masking, large-scale random masking, small-scale channel random masking, and large-scale channel random masking can be used to generate a random mask matrix to dynamically mask specific areas of the radar, infrared, and optical remote sensing images. In this way, a random masked image can be generated that specifies which areas in the radar remote sensing image, infrared remote sensing image, and optical remote sensing image will be masked to simulate data missing or noise. The mask matrix is ​​usually a binary matrix that marks which pixels need to be masked, usually 0, indicating that the area is masked, while other pixels remain unchanged. The randomly masked images of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image will be used to process the input image to help the model learn how to effectively extract features when some information is missing.

[0070] In some embodiments, small-scale random masking refers to randomly selecting small, typically small, areas within an image for masking (occlusion). The masked area is set to zero or a constant value to simulate missing information. This method is often used for data augmentation or training deep learning models to make the model more robust to localized loss. Large-scale random masking is similar to small-scale random masking, but the masked area is larger and may cover a portion of the image or multiple target areas. The size of the mask is randomly selected. Small-scale channel random masking refers to random masking within the channel dimension of the image. That is, some channels are randomly selected and then masked over a small range. The masked channels may randomly set to zero or change the pixel values ​​at a certain location. Large-scale channel random masking is similar to small-scale channel random masking, but the masked channels are larger, potentially masking certain important channels of the entire image or multiple channels simultaneously. This multi-scale masking strategy is primarily used to enhance the robustness of deep learning models, particularly in image processing and remote sensing image analysis, to simulate different types of data loss or occlusion. By introducing these masking strategies during the training process, the model can learn how to deal with the situation where some information is missing, thereby improving its generalization ability in practical applications.

[0071] In some embodiments, the step of obtaining a radar remote sensing image containing each target object and its corresponding infrared remote sensing image and optical remote sensing image includes:

[0072] Step S101: responding to a data preprocessing instruction, for the optical remote sensing image, using geometric transformation to eliminate illumination and viewing angle interference, and combining image denoising technology to suppress cloud and noise;

[0073] Step S102: For radar remote sensing images, speckle noise suppression and terrain radiometric correction are used to extract backscatter coefficient features, and a vertex suppression algorithm is used to eliminate interference from overlapping and shadow areas.

[0074] Step S103: For the infrared remote sensing image, the temperature difference contrast between the target object and the background is enhanced based on thermal radiation calibration and dynamic range compression.

[0075] Specifically, geometric transformations include geometric correction and radiometric correction, while image denoising techniques include mean filtering. Optical remote sensing images are easily affected by lighting conditions (such as the sun's angle) and viewing angle (such as the shooting angle), resulting in overbrightness or underdarkness in certain areas of the image, which in turn affects target recognition and detection. Geometric transformations eliminate these effects by applying geometric corrections (such as radiometric correction and orthorectification) to the image. Geometric correction adjusts the image's geometry to conform to the geographic coordinate system, eliminating image distortion caused by factors such as the sensor's viewing angle and terrain. Radiometric correction adjusts the image's lighting conditions to enable effective comparisons across different times, locations, and weather conditions, eliminating missing information and improving its generalization capabilities in practical applications. Clouds often obstruct or interfere with the observation of target objects in optical remote sensing images. Cloud removal techniques (such as cloud detection and removal based on multispectral imagery) can effectively mitigate the effects of clouds. Optical remote sensing images may be subject to noise from sensor noise or atmospheric influences (such as atmospheric scattering). Denoising techniques (such as median filtering and wavelet denoising) are used to remove this noise and improve image quality.

[0076] Furthermore, radar remote sensing images often contain speckle noise, a type of random noise caused by scattering characteristics that affects image quality. Speckle noise suppression techniques (such as filtering, mean filtering, and band shifting) help reduce noise interference and extract clearer target information. Radar images are affected by terrain, especially in mountainous or rugged areas. Topographic variations can lead to differences in radiation intensity at different locations. Terrain radiometric correction techniques (such as using digital elevation models) can correct for this terrain effect, thereby more accurately reflecting the backscattering characteristics of the surface.

[0077] Radar remote sensing images acquire ground object information by analyzing the reflection of radar waves (backscatter) from surface objects. By correcting the backscatter characteristics, the physical characteristics of the target object, such as surface roughness, humidity, and vegetation cover, can be extracted. For example, by processing SAR images, the backscatter coefficients of different ground objects can be obtained, which helps to distinguish different types of ground objects. In radar remote sensing images, due to the shooting angle of the sensor or the influence of the terrain, there may be overlapping or shadowed areas of ground objects, which may not provide effective ground object information. The vertex suppression algorithm is a technology that suppresses the influence of these areas by analyzing the geometric shape of objects in the image. The vertex suppression algorithm improves the usability and quality of the image by identifying and removing these areas.

[0078] Furthermore, infrared remote sensing images acquire information by receiving thermal radiation from ground objects. However, due to interference from the atmosphere and the sensor, raw infrared images may exhibit deviations. Thermal radiation calibration technology is used to correct images according to known radiation standards to ensure the accuracy of thermal radiation in the image. The calibration process typically involves matching the digital values ​​of the image with the actual radiation values ​​(or temperature) of the ground objects to obtain accurate temperature information. Infrared images can have a very wide temperature range, making it difficult to distinguish targets with small temperature differences from the background. Dynamic range compression adjusts the brightness and contrast of the image to enhance the temperature contrast, thereby enhancing the difference between the target object and the background. This technology helps improve the ability of low-temperature targets (such as human temperature and vegetation) to be recognized against the background.

[0079] Step S20: In the feature extraction network of the encoder, a local window self-attention mechanism is adopted based on the Swin Transformer model to capture local detail features corresponding to each target object in the optical remote sensing image according to the encoded data corresponding to the randomly masked image of the optical remote sensing image; and a global attention mechanism is adopted based on the Vision Transformer model to extract global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image according to the encoded data corresponding to the randomly masked image of the radar remote sensing image and the infrared remote sensing image;

[0080] Acquire a radar remote sensing image containing each target object and its corresponding infrared remote sensing image and optical remote sensing image, use a preset multi-scale masking strategy to generate a random mask matrix to dynamically mask specific areas of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image to determine the randomly masked images of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image. Then, in a feature extraction network of an encoder, based on a SwinTransformer model, a local window self-attention mechanism is used to capture local detail features corresponding to each target object in the optical remote sensing image according to the encoded data corresponding to the randomly masked image of the optical remote sensing image. Based on a Vision Transformer model, a global attention mechanism is used to extract global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image according to the encoded data corresponding to the randomly masked images of the radar remote sensing image and the infrared remote sensing image.

[0081] In some embodiments, the step of capturing local detail features corresponding to each target object in the optical remote sensing image based on the encoded data corresponding to the randomly masked image of the optical remote sensing image using a local window self-attention mechanism based on the Swin Transformer model includes:

[0082] Step S201: inputting the encoded data corresponding to the random masked image of the optical remote sensing image into the SwinTransformer model, and dividing the optical remote sensing image into a plurality of local windows according to a preset size, wherein each local window represents a local area;

[0083] Step S202: Within each local window, the Swin Transformer model uses a self-attention mechanism to perform local self-attention calculations on the unmasked area features based on the encoded data corresponding to the randomly masked image of the optical remote sensing image to determine the relationship between pixels within the window. Each pixel is assigned different weights based on the features of other pixels to capture the spatial relationship and detailed features between the target objects within the local area.

[0084] Specifically, the Swin Transformer is a deep learning model based on the self-attention mechanism. It uses a "windowed" self-attention mechanism, making it particularly suitable for tasks with localized and large-scale data. The core advantage of the Swin Transformer lies in its ability to process images through localized windowing, enabling the model to efficiently learn local features.

[0085] When the encoded data corresponding to the random masked image of the optical remote sensing image is input into SwinTransformer, the optical remote sensing image is first divided into multiple local windows according to a preset size (eg, 7x7 or 8x8), each window representing a local area.

[0086] Within each window, the Swin Transformer uses a self-attention mechanism to calculate the relationships between pixels within the window, assigning different weights to each pixel based on the characteristics of other pixels, thereby capturing the spatial relationships and characteristics within the local region. Based on the random masking of the optical remote sensing image, the corresponding input feature data is masked. During training, the features of the masked areas are "ignored", and local self-attention calculations rely solely on the features of the unmasked areas. This strategy helps the model enhance its robustness to locally missing or occluded areas.

[0087] Within each local window, the Swin Transformer model learns local region features based on a self-attention mechanism. This self-attention calculation allows the model to focus on key information within the local region, effectively capturing the detailed features of the target object, despite potential information loss in some areas due to masking. Through a multi-level local window attention mechanism, the model not only learns the detailed features of each local region but also integrates these local features at a deeper level, capturing contextual information across local regions and achieving a more comprehensive understanding of the target object.

[0088] The local window self-attention mechanism based on the Swin Transformer model can effectively capture the local details of target objects in optical remote sensing images. Through masking strategies and self-attention mechanisms, the model can learn useful local features from unmasked areas even when some areas are missing, and can effectively perform cross-modal feature fusion, thereby improving the performance of remote sensing image analysis tasks.

[0089] In some embodiments, the Vision Transformer model uses a global attention mechanism to extract global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image based on the encoded data corresponding to the randomly masked image of the radar remote sensing image and the infrared remote sensing image; specifically, the radar remote sensing image and the infrared remote sensing image are first divided into small blocks (patches), and each radar remote sensing image or infrared remote sensing image is divided into blocks of a fixed size (for example, 16x16 or 32x32 small blocks), so that each small block becomes an input unit that can be processed by the Vision Transformer model; each small block will be mapped to an embedded vector through a linear transformation. In this way, each small block of the image (for example, a 16x16 image block) will be converted into a one-dimensional vector representation as the input of the subsequent Vision Transformer model.

[0090] In the Vision Transformer model, the self-attention mechanism globally computes relationships between image patches, with each patch updating its representation through interaction with other patches. This means that each patch dynamically adjusts its weight based on global information and weights it according to the characteristics of other patches. For each patch, the Vision Transformer model calculates an attention weight relative to all other patches. These weights are calculated using an inner product operation and a softmax function based on the content and position of the patch. The representation of each patch is aggregated with the weighted information from other patches to form a globally context-aware representation.

[0091] Based on the encoded data corresponding to the randomly masked images of the radar remote sensing image and the infrared remote sensing image, the masked areas can be ignored or assigned lower attention weights when calculating self-attention. This allows the model to avoid over-reliance on masked areas and focus on unmasked areas. Through the multi-layer structure of VisionTransformer, the network updates and adjusts image patches at each layer to gradually extract high-level semantic features of the image.

[0092] Through a multi-layer global attention mechanism, the Vision Transformer model gradually integrates information from different image patches and learns the dependencies between patches. In particular, for radar and infrared remote sensing imagery, the Vision Transformer model is able to learn different types of features, such as the shape, size, motion characteristics (for radar remote sensing imagery), and thermal radiation information (for infrared remote sensing imagery).

[0093] Finally, after processing through the multi-layer self-attention mechanism, the model will obtain a set of high-dimensional global feature vectors, which represent the global semantic information of each target object in the image. This information can reflect the specific location, category, structure, etc. of the target object.

[0094] Step S30: In the cross-modal fusion layer of the encoder, a gated cross-attention mechanism is used to fuse the local detail features corresponding to each target object with the global semantic information corresponding to the target object to determine a multi-source fusion feature corresponding to each target object;

[0095] Based on the Vision Transformer model, a global attention mechanism is used to extract global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image based on the encoded data corresponding to the randomly masked image. Then, in the cross-modal fusion layer of the encoder, a gated cross-attention mechanism is used to fuse the local detail features corresponding to each target object with the global semantic information corresponding to the target object to determine a multi-source fusion feature corresponding to each target object.

[0096] In some embodiments, the step of using a gated cross-attention mechanism to fuse local detail features corresponding to each target object with global semantic information corresponding to the target object to determine a multi-source fusion feature corresponding to each target object includes:

[0097] Step S301: Acquire local detail features corresponding to each target object in the optical remote sensing image and global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image, so as to generate a query matrix corresponding to the local detail features and a key matrix corresponding to the global semantic information;

[0098] Step S302: Using a preset Sigmoid gating function to perform weighting according to the matching degree of the query matrix and the key matrix, so that features of different modalities are weightedly fused in different regions;

[0099] Step S303: splicing the gated weighted features together to integrate the information of each modality into the same space to preserve their local detail features and global semantic information;

[0100] Step S304: After the feature splicing, the spliced ​​feature map is recalibrated through channel recalibration to determine a unified cross-modal representation, wherein the cross-modal representation contains local detail features and global semantic information of each target object.

[0101] Specifically, in the encoder's cross-modal fusion layer, the main purpose is to fuse the local detail features extracted from radar, infrared, and optical remote sensing images with global semantic information to obtain a multi-source fusion feature. The gated cross-modal attention mechanism is a cross-modal attention mechanism that introduces a gating mechanism. It optimizes the fusion of multi-source data by effectively controlling the interaction between local detail features and global semantic information. Its workflow includes:

[0102] Local detail features and global semantic information are mapped to spaces of the same dimension respectively, in order to ensure that local detail features and global semantic information can be fused in the same feature space. The core of the gating mechanism is to dynamically adjust the degree of fusion of local detail features and semantic information by learning a gating variable (such as attention weight). Specifically, the gating mechanism generates a "gating coefficient" for each target object, which controls the fusion ratio of local detail features extracted from optical remote sensing images and semantic information extracted from radar and infrared images. The gating coefficient can be obtained by calculating the similarity of the input features (such as through a self-attention mechanism or other metrics).

[0103] Then, the gating coefficient is used to weight the local detail features and semantic information, allowing them to be selectively fused according to the actual situation. This step changes dynamically for different target objects to ensure that the feature fusion method for each target object is targeted.

[0104] Building on the gating mechanism, the cross-attention mechanism further strengthens the interaction between different modalities. Under this mechanism, local detail features from optical images weight semantic information from radar and infrared images, and vice versa. The goal of cross-attention is to achieve more efficient information fusion by alternately focusing on features from different modalities.

[0105] Finally, local detail features and global semantic information are fused through a cross-attention mechanism and a gating mechanism to generate the final multi-source fused features. These fused features contain detailed information about the target object as well as semantic context, providing a more comprehensive and accurate feature representation for downstream tasks such as object detection and change detection.

[0106] Furthermore, local detail features help capture image details, while global semantic information provides an understanding of overall structure and relationships. The gated cross-attention mechanism flexibly combines these two types of information, improving the quality of the fused features. After feature fusion, each target object can simultaneously utilize local details and global semantic information, resulting in a more precise feature representation.

[0107] The gated cross-attention mechanism dynamically adjusts the fusion of local detail features and global semantic information, using a gating mechanism to control the weighting of cross-attention, ensuring that information from different modalities is effectively combined, thereby generating high-quality multi-source fusion features. These features provide richer and more accurate feature representations for subsequent remote sensing image analysis tasks.

[0108] Step S40: reconstructing the multi-source fusion features in the decoder to determine the reconstruction loss between the reconstructed image and the original image, training to a certain number of iterations to minimize the reconstruction loss, and completing the construction and pre-training of the visual large model based on the multi-source remote sensing image;

[0109] In the cross-modal fusion layer of the encoder, a gated cross-attention mechanism is used to fuse the local detail features corresponding to each target object with the global semantic information corresponding to the target object to determine the multi-source fusion features corresponding to each target object. After that, the multi-source fusion features are reconstructed in the decoder to determine the reconstruction loss between the reconstructed image and the original image. The training is performed to a certain number of iterations to minimize the reconstruction loss, thereby completing the construction of a large visual model based on multi-source remote sensing images and its pre-training. The reconstruction loss represents the pixel-by-pixel mean square error between the reconstructed image output by the decoder and the original images of the radar remote sensing image, infrared remote sensing image, and optical remote sensing image corresponding to the reconstructed image. The reconstructed image represents a remote sensing image that fuses the local detail features corresponding to each target object and the global semantic information.

[0110] Specifically, in the decoder, the multi-source fused features are reconstructed. The purpose of this process is to convert the fused features back to the form of the original image to evaluate the performance of the model. By calculating the difference between the reconstructed image and the original image (i.e., the reconstruction loss), the model can learn how to better extract and fuse information from different modalities. Through iterative training, the model is gradually optimized until the reconstruction loss is as small as possible, completing the construction of a large visual model based on multi-source remote sensing images and its pre-training. The training at this stage is to pre-train the encoder so that it can show better generalization capabilities in subsequent downstream interpretation tasks.

[0111] Step S50: Add a specific task head to the pre-trained encoder for fine-tuning and apply it to downstream interpretation tasks, which include but are not limited to target detection tasks, target recognition tasks or change detection tasks, wherein the task head includes but is not limited to target recognition head, target detection head, and change detection head.

[0112] The multi-source fusion features are reconstructed in the decoder to determine the reconstruction loss between the reconstructed image and the original image, and trained to a certain number of iterations to minimize the reconstruction loss. After completing the construction of the large visual model based on multi-source remote sensing images and its pre-training, the pre-trained encoder is fine-tuned by adding a specific task head to the pre-trained encoder for application to downstream interpretation tasks, including but not limited to target detection tasks, target recognition tasks or change detection tasks, wherein the task head includes but is not limited to a target recognition head, a target detection head, and a change detection head. Specifically, the pre-trained encoder can be used for a series of downstream remote sensing image interpretation tasks, such as target detection, target recognition, change detection, etc. These tasks require the model to be able to understand and recognize various targets in remote sensing images, such as buildings, roads, vegetation, etc.

[0113] In some embodiments, a dedicated task output head is designed based on task requirements to perform specific downstream tasks. After fine-tuning the pre-trained encoder with the specific task head, it is applied to downstream interpretation tasks, including but not limited to object detection tasks, object recognition tasks, or change detection tasks, including the following steps:

[0114] For target detection tasks, the target location and category probability are generated through the anchor box adaptation mechanism;

[0115] For the change detection task, the input data is processed by the classification network, and the probability distribution of each pixel belonging to different change categories is output. Then, the change category of the pixel is determined based on the maximum probability.

[0116] For the target recognition task, a fully connected layer classification head is designed to determine the category of the target with the highest probability based on the probability distribution.

[0117] In some embodiments, the multi-source remote sensing image collection is represented as:

[0118]

[0119] Among them, X (k) It represents the original input image data of the kth modality, K represents the modality number, k=1,2,3,....K, for example, k=1 represents SAR image, k=2 can represent infrared remote sensing image, and k=3 represents optical remote sensing image.

[0120] The expression for generating random masked images is:

[0121]

[0122] in, represents the image after the mask operation, and M represents the random mask matrix.

[0123] Based on the Swin Transformer model, a local window self-attention mechanism is used to capture the local detail features corresponding to each target object in the optical remote sensing image according to the encoded data corresponding to the random masked image of the optical remote sensing image. The local window self-attention is calculated as:

[0124]

[0125] Among them, Q represents the query matrix, K represents the key matrix, V represents the value matrix, Attention(Q, W) represents the calculation result of the local window self-attention, the Softmax function is used to calculate the similarity between Q and K, and finally the attention output is obtained after weighting the similarity by V, QW Q (KW K ) T Represents the matrix multiplication of Q and K, used to calculate their similarity; VW V Represents the result after weighting of the V matrix.

[0126] Based on the Vision Transformer model, a global attention mechanism is used to extract the global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image according to the encoded data corresponding to the random masked image of the radar remote sensing image and the infrared remote sensing image. The global attention output is:

[0127] Z (k) =LayerNorm(MultiHead(E (k) )+E (k) ),

[0128] Among them, LayerNorm represents layer normalization, MultiHead represents the multi-head attention mechanism, which is used to enhance the model's ability to pay attention to different information sources. It splices or weightedly merges the outputs of multiple attention heads. (k) represents the embedding representation of the kth modality, which is obtained by Perform embedding operation to obtain .

[0129] In the cross-modal fusion layer of the encoder, a gated cross-attention mechanism is used to fuse the local detail features corresponding to each target object with the global semantic information corresponding to the target object to determine the multi-source fusion features corresponding to each target object. The gated cross-attention mechanism is defined as:

[0130]

[0131] Here, σ represents the Sigmoid gate function, which is used to weight the interaction between different modalities of cross-attention. Its output is a value between 0 and 1, which is used to control the influence of each modality. It represents the feature concatenation operation, which is used to fuse information from different modalities into a unified representation. The concatenated feature map can contain information from multiple modalities. Represents the feature map after gated.

[0132] The expression for channel recalibration is:

[0133]

[0134] in, represents the feature map after channel recalibration; γ represents the scaling parameter in the channel recalibration operation; β represents the translation parameter in the channel recalibration operation.

[0135] The reconstruction loss between the reconstructed image output by the decoder and each original image is expressed as:

[0136]

[0137] As can be seen from the above embodiments, compared with the prior art, the present application addresses the problems in the prior art where the feature expression of a single sensor model is limited to specific physical characteristics, and is unable to fuse complementary information from multi-source data, resulting in insufficient target feature diversity in cross-source scenarios, and the model architecture lacks a unified multi-source feature fusion mechanism. The present application includes but is not limited to the following beneficial effects:

[0138] First, this application breaks through the limitations of single-sensor feature representation. Through collaborative modeling of multi-source remote sensing imagery, this application uses data from different sensor types (optical, SAR, and infrared) for deep feature mining, fully integrating the complementary information from various sensor data types. Through this cross-source collaborative modeling, the model can capture the multidimensional characteristics of the target object, no longer limited to the physical characteristics of a single sensor, effectively improving the accuracy of target recognition and object classification.

[0139] Secondly, existing technologies usually adopt a staged processing architecture, in which different sensor data are processed independently and then shallowly fused. This approach often leads to information loss and cannot fully explore the deep correlation of cross-modal features. The architecture based on Swin Transformer and Vision Transformer proposed in this application can process the local detail features of optical images and the global semantic information of radar and infrared images respectively in the encoder, and realize deep feature fusion through the gated cross-attention mechanism in the cross-modal fusion layer. The unified large model architecture enables end-to-end joint learning of multi-source data, eliminates the information loss in traditional technologies, and greatly improves the performance of cross-source remote sensing image interpretation.

[0140] Third, existing transfer learning methods usually rely on labeled data in the target domain. When new sensors or cross-source combination data appear, the model needs to be re-labeled and trained, which limits the generalization ability of the model. Especially when faced with multi-source heterogeneous data, the adaptability of existing models is poor. This application can reduce the dependence on labeled data in the target domain by designing a self-supervised pre-training strategy, so that the model can maintain good generalization capabilities in cross-source and cross-domain scenarios. Especially in the absence of sensor or cross-source combination data, the zero-sample capability of the model has been significantly improved. This makes this application highly adaptable and flexible, and can cope with ever-changing remote sensing image data and scenes.

[0141] Fourthly, existing traditional models often have limited parameters and insufficient ability to learn high-dimensional features when processing multi-source remote sensing data, which makes it difficult to meet the needs of large-scale data processing. This application introduces powerful network architectures such as SwinTransformer and Vision Transformer, which can effectively learn in large-scale parameter spaces. These models have efficient feature extraction capabilities, especially when processing multi-source heterogeneous data, and can capture rich contextual information and cross-modal features to meet the needs of large-scale remote sensing data processing and cross-domain generalization.

[0142] Fifth, traditional training methods usually rely on large amounts of labeled data for supervised learning. However, in the case of cross-source data, labeled data is difficult to obtain and expensive, and the performance of existing models in this regard is also limited. This application uses a self-supervised pre-training mechanism to enable the model to automatically learn the deep features of multi-source remote sensing images without labeled data. Self-supervised learning not only reduces the dependence on labeled data, but also improves the model's ability to abstract potential patterns in remote sensing data, further enhancing its feature expression and recognition capabilities.

[0143] See also Figure 3, a device for constructing a large visual model based on multi-source remote sensing images is provided to meet one of the purposes of the present application, including an occluded image determination module 1100, a semantic information extraction module 1200, a fusion feature determination module 1300, an encoder pre-training module 1400 and a downstream task execution module 1500. Among them, the occluded image determination module 1100 is configured to obtain a radar remote sensing image containing each target object and its corresponding infrared remote sensing image and optical remote sensing image, and adopt a preset multi-scale masking strategy to generate a random mask matrix to dynamically mask specific areas of the radar remote sensing image, the infrared remote sensing image and the optical remote sensing image to determine the random occluded image of the radar remote sensing image, the infrared remote sensing image and the optical remote sensing image; the semantic information extraction module 1200 is configured to use a local window self-attention mechanism based on the Swin Transformer model in the feature extraction network of the encoder according to the encoded data corresponding to the random occluded image of the optical remote sensing image, so as to capture the local detail features corresponding to each target object in the optical remote sensing image, based on the Vision The Transformer model uses a global attention mechanism to extract global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image based on the encoded data corresponding to the randomly masked image of the radar remote sensing image and the infrared remote sensing image. A fusion feature determination module 1300 is configured to use a gated cross-attention mechanism in the cross-modal fusion layer of the encoder to fuse the local detail features corresponding to each target object with the global semantic information corresponding to the target object to determine the multi-source fusion features corresponding to each target object. An encoder pre-training module 1400 is configured to reconstruct the multi-source fusion features in the decoder to determine the reconstruction loss between the reconstructed image and the original image, and train to a certain number of iterations to minimize the reconstruction loss, thereby completing the construction and pre-training of a large visual model based on multi-source remote sensing images. A downstream task execution module 1500 is configured to add a specific task head to the pre-trained encoder for fine-tuning, and then apply it to downstream interpretation tasks, which include but are not limited to target detection tasks, target recognition tasks, or change detection tasks, wherein the task head includes but is not limited to target recognition head, target detection head, and change detection head.

[0144] Based on any embodiment of this application, please refer to Figure 4 Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 4As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions, and the database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a method for constructing a large visual model based on multi-source remote sensing images. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the method for constructing a large visual model based on multi-source remote sensing images of the present application. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0145] In this embodiment, the processor is used to execute Figure 3 The memory stores the program code and various data required to execute the specific functions of each module in the multi-source remote sensing imagery-based visual model construction device. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules in the multi-source remote sensing imagery-based visual model construction device of this application. The server can call the server's program code and data to execute the functions of all modules.

[0146] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for constructing a large visual model based on multi-source remote sensing images described in any embodiment of the present application.

[0147] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the method for constructing a large visual model based on multi-source remote sensing images described in any embodiment of the present application.

[0148] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0149] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for constructing a large visual model based on multi-source remote sensing images, characterized in that: include: Acquire a radar remote sensing image containing each target object and its corresponding infrared remote sensing image and optical remote sensing image, and use a preset multi-scale masking strategy to generate a random mask matrix to dynamically mask specific areas of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image to determine a randomly masked image of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image; In the feature extraction network of the encoder, a local window self-attention mechanism is adopted based on the Swin Transformer model according to the encoded data corresponding to the randomly masked image of the optical remote sensing image to capture the local detail features corresponding to each target object in the optical remote sensing image. A global attention mechanism is adopted based on the Vision Transformer model according to the encoded data corresponding to the randomly masked images of the radar remote sensing image and the infrared remote sensing image to extract the global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image. In the cross-modal fusion layer of the encoder, a gated cross-attention mechanism is used to fuse the local detail features corresponding to each target object with the global semantic information corresponding to the target object to determine the multi-source fusion features corresponding to each target object; The multi-source fusion features are reconstructed in the decoder to determine the reconstruction loss between the reconstructed image and the original image, and training is performed to a certain number of iterations to minimize the reconstruction loss, thereby completing the construction and pre-training of a large visual model based on multi-source remote sensing images; The pre-trained encoder is fine-tuned by adding a specific task head and then applied to downstream interpretation tasks, including but not limited to target detection tasks, target recognition tasks or change detection tasks, where the task head includes but is not limited to target recognition head, target detection head, and change detection head.

2. The method for constructing a large visual model based on multi-source remote sensing images according to claim 1, characterized in that: The steps of obtaining radar remote sensing images containing each target object and its corresponding infrared remote sensing image and optical remote sensing image include: In response to the data preprocessing instruction, for the optical remote sensing image, geometric transformation is used to eliminate the interference of illumination and viewing angle, and image denoising technology is combined to suppress cloud and noise; For radar remote sensing images, speckle noise suppression and terrain radiation correction are used to extract backscatter coefficient features, and vertex suppression algorithm is used to eliminate interference from overlapping and shadow areas. For infrared remote sensing images, the temperature difference contrast between the target object and the background is enhanced based on thermal radiation calibration and dynamic range compression.

3. The method for constructing a large visual model based on multi-source remote sensing images according to claim 1, characterized in that: The steps of capturing local detail features corresponding to each target object in the optical remote sensing image based on the encoded data corresponding to the randomly masked image of the optical remote sensing image by using a local window self-attention mechanism based on the Swin Transformer model include: Inputting the encoded data corresponding to the randomly masked image of the optical remote sensing image into a Swin Transformer model, dividing the optical remote sensing image into a plurality of local windows according to a preset size, wherein each local window represents a local area; Within each local window, the Swin Transformer model uses a self-attention mechanism to perform local self-attention calculations on the unmasked area features based on the encoded data corresponding to the randomly masked image of the optical remote sensing image to determine the relationship between pixels within the window. Each pixel is assigned different weights based on the characteristics of other pixels to capture the spatial relationship and detailed features between the target objects within the local area.

4. The method for constructing a large visual model based on multi-source remote sensing images according to claim 1, characterized in that: The step of using a gated cross attention mechanism to fuse the local detail features corresponding to each target object with the global semantic information corresponding to the target object to determine the multi-source fusion features corresponding to each target object includes: Acquire local detail features corresponding to each target object in the optical remote sensing image and global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image, so as to generate a query matrix corresponding to the local detail features and a key matrix corresponding to the global semantic information; A preset Sigmoid gating function is used to perform weighting according to the matching degree of the query matrix and the key matrix, so that the features of different modalities are weighted and fused in different areas; The gated weighted features are concatenated to integrate the information of each modality into the same space to preserve their local detail features and global semantic information; After feature splicing, the spliced ​​feature maps are recalibrated through channel recalibration to determine a unified cross-modal representation, where the cross-modal representation contains local detail features and global semantic information of each target object.

5. The method for constructing a large visual model based on multi-source remote sensing images according to claim 1, characterized in that: After fine-tuning the pre-trained encoder with a specific task head, it can be applied to downstream interpretation tasks, including but not limited to object detection tasks, object recognition tasks, or change detection tasks. The steps include: For target detection tasks, the target location and category probability are generated through the anchor box adaptation mechanism; For the change detection task, the input data is processed by the classification network, and the probability distribution of each pixel belonging to different change categories is output. Then, the change category of the pixel is determined based on the maximum probability. For the target recognition task, a fully connected layer classification head is designed to determine the category of the target with the highest probability based on the probability distribution.

6. The method for constructing a large visual model based on multi-source remote sensing images according to any one of claims 1 to 5, characterized in that: The multi-scale masking strategy includes one or more of small-size random masking, large-size random masking, small-size channel random masking, and large-size channel random masking.

7. The method for constructing a large visual model based on multi-source remote sensing images according to any one of claims 1 to 5, characterized in that: The radar remote sensing image includes a SAR image; The reconstruction loss represents the pixel-by-pixel mean square error between the reconstructed image output by the decoder and the original image of the radar remote sensing image, infrared remote sensing image and optical remote sensing image corresponding to the reconstructed image; The reconstructed image represents a remote sensing image that fuses local detail features corresponding to each target object and global semantic information.

8. A device for constructing a large visual model based on multi-source remote sensing images, characterized in that: include: a masked image determination module configured to obtain a radar remote sensing image containing each target object and its corresponding infrared remote sensing image and optical remote sensing image, and to use a preset multi-scale masking strategy to generate a random mask matrix to dynamically mask specific areas of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image to determine a random masked image of the radar remote sensing image, the infrared remote sensing image, and the optical remote sensing image; The semantic information extraction module is configured to, in a feature extraction network of an encoder, capture local detail features corresponding to each target object in the optical remote sensing image based on the encoded data corresponding to the randomly masked image of the optical remote sensing image using a local window self-attention mechanism based on a Swin Transformer model, and extract global semantic information corresponding to each target object in the radar remote sensing image and the infrared remote sensing image based on the encoded data corresponding to the randomly masked image of the radar remote sensing image and the infrared remote sensing image using a global attention mechanism based on a Vision Transformer model; a fusion feature determination module configured to use a gated cross-attention mechanism in a cross-modal fusion layer of the encoder to perform feature fusion on the local detail features corresponding to each target object and the global semantic information corresponding to the target object to determine a multi-source fusion feature corresponding to each target object; An encoder pre-training module is configured to reconstruct the multi-source fusion features in the decoder to determine the reconstruction loss between the reconstructed image and the original image, train to a certain number of iterations to minimize the reconstruction loss, and complete the construction and pre-training of a large visual model based on multi-source remote sensing images; The downstream task execution module is configured to add a specific task head to the pre-trained encoder for fine-tuning before applying it to downstream interpretation tasks, which include but are not limited to target detection tasks, target recognition tasks or change detection tasks, wherein the task head includes but is not limited to target recognition head, target detection head, and change detection head.

9. An electronic device comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images