Unmanned aerial vehicle image super-resolution reconstruction method and system based on multi-modal heterogeneity, and medium

Abstract

The application discloses a multi-modal heterogeneous-based unmanned aerial vehicle image super-resolution reconstruction method and system and a medium, relates to the technical field of image processing, and collects multi-channel images of a first resolution and reference panchromatic images of a second resolution, generates a paired data set after preprocessing, extracts multi-channel feature maps and panchromatic feature maps from the paired data set based on a MultiHetSRNet model, generates a multi-modal heterogeneous graph by modeling nodes of a graph structure, and generates a relationship mode matrix describing spatial relationships and cross-modal relationships between nodes; target multi-channel images are reconstructed by aggregating local features and global features according to the relationship mode matrix; the scheme is improved in the method on the basis of the prior art, multi-channel images of an unmanned aerial vehicle and remote sensing reference panchromatic images are fused, complex spatial relationships are modeled by using a multi-modal heterogeneous graph, the robustness of a model is improved in combination with multiple degradation modes, and high-quality super-resolution reconstruction is realized.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method, system, and medium for super-resolution reconstruction of UAV images based on multimodal heterogeneity. Background Technology

[0002] Unmanned aerial vehicle (UAV) imagery has wide applications in urban planning, agricultural monitoring, and environmental management. However, limited by sensor resolution, low-resolution imagery struggles to meet high-precision requirements. Super-resolution (SR) technology, which reconstructs high-resolution (HR) imagery from low-resolution (LR) imagery, has become a key technology for solving this problem. The panchromatic (PAN) band in remote sensing imagery typically has high spatial resolution and can be used as a reference to improve super-resolution performance.

[0003] Super-resolution (SR) technology is a key task in the field of computer vision. Its goal is to reconstruct high-resolution (HR) images from low-resolution (LR) images to restore the image's detail and sharpness. This technology has wide applications in satellite remote sensing, medical imaging, security monitoring, and high-definition displays.

[0004] Existing super-resolution methods mainly include those based on convolutional neural networks (CNN), Transformer networks, and generative adversarial networks (GAN) for super-resolution reconstruction. These methods have achieved certain results in single-modal image super-resolution tasks, but they have problems such as insufficient spatial relationship modeling, insufficient utilization of modal information, and insufficient balance between local and global consistency; as detailed below.

[0005] 1. Insufficient spatial relationship modeling: CNNs rely on fixed convolution kernels, making it difficult to effectively capture complex and irregular spatial structures (such as building and vegetation textures) in UAV imagery; while Transformers can model global dependencies, their computational complexity is high.

[0006] 2. Insufficient utilization of modal information: Satellite remote sensing images can usually acquire high-resolution panchromatic (PAN) bands, but existing methods rarely consider the spatial information of panchromatic images as a reference, resulting in limited ability to recover high-frequency details.

[0007] 3. Insufficient balance between local and global consistency: Existing methods tend to neglect global semantic consistency when restoring local texture details, which affects the overall quality of the reconstructed image.

[0008] In recent years, Graph Neural Networks (GNNs) have demonstrated outstanding performance in non-Euclidean space modeling, particularly in panchromatic sharpening tasks, where heterogeneous graph-based methods have significantly improved fusion results by modeling spatial-spectral relationships. However, these methods primarily target multispectral image fusion and are difficult to directly apply to super-resolution tasks of single-modality RGB imagery, and they do not fully utilize the reference information of panchromatic images. Therefore, there is an urgent need for a UAV imagery super-resolution method that can combine panchromatic imagery references, flexibly model complex spatial relationships, and balance local and global consistency. Summary of the Invention

[0009] The technical problem this invention aims to solve is that while existing super-resolution methods have achieved certain results in single-modal image super-resolution tasks, the single degradation method leads to weak model generalization ability, high-cost data acquisition limits practicality, and insufficient modeling of complex spatial relationships. This invention aims to provide a method, system, and medium for UAV image super-resolution reconstruction based on multimodal heterogeneity. It improves upon existing technologies by fusing multi-channel UAV imagery with remote sensing reference panchromatic imagery, using multimodal heterogeneous graphs to model complex spatial relationships, and combining various degradation methods to enhance model robustness, thereby achieving high-quality super-resolution reconstruction.

[0010] This invention is achieved through the following technical solution:

[0011] This solution provides a UAV imagery super-resolution reconstruction method based on multimodal heterogeneity, including:

[0012] Acquire a multi-channel image at a first resolution and a reference panchromatic image at a second resolution, and preprocess the multi-channel image and the reference panchromatic image to generate a paired dataset; the values ​​of the first resolution are lower than the values ​​of the second resolution.

[0013] The paired dataset is input into the MultiHetSRNet model to reconstruct the target multi-channel image:

[0014] Feature extraction is performed on the paired dataset to obtain multi-channel feature maps and panchromatic feature maps, and the multi-channel feature maps and panchromatic feature maps are concatenated to generate an initial feature representation;

[0015] Nodes of the graph structure are modeled based on multi-channel feature maps and panchromatic feature maps, and multimodal heterogeneous graphs are generated based on the graph structure nodes, as well as relational pattern matrices describing the spatial and cross-modal relationships between nodes;

[0016] Based on the relational pattern matrix, local and global features are aggregated to reconstruct the target multi-channel image.

[0017] A further optimized solution involves preprocessing multi-channel and panchromatic images to generate paired datasets; including the following methods:

[0018] The multi-channel image is downsampled to generate a first multi-channel image with a third resolution; the value of the second resolution is lower than the value of the third resolution.

[0019] Georegistration and pixel alignment are performed on the first multi-channel image and the reference panchromatic image;

[0020] The pixel values ​​of the multi-channel image, the reference panchromatic image, and the first multi-channel image are normalized, and the multi-channel image, the reference panchromatic image, and the first multi-channel image are cropped according to the first pixel, the second pixel, and the third pixel, respectively.

[0021] Data augmentation is performed on the cropped images to generate paired datasets.

[0022] A further optimized approach involves extracting features from the paired dataset to obtain multi-channel feature maps and panchromatic feature maps, and then concatenating the multi-channel feature maps and panchromatic feature maps to generate an initial feature representation; including the following methods:

[0023] Multi-channel feature maps and panchromatic feature maps are extracted from the first multi-channel image and the reference panchromatic image respectively based on the same convolutional layer located in different branches; both the panchromatic feature map and the multi-channel feature map are at the third resolution;

[0024] The panchromatic feature map and the multi-channel feature map are concatenated along the channel dimension to generate the initial feature representation, and the panchromatic feature map and the multi-channel feature map are merged to generate the combined feature map.

[0025] A further optimized scheme involves modeling nodes of a graph structure based on multi-channel feature maps and panchromatic feature maps, generating a multimodal heterogeneous graph based on the graph structure nodes, and a relational pattern matrix describing the spatial and cross-modal relationships between nodes; including the following methods:

[0026] Each pixel feature of the multi-channel feature map is defined as a multi-channel node, resulting in n = H × W multi-channel nodes. Each pixel feature of the panchromatic feature map is defined as a panchromatic node, resulting in m = H × W panchromatic nodes.

[0027] For each multi-channel node and panchromatic node, the k-nearest neighbor algorithm is used to select a nearest neighbor nodes to construct multi-channel spatial neighborhood edges and panchromatic spatial neighborhood edges; cross-modal edges are constructed by connecting multi-channel nodes and panchromatic nodes based on spatial coordinate mapping; multi-channel spatial neighborhood edge adjacency matrix, panchromatic spatial neighborhood edge adjacency matrix, and cross-modal edge adjacency matrix are generated;

[0028] Logical operations are performed on the multi-channel spatial neighborhood edge adjacency matrix, the full-color spatial neighborhood edge adjacency matrix, and the cross-modal edge adjacency matrix to generate a relational pattern matrix describing the spatial and cross-modal relationships between nodes.

[0029] A further optimization is that the relational schema matrix includes single schemas and combined schemas;

[0030] The single pattern includes different adjacency matrices that each constitute a relation pattern;

[0031] The combined pattern includes generating a combined relation pattern using any z different adjacency matrices through XNOR operation and bitwise AND operation; z≥1.

[0032] A further optimized solution involves a method for aggregating local and global features based on the relational pattern matrix and reconstructing a multi-channel image.

[0033] The local relation matrix is ​​generated by weighted summation of all relation pattern matrices. The neighbor node features are aggregated from the local relation matrix based on the first layer graph convolutional network. The neighbor node features are then subjected to a first weight transformation and nonlinear activation. Finally, the local features are generated by a second weight transformation using the second layer graph convolutional network.

[0034] A global relation matrix is ​​generated by calculating cosine similarity based on all relation pattern matrices. The neighbor node features are aggregated from the full relation matrix using a third-layer graph convolutional network. The neighbor node features are then subjected to a third weight transformation and nonlinear activation. Finally, a fourth-layer graph convolutional network performs a fourth weight transformation to generate global features.

[0035] The local and global features are fused to obtain unified node features, and the fusion is optimized based on contrastive learning loss.

[0036] A further optimized solution is that the reconstruction method for the target multi-channel image includes:

[0037] Multi-channel node features are extracted from unified node features and reconstructed into a multi-channel node feature map.

[0038] The PixelShuffle sub-pixel convolution reduces the number of channels in the multi-channel node feature map and rearranges it into spatial dimensions. The features are then adjusted through convolutional layers to obtain the target multi-channel image.

[0039] A further optimization scheme involves using pixel-level loss, perceptual loss, full-color guided loss, contrastive learning loss, and total loss as loss functions for the MultiHetSRNet model, and training and optimizing it based on the Adam optimizer.

[0040] This solution also provides a UAV image super-resolution reconstruction system based on multimodal heterogeneity, used to implement the aforementioned UAV image super-resolution reconstruction method based on multimodal heterogeneity; the system includes:

[0041] The acquisition module is used to acquire a multi-channel image at a first resolution and a reference panchromatic image at a second resolution; the value of the first resolution is lower than the value of the second resolution.

[0042] The preprocessing module is used to preprocess multi-channel images and reference panchromatic images to generate paired datasets;

[0043] The MultiHetSRNet model is used to reconstruct multi-channel images of a target based on a paired dataset; the MultiHetSRNet model includes:

[0044] The feature extraction module is used to extract features from the paired dataset to obtain a multi-channel feature map and a panchromatic feature map, and to concatenate the multi-channel feature map and the panchromatic feature map to generate an initial feature representation;

[0045] The multimodal heterogeneous module is used to model the nodes of the graph structure based on the multi-channel feature map and the panchromatic feature map, and to generate a multimodal heterogeneous graph based on the graph structure nodes, as well as a relational pattern matrix describing the spatial and cross-modal relationships between nodes;

[0046] The reconstruction module is used to aggregate local and global features based on the relational pattern matrix and reconstruct the target multi-channel image.

[0047] This solution also provides a computer-readable medium storing a computer program, characterized in that the computer program, when executed by a processor, can implement the above-described method for super-resolution reconstruction of UAV images based on multimodal heterogeneity.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] 1. This invention provides a method, system, and medium for super-resolution reconstruction of UAV imagery based on multimodal heterogeneity; it acquires multi-channel imagery at a first resolution and a reference panchromatic imagery at a second resolution, preprocesses them to generate paired datasets; it extracts multi-channel feature maps and panchromatic feature maps from the paired datasets based on the MultiHetSRNet model, and generates multimodal heterogeneous graphs by modeling nodes in a graph structure, as well as a relational pattern matrix describing the spatial and cross-modal relationships between nodes; it reconstructs the target multi-channel imagery by aggregating local and global features based on the relational pattern matrix; this scheme improves upon existing technologies by fusing multi-channel UAV imagery with remote sensing reference panchromatic imagery, using multimodal heterogeneous graphs to model complex spatial relationships, and combining various degradation methods to improve model robustness, thereby achieving high-quality super-resolution reconstruction.

[0050] 2. The present invention provides a method, system, and medium for super-resolution reconstruction of UAV images based on multimodal heterogeneity. By introducing a high-resolution remote sensing reference panchromatic image as an independent modality, reliable prior information on spatial details is provided for the MultiHetSRNet model. Even if the input low-resolution UAV image has complex degradation such as blurring and noise, the MultiHetSRNet model can learn clear structural and textural features from the high-quality panchromatic image, thereby reducing its over-reliance on low-quality input data modes.

[0051] 3. The present invention provides a method, system and medium for super-resolution reconstruction of UAV images based on multimodal heterogeneity; by constructing a multimodal heterogeneous graph and generating a relational pattern matrix, the MultiHetSRNet model can adaptively learn the relationship between modes under different degradation conditions, thereby dynamically adjusting the fusion and reconstruction strategy for different inputs, which greatly enhances the model's adaptability to diverse degradation modes, i.e., its generalization ability.

[0052] 4. The present invention provides a method, system, and medium for super-resolution reconstruction of UAV images based on multimodal heterogeneity; the MultiHetSRNet model extracts high-frequency spatial information from readily available panchromatic images and transfers it to low-resolution UAV images. The training process no longer relies entirely on the difficult mapping from "low quality" to "high quality", but achieves image reconstruction by learning the correlation between two modalities, thus reducing the requirements for data quality; the feature map is regarded as a node of a graph structure, which innovatively transforms the image spatial relationship problem into a node relationship modeling problem of graph neural networks (GNN); the MultiHetSRNet model can simultaneously capture the long-range spatial dependence (global structure) and local detail correlation within the image, thereby guiding the feature fusion process and generating higher resolution results with better structural consistency and spectral fidelity. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0054] Figure 1 This is a schematic diagram of the process of UAV image super-resolution reconstruction method based on multimodal heterogeneity;

[0055] Figure 2 This is a schematic diagram illustrating the principle of UAV image super-resolution reconstruction based on multimodal heterogeneity.

[0056] Figure 3 This is a schematic diagram of the MultiHetSRNet model architecture;

[0057] Figure 4 A schematic diagram of the preprocessing process for multi-channel and panchromatic images;

[0058] Figure 5 This is a schematic diagram of the structure of a UAV image super-resolution reconstruction system based on multimodal heterogeneity. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0060] Existing super-resolution methods have achieved certain results in single-modal image super-resolution tasks, but the use of a single degradation method leads to problems such as weak model generalization ability, high-cost data acquisition limiting practicality, and insufficient modeling of complex spatial relationships. Therefore, this solution provides the following embodiments to address the above technical problems:

[0061] Example 1: This example provides a method for super-resolution reconstruction of UAV imagery based on multimodal heterogeneity, such as... Figure 1 and Figure 2 As shown, it includes:

[0062] Step 1: Acquire a multi-channel image at a first resolution and a reference panchromatic image at a second resolution. Preprocess the multi-channel and reference panchromatic images to generate a paired dataset. The values ​​of the first resolution images are lower than those of the second resolution images. In this embodiment, the resolution of the multi-channel image is 0.125m, and the resolution of the reference panchromatic image is 0.3m. The reference panchromatic image is a remote sensing panchromatic image of the same study area and date as the multi-channel image; it is a single-channel grayscale image containing high-resolution spatial information. The multi-channel image is an RGB image acquired by a UAV containing three color channels (red, green, and blue), with dimensions of 4H×4W×3; where H represents height and W represents width.

[0063] In this step, multi-channel and panchromatic images are preprocessed to generate paired datasets; such as... Figure 4 As shown, the methods include:

[0064] S11, downsample the multi-channel image to generate a first multi-channel image with a third resolution; the value of the second resolution is lower than that of the third resolution; the resolution of the first multi-channel image is 0.5m, and the size is H×W×3, which is used to simulate the low-resolution image actually collected by the UAV; specifically, the bilinear interpolation method can be used to downsample the 0.125m resolution RGB image to 0.5m resolution to ensure spatial correspondence; bilinear interpolation calculates the target pixel value by weighted average of adjacent pixels, simulating resolution reduction while retaining color information.

[0065] S12, perform georegistration and pixel alignment on the first multi-channel image and the reference panchromatic image; specifically, the GDAL tool can be used to georegister the 0.5m resolution RGB image and the 0.3m resolution panchromatic image, and pixel-level alignment is achieved through control point matching and affine transformation.

[0066] S13, normalize the pixel values ​​of the multi-channel image, the reference panchromatic image and the first multi-channel image, and crop the multi-channel image, the reference panchromatic image and the first multi-channel image according to the first pixel, the second pixel and the third pixel respectively; specifically, through histogram matching, correct the illumination or color deviation between the RGB image and the panchromatic image caused by the difference in the acquisition date, and ensure spectral consistency.

[0067] The image pixel values ​​are normalized to [0, 1] and cropped into small blocks, where the first pixel is 32×32, the second pixel is 53×53 pixels, and the third pixel is 128×128, which facilitates the training of the MultiHetSRNet model.

[0068] Data augmentation is performed on the cropped images to generate paired datasets. Specifically, random flipping, rotation (90°, 180°, 270°) and color jitter (brightness, contrast, saturation adjustment) can be applied to increase dataset diversity and improve model robustness.

[0069] like Figure 3 As shown, the paired dataset is input into the MultiHetSRNet model to reconstruct the target multi-channel image:

[0070] Step two involves extracting features from the paired dataset to obtain multi-channel feature maps and panchromatic feature maps, and then concatenating the multi-channel feature maps and panchromatic feature maps to generate an initial feature representation. This step specifically includes the following methods:

[0071] S21, based on the same convolutional layer located in different branches, respectively extract from the first multi-channel image (size H×W×3) and the reference panchromatic image (size H×W×3). Extract the multi-channel feature map F. RGB ∈R H×W×64 and panchromatic feature map F PAN∈R H×W×64 Both the panchromatic feature map and the multi-channel feature map are at third resolution; the panchromatic feature map F PAN ∈R H×W×64 Downsampling improves resolution and multi-channel features. Figure 1 Both are 0.5m.

[0072] The convolutional layers have 3×3 kernels, 64 channels per layer, a stride of 1, and padding of 1.

[0073] S22, concatenate the panchromatic feature map and the multi-channel feature map along the channel dimension to generate the initial feature representation F=Concat(F RGB F PAN R H×W×128 The panchromatic feature map and the multi-channel feature map are merged to generate a combined feature map.

[0074] Concat() represents a concatenation operation along the channel dimension, merging the multi-channel feature map (64 channels) and the panchromatic feature map (64 channels) into a combined feature map F with 128 channels, maintaining the size as H × W; providing multimodal features for subsequent heterogeneous map construction.

[0075] Step 3 involves modeling the nodes of the graph structure based on multi-channel feature maps and panchromatic feature maps, and generating a multimodal heterogeneous graph and a relational pattern matrix describing the spatial and cross-modal relationships between nodes based on the graph structure nodes. This step specifically includes the following methods:

[0076] S31, define each pixel feature of the multi-channel feature map as a multi-channel node, resulting in n = H × W multi-channel nodes, denoted as V. RGB Each pixel feature of the panchromatic feature map is defined as a panchromatic node, resulting in m = H × W panchromatic nodes, denoted as V. PAN The node set is V=V RGB ∪V PAN The total number of nodes is n + m;

[0077] S32, for each multi-channel node and panchromatic node, the k-nearest neighbor algorithm is used to select a (in this embodiment, a is 8) nearest neighbor nodes to construct multi-channel spatial neighborhood edges and panchromatic spatial neighborhood edges; cross-modal edges are constructed by connecting multi-channel nodes and panchromatic nodes based on spatial coordinate mapping; multi-channel spatial neighborhood edge adjacency matrix, panchromatic spatial neighborhood edge adjacency matrix, and cross-modal edge adjacency matrix are generated; specifically, the k-nearest neighbor algorithm uses the Euclidean distance of the feature vectors, and the calculation formula is:

[0078] ;

[0079] Where d(x) i x j) represents the eigenvectors x of nodes i and j. i x j The Euclidean distance between them is used to select the nearest neighbor node and construct the spatial neighborhood edges; x i x j The feature vectors representing nodes i and j have a length of 64 (the number of feature channels); k is the feature vector dimension index, k=1, 2, ..., 64; cross-modal edges are determined through spatial coordinate mapping, for example, the correspondence between RGB pixels (i, j) and panchromatic pixels (i', j') is based on the resolution ratio (0.5m:0.3m). To reduce computational complexity, the maximum number of edges per node is limited (k=8), generating a sparse adjacency matrix.

[0080] Construct three types of adjacency matrices to represent different types of edges A r ∈R (n+m)×(n+m) r = 1, 2, 3; where A r Let r = 1 represent a multi-channel spatial neighborhood edge, r = 2 represent a panchromatic spatial neighborhood edge, and r = 3 represent a cross-modal edge. Matrix element A r (i, j) = 1 indicates that there is an edge between node i and node j, A r (i, j) = 0 indicates no borders; the matrix size (n+m) × (n+m) reflects the total number of nodes.

[0081] S33 performs logical operations on the multi-channel spatial neighborhood edge adjacency matrix, the full-color spatial neighborhood edge adjacency matrix, and the cross-modal edge adjacency matrix to generate a relational pattern matrix describing the spatial and cross-modal relationships between nodes.

[0082] The relation schema matrix includes single schemas and composite schemas;

[0083] A single schema consists of different adjacency matrices that each constitute a relation schema;

[0084] The combination pattern includes generating a combined relation pattern using any z different adjacency matrices through XNOR and bitwise AND operations; z≥1; in this embodiment, it is based on three adjacency matrices A r Seven relation schema matrices are generated through logical operations, including:

[0085] Single mode: Directly use A1 (multi-channel spatial neighborhood edge adjacency matrix mode), A2 (full-color space neighborhood edge adjacency matrix mode), and A3 (cross-modal edge adjacency matrix mode).

[0086] Combinatorial patterns: Combinatorial relationships are generated through XNOR operations and bitwise AND operations, for example: , , and ;

[0087] in, This indicates a bitwise AND operation. Indicates element-level logical AND; This indicates a bitwise OR operation. Indicates the logical NOT operation; This indicates a relationship pattern that simultaneously has RGB spatial neighborhood edges and cross-modal edges. This indicates that node i and node j simultaneously satisfy both RGB neighborhood and cross-modal relationships. This indicates a relationship that simultaneously possesses both a neighborhood edge in the full color space and a cross-modal edge. This indicates a relationship that simultaneously possesses neighborhood edges in both RGB and full color space; This indicates a relationship without any edges (inverted by bitwise OR operation), capturing relationships between nodes that are not directly connected.

[0088] Step four: Aggregate local and global features based on the relational pattern matrix and reconstruct the target multi-channel image.

[0089] Step four involves aggregating local and global features based on the relational pattern matrix, specifically including the following methods:

[0090] S41, a local relation matrix is ​​generated by weighted summation of all relation pattern matrices. The neighbor node features are aggregated from the local relation matrix based on the first layer graph convolutional network. The neighbor node features are then subjected to a first weight transformation and nonlinear activation. The second layer graph convolutional network then performs a second weight transformation to generate local features.

[0091] The specific weighted aggregation relation pattern matrix generates a local relation matrix. :

[0092] , where the local relation matrix The size is (n+m)×(n+m); Let r represent the matrix of the r-th relation pattern, with size (n+m)×(n+m); r=1,2,…,7; The weights of the r-th relation schema matrix are scalars, ranging from [0, 1], and satisfy the following conditions: ;

[0093] The weights are generated by weighted summation of matrices representing seven relation schemas. Through learnable parameter optimization, the initial values ​​are uniformly distributed (e.g. This matrix reflects the local connection patterns between nodes.

[0094] A two-layer graph convolutional network (GCN) is used to process local relations and generate local features H. Local :

[0095] ;

[0096] Among them, local feature H Local The size of X is (n+m)×d (d=64 is the output feature dimension); X is the input node feature matrix with a size of (n+m)×128, which is reshaped by the initial feature representation F; W1 and W2 are the weight matrices of the two-layer graph convolutional network with sizes of 128×d and d×64, respectively. Represents the ReLU activation function; .

[0097] The first layer of graph convolutional network passes through The features X of neighboring nodes are aggregated, transformed by the weight matrix W1, and then activated by nonlinearity. (ReLU) and the second-layer weight matrix W2 generate the final local representation.

[0098] S42, based on the cosine similarity calculation of all relation pattern matrices, a global relation matrix is ​​generated. The neighbor node features are aggregated from the full relation matrix based on the third layer graph convolutional network. After the neighbor node features are subjected to the third weight transformation, nonlinear activation is performed. Then, the fourth layer graph convolutional network performs the fourth weight transformation to generate global features.

[0099] Specifically, the global similarity matrix is ​​calculated using cosine similarity. :

[0100] ;

[0101] Global similarity matrix The size is (n+m)×(n+m), and the element (i, j) represents the cosine similarity of the feature vectors of node i and node j, reflecting the global feature association; x i x j Let i represent the feature vectors of nodes i and j, with a length of 128.

[0102] Similarly, a two-layer graph convolutional network (GCN) is used to process global relations and generate global features H. Global :

[0103] ;

[0104] Global feature H Global The size is (n+m)×64, and W3 and W4 represent the weight matrices of the graph convolutional network, with sizes of 128×d and d×64, respectively; node features are aggregated through the global similarity matrix, a process similar to local aggregation.

[0105] S43, the local features and global features are fused to obtain unified node features, and the fusion is optimized based on contrastive learning loss.

[0106] Integrating local and global representations H represents a unified node feature representation with a size of (n+m)×64. By averaging local and global features, the model balances detail and overall consistency, and the fusion is optimized using contrastive learning loss.

[0107] ;

[0108] in, Indicates contrastive learning loss; h i The uniform feature vector of node i has a length of 64; h represents the positive sample features (nearest nodes) of node i; k Indicates excluding h i The feature vectors of samples other than those in the negative sample are used as negative sample features; sim(·) represents the calculation of cosine similarity; It represents the temperature parameter, a scalar, and controls the loss distribution.

[0109] The reconstruction methods for the target multi-channel image in step four include:

[0110] G41 extracts multi-channel node features from unified node features and reshapes them into a multi-channel node feature map; from unified node features... Extract the RGB node features (first n = H × W rows) and reshape them into a multi-channel node feature map F. out :F out ∈R H ×W×64 F out This is a feature map reshaped from RGB node features, with dimensions H×W×64. It contains only RGB node features, excluding pancolor nodes, and is ready for upsampling.

[0111] G42, based on sub-pixel convolution PixelShuffle, reduces the number of channels in the multi-channel node feature map and rearranges it into spatial dimensions, and then adjusts the features through convolutional layers to obtain the target multi-channel image.

[0112] A target multi-channel image F is generated by upsampling by 4x using sub-pixel convolution PixelShuffle. up F up ∈R 4H ×4W×3 PixelShuffle rearranges the channels of the input feature map to represent spatial dimensions, generating a high-resolution 4H×4W feature map with the number of channels reduced to 3 (RGB color channels). A 3×3 convolutional layer (stride 1, padding 1) is then used to adjust the features, generating the final output. ∈R 4H×4W×3 , For the reconstructed 0.125m resolution RGB image, with dimensions of 4H×4W×3, convolution was used to refine the features to ensure color and detail consistency.

[0113] The MultiHetSRNet model uses pixel-level loss, perceptual loss, full-color guided loss, contrastive learning loss, and total loss as loss functions, and is trained and optimized based on the Adam optimizer.

[0114] Pixel-level loss is used to calculate the L1 norm difference between the reconstructed image and the multi-channel image to facilitate accurate pixel-level matching; pixel-level loss for: ;

[0115] in, This represents the reconstructed image, GT represents the multi-channel image; |*|1 represents the L1 norm, which is the sum of absolute differences pixel by pixel.

[0116] Perceptual loss is used to calculate the differences in high-level features between reconstructed images and multi-channel images, thereby promoting their similarity at a high-level semantic level. for: ;

[0117] Among them, C j H j W j These represent the number of channels, height, and width of the feature map, respectively. This represents the square of the L2 norm.

[0118] Panchromatic guided loss downsamples the reconstructed image to 0.3m resolution (scaling ratio 5 / 12) and calculates the L1 norm difference with the input panchromatic image (PAN), thereby promoting spatial detail consistency between the reconstructed image and the panchromatic image. : ;

[0119] in, This indicates a downsampling operation with a scaling factor of 5 / 12 (0.125m to 0.3m); PAN indicates a panchromatic image with a size of... .

[0120] Contrastive learning loss promotes feature similarity among positive samples and excludes negative samples. The contrastive learning loss is... :

[0121] ;

[0122] Where log represents exponentiation; exp() represents the exponential function; k represents the index of all samples except i, and the specific denominator is... It is for all and Different samples similarity Summation, It is a temperature parameter;

[0123] Total loss for:

[0124] ;

[0125] The total loss is the weighted sum of the individual losses, with the weights (1, 0.1, 0.1, 0.03) fine-tuned experimentally to balance pixel precision, semantic consistency, spatial detail, and feature fusion.

[0126] Using the Adam optimizer, with an initial learning rate of 1×10⁻⁶. -4 The degradation factor is 0.85 every 3000 iterations. The batch size is set to 4, and the number of iterations is 30000. The feature extraction and image reconstruction modules are pre-trained on the DIV2K dataset (a general image super-resolution dataset), and the model parameters are initialized. Fine-tuning is performed on a specific dataset containing: 0.5m resolution UAV RGB imagery (generated by downsampling from 0.125m resolution RGB imagery); 0.3m resolution remote sensing panchromatic imagery; and 0.125m resolution ground truth RGB imagery.

[0127] Example 2: This example provides a UAV image super-resolution reconstruction system based on multimodal heterogeneity, used to implement the UAV image super-resolution reconstruction method based on multimodal heterogeneity described in Example 1; such as Figure 5 As shown, the system includes:

[0128] The acquisition module is used to acquire a multi-channel image at a first resolution and a reference panchromatic image at a second resolution; the value of the first resolution is lower than the value of the second resolution.

[0129] The preprocessing module is used to preprocess multi-channel images and reference panchromatic images to generate paired datasets;

[0130] The MultiHetSRNet model is used to reconstruct multi-channel images of a target based on a paired dataset; the MultiHetSRNet model includes:

[0131] The feature extraction module is used to extract features from the paired dataset to obtain a multi-channel feature map and a panchromatic feature map, and to concatenate the multi-channel feature map and the panchromatic feature map to generate an initial feature representation;

[0132] The multimodal heterogeneous module is used to model the nodes of the graph structure based on the multi-channel feature map and the panchromatic feature map, and to generate a multimodal heterogeneous graph based on the graph structure nodes, as well as a relational pattern matrix describing the spatial and cross-modal relationships between nodes;

[0133] The reconstruction module is used to aggregate local and global features based on the relational pattern matrix and reconstruct the target multi-channel image.

[0134] Example 3: This example provides a computer-readable medium storing a computer program, which, when executed by a processor, can implement the UAV image super-resolution reconstruction method based on multimodal heterogeneity as described in Example 1; specifically, the following steps are performed:

[0135] Step 1: Acquire a multi-channel image at a first resolution and a reference panchromatic image at a second resolution, and preprocess the multi-channel image and the reference panchromatic image to generate a paired dataset; the first resolution is lower than the second resolution.

[0136] Step 2: Extract features from the paired dataset to obtain multi-channel feature maps and panchromatic feature maps, and then concatenate the multi-channel feature maps and panchromatic feature maps to generate an initial feature representation;

[0137] Step 3: Model the nodes of the graph structure based on the multi-channel feature map and the panchromatic feature map, and generate a multimodal heterogeneous graph and a relational pattern matrix describing the spatial and cross-modal relationships between the nodes based on the graph structure nodes;

[0138] Step four: Aggregate local and global features based on the relational pattern matrix and reconstruct the target multi-channel image.

[0139] Steps two through four are implemented using the MultiHetSRNet model.

[0140] In this embodiment, on a test set of UAV imagery (including urban, rural, and vegetated scenes), the proposed method outperforms the current mainstream remote sensing imagery super-resolution models EDSR and TTST. PSNR is improved by 1.2-2.5 dB, reflecting higher pixel-level reconstruction accuracy; SSIM is improved by 0.03-0.07, indicating better structural similarity; LPIPS is reduced by 0.05-0.1, showing superior perceived quality; panchromatic consistency is reduced by 15%-20% in L1 error, validating the effectiveness of panchromatic guidance.

[0141] The table below shows the performance comparison between our model and EDSR and TTST on a UAV imagery test set (approximately 1934 images, including urban, rural, and vegetated scenes). Evaluation metrics include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Learned Perceptual Patch Similarity (LPIPS), and Panchromatic Consistency (L1 error).

[0142] Table 1 Performance Comparison of Different Models

[0143]

[0144] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for super-resolution reconstruction of UAV imagery based on multimodal heterogeneity, characterized in that, include: Acquire a multi-channel image at a first resolution and a reference panchromatic image at a second resolution, and preprocess the multi-channel image and the reference panchromatic image to generate a paired dataset; the values ​​of the first resolution are lower than the values ​​of the second resolution. The paired dataset is input into the MultiHetSRNet model to reconstruct the target multi-channel image: Feature extraction is performed on the paired dataset to obtain multi-channel feature maps and panchromatic feature maps, and the multi-channel feature maps and panchromatic feature maps are concatenated to generate an initial feature representation; Graph structure nodes are modeled based on multi-channel feature maps and panchromatic feature maps, and multimodal heterogeneous graphs are generated based on the graph structure nodes, as well as relational pattern matrices describing the spatial and cross-modal relationships between nodes; Based on the relational pattern matrix, local and global features are aggregated, and the target multi-channel image is reconstructed. The method for preprocessing multi-channel and panchromatic images to generate paired datasets includes: The multi-channel image is downsampled to generate a first multi-channel image with a third resolution; the value of the second resolution is lower than the value of the third resolution. Georegistration and pixel alignment are performed on multi-channel imagery and reference panchromatic imagery; The pixel values ​​of the multi-channel image, the reference panchromatic image, and the first multi-channel image are normalized, and the multi-channel image, the reference panchromatic image, and the first multi-channel image are cropped according to the first pixel, the second pixel, and the third pixel, respectively. The first pixel is 32×32, the second pixel is 53×53 pixels, and the third pixel is 128×128. Data augmentation is performed on the cropped images to generate paired datasets; Feature extraction is performed on the paired dataset to obtain multi-channel feature maps and panchromatic feature maps, and the multi-channel feature maps and panchromatic feature maps are concatenated to generate an initial feature representation; including the following methods: Multi-channel feature maps and panchromatic feature maps are extracted from the first multi-channel image and the reference panchromatic image respectively based on the same convolutional layer located in different branches; both the panchromatic feature map and the multi-channel feature map are at the third resolution; The panchromatic feature map and the multi-channel feature map are concatenated along the channel dimension to generate the initial feature representation, and the panchromatic feature map and the multi-channel feature map are merged to generate the combined feature map. The method involves modeling nodes based on multi-channel feature maps and panchromatic feature maps, generating a multimodal heterogeneous graph based on the graph structure nodes, and a relational pattern matrix describing the spatial and cross-modal relationships between nodes. Each pixel feature of the multi-channel feature map is defined as a multi-channel node, resulting in n = H × W multi-channel nodes. Each pixel feature of the panchromatic feature map is defined as a panchromatic node, resulting in m = H × W panchromatic nodes. For each multi-channel node and panchromatic node, the k-nearest neighbor algorithm is used to select a nearest neighbor nodes to construct multi-channel spatial neighborhood edges and panchromatic spatial neighborhood edges; cross-modal edges are constructed by connecting multi-channel nodes and panchromatic nodes based on spatial coordinate mapping; multi-channel spatial neighborhood edge adjacency matrix, panchromatic spatial neighborhood edge adjacency matrix, and cross-modal edge adjacency matrix are generated; Logical operations are performed on the multi-channel spatial neighborhood edge adjacency matrix, the full-color spatial neighborhood edge adjacency matrix, and the cross-modal edge adjacency matrix to generate a relational pattern matrix describing the spatial and cross-modal relationships between nodes; The relation schema matrix includes single schemas and combined schemas; A single schema consists of different adjacency matrices that each constitute a relation schema; Combinatorial patterns include generating a combined relational pattern using any z distinct adjacency matrices through XNOR and bitwise AND operations; z≥1; The method involves aggregating local and global features based on a relational pattern matrix and reconstructing a multi-channel image. The local relation matrix is ​​generated by weighted summation of all relation pattern matrices. The neighbor node features are aggregated from the local relation matrix based on the first layer graph convolutional network. The neighbor node features are then subjected to a first weight transformation and nonlinear activation. Finally, the local features are generated by a second weight transformation using the second layer graph convolutional network. The features of neighboring nodes are aggregated based on the following formula: , where the local relation matrix The size is (n+m)×(n+m); Let r represent the matrix of the r-th relation pattern, with size (n+m)×(n+m); r=1,2,…,7; The weights of the r-th relation schema matrix are scalars, ranging from [0, 1], and satisfy the following conditions: Weights are generated by weighted summation of the matrices of seven relation patterns. ; Local feature H is generated based on the following formula. Local : ； Where X is the input node feature matrix, which is reshaped by the concatenated features F; W1 and W2 are the weight matrices of the two-layer graph convolutional network; Represents the ReLU activation function; ; The first layer of graph convolutional network passes through The features X of neighboring nodes are aggregated, transformed by the weight matrix W1, and then activated by nonlinearity. (ReLU) and the second-layer weight matrix W2 generate the final local features; A global relation matrix is ​​generated by calculating cosine similarity based on all relation pattern matrices. The neighbor node features are aggregated from the full relation matrix using a third-layer graph convolutional network. The neighbor node features are then subjected to a third weight transformation and nonlinear activation. Finally, a fourth-layer graph convolutional network performs a fourth weight transformation to generate global features. Calculate the global similarity matrix using cosine similarity. : ； Where the element (i, j) represents the cosine similarity of the feature vectors of node i and node j; x i , x j Represents the feature vectors of nodes i and j; Global feature H is generated based on the following formula. Global : ； Where W3 and W4 represent the weight matrices of the graph convolutional network; The local and global features are fused to obtain unified node features, and the fusion is optimized based on contrastive learning loss; The method for reconstructing the target multi-channel image includes: Multi-channel node features are extracted from unified node features and reconstructed into a multi-channel node feature map. The PixelShuffle sub-pixel convolution reduces the number of channels in the multi-channel node feature map and rearranges it into spatial dimensions. The features are then adjusted through convolutional layers to obtain the target multi-channel image.

2. The UAV image super-resolution reconstruction method based on multimodal heterogeneity according to claim 1, characterized in that, The MultiHetSRNet model uses pixel-level loss, perceptual loss, full-color guided loss, contrastive learning loss, and total loss as loss functions, and is trained and optimized based on the Adam optimizer.

3. A UAV image super-resolution reconstruction system based on multimodal heterogeneity, characterized in that, The system is used to implement the UAV image super-resolution reconstruction method based on multimodal heterogeneity as described in claim 1 or 2; the system includes: The acquisition module is used to acquire a multi-channel image at a first resolution and a reference panchromatic image at a second resolution; the value of the first resolution is lower than the value of the second resolution. The preprocessing module is used to preprocess multi-channel images and reference panchromatic images to generate paired datasets; The MultiHetSRNet model is used to reconstruct multi-channel images of a target based on a paired dataset; the MultiHetSRNet model includes: The feature extraction module is used to extract features from the paired dataset to obtain a multi-channel feature map and a panchromatic feature map, and to concatenate the multi-channel feature map and the panchromatic feature map to generate an initial feature representation; The multimodal heterogeneous module is used to model the nodes of the graph structure based on the multi-channel feature map and the panchromatic feature map, and to generate a multimodal heterogeneous graph based on the graph structure nodes, as well as a relational pattern matrix describing the spatial and cross-modal relationships between nodes; The reconstruction module is used to aggregate local and global features based on the relational pattern matrix and reconstruct the target multi-channel image.

4. A computer-readable medium having a computer program stored thereon, characterized in that, The computer program, when executed by a processor, can implement the UAV image super-resolution reconstruction method based on multimodal heterogeneity as described in claim 1 or 2.

Images