A remote sensing image multi-source heterogeneous data fusion processing method and system

Abstract

The application discloses a kind of remote sensing image multi-source heterogeneous data fusion processing method and system, comprising: extracting global information using convolutional neural network to remote sensing image data, extract image local feature by clipping operation, capture local feature information in remote sensing image data;The mutual information between different modal variables is extracted by constructing cross-time domain attention mechanism through cyclic matrix to the local feature information, and the cross-time domain key information of high correlation is screened;Cross-time domain perception hierarchical aggregation module is established to the cross-time domain key information, and the detail information and edge information of remote sensing image are obtained;Using asymptotic fusion strategy to obtain global fusion data, introduce loss function to narrow semantic gap;Using the interactive network model of mixed contrast learning to repair missing information, obtain complete real-time remote sensing image multi-source heterogeneous fusion data, by constructing multi-collaborative deep fusion framework, efficient, high-precision fusion processing to multi-source remote sensing data is realized.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image data fusion technology, specifically to a method and system for fusion processing of multi-source heterogeneous remote sensing image data. Background Technology

[0002] With the rapid development of remote sensing technology, multi-source remote sensing data exhibits diverse characteristics in terms of spatial resolution, temporal resolution, and spectral resolution. These data are highly complementary and rich in information, playing an important role in fields such as urban planning, environmental monitoring, and disaster early warning. In order to fully utilize the information potential of multi-source remote sensing data and improve the accuracy and application efficiency of remote sensing image interpretation, multi-source heterogeneous data fusion technology has become one of the key research directions.

[0003] However, existing remote sensing image fusion methods still have many shortcomings, mainly including the following aspects:

[0004] (1) Traditional methods have a weak ability to model the semantic consistency of multi-source heterogeneous data. Currently, most fusion methods rely on manually designed features or simple pixel-level overlay strategies, which are difficult to effectively capture the complex semantic relationships between data acquired by different sensors, resulting in inconsistent semantic expressions in the fusion results and affecting the accuracy of data monitoring.

[0005] (2) There is a lack of modeling mechanism for dynamic evolution in the time dimension. Existing technologies often process remote sensing images of each time phase independently, without fully exploring the trend information of ground targets changing over time. This results in poor stability of the fusion results in dynamic scenarios, making it difficult to meet the needs of long-term remote sensing monitoring.

[0006] (3) Insufficient preservation of local detail information and blurred edge structure. In the process of multi-source data fusion, many algorithms focus more on the overall feature expression and ignore the effective extraction and preservation of local details of the image, resulting in blurred boundaries and unclear outlines of ground features in the fused remote sensing image, which affects the identification and interpretation of key ground features. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for fusion processing of multi-source heterogeneous remote sensing image data, in order to solve the technical problems in the prior art, such as difficulty in effectively capturing the complex semantic relationships between data acquired by different sensors, resulting in inconsistent semantic expression of fusion results, lack of modeling mechanism for dynamic evolution in the time dimension, insufficient preservation of local detail information, and blurred edge structure.

[0008] To solve the above-mentioned technical problems, the present invention specifically provides the following technical solution:

[0009] In a first aspect, the present invention provides a method for fusing multi-source heterogeneous remote sensing image data, comprising the following steps:

[0010] A convolutional neural network is used as an encoder to extract global information from time-continuous remote sensing image data. Local image features are extracted through cropping operations. A lightweight multi-scale pyramid model combined with a feature semantic extraction module is used to capture local feature information in the remote sensing image data.

[0011] The local feature information is used to construct a cross-temporal attention mechanism through a cyclic matrix to extract the mutual information between different modal variables and to filter highly correlated cross-temporal key information;

[0012] A cross-temporal perception hierarchical aggregation module is established for the cross-temporal key information, and the local feature information is aggregated on the corresponding global information according to the temporal domain to obtain the detailed information and edge information of the remote sensing image.

[0013] The detailed and edge information of the remote sensing image is fused according to the perception layer level using a progressive fusion strategy to obtain global fused data. A loss function is introduced into the global fused data to reduce the semantic gap and obtain semantic fusion information.

[0014] An interactive network model employing hybrid contrastive learning is used to repair missing information in the semantic fusion information, thereby obtaining complete real-time remote sensing image multi-source heterogeneous fusion data.

[0015] As a preferred embodiment of the present invention, a convolutional neural network is used as an encoder to extract global information from temporally continuous remote sensing image data, including:

[0016] Multi-source remote sensing image data are sequentially input into a deep convolutional neural network in chronological order. A multi-layered feature extraction module is constructed with convolutional layers as the backbone. By stacking convolutional layers and pooling layers, the spatial feature representation of the image is gradually extracted.

[0017] An attention mechanism is used to capture the semantic information of remote sensing images in the spatial feature representation. Each remote sensing image is mapped into a high-dimensional feature vector, preserving its key spatial distribution patterns and temporal evolution trends, and outputting a high-level feature vector.

[0018] Extract the global feature vector from the high-level feature vector, and use the global feature vector as global information in the multi-source heterogeneous data fusion stage.

[0019] As a preferred embodiment of the present invention, local image features are extracted through cropping operations, and multi-scale hierarchical aggregated features are obtained by applying a lightweight multi-scale pyramid model to the local image features, including:

[0020] In the convolutional layer, a lightweight multi-scale pyramid model is used to segment local information from global information. This lightweight multi-scale pyramid model has K channels, and each channel is assigned a corresponding dilation rate. ;

[0021] The number of scales in the lightweight multi-scale pyramid model is set to M, and the size of each channel is set to... ;

[0022] The size of the convolutional layer is constrained using a pyramid model, and the three layers are... The convolutional blocks are connected together to aggregate local features of the image;

[0023] By hierarchical feature aggregation, the feature information of each feature pyramid channel of the pyramid model is added one after another to obtain multi-scale hierarchical aggregated features.

[0024] As a preferred embodiment of the present invention, the multi-scale hierarchical aggregation features are combined with a feature semantic extraction module to capture local feature information in remote sensing image data, including:

[0025] The multi-scale hierarchical aggregated features are processed by a feature semantic extraction module to obtain local feature information. The feature semantic extraction module consists of three different convolutions: pointwise convolution, ordinary convolution, and dilated convolution. The pointwise convolution, ordinary convolution, and dilated convolution are applied to the multi-scale hierarchical aggregated features based on the time domain.

[0026] The features after each convolution are passed through a normalization layer to output aggregated concatenated features with the same target dimension. These aggregated concatenated features are then decoded using a multi-scale attention mechanism to output local feature information, the expression of which is:

[0027] ;

[0028] ;

[0029] in, Represents coding features, , These represent the ReLU and Sigmoid activation function weights of the convolutional layer, respectively. Indicates aggregated cascade features, This represents the local feature information output after decoding through a multi-scale attention mechanism. , }、 These correspond to the convolution operations of pointwise convolution, ordinary convolution, and dilated convolution, respectively. This represents the feature aggregation function.

[0030] As a preferred embodiment of the present invention, the local feature information is extracted using a cross-temporal attention mechanism constructed through a cyclic matrix to extract the mutual information between different modal variables, and highly correlated cross-temporal key information is filtered, including:

[0031] The local feature information extracted from multiple time points is organized into a feature sequence in chronological order. , Represents the local feature map at time step t;

[0032] Based on the feature sequence Construct a cyclic matrix M to establish the relationships between features at different time points. The expression for the cyclic matrix M is:

[0033] ;

[0034] The similarity metric for the circular matrix M is calculated using cosine similarity. , Let i and j represent the local feature maps at the i-th and j-th time steps, respectively.

[0035] An attention weight matrix is ​​constructed using the cyclic matrix to quantify the feature distribution across time points. The expression for the attention weight matrix is ​​as follows:

[0036] ;

[0037] in, This represents the cyclic matrix between the current time step and the i-th time step. This represents the attention weight between the current time step and the i-th time step;

[0038] The feature sequence is based on the attention weight matrix. Weighted summation yields the enhanced cross-temporal feature representation.

[0039] ;

[0040] in, This represents the time-domain characteristics at time j. This represents the attention weight matrix at times i and j. Indicates the time length of the feature sequence. This represents the cross-temporal feature representation at time i after enhancement;

[0041] The attention weights are sorted according to the attention weight matrix, and the time points with higher weights and their corresponding cross-temporal feature representations are selected to extract highly correlated cross-temporal key information.

[0042] As a preferred embodiment of the present invention, a cross-temporal sensing hierarchical aggregation module is established for the cross-temporal key information, and the local feature information is aggregated onto the corresponding global information according to the temporal domain to obtain the detailed information and edge information of the remote sensing image, including:

[0043] The cross-temporal feature representation at time i is used as the input of the local feature, and the global feature at the current time point is also input.

[0044] Construct a cross-temporal perception module based on an attention mechanism, and calculate the attention weights between the cross-temporal feature representation at the current time and the global features at different times;

[0045] By incorporating global information from historical moments into the local feature representation of the current frame through a weighted fusion function, an enhanced local feature representation with temporal memory capability is obtained.

[0046] The enhanced local feature representation is input into the convolutional layer of the feature semantic extraction module. The U-Net hierarchical aggregation architecture is used to sample the enhanced local feature representation layer by layer and concatenate it with the global features of the corresponding layer.

[0047] An attention gating module is embedded in the convolutional layer of the feature semantic extraction module to filter and enhance information that is sensitive to details and edges; and to obtain the corresponding detail information and edge information.

[0048] As a preferred embodiment of the present invention, the detailed information and edge information of the remote sensing image are fused according to the perception layer hierarchy using a progressive fusion strategy to obtain global fused data, including:

[0049] Based on the detailed information and edge information, a multi-level perceptual feature map is constructed using a deep convolutional neural network, and multi-level feature maps are extracted to form a perceptual hierarchical structure from shallow to deep.

[0050] Attention-weighted fusion modules are introduced at each of the perception hierarchical levels to fuse the detailed and edge information extracted at the current level with the global features of the corresponding level. The output of each fusion layer is used as the input for the next fusion layer to obtain progressively enhanced fusion data.

[0051] An edge-guided fusion strategy is used to fuse the fused data, fusing the edge and texture information of the image to obtain local details and edge features of the remote sensing image;

[0052] The fusion ratio is adjusted in real time at the perception layer level, and global fusion data is obtained after multi-level progressive fusion.

[0053] As a preferred embodiment of the present invention, a loss function is introduced into the globally fused data to reduce the semantic gap and obtain semantic fusion information, including:

[0054] The cross-entropy loss function is used to calculate the semantic relevance at each of the perceptual hierarchical levels for the globally fused data. The expression is as follows:

[0055] ;

[0056] in, , These represent the weight coefficients at the primary perception layer and the secondary perception layer, respectively. Indicates the value of the weight parameter. , These represent the actual values ​​of the fused data mapped to the primary perception layer and the secondary perception layer, respectively. This indicates the correlation between the images contained in the attention weight matrix.

[0057] As a preferred embodiment of the present invention, an interactive network model employing hybrid contrastive learning is used to repair missing information in the semantic fusion information, thereby obtaining complete real-time remote sensing image multi-source heterogeneous fusion data, including:

[0058] An interactive neural network structure with two branches, a main branch and an auxiliary branch, is constructed. The main branch is used to extract the deep representation of the current semantic fusion information, and the auxiliary branch is used to extract the global features contained in the cross-temporal key information.

[0059] A hybrid contrastive learning strategy is introduced between the two branches, the main and the auxiliary branches, to compare and learn remote sensing image data at the same time and to quantify the sample similarity by maximizing the loss function.

[0060] A cross-attention mechanism is used to extract potential information that can be used to repair missing regions at each of the current perception hierarchies.

[0061] Based on contrastive learning, semantic consistency representations are obtained, and content reconstruction is performed on missing or low-confidence regions in the semantic fusion graph by combining contextual information.

[0062] The repaired semantic fusion feature map is output and mapped back to pixel space to form complete remote sensing image fusion data.

[0063] A second aspect of the present invention provides a remote sensing image multi-source heterogeneous data fusion processing system, comprising:

[0064] The data acquisition module is used to acquire multi-source remote sensing image data from different sensor types, different time series, and different spatial resolutions;

[0065] The feature extraction module is connected to the data acquisition module and uses a convolutional neural network as an encoder to process the time-continuous remote sensing image data and extract the global feature information of the image.

[0066] The local feature enhancement module is connected to the feature extraction module. It extracts the region of interest through image cropping and uses a lightweight multi-scale pyramid model combined with the feature semantic extraction module to capture local details and edge information in remote sensing images.

[0067] The cross-temporal modeling module is connected to the local feature enhancement module. It constructs a cross-temporal attention mechanism based on a cyclic matrix, extracts the interrelationships between variables in different time phases and different modalities, and filters highly correlated cross-temporal key information.

[0068] The hierarchical aggregation module is connected to the cross-temporal modeling module to establish a cross-temporal perception hierarchical aggregation structure, which integrates local feature information into the corresponding global features layer by layer according to temporal information to enhance the detail expression capability of remote sensing images.

[0069] The progressive fusion module, connected to the hierarchical aggregation module, adopts a progressive fusion strategy to perform feature fusion at each perception layer level to generate global fused data with high spatial resolution and semantic consistency.

[0070] The semantic repair module, connected to the progressive fusion module, introduces an interactive network model of hybrid contrastive learning to intelligently repair missing or abnormal regions in the semantic fusion information.

[0071] The loss optimization module, connected to the semantic repair module, optimizes and trains the global fused data by designing a multi-task loss function to obtain efficient fused data.

[0072] Compared with the prior art, the present invention has the following advantages:

[0073] This invention extracts high-level semantic global features from remote sensing images by combining a feature extraction module with convolutional neural networks and attention mechanisms. A local feature enhancement module captures local details and edge information through image cropping and a lightweight multi-scale pyramid model, achieving effective perception of the multi-layered structure of remote sensing images and significantly improving the spatial resolution and semantic clarity of the fusion results. A hierarchical aggregation module establishes a cross-temporal perception hierarchical aggregation structure, fusing local features layer by layer into corresponding global features based on temporal information. This further strengthens the information complementarity of remote sensing images in both time and space dimensions, thereby more accurately restoring the details and edge information of the images. This avoids information redundancy or conflict problems that may occur in traditional methods, improving the controllability of the fusion process and the accuracy of the results, while also considering computational efficiency. By constructing a deep fusion framework that coordinates "global-local-temporal-semantic," efficient and high-precision fusion processing of multi-source remote sensing data is achieved. Attached Figure Description

[0074] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0075] Figure 1 This is a flowchart of a remote sensing image multi-source heterogeneous data fusion processing method provided in an embodiment of the present invention;

[0076] Figure 2 This is a block diagram of a remote sensing image multi-source heterogeneous data fusion processing system provided in an embodiment of the present invention. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] like Figure 1 and Figure 2 As shown, this invention provides a method for fusing multi-source heterogeneous remote sensing image data, comprising the following steps:

[0079] A convolutional neural network is used as an encoder to extract global information from time-continuous remote sensing image data. Local image features are extracted through cropping operations. A lightweight multi-scale pyramid model combined with a feature semantic extraction module is used to capture local feature information in the remote sensing image data.

[0080] In this embodiment, a convolutional neural network is used as an encoder to extract global information from remote sensing images, and image cropping is combined to obtain local regional features. Then, a lightweight multi-scale pyramid model is used in combination with a feature semantic extraction module to capture local feature information, thereby realizing multi-level perception of detailed structure, edge information and semantic content in remote sensing images, which significantly enhances data expression ability and fusion accuracy.

[0081] The local feature information is used to construct a cross-temporal attention mechanism through a cyclic matrix to extract the mutual information between different modal variables and to filter highly correlated cross-temporal key information;

[0082] In this embodiment, a cross-temporal attention mechanism is constructed using a cyclic matrix to effectively model the dynamic relationship between different time points, extract the mutual information between different modal variables, and filter out highly correlated cross-temporal key information, thereby improving the model's ability and stability in modeling time-series remote sensing data. This approach is suitable for processing multi-source heterogeneous data with temporal continuity.

[0083] A cross-temporal perception hierarchical aggregation module is established for the cross-temporal key information, and the local feature information is aggregated on the corresponding global information according to the temporal domain to obtain the detailed information and edge information of the remote sensing image.

[0084] In this embodiment, by establishing a cross-temporal sensing hierarchical aggregation module, local features are aggregated layer by layer to the corresponding global features based on temporal information, realizing the fusion and complementarity of information in the temporal and spatial dimensions, and further improving the detail restoration capability and edge clarity of remote sensing images in the process of temporal evolution.

[0085] The detailed and edge information of the remote sensing image is fused according to the perception layer level using a progressive fusion strategy to obtain global fused data. A loss function is introduced into the global fused data to reduce the semantic gap and obtain semantic fusion information.

[0086] In this embodiment, a progressive fusion strategy based on perception hierarchy is adopted, which sequentially fuses detail and edge information according to the hierarchical structure. This avoids information conflict or redundancy problems that may occur in traditional fusion methods, improves the controllability and robustness of the fusion result, and reduces computational complexity and improves algorithm efficiency.

[0087] In this embodiment, a loss function is used to optimize the fusion process, effectively narrowing the semantic gap between different modal data and enhancing the consistency and interpretability of the fusion results at the high-level semantic level.

[0088] An interactive network model employing hybrid contrastive learning is used to repair missing information in the semantic fusion information, thereby obtaining complete real-time remote sensing image multi-source heterogeneous fusion data.

[0089] In this embodiment, an interactive network model of hybrid contrastive learning is used to intelligently repair the missing parts in the semantic fusion information. This not only improves the integrity and accuracy of the fused data, but also enhances the system's adaptability and real-time response performance, making it suitable for remote sensing monitoring and decision support in complex environments.

[0090] A convolutional neural network is used as an encoder to extract global information from temporally continuous remote sensing image data, including:

[0091] Multi-source remote sensing image data are sequentially input into a deep convolutional neural network in chronological order. A multi-layered feature extraction module is constructed with convolutional layers as the backbone. By stacking convolutional layers and pooling layers, the spatial feature representation of the image is gradually extracted.

[0092] In this embodiment, by inputting multi-source remote sensing image data into a deep convolutional neural network in chronological order, and combining the stacked structure of convolutional and pooling layers, the spatial feature representation of the image can be extracted layer by layer, while preserving its temporal evolution trend, providing a high-dimensional semantic representation containing dynamic change information for subsequent fusion processing.

[0093] An attention mechanism is used to capture the semantic information of remote sensing images in the spatial feature representation. Each remote sensing image is mapped into a high-dimensional feature vector, preserving its key spatial distribution patterns and temporal evolution trends, and outputting a high-level feature vector.

[0094] In this embodiment, an attention mechanism is introduced on the basis of spatial feature extraction, which can effectively filter out key areas with significant semantic meaning in remote sensing images, improve the model's ability to identify complex ground objects, and make each image ultimately mapped into a high-quality high-dimensional feature vector with stronger discriminativeness and fusionability.

[0095] Extract the global feature vector from the high-level feature vector, and use the global feature vector as global information in the multi-source heterogeneous data fusion stage.

[0096] In this embodiment, the high-level feature vector not only contains the key spatial distribution patterns of the original image, but also integrates the semantic consistency and change information between multiple temporal phases, forming a unified and compact global feature representation.

[0097] In this embodiment, the global feature vector has good generalization ability and modal adaptability, and can establish semantic associations between data from different sensors, different resolutions or different imaging methods, thereby improving the consistency and compatibility of multi-source heterogeneous remote sensing images in the fusion process.

[0098] Local image features are extracted through cropping operations, and a lightweight multi-scale pyramid model is used to obtain multi-scale hierarchical aggregated features from these local image features, including:

[0099] In the convolutional layer, a lightweight multi-scale pyramid model is used to segment local information from global information. This lightweight multi-scale pyramid model has K channels, and each channel is assigned a corresponding dilation rate. ;

[0100] In this embodiment, by cropping the original image or global feature map, focusing on key areas, local features of the image are effectively extracted, enhancing the model's ability to perceive details such as small targets, edge structures, and complex land cover types in remote sensing images, and improving the spatial resolution and semantic clarity of the fusion results.

[0101] The number of scales in the lightweight multi-scale pyramid model is set to M, and the size of each channel is set to... ;

[0102] The size of the convolutional layer is constrained using a pyramid model, and the three layers are... The convolutional blocks are connected together to aggregate local features of the image;

[0103] In this embodiment, a pyramid model is used to construct multi-scale convolutional blocks, and three convolutional modules of different scales are cascaded to form a feature representation structure from coarse to fine. This realizes information complementarity and fusion between features at different levels, and significantly enhances the model's ability to identify and represent multi-scale targets in remote sensing images.

[0104] By hierarchical feature aggregation, the feature information of each feature pyramid channel of the pyramid model is added one after another to obtain multi-scale hierarchical aggregated features.

[0105] In this embodiment, by successively adding and fusing the feature information output from each level of the pyramid, key structural information at different scales is preserved, avoiding the information loss problem that may be caused by single-scale feature extraction, and improving the consistency and anti-interference ability of feature representation.

[0106] The multi-scale hierarchical aggregation features are combined with the feature semantic extraction module to capture local feature information in remote sensing image data, including:

[0107] The multi-scale hierarchical aggregated features are processed by a feature semantic extraction module to obtain local feature information. The feature semantic extraction module consists of three different convolutions: pointwise convolution, ordinary convolution, and dilated convolution. The pointwise convolution, ordinary convolution, and dilated convolution are applied to the multi-scale hierarchical aggregated features based on the time domain.

[0108] In this embodiment, the feature semantic extraction module uses three different structures—pointwise convolution, ordinary convolution, and dilated convolution—to process multi-scale hierarchical aggregated features in parallel. This allows the module to capture multi-level semantic information such as channel compression, spatial structure, and large-scale context, thereby improving the model's recognition accuracy and generalization ability for complex ground objects in remote sensing images.

[0109] The features after each convolution are passed through a normalization layer to output aggregated concatenated features with the same target dimension. These aggregated concatenated features are then decoded using a multi-scale attention mechanism to output local feature information, the expression of which is:

[0110] ;

[0111] ;

[0112] in, Represents coding features, , These represent the ReLU and Sigmoid activation function weights of the convolutional layer, respectively. Indicates aggregated cascade features, This represents the local feature information output after decoding through a multi-scale attention mechanism. }、 }、 These correspond to the convolution operations of pointwise convolution, ordinary convolution, and dilated convolution, respectively. This represents the feature aggregation function.

[0113] In this embodiment, by normalizing the outputs of the three convolution operations and concatenating them into aggregated cascaded features of a unified dimension, the consistency and stability of feature representation are enhanced, providing high-quality input for subsequent attention decoding and effectively avoiding the dimension mismatch problem in the multi-scale feature fusion process.

[0114] The local feature information is used to construct a cross-temporal attention mechanism using a cyclic matrix to extract the mutual information between different modal variables, and to filter highly correlated cross-temporal key information, including:

[0115] The local feature information extracted from multiple time points is organized into a feature sequence in chronological order. , Represents the local feature map at time step t;

[0116] Based on the feature sequence Construct a cyclic matrix M to establish the relationships between features at different time points. The expression for the cyclic matrix M is:

[0117] ;

[0118] The similarity metric for the circular matrix M is calculated using cosine similarity. , Let i and j represent the local feature maps at the i-th and j-th time steps, respectively.

[0119] In this embodiment, local feature maps at multiple time points are organized sequentially into a feature sequence, and a cyclic matrix based on cosine similarity is constructed. This can effectively model the evolution of remote sensing images in the time dimension and reveal the potential semantic associations between different time points.

[0120] An attention weight matrix is ​​constructed using the cyclic matrix to quantify the feature distribution at each time point. The expression for the attention weight matrix is ​​as follows:

[0121] ;

[0122] in, This represents the cyclic matrix between the current time step and the i-th time step. This represents the attention weight between the current time step and the i-th time step;

[0123] In this embodiment, an attention weight matrix is ​​constructed based on a cyclic matrix to quantify the differences in feature distribution between time points. Furthermore, an enhanced cross-temporal feature representation is generated through a weighted summation method. This not only preserves the continuity of temporal information but also highlights the contribution of highly correlated moments, significantly improving the model's time perception capability.

[0124] The feature sequence is based on the attention weight matrix. Weighted summation yields the enhanced cross-temporal feature representation.

[0125] ;

[0126] in, This represents the time-domain characteristics at time j. This represents the attention weight matrix at times i and j. Indicates the time length of the feature sequence. This represents the cross-temporal feature representation at time i after enhancement;

[0127] The attention weights are sorted according to the attention weight matrix, and the time points with higher weights and their corresponding cross-temporal feature representations are selected to extract highly correlated cross-temporal key information.

[0128] In this embodiment, by sorting the attention weights, the time points with higher weights and their corresponding cross-temporal feature representations are selected, which can quickly locate multimodal variable information that is highly relevant to the current moment, reduce redundant calculations, and improve the response speed and accuracy of the fusion system.

[0129] A cross-temporal sensing hierarchical aggregation module is established for the aforementioned cross-temporal key information. This module aggregates the local feature information onto the corresponding global information based on the temporal domain, thereby obtaining detailed and edge information of the remote sensing image, including:

[0130] The cross-temporal feature representation at time i is used as the input of the local feature, and the global feature at the current time point is also input.

[0131] Construct a cross-temporal perception module based on an attention mechanism, and calculate the attention weights between the cross-temporal feature representation at the current time and the global features at different times;

[0132] In this embodiment, an attention-based cross-temporal perception module is constructed to calculate the attention weights between the global features at the current time and at different times. A weighted fusion function is then used to introduce the global information of historical times into the local feature expression of the current frame, thereby realizing the memory function of time series data. This method effectively captures the dynamic change patterns in the time dimension and improves the model's ability to understand complex spatiotemporal scenes.

[0133] By incorporating global information from historical moments into the local feature representation of the current frame through a weighted fusion function, an enhanced local feature representation with temporal memory capability is obtained.

[0134] The enhanced local feature representation is input into the convolutional layer of the feature semantic extraction module. The U-Net hierarchical aggregation architecture is used to sample the enhanced local feature representation layer by layer and concatenate it with the global features of the corresponding layer.

[0135] In this embodiment, under the U-Net hierarchical aggregation architecture, the enhanced local feature representation is sampled layer by layer and concatenated with the global features of the corresponding layer, realizing an effective combination of local details and global context information. This hierarchical feature fusion method not only preserves rich spatial structure information, but also enhances the model's understanding and adaptability to a wide range of environmental backgrounds.

[0136] An attention gating module is embedded in the convolutional layer of the feature semantic extraction module to filter and enhance information that is sensitive to details and edges; and to obtain the corresponding detail information and edge information.

[0137] In this embodiment, the attention gating module embedded in the convolutional layer of the feature semantic extraction module can intelligently filter and enhance information that is sensitive to details and edges. This enables the model to significantly improve the recognition accuracy of fine structures in remote sensing images while maintaining overall semantic consistency.

[0138] The detailed and edge information of the remote sensing image is fused using a progressive fusion strategy according to the perception layer hierarchy to obtain global fused data, including:

[0139] Based on the detailed information and edge information, a multi-level perceptual feature map is constructed using a deep convolutional neural network, and multi-level feature maps are extracted to form a perceptual hierarchical structure from shallow to deep.

[0140] In this embodiment, by extracting multi-level feature maps based on deep convolutional neural networks and forming a perception hierarchy from shallow to deep, it is possible to effectively capture the complete information chain from low-level edge textures to high-level semantic objects in remote sensing images.

[0141] Attention-weighted fusion modules are introduced at each of the perception hierarchical levels to fuse the detailed and edge information extracted at the current level with the global features of the corresponding level. The output of each fusion layer is used as the input for the next fusion layer to obtain progressively enhanced fusion data.

[0142] In this embodiment, an attention-weighted fusion module is introduced at each layer of perception hierarchy to adaptively fuse the details and edge information of the current layer with the global features of the corresponding layer. This not only enhances the information retention of key areas but also effectively suppresses the interference of redundant or conflicting information, significantly improving the accuracy and consistency of the fusion results.

[0143] An edge-guided fusion strategy is used to fuse the fused data, fusing the edge and texture information of the image to obtain local details and edge features of the remote sensing image;

[0144] The fusion ratio is adjusted in real time at the perception layer level, and global fusion data is obtained after multi-level progressive fusion.

[0145] In this embodiment, the output of each fusion layer serves as the input for the next fusion layer, thereby achieving gradual optimization and enhancement of information across different levels. This results in a good feature transfer and accumulation effect, giving the final fusion result stronger structural integrity and semantic coherence.

[0146] A loss function is introduced into the globally fused data to reduce the semantic gap and obtain semantic fusion information, including:

[0147] The cross-entropy loss function is used to calculate the semantic relevance at each of the perceptual hierarchical levels for the globally fused data. The expression is as follows:

[0148] ;

[0149] in, , These represent the weight coefficients at the primary perception layer and the secondary perception layer, respectively. Indicates the value of the weight parameter. , These represent the actual values ​​of the fused data mapped to the primary perception layer and the secondary perception layer, respectively. This indicates the correlation between the images contained in the attention weight matrix.

[0150] In this embodiment, by introducing the cross-entropy loss function, the semantic correlation between the main layer and the sub-layer is calculated at each layer of the perception hierarchy. This can effectively quantify the semantic deviation between remote sensing data of different modalities or different time points, thereby guiding the model to learn a more consistent high-level semantic representation and significantly improving the fusion quality of multi-source heterogeneous data.

[0151] An interactive network model employing hybrid contrastive learning is used to repair missing information in the semantic fusion information, thereby obtaining complete real-time remote sensing image multi-source heterogeneous fusion data, including:

[0152] An interactive neural network structure with two branches, a main branch and an auxiliary branch, is constructed. The main branch is used to extract the deep representation of the current semantic fusion information, and the auxiliary branch is used to extract the global features contained in the cross-temporal key information.

[0153] In this embodiment, by designing an interactive neural network architecture that includes a main branch and auxiliary branches, the deep representation of the current semantic fusion information and the global features of key cross-temporal information are extracted respectively, realizing the joint modeling of spatial semantics and temporal evolution information, and enhancing the ability to understand dynamic changes and static structures in remote sensing images.

[0154] A hybrid contrastive learning strategy is introduced between the two branches, the main and the auxiliary branches, to compare and learn remote sensing image data at the same time and to quantify the sample similarity by maximizing the loss function.

[0155] In this embodiment, a hybrid contrastive learning mechanism is introduced between the main and auxiliary branches. By maximizing the similarity between samples of remote sensing images from different perspectives or modalities at the same time, the semantic consistency and discriminative ability of feature representation can be effectively enhanced, and the robustness of the model to interference factors such as noise and occlusion can be significantly improved.

[0156] A cross-attention mechanism is used to extract potential information that can be used to repair missing regions at each of the current perception hierarchies.

[0157] In this embodiment, the cross-attention mechanism is used to extract contextual potential information that can be used to repair missing regions at the perception hierarchical level, and to accurately locate the feature expressions that are semantically related to the missing regions, thereby achieving high-quality content reconstruction and improving the detail integrity and visual quality of remote sensing images.

[0158] Based on contrastive learning, semantic consistency representations are obtained, and content reconstruction is performed on missing or low-confidence regions in the semantic fusion graph by combining contextual information.

[0159] The repaired semantic fusion feature map is output and mapped back to pixel space to form complete remote sensing image fusion data.

[0160] In this embodiment, the repaired semantic fusion feature map has higher integrity and accuracy, and can be further mapped back to pixel space to generate complete remote sensing image fusion data, so that the remote sensing image fusion data achieves a high level in terms of structural integrity, semantic consistency and visual quality.

[0161] Second embodiment: A remote sensing image multi-source heterogeneous data fusion processing system, comprising:

[0162] The data acquisition module is used to acquire multi-source remote sensing image data from different sensor types, different time series, and different spatial resolutions;

[0163] The feature extraction module is connected to the data acquisition module and uses a convolutional neural network as an encoder to process the time-continuous remote sensing image data and extract the global feature information of the image.

[0164] The local feature enhancement module is connected to the feature extraction module. It extracts the region of interest through image cropping and uses a lightweight multi-scale pyramid model combined with the feature semantic extraction module to capture local details and edge information in remote sensing images.

[0165] The cross-temporal modeling module is connected to the local feature enhancement module. It constructs a cross-temporal attention mechanism based on a cyclic matrix, extracts the interrelationships between variables in different time phases and different modalities, and filters highly correlated cross-temporal key information.

[0166] The hierarchical aggregation module is connected to the cross-temporal modeling module to establish a cross-temporal perception hierarchical aggregation structure, which integrates local feature information into the corresponding global features layer by layer according to temporal information to enhance the detail expression capability of remote sensing images.

[0167] The progressive fusion module, connected to the hierarchical aggregation module, adopts a progressive fusion strategy to perform feature fusion at each perception layer level to generate global fused data with high spatial resolution and semantic consistency.

[0168] The semantic repair module, connected to the progressive fusion module, introduces an interactive network model of hybrid contrastive learning to intelligently repair missing or abnormal regions in the semantic fusion information.

[0169] The loss optimization module, connected to the semantic repair module, optimizes and trains the global fused data by designing a multi-task loss function to obtain efficient fused data.

[0170] In this embodiment, the entire system has the ability to fully support multi-source heterogeneous data input and collaborative modeling. It significantly improves information expression capabilities through a global-local collaborative feature extraction mechanism, strengthens the complementarity of spatiotemporal information by combining cross-temporal attention modeling and hierarchical aggregation structure, and improves fusion accuracy and efficiency by adopting a progressive fusion strategy. It avoids information redundancy or conflict problems that may occur in traditional methods, and improves the controllability of the fusion process and the accuracy of the results.

[0171] This invention extracts high-level semantic global features from remote sensing images by combining a feature extraction module with convolutional neural networks and attention mechanisms. A local feature enhancement module captures local details and edge information through image cropping and a lightweight multi-scale pyramid model, achieving effective perception of the multi-layered structure of remote sensing images and significantly improving the spatial resolution and semantic clarity of the fusion results. A hierarchical aggregation module establishes a cross-temporal perception hierarchical aggregation structure, fusing local features layer by layer into corresponding global features based on temporal information. This further strengthens the information complementarity of remote sensing images in both time and space dimensions, thereby more accurately restoring the details and edge information of the images. This avoids information redundancy or conflict problems that may occur in traditional methods, improving the controllability of the fusion process and the accuracy of the results, while also considering computational efficiency. By constructing a deep fusion framework that coordinates "global-local-temporal-semantic," efficient and high-precision fusion processing of multi-source remote sensing data is achieved.

[0172] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. A remote sensing image multi-source heterogeneous data fusion processing method, characterized in that, The method comprises the following steps: The time-continuous remote sensing image data adopts a convolutional neural network as an encoder to extract global information, extracts image local features through cropping operations, and adopts a lightweight multi-scale pyramid model combined with a feature semantic extraction module to capture local feature information in the remote sensing image data; The local feature information is extracted through a cyclic matrix to construct a cross-time domain attention mechanism to extract mutual information between different modal variables and filter highly correlated cross-time domain key information; The cross-time domain key information is established to build a cross-time domain perception hierarchical aggregation module, the local feature information is aggregated on the corresponding global information according to the time domain, and the detail information and edge information of the remote sensing image are obtained; The detail information and edge information of the remote sensing image are fused according to the perception hierarchical level respectively by using an asymptotic fusion strategy, global fusion data is obtained, a loss function is introduced to the global fusion data to narrow the semantic gap, and semantic fusion information is obtained; A hybrid contrast learning interactive network model is used to repair missing information in the semantic fusion information, and complete real-time remote sensing image multi-source heterogeneous fusion data is obtained; The local feature information is extracted through a cyclic matrix to construct a cross-time domain attention mechanism to extract mutual information between different modal variables and filter highly correlated cross-time domain key information, comprising: organize the local feature information extracted at multiple time points into a feature sequence in time sequence , denotes the local feature map at the t-th time step According to the characteristic sequence A circulant matrix M is constructed to build the mutual relationship between the characteristics at different time, and the expression of the circulant matrix M is: ； Wherein, the similarity measure of the circulant matrix M is calculated by cosine similarity, , respectively represent the local feature maps of the i, j-th time step. An attention weight matrix is constructed through the cyclic matrix to quantify the feature distribution between each time point, and the expression of the attention weight matrix is: ； wherein, represents a circulant matrix between the current time and the i-th time, represents an attention weight between the current time and the i-th time; performing weighted sum on the feature sequence according to the attention weight matrix performing weighted sum on the feature sequence according to the attention weight matrix ； wherein, denotes the time-domain feature at time j, denotes the attention weight matrix at time i, j, denotes the time length of the feature sequence, denotes the cross-time domain feature representation at time i after enhancement; The attention weights are sorted according to the attention weight matrix, and the time points with higher weights and the corresponding cross-time domain feature representations are filtered out to extract highly correlated cross-time domain key information.

2. The remote sensing image multi-source heterogeneous data fusion processing method according to claim 1, wherein The time-continuous remote sensing image data adopts a convolutional neural network as an encoder to extract global information, comprising: The multi-source remote sensing image data is sequentially input into a deep convolutional neural network in time sequence, a multi-level feature extraction module is constructed with a convolutional layer as the main body, the spatial feature expression of the image is gradually extracted through stacked convolutional layers and pooling layers, and the semantic information of the remote sensing image is captured by using an attention mechanism, each remote sensing image is mapped into a high-dimensional feature vector, the key spatial distribution rule and time evolution trend are retained, and a high-level feature vector is outputted; The global feature vector of the high-level feature vector is extracted, and the global feature vector is taken as the global information in the multi-source heterogeneous data fusion stage.

3. The remote sensing image multi-source heterogeneous data fusion processing method according to claim 2, wherein The image local features are extracted through cropping operations, and a lightweight multi-scale pyramid model is used to obtain multi-scale hierarchical aggregation features, comprising: The feature information of each feature pyramid channel of the pyramid model is added successively through hierarchical feature aggregation to obtain multi-scale hierarchical aggregation features. The convolutional layer adopts a lightweight multi-scale pyramid model for local information segmentation on the global information, the lightweight multi-scale pyramid model is provided with K channels, each channel is provided with a corresponding expansion rate ; The number of scales of the lightweight multi-scale pyramid model is set as M, and the size of each channel is set as ; The size of the convolutional layer is constrained by a pyramid model, three convolutional blocks are connected to aggregate local features of the image; 4. The remote sensing image multi-source heterogeneous data fusion processing method according to claim 3, wherein ​ The multi-scale hierarchical aggregation feature combination feature semantic extraction module captures local feature information in remote sensing image data, including: The multi-scale hierarchical aggregation feature is obtained by a feature semantic extraction module to obtain local feature information, and the feature semantic extraction module is composed of three different convolutions, namely point-by-point convolution, ordinary convolution and dilated convolution. Each convolutional feature is output through a normalization layer to output an aggregated cascaded feature with the same target dimension, and the aggregated cascaded feature is output after decoding by a multi-scale attention mechanism, and its expression is: ； ； wherein, represents an encoding feature, , respectively represent the Relu and Sigmoid activation functions weights of the convolution layer, represents an aggregated concatenated feature, represents the local feature information output after decoding by the multi-scale attention mechanism, }、 、 } respectively correspond to the convolution operations of the point-wise convolution, the ordinary convolution and the dilated convolution, represents a feature aggregation function.

5. The remote sensing image multi-source heterogeneous data fusion processing method according to claim 4, characterized in that, A cross-time domain perception hierarchical aggregation module is established for the cross-time domain key information, and the local feature information is aggregated on the corresponding global information according to the time domain to obtain the detail information and edge information of the remote sensing image, including: The cross-time domain feature representation at the i time is input as the local feature, and the global feature at the current time point is also input; A cross-time domain perception module based on an attention mechanism is constructed to calculate the attention weight between the cross-time domain feature representation at the current time and the global feature at different times; The global information at the historical time is introduced into the local feature expression of the current frame through a weighted fusion function to obtain an enhanced local feature representation with time sequence memory capability; The enhanced local feature representation is input into the convolution layer of the feature semantic extraction module, and a U-Net hierarchical aggregation architecture is adopted to sample the enhanced local feature representation layer by layer and splice it with the global feature at the corresponding level; An attention gate module is embedded in the convolution layer of the feature semantic extraction module to filter and strengthen information sensitive to details and edges.

6. The remote sensing image multi-source heterogeneous data fusion processing method according to claim 5, characterized in that, The detail information and edge information of the remote sensing image are fused according to the perception hierarchical level by an asymptotic fusion strategy to obtain global fusion data, including: A multi-level perception feature map is constructed based on a deep convolutional neural network according to the detail information and edge information to extract multi-level feature maps and form a perception hierarchical level from shallow to deep; An attention weighted fusion module is introduced on each perception hierarchical level to fuse the detail information and edge information extracted at the current level with the global feature at the corresponding level, and the output after fusion at each level is input into the next layer for fusion to obtain layer-by-layer enhanced fusion data; An edge-guided fusion strategy is adopted for the fusion data to fuse the edge and texture information of the image to obtain the local details and edge features of the remote sensing image; The fusion ratio is adjusted in real time on the perception hierarchical level, and the global fusion data is obtained after multi-level progressive fusion.

7. The remote sensing image multi-source heterogeneous data fusion processing method according to claim 6, characterized in that, The loss function is introduced to the global fusion data to narrow the semantic gap and obtain semantic fusion information, including: The cross-entropy loss function is used to calculate the relevance of the corresponding semantics on each layer of the perception hierarchical level, and its expression is: ； wherein, , respectively represent the weight coefficients on the primary and secondary perceptual hierarchical levels, represent the weight parameter values, , respectively represent the ground truth values of the fused data mapped onto the primary and secondary perceptual hierarchical levels, represent the correlation between images contained in the attention weight matrix.

8. The multi-source heterogeneous data fusion processing method of remote sensing images according to claim 7, characterized in that, The interactive network model using hybrid contrast learning is used to repair the missing information in the semantic fusion information, and complete real-time remote sensing image multi-source heterogeneous fusion data is obtained, including: An interactive neural network structure including a main branch and an auxiliary branch is constructed, wherein the main branch is used to extract deep representations of the current semantic fusion information, and the auxiliary branch is used to extract global features contained in the cross-time domain key information; A hybrid contrast learning strategy is introduced between the main branch and the auxiliary branch to compare and learn the remote sensing image data at the same time, and the sample similarity is maximized by the loss function; The cross-attention mechanism is used to extract potential information of each layer of the perception hierarchical level at the current time, which can be used to repair the missing area; Based on the contrast learning, a semantic consistency representation is obtained, and the content of the missing or low-confidence area in the semantic fusion graph is reconstructed based on the context information; The repaired semantic fusion feature map is outputted and mapped back to the pixel space to form complete remote sensing image fusion data.

9. A remote sensing image multi-source heterogeneous data fusion processing system, characterized in that, including: A data acquisition module is configured to acquire multi-source remote sensing image data from different sensor types, different time sequences, and different spatial resolutions; A feature extraction module is connected with the data acquisition module and uses a convolutional neural network as an encoder to process time-continuous remote sensing image data and extract global feature information of the image; A local feature enhancement module is connected with the feature extraction module, extracts the region of interest through image cropping, and captures local details and edge information in the remote sensing image using a lightweight multi-scale pyramid model combined with a feature semantic extraction module; A cross-time domain modeling module is connected with the local feature enhancement module, constructs a cross-time domain attention mechanism based on a circulant matrix, extracts the mutual relationship between different time phases and different modal variables, and filters highly correlated cross-time domain key information; A hierarchical aggregation module is connected with the cross-time domain modeling module, establishes a cross-time domain perception hierarchical aggregation structure, and gradually fuses local feature information to corresponding global features according to time domain information to enhance the detail expression capability of the remote sensing image; An asymptotic fusion module is connected with the hierarchical aggregation module, uses an asymptotic fusion strategy to perform feature fusion at each perception hierarchical level to generate global fusion data with high spatial resolution and semantic consistency; A semantic repair module is connected with the asymptotic fusion module, introduces an interactive network model using hybrid contrast learning to intelligently repair missing or abnormal areas in the semantic fusion information; A loss optimization module is connected with the semantic repair module, optimizes and trains the global fusion data by designing a multi-task loss function to obtain efficient fusion data. Wherein, the local feature information is extracted through a cyclic matrix to construct a cross-time domain attention mechanism to extract mutual information between different modal variables, and to screen highly correlated cross-time domain key information, including: organize the local feature information extracted at multiple time points into a feature sequence in time sequence , denotes the local feature map at the t-th time step According to the characteristic sequence A circulant matrix M is constructed to build the mutual relationship between the characteristics at different time, and the expression of the circulant matrix M is: ； Wherein, the similarity measure of the circulant matrix M is calculated by cosine similarity, , respectively represent the local feature maps of the i, j-th time step. An attention weight matrix is constructed through the cyclic matrix to quantify the feature distribution between each time point, and the expression of the attention weight matrix is: ； wherein, represents a circulant matrix between the current time and the i-th time, represents an attention weight between the current time and the i-th time; performing weighted sum on the feature sequence according to the attention weight matrix performing weighted sum on the feature sequence according to the attention weight matrix ； wherein, denotes the time-domain feature at time j, denotes the attention weight matrix at time i, j, denotes the time length of the feature sequence, denotes the cross-time domain feature representation at time i after enhancement; The attention weights are sorted according to the attention weight matrix, and the time points with higher weights and the corresponding cross-time domain feature representations are screened out to extract highly correlated cross-time domain key information.

Images