A sequence image fusion method for rotation synthetic aperture computational imaging

Abstract

The application discloses a sequence image fusion method for rotation synthetic aperture computational imaging, and is based on the imaging mechanism of a rotation synthetic aperture optical remote sensing system, and proposes an end-to-end image fusion network based on a visual Transformer. Intra-frame self-attention calculation in a space-time information extraction module can more effectively process information of objects of different scales in a remote sensing image. At the same time, an inter-frame mutual attention is used to replace an explicit alignment module, so that the correlation between pixels at similar positions in different frames can be adaptively captured, and the generation of artifacts can be reduced. A visual sliding window Transformer module is used in a space-time information fusion module of the fusion network, time domain information is fully fused through the strong modeling capability of the Transformer itself, additional information in a low-quality image sequence can be fully utilized, and characteristics prior and data input are provided for actual on-orbit application of the rotation synthetic aperture system.

Description

Technical Field

[0001] This invention belongs to the field of optical image processing and relates to an optical remote sensing image fusion method, specifically a method for multi-rotation phase sequence image fusion for a novel rotating synthetic aperture computational imaging system. Background Technology

[0002] Remote sensing images from platforms such as drones, airships, and satellites offer advantages such as wide-area coverage, high temporal resolution, and flexible mission planning. In today's information age, they are widely used in fields such as weather forecasting, environmental monitoring, agricultural surveys, and urban planning, providing precise data support for various industries and becoming an indispensable tool in many sectors. Their contribution to continuously improving the quality of life for the public and the overall level of scientific and technological services is self-evident, and their significance to my country's national economic development is self-evident.

[0003] Remote sensing platforms primarily rely on large-aperture primary mirrors to acquire high spatial resolution and high imaging quality images. Traditional refractive / reflective optical systems centered around a large-aperture monolithic primary mirror are severely limited by the platform's load capacity, as well as their own system complexity, size, weight, and cost, making it difficult to meet the application requirements of high reliability, lightweight design, and low cost. Rotational synthetic aperture computing (RSC) imaging technology, employing a rectangular primary mirror with a large aspect ratio, rotates the primary mirror around its center during imaging. This allows the system to achieve rectangular equivalent aperture imaging in multiple directions in a time-division manner. It offers advantages such as lightweight primary mirrors, no need for primary mirror stitching, and no need for real-time on-orbit surface shape maintenance, significantly reducing the overall design, manufacturing, and testing complexity of the system. It is currently one of the most promising technological approaches for realizing large-aperture, high-resolution cameras and has become an important development direction for future multi-platform high-resolution optical remote sensing cameras.

[0004] In the temporal imaging process of a rotating synthetic aperture imaging system, the rotation of the primary mirror causes periodic changes in image quality, with higher image quality along the longer side than along the shorter side. Due to variations in the primary mirror's point spread function with the spatial distribution of the field of view, as well as factors such as satellite platform vibration, primary mirror rotation error, detector sampling, and noise during imaging, even two frames from the same rotational position of the primary mirror will have different image qualities. This complex image quality degradation mechanism cannot be addressed using existing imaging quality recovery methods. It is necessary to address the underlying physical mechanisms of this imaging system and deeply integrate image fusion methods to restore and enhance the system's imaging quality, enabling images to meet interpretation requirements. Summary of the Invention

[0005] The purpose of this invention is to provide a sequential image fusion method for rotational synthetic aperture computational imaging. This method addresses the image interpretation requirements of rotational synthetic aperture imaging systems by treating the image synthesis of such systems as a multi-frame image fusion problem. It proposes an end-to-end image fusion network based on deep learning, which calculates intra-frame self-attention and inter-frame mutual attention to balance the extraction and fusion of spatial and temporal information, thereby improving image quality.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A sequence image fusion method for rotational synthetic aperture computational imaging includes the following steps:

[0008] Step 1: Construct an end-to-end image fusion network based on Visual Transformer

[0009] The end-to-end image fusion network based on the visual Transformer includes a spatiotemporal information extraction module, a spatiotemporal information fusion module, and two convolutional layers, wherein:

[0010] The spatiotemporal information extraction module includes intra-frame self-attention, inter-frame mutual attention, and a two-dimensional convolutional layer. In this module, window-based intra-frame self-attention calculates and extracts spatial dimension information within the frame to capture the self-similarity of the spatial dimension of the remote sensing image. Temporal inter-frame mutual attention replaces the explicit registration and motion compensation process, thereby more accurately utilizing the sub-pixel information between each frame image. The two-dimensional convolutional layer is used for information fusion and adjusts the number of channels in the module output.

[0011] The spatiotemporal information fusion module consists of an even number of video sliding window Transformer modules and a patch extension layer. One of every two video sliding window Transformer modules is used to calculate the multi-head self-attention of the 3D window, and the other is used to calculate the multi-head self-attention of the 3D moving window. The patch extension layer is used to upsample the feature map obtained by the video sliding window Transformer module.

[0012] The two convolutional layers are used to adjust the channel dimensions to generate a single-frame fused image;

[0013] Step 2: Use the window-based intra-frame self-attention calculation in the spatiotemporal information extraction module to extract intra-frame spatial dimension information in order to capture the self-similarity of the spatial dimension of the remote sensing image. The specific steps are as follows:

[0014] Step 21: For an input vector of size T×H×W×C Window self-attention first divides the input feature into M×M non-overlapping local windows, i.e. For each window, the input features are: The input characteristics of all windows are represented as follows: Where: ∈ indicates belonging to Let T, H, W, and C represent the frame sequence length, image width, image height, and number of feature channels, respectively. Let be the number of windows, and i represent the index of the window; then, for each window, a standard multi-head self-attention matrix Q is computed, resulting in the query, key, and value matrix. w K w and V w They are respectively:

[0015]

[0016]

[0017]

[0018] In the formula, h represents the number of heads, and j represents the index of the attention head;

[0019] Step 22: Perform attention calculations within each non-overlapping local window to obtain the following features:

[0020]

[0021] In the formula: Attention(·) represents the calculation of attention, and softmax(·) represents the normalized exponential function.

[0022] Steps two and three: Features To reconstruct the shape of the input features, the following concatenation method is employed: features from different non-overlapping windows are concatenated in the spatial dimension, and features from different heads are concatenated in the channel dimension.

[0023] Step 3: Use temporal mutual attention to extract temporal information instead of explicit registration, and establish connections for misaligned pixels to adaptively extract information from different frames. The specific steps are as follows:

[0024] Step 31: For each window, Let represent the input vector and query matrix at time t in the sequence, respectively, 1 ≤ t ≤ T, where T represents the rotation period of the primary mirror. In the calculation of inter-frame mutual attention in the temporal domain, the input features are obtained through the window at the same position in each frame. The key-value matrix is ​​calculated as follows:

[0025]

[0026]

[0027] In the formula, p represents the index of the frame sequence;

[0028] Step 32: Calculate mutual attention to obtain the output features:

[0029]

[0030] Step 33: Features Concatenate along the channel dimension to generate the feature vector at time t. It reflects the correlation between elements in the reference frame and adjacent frames;

[0031] Steps three and four: Along the time dimension Perform splicing to restore the shape of the input features;

[0032] Step 4: Use the Transformer module of the video sliding window to fuse the features extracted in Step 2 and Step 3, perform implicit motion estimation and distortion of the image at the feature level, and avoid information loss and artifact generation.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] (1) Based on the characteristics of remote sensing images and the physical mechanism of the rotating synthetic aperture system, intra-frame self-attention is used to capture the long-distance dependence of remote sensing images, which is conducive to extracting richer contextual information and realizing the full extraction of intra-frame self-similar information.

[0035] (2) Based on the imaging mechanism of the rotating synthetic aperture optical remote sensing system, the explicit registration operation is abandoned and the inter-frame mutual attention is used directly to extract the information in the time dimension, so as to reduce the possibility of artifact generation and realize the fusion of high confidence rotating synthetic aperture system images.

[0036] (3) Using the spatiotemporal information extracted by intra-frame self-attention and inter-frame mutual attention, the video sliding window Transformer is used for fusion. This not only implicitly establishes connections for misaligned pixels to adaptively save information from different frames, but also to a certain extent is equivalent to implicit motion estimation and distortion of the image at the feature level, which significantly improves the quality of the fused image. Attached Figure Description

[0037] Figure 1 This is a diagram of an end-to-end image fusion network structure based on the Visual Transformer.

[0038] Figure 2 This is a spatiotemporal information extraction module;

[0039] Figure 3 Flowchart for inter-frame mutual attention calculation;

[0040] Figure 4 For video sliding window Transformer module;

[0041] Figure 5 The image shows the degraded image sequence (left) and the fused result (right). Detailed Implementation

[0042] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0043] This invention provides a sequence image fusion method for computational imaging with rotating synthetic aperture (RSA). Based on the imaging mechanism of RSA optical remote sensing systems, the method proposes an end-to-end image fusion network based on a visual Transformer. The intra-frame self-attention computation in the spatiotemporal information extraction module of this network can more effectively process information about objects of different scales in remote sensing images. Simultaneously, using inter-frame mutual attention instead of an explicit alignment module can adaptively capture the correlation between pixels in similar positions across different frames and reduce artifact generation. The spatiotemporal information fusion module of the fusion network uses a visual sliding window Transformer module, leveraging the powerful modeling capabilities of the Transformer to fully fuse temporal information. This allows for the full utilization of additional information in low-quality image sequences, providing prior characteristics and data input for the practical on-orbit application of RSA systems. Specifically, the method includes the following steps:

[0044] Step 1: Use window-based self-attention calculation in the spatiotemporal information extraction module to extract spatial dimension information within the frame in order to capture the self-similarity of the spatial dimension of the remote sensing image.

[0045] Convolutional neural network (CNN)-based methods excel in natural image processing. However, CNNs have limited receptive fields, hindering their ability to capture non-local self-similarity and model long-range spatial dependencies in remote sensing images. In contrast, the attention mechanism in the Visual Transformer can compute correlations between pixels. It not only excels at modeling long-range spatial dependencies within remote sensing image frames but also at capturing self-similarity from adjacent frames. This spatiotemporal information can provide crucial clues for image synthesis using rotational synthetic aperture systems. Therefore, this invention uses only a few two-dimensional convolutional layers (Conv2d) at the beginning and end of the fusion network to change the number of feature channels, and uses methods such as... Figure 2The spatio-temporal information extraction block (SIEB) and the patch merging layer shown extract hierarchical spatio-temporal domain information.

[0046] The following details the window-based self-attention mechanism in the spatiotemporal information extraction module. Specifically, for an input vector of size T×H×W×C... (where ∈ indicates belonging to; Let T represent the vector space; T, H, W, and C represent the frame sequence length, image width, image height, and number of feature channels, respectively. Window self-attention first divides the input features into M×M non-overlapping local windows, i.e. in, Let be the number of windows. For each window, the input features are: Where C is the number of channels. Therefore, the input features of all windows can be represented as: Where i represents the window index. Then, standard multi-head self-attention is computed for each window. Let h represent the number of heads, then the query, key, and value matrix Q is... w K w and V w They are respectively:

[0047]

[0048] In the formula, j represents the index of the attention head.

[0049] Then, attention is calculated in each non-overlapping local window to obtain the following features:

[0050]

[0051] In the formula: Attention(·) represents the calculation of attention, and softmax(·) represents the normalized exponential function.

[0052] Finally, the features The input features are then concatenated to restore their shape. Specifically, features from different non-overlapping windows are concatenated in the spatial dimension, and features from different headers are concatenated in the channel dimension.

[0053] Step 2: Use temporal mutual attention to extract temporal information instead of explicit registration to establish connections for misaligned pixels in order to adaptively extract information from different frames.

[0054] Due to the rotation of the primary mirror and various coupled factors during the imaging process, the information content of different adjacent frames is not equal, making the inter-frame temporal relationship crucial in image fusion. However, because geostationary staring satellites have high platform stability and temporal resolution, the inter-frame displacement is at the sub-pixel level. This makes traditional explicit registration and motion compensation operations unfavorable for extracting temporal information, as inaccurate motion estimation and image warping operations can lead to the accumulation of errors. Specifically, inaccurate motion estimation can be considered as a combination of the actual inter-frame displacement and random error terms, while warping operations based on random errors can damage sub-pixel level information. Furthermore, inaccurate motion compensation operations can affect information extraction in subsequent stages, and this error propagation obviously affects the subsequent information fusion process. Thanks to the self-attention mechanism, the visual Transformer can achieve good results even without alignment. For sub-pixel level inter-frame displacement, since window-based attention does not have a local inductive bias when processing pixels within a local window, it can implicitly establish accurate connections for misaligned pixels. This dynamic aggregation of adjacent frames at the sub-pixel level is a necessary condition for efficient fusion of temporal information.

[0055] Therefore, this invention introduces attention calculation into the temporal dimension within the spatiotemporal information extraction module. Specifically, instead of using an explicit alignment module, this invention fully leverages the powerful modeling capabilities of the visual Transformer itself to capture the self-similarity between pixels in the temporal dimension. Figure 3 As shown, the computation of inter-frame mutual attention consists of a layer normalization (LN), a temporal mutual attention (TMA), and a feed-forward network (FFN). The computation of inter-frame mutual attention is similar to that of window-based intra-frame self-attention. The difference lies in that, in the computation of inter-frame mutual attention, each head is used to focus on the high-resolution information preserved in different directions in images acquired at other time steps.

[0056] Specifically, for each window, Let represent the input vector and query matrix at time t (1≤t≤T) in the sequence, respectively. In the calculation of inter-frame mutual attention in the temporal domain, the input features are obtained through windows at the same position in each frame. The key-value matrix is ​​calculated as follows:

[0057]

[0058] In the formula, p represents the index of the frame sequence.

[0059] Next, mutual attention is calculated to obtain the output features:

[0060]

[0061] Features Concatenate along the channel dimension to generate the feature vector at time t. Because of K w and V w Generated from adjacent frames, so This reflects the correlation between elements in the reference frame and adjacent frames. Specifically, by calculating inter-frame mutual attention, during the information extraction process along the temporal dimension, similar image features can be explored in adjacent frames for subsequent fusion modules. Finally, along the temporal dimension... The input features are then stitched together to restore their shape.

[0062] Step 3: Use the Transformer module (video sliding window) to fuse the features extracted in Step 1 and Step 2. This performs implicit motion estimation and distortion of the image at the feature level, avoiding information loss and artifacts. The degraded image sequence and fusion result are shown below. Figure 5 As shown.

[0063] After extracting information from the spatial and temporal domains, this invention uses a spatiotemporal information fusion module to adaptively fuse the extracted features from each frame. The calculations of temporal mutual attention and spatial self-attention in steps one and two ensure that local self-attention in the spatiotemporal information fusion module can effectively approximate global spatiotemporal self-attention. This spatiotemporal information fusion module consists of… Figure 4The diagram shows an even number of VideoSwinTransformer blocks (VSTBs) and a Patch Expanding Layer. The VideoSwinTransformer block is constructed by replacing the multi-head self-attention module in the standard global Transformer with either 3D Window Based Multi-head Self-attention (3D W-MAS) or 3D Shifted Window Based Multi-head Self-attention (3D SW-MAS). It consists of a layer normalization layer, a 3D multi-head self-attention layer, and a feedforward network. Therefore, in the fusion network of this invention, two VideoSwinTransformer blocks are always used consecutively, one calculating the 3D W-MAS and the other calculating the 3D Shifted Window Based Multi-head Self-attention. Following this, the image fusion network has two convolutional layers to adjust the channel dimensions to generate a single-frame fused image.

Claims

1. A sequential image fusion method for rotational synthetic aperture computational imaging, characterized in that... The method includes the following steps: Step 1: Construct an end-to-end image fusion network based on Visual Transformer The end-to-end image fusion network based on the visual Transformer includes a spatiotemporal information extraction module, a spatiotemporal information fusion module, and two convolutional layers, wherein: The spatiotemporal information extraction module includes intra-frame self-attention, temporal inter-frame mutual attention, and a two-dimensional convolutional layer. In this module, window-based intra-frame self-attention calculates and extracts spatial dimension information within the frame to capture the self-similarity of the spatial dimension of the remote sensing image; temporal inter-frame mutual attention replaces the explicit registration and motion compensation process, thereby more accurately utilizing the sub-pixel information between each frame image; the two-dimensional convolutional layer is used for information fusion and adjusts the number of channels in the module output. The spatiotemporal information fusion module consists of an even number of video sliding window Transformer modules and a patch extension layer. One of every two video sliding window Transformer modules is used to calculate the multi-head self-attention of the 3D window, and the other is used to calculate the multi-head self-attention of the 3D moving window. The patch extension layer is used to upsample the feature map obtained by the video sliding window Transformer module. The two convolutional layers are used to adjust the channel dimensions to generate a single-frame fused image; Step 2: Use the window-based intra-frame self-attention calculation in the spatiotemporal information extraction module to extract the spatial dimension information within the frame in order to capture the self-similarity of the spatial dimension of the remote sensing image; Step 3: Use temporal inter-frame mutual attention calculation to extract temporal dimension information instead of explicit registration operation, establish connections for misaligned pixels, and adaptively extract information from different frames; Step 4: Use the video sliding window Transformer module to fuse the features extracted in Step 2 and Step 3, and perform implicit motion estimation and distortion of the image at the feature level to avoid information loss and artifact generation; the video sliding window Transformer module is constructed by replacing the multi-head self-attention module in the standard global Transformer with a 3D window multi-head self-attention or a 3D moving window multi-head self-attention.

2. The sequential image fusion method for rotational synthetic aperture computational imaging according to claim 1, characterized in that... The specific steps of step two are as follows: Step 21: For an input vector of size T×H×W×C Window self-attention first divides the input vector into M×M non-overlapping local windows, i.e. For each window, the input features are The input characteristics of all windows are represented as ,in: Indicates belonging to, Let T, H, W, and C represent the frame sequence length, image width, image height, and number of feature channels, respectively. Let i be the number of windows and i be the index of the window; then, for each window, a standard multi-head self-attention matrix is ​​computed, resulting in the query, key, and value matrix. , and They are respectively: In the formula, h represents the number of heads, and j represents the index of the attention head; Step 22: Perform attention calculations within each non-overlapping local window to obtain the following features: In the formula: The calculation representing attention, Represents the normalized exponential function, ; Steps two and three: Features The input features are then stitched together to restore their shape.

3. The sequential image fusion method for rotational synthetic aperture computational imaging according to claim 2, characterized in that... In steps two and three, the specific splicing methods are as follows: features from different non-overlapping windows are spliced ​​in the spatial dimension, and features from different headers are spliced ​​in the channel dimension.

4. The sequential image fusion method for rotational synthetic aperture computational imaging according to claim 1, characterized in that... The specific steps of step three are as follows: Step 31: For each window, Let represent the input vector and query matrix at time t in the sequence, respectively. T represents the rotation period of the primary mirror. In the calculation of inter-frame mutual attention in the temporal domain, the input features are obtained through the window at the same position in each frame. The key-value matrix is ​​calculated as follows: In the formula, p represents the index of the frame sequence; Step 32: Calculate mutual attention to obtain the output features: Step 33: Features Concatenate along the channel dimension to generate the feature vector at time t. , It reflects the correlation between elements in the reference frame and adjacent frames; Steps three and four: Along the time dimension The input features are then stitched together to restore their shape.

Images