SAR (Synthetic Aperture Radar) video stream enhancement method and device, equipment and medium

Abstract

The invention provides an SAR (Synthetic Aperture Radar) video stream enhancement method, device and equipment and a medium, and the method comprises the steps: obtaining a continuous three-frame SAR image sequence, extracting multi-scale spatial-temporal features through a feature extraction network, adaptively fusing clear region information of each frame through an attention mechanism to suppress blurring, calculating an inter-frame motion vector through an optical stream estimation module, and obtaining a multi-scale SAR image sequence. Accurate alignment and compensation of adjacent frames are achieved, and finally a high-quality clear image is reconstructed through an image fusion network. According to the method, the problem of SAR image blurring caused by attitude disturbance and trajectory offset of motion platforms such as an unmanned aerial vehicle is effectively solved, the image detail reduction capability and the visual quality are remarkably improved, and the method can be widely applied to the fields with high image definition requirements such as low-altitude remote sensing, disaster monitoring and automatic driving.

Description

Technical Field

[0001] This invention relates to the field of video enhancement technology, and in particular to a SAR video stream enhancement method, apparatus, device, and medium. Background Technology

[0002] With the rapid development of the low-altitude economy and UAV technology, the application of Synthetic Aperture Radar (SAR) on UAV platforms has gradually become widespread, becoming a core technology for critical tasks such as high-resolution imaging, terrain mapping, and emergency monitoring. However, due to factors such as attitude instability, limited wind resistance, and trajectory deviation during UAV flight, severe motion blur often occurs in SAR video streams. This blur not only weakens image details but also interferes with subsequent image interpretation and target recognition, reducing the availability and robustness of the entire system.

[0003] Theoretically, a drone should fly at a constant speed along a straight path. However, in practice, due to wind, the drone may experience shaking during a certain section of flight, deviating from the intended route and forming an actual flight path that deviates from the intended straight path. This deviation causes the target's echo phase trajectory to no longer match the preset model, leading to the failure of the matched filter's phase compensation and resulting in azimuth blurring. Simultaneously, since the drone is not flying at a strictly constant speed, the Doppler frequency of the echo no longer exhibits a linear change; the Doppler center frequency shifts, and its trajectory may even become curved or asymmetrical, ultimately causing the target point in the image to appear elongated or broken. The combined effect of these factors results in significant motion blur in the entire SAR image.

[0004] During SAR imaging, the system needs to resample and correct echo signals acquired at different times according to the expected temporal distribution (e.g., range cell migration correction (RCMC)) to ensure that signals at each moment can be accurately superimposed in the image. However, due to the platform's non-uniform flight speed, the sampling interval on the time axis is no longer uniform, and the Doppler history of the same target at different time points is not the same. This causes the compression window to malfunction, resulting in phase errors, which in turn cause the focus point to shift, ultimately leading to a blurred target position and edges in the image. Platform jitter also prevents effective phase correction, reduces echo signal coherence, and causes coherence integration to fail. In the middle, if the error Δφ i The signal is very large, and the echoes will cancel each other out, causing the image energy to disperse and resulting in blurring.

[0005] Traditional video deblurring methods mainly include blind deblurring algorithms based on traditional image processing and convolutional neural network methods based on deep learning (such as DeblurGAN, EDVR, etc.). These methods usually only utilize single-frame information or employ simple inter-frame fusion mechanisms, making it difficult to fully model the complex effects of motion trajectories on image blurring. Especially when dealing with dynamic blur or nonlinear jitter, the recovery effect is relatively limited. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a SAR video stream enhancement method, apparatus, device and medium, which aims to solve the coupling problem of motion blur and structural distortion in dynamic scenes.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a SAR video stream enhancement method, comprising the following steps:

[0008] Obtain a continuous SAR image sequence containing the current blurred frame, the previous frame, and the next frame;

[0009] The continuous SAR image sequence is input into a feature extraction network to extract multi-scale spatial features of each frame of the continuous SAR image sequence, resulting in multiple sets of temporal feature groups.

[0010] The multiple sets of temporal feature groups are input into the attention fusion module. A spatial attention weight map is generated based on the local sharpness of each frame in the continuous SAR image sequence. The input features are weighted position by position to enhance the features of sharp regions and suppress the features of blurry regions. Multiple sets of fused attention features are output.

[0011] The multiple sets of fused attention features are input into the image reconstruction network for upsampling and feature fusion processing to reconstruct the preliminary deblurred image of the current frame.

[0012] Based on the current blurred frame, the previous frame, and the next frame, the backward optical flow between the current frame and the previous frame, and the forward optical flow between the current frame and the next frame are estimated respectively.

[0013] Based on the forward optical flow and the backward optical flow, the previous frame and the next frame are subjected to reverse sampling transformation respectively to obtain a forward-aligned image and a backward-aligned image that are aligned with the current frame's viewpoint.

[0014] The preliminary deblurred image, the forward-aligned image, and the backward-aligned image are concatenated by channels and input into an image fusion network for feature fusion and residual reconstruction, outputting the final deblurred image.

[0015] Furthermore, the extraction of multi-scale spatial features from each frame of the continuous SAR image sequence specifically includes:

[0016] An encoder structure is used to perform multiple downsampling processes on each frame of the image to extract spatial features at different resolution levels.

[0017] Features of three frames in the continuous SAR image sequence at the same resolution level are stacked to form a feature group containing temporal information.

[0018] Furthermore, the step of generating a spatial attention weight map based on the local sharpness of each frame in the continuous SAR image sequence specifically includes:

[0019] The input features are subjected to average pooling and max pooling in the channel dimension to obtain two single-channel spatial feature maps.

[0020] Two single-channel spatial feature maps are stitched together along the channel dimension and then input into a convolutional layer to extract the fused spatial features;

[0021] The fused spatial features are normalized using the Sigmoid function to obtain the spatial attention weight map α_spatial;

[0022] The spatial attention weight map α_spatial is multiplied element-wise with the original features in spatial location to highlight the feature contribution of clear regions.

[0023] Furthermore, the reconstructing of the preliminary deblurred image of the current frame specifically includes:

[0024] A decoder structure is used to perform multiple upsampling processes on the fused attention features;

[0025] Skip connections are used to fuse features of the same scale in the encoder with features in the decoder to restore image details and spatial structure.

[0026] Furthermore, the estimation of the backward optical flow between the current frame and the previous frame, and the forward optical flow between the current frame and the next frame, specifically includes:

[0027] An optical flow estimation network is used to receive the current frame and the previous frame, and the current frame and the next frame as inputs, respectively.

[0028] The output represents the optical flow field of pixel displacement, which is used to characterize motion information between adjacent frames.

[0029] Furthermore, the step of performing reverse sampling transformation on the previous frame and the next frame respectively specifically includes:

[0030] Pixel-level resampling is performed on the previous or next frame image based on the displacement vector of each pixel in the optical flow field.

[0031] Transform the image of the previous or next frame to the coordinate system of the current frame to achieve inter-frame alignment.

[0032] Furthermore, the image fusion network performs feature fusion and residual reconstruction, specifically including:

[0033] Feature extraction and fusion of the stitched multi-channel image are performed using a multi-layer convolutional neural network.

[0034] A residual learning mechanism is used to reconstruct image details and output high-quality, clear images.

[0035] The present invention also provides a SAR video stream enhancement device, comprising:

[0036] The image acquisition module is used to acquire a continuous SAR image sequence containing the current blurred frame, the previous frame, and the next frame.

[0037] The feature extraction module is used to input the continuous SAR image sequence into the feature extraction network, extract the multi-scale spatial features of each frame of the continuous SAR image sequence, and obtain multiple sets of temporal feature groups;

[0038] The attention fusion module is used to input the multiple sets of temporal feature groups into the attention fusion module, generate a spatial attention weight map based on the local sharpness of each frame image in the continuous SAR image sequence, weight the input features position by position to enhance the features of sharp regions and suppress the features of blurry regions, and output multiple sets of fused attention features.

[0039] The image reconstruction module is used to input the multiple sets of fused attention features into the image reconstruction network, perform upsampling and feature fusion processing, and reconstruct the preliminary deblurred image of the current frame.

[0040] The optical flow estimation module is used to estimate the backward optical flow between the current frame and the previous frame, and the forward optical flow between the current frame and the next frame, respectively, based on the current blurred frame, the previous frame, and the next frame.

[0041] The image alignment module is used to perform reverse sampling transformation on the previous frame and the next frame respectively according to the forward optical flow and the backward optical flow to obtain a forward-aligned image and a backward-aligned image that are aligned with the viewpoint of the current frame.

[0042] The image fusion module is used to perform channel stitching on the preliminary deblurred image, the forward aligned image, and the backward aligned image, and input them into the image fusion network for feature fusion and residual reconstruction, and output the final deblurred image.

[0043] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the SAR video stream enhancement method as described above.

[0044] The present invention also provides a storage medium storing a computer program that, when executed by a processor, can implement the SAR video stream enhancement method described above.

[0045] The beneficial effects of this invention are as follows: Multi-scale spatial coding is constructed through temporal feature extraction; a weighted attention mechanism is used to dynamically fuse clear region features from consecutive frames, effectively suppressing phase mismatch and Doppler distortion caused by UAV attitude disturbances and trajectory offsets; optical flow-guided bidirectional motion compensation can accurately correct inter-frame pixel misalignment, overcoming nonlinear motion blur that is difficult to handle with traditional methods; finally, a multi-source feature fusion mechanism eliminates dynamic afterimages while preserving structural integrity, significantly improving image edge sharpness and texture realism. This method breaks through the coupling problem of motion blur and structural distortion in dynamic scenes, greatly enhancing the imaging robustness and usability of SAR video streams in complex environments. Attached Figure Description

[0046] The specific structure of the present invention will now be described in detail with reference to the accompanying drawings.

[0047] Figure 1 This is a flowchart of the SAR video stream enhancement method according to an embodiment of the present invention;

[0048] Figure 2 This is a network structure diagram of an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the frame processor module structure according to an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the weighted attention mechanism module structure according to an embodiment of the present invention;

[0051] Figure 5 This is a comparison chart of experimental results from embodiments of the present invention;

[0052] Figure 6 This is a block diagram of a SAR video stream enhancement device according to an embodiment of the present invention;

[0053] Figure 7 This is a schematic block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0056] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0057] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0058] like Figure 1 As shown, an embodiment of the present invention is: a SAR video stream enhancement method, comprising the following steps:

[0059] S1. Obtain a continuous SAR image sequence containing the current blurred frame, the previous frame, and the next frame.

[0060] S2. Input the continuous SAR image sequence into the feature extraction network to extract the multi-scale spatial features of each frame of the continuous SAR image sequence, and obtain multiple sets of temporal feature groups;

[0061] In step S2, extracting the multi-scale spatial features of each frame in the continuous SAR image sequence specifically includes:

[0062] An encoder structure is used to perform multiple downsampling processes on each frame of the image to extract spatial features at different resolution levels.

[0063] Features of three frames in the continuous SAR image sequence at the same resolution level are stacked to form a feature group containing temporal information.

[0064] In this embodiment, as Figure 2The diagram shows the network structure used in the SAR video stream enhancement method. The input is a stack of three consecutive blurred SAR images, which are fed into an encoder designed based on the U-Net structure. The encoder consists of four downsampling modules and a bottleneck layer, which can extract multi-scale spatial features of the image. For each frame, the encoder extracts five sets of features at different resolutions (denoted as feat1 to bottleneck), and stacks the corresponding features from the three frames to construct a temporal feature set.

[0065] The encoder uses LeakyReLU (0.2) activation function, which allows small gradients to pass even when the input is negative, thus reducing gradient vanishing and preserving more feature information in deep networks. Dropout2d is added after two convolutions within each convolutional block to randomly mask some spatial locations of feature channels. This significantly increases regularization and enhances the model's generalization ability, especially effective when data is limited and overfitting is prevented. Features (feat1, feat2, feat3, feat4) are preserved after pooling; these skip connection features have a lower spatial resolution than traditional U-Net. This results in lower memory and computational cost, making it particularly suitable for high-resolution inputs (e.g., 512×256). ConvTranspose2d is used for upsampling, which both amplifies the resolution and simultaneously learns the upsampled convolution weights, resulting in smoother reconstruction that better meets task requirements.

[0066] The encoder and decoder in this scheme share the same _conv_block design: each block has two 3×3 convolutions + GroupNorm + LeakyReLU + Dropout2d, which ensures that the normalization, activation and regularization methods of features at different levels are consistent.

[0067] S3. Input the multiple sets of temporal feature groups into the attention fusion module. Based on the local sharpness of each frame image in the continuous SAR image sequence, dynamically calculate and allocate channel weights to enhance the features of sharp regions and suppress the features of blurry regions, and output multiple sets of fused attention features.

[0068] S3. Input the multiple sets of temporal feature groups into the attention fusion module, generate a spatial attention weight map based on the local sharpness of each frame image in the continuous SAR image sequence, weight the input features position by position to enhance the features of clear areas and suppress the features of blurry areas, and output multiple sets of fused attention features.

[0069] In step S3, a spatial attention weight map is generated based on the local sharpness of each frame in a continuous SAR image sequence. Specifically, as follows... Figure 3As shown, the FrameProcessor module of this embodiment processes the input multiple sets of temporal features. First, global average pooling and global max pooling are performed along the channel dimension to obtain two single-channel spatial feature maps, FavgFavg and FmaxFmax. Then, the two feature maps are concatenated along the channel dimension to form a two-channel spatial feature map FconcatFconcat, which is then input into a convolutional layer with a kernel size of 7×7 for feature fusion to obtain a fused spatial saliency distribution map. Finally, the sigmoid function is used for normalization to generate a spatial attention weight map αspatial with values ​​ranging from 0 to 10 to 1.

[0070] like Figure 4 As shown, the weighted attention mechanism module of this embodiment of the invention utilizes the aforementioned αspatial to multiply the original feature map FF element-wise in the spatial dimension (⊙), thereby enhancing the response intensity of sharp regions and suppressing interference from blurry regions at the feature level, achieving feature enhancement for regions with high local sharpness. This step, through the synergistic effect of the frame processor module and the weighted attention mechanism module, effectively improves the discriminative power and detail preservation capability of continuous SAR image sequence features, providing high-quality feature support for subsequent multi-frame fusion and target recognition.

[0071] In this embodiment, at each scale, the fused features of the three frames are fed into an independent attention fusion module for the above processing. The attention fusion module adaptively generates a spatial attention weight map based on local sharpness, automatically enhancing the response of sharp regions and suppressing interference from blurry regions. The output consists of five sets of fused features, denoted as attention feature 1 to the attention bottleneck layer. These features retain the sharpest pixel information from the three frames, effectively reducing the cumulative damage caused by blur during feature propagation and providing high-quality feature support for subsequent image reconstruction. S4. The multiple sets of fused attention features are input into the image reconstruction network for upsampling and feature fusion processing to reconstruct the preliminary deblurred image of the current frame.

[0072] In step S4, the reconstructing of the preliminary deblurred image of the current frame specifically includes:

[0073] A decoder structure is used to perform multiple upsampling processes on the fused attention features;

[0074] Skip connections are used to fuse features of the same scale in the encoder with features in the decoder to restore image details and spatial structure.

[0075] In this embodiment, the aforementioned fused features are input into the decoder for image reconstruction. This decoder is structurally symmetrical to the encoder, containing multiple upsampling layers and skip connections, enabling it to progressively restore the spatial resolution of the image while preserving original structural details. In this scheme, the bottleneck layer and feature 4 (which have the same resolution) are directly concatenated along the channel dimension in the first decoding layer, followed by convolutional fusion without upsampling. This approach allows the two deepest features to be fully fused at low resolution first, which helps reduce computational load and mix large-scale semantic information.

[0076] The final output is a preliminary deblurred image, denoted as Raw Deblur Image. This image has removed most of the static blur, but may still have residual motion errors.

[0077] S5. Based on the current blurred frame, the previous frame, and the next frame, estimate the backward optical flow between the current frame and the previous frame, and the forward optical flow between the current frame and the next frame, respectively.

[0078] In step S5, estimating the backward optical flow between the current frame and the previous frame, and the forward optical flow between the current frame and the next frame, specifically includes:

[0079] An optical flow estimation network is used to receive the current frame and the previous frame, and the current frame and the next frame as inputs, respectively.

[0080] The output represents the optical flow field of pixel displacement, which is used to characterize motion information between adjacent frames.

[0081] In this embodiment, to further improve inter-frame consistency and motion compensation, the system incorporates an optical flow pyramid module. This module takes the current frame and the previous / next frames as input to estimate the forward and backward optical flows, respectively.

[0082] S6. Based on the forward optical flow and the backward optical flow, perform reverse sampling transformation on the previous frame and the next frame respectively to obtain a forward-aligned image and a backward-aligned image that are aligned with the current frame's viewpoint.

[0083] In step S6, the inverse sampling transformation of the previous frame and the next frame specifically includes:

[0084] Pixel-level resampling is performed on the previous or next frame image based on the displacement vector of each pixel in the optical flow field.

[0085] Transform the image of the previous or next frame to the coordinate system of the current frame to achieve inter-frame alignment.

[0086] In this embodiment, by performing a backsampling operation, the images of the previous / next frame are transformed to the perspective of the current frame to obtain aligned forward deblurred and backward deblurred images.

[0087] S7. The preliminary deblurred image, the forward-aligned image, and the backward-aligned image are concatenated by channel stitching and input into an image fusion network for feature fusion and residual reconstruction, and the final deblurred image is output.

[0088] In step S7, the image fusion network performs feature fusion and residual reconstruction, specifically including:

[0089] Feature extraction and fusion of the stitched multi-channel image are performed using a multi-layer convolutional neural network.

[0090] A residual learning mechanism is used to reconstruct image details and output high-quality, clear images.

[0091] In this embodiment, three images—the initial deblurred image, the aligned forward image, and the backward image—are stitched together along the channel dimension and fed into the fusion network. The fusion network consists of multiple layers of residual convolutional blocks, which can effectively integrate information redundancy in multi-source images and suppress error propagation, ultimately outputting a high-quality deblurred image with significantly improved clarity, exhibiting good visual consistency and structural restoration capabilities.

[0092] In one specific embodiment, the SAR video stream enhancement method further includes a training and optimization step:

[0093] A weighted multi-task loss function is constructed to jointly optimize structural similarity index, pixel accuracy error, and inter-frame alignment consistency.

[0094] The ability to restore image texture details is improved by using adversarial generation mechanisms.

[0095] In this embodiment, to achieve optimal network training results, a weighted multi-task loss function is designed, comprehensively considering image structure restoration, pixel accuracy, and inter-frame alignment consistency. This loss function consists of three parts:

[0096] Structural Similarity Loss (SSIM): This measure is used to assess the structural similarity between the deblurred image and the target image. The SSIM value ranges from -1 to 1, with values ​​closer to 1 indicating better structure preservation. This loss considers image brightness, contrast, and structural distribution, and is an important indicator for evaluating image perceptual quality. The formula is as follows:

[0097]

[0098] Where, μ x and μ y These are the means of the X and Y axes of the image, respectively. and It is their variance, σ xy These are their covariances, and C1 and C2 are constants used to avoid zero in the denominator. The SSIM value ranges from [-1, 1], and a larger value indicates that the images are more structurally similar.

[0099] Mean Squared Error Loss (MSE): Reflects pixel-level reconstruction accuracy, i.e., the average squared error between the output image and the target image at each pixel. A smaller MSE value indicates higher image reconstruction accuracy. The formula is:

[0100]

[0101] Where, x i and y i Let x and y represent the i-th pixels of image x and the target image y, respectively, where N is the total number of pixels in the image. The smaller the MSE value, the smaller the difference between the reconstructed image and the target image at the pixel level.

[0102] Optical flow consistency loss: Ensures that the forward and backward optical flow estimates are consistent with the actual optical flow, improving inter-frame alignment accuracy. Specifically, the optical flow loss measures whether the image changes obtained through optical flow estimation are consistent, as shown in the following formula:

[0103]

[0104] Where I(x) represents the image x at position x i The pixel value at F(x) i ) represents the optical flow field at position x i The displacement. This loss improves the accuracy of inter-frame alignment by calculating the consistency of forward and backward optical flow, and the L1 norm represents the absolute error between pixels.

[0105] The final training loss function is a weighted sum of the three losses mentioned above, as shown in the following formula:

[0106] Loss=α·SSIM(x,y)+β·MSE(x,y)+γ·FlowLoss(x,y,F)

[0107] Here, α, β, and γ are weighting coefficients used to control the impact of different loss terms on the total loss. Through this weighted multi-task loss function, the network can achieve a good balance between image structure consistency, pixel reconstruction accuracy, and inter-frame alignment consistency.

[0108] In summary, this invention, through an innovative combination of multi-scale feature extraction, multi-frame attention fusion mechanism, optical flow-guided correction, and residual fusion reconstruction techniques, successfully solves the problem of difficult-to-recover blurred SAR images in dynamic scenes. It is particularly suitable for SAR imaging tasks on low-altitude unmanned aerial vehicle platforms, enhancing the system's anti-disturbance capability while ensuring high-resolution image quality, providing a solid foundation for subsequent applications such as target recognition and image understanding.

[0109] As shown in Table 1, in order to verify the overall effect of the network, a U-shaped network and a generative adversarial network were used to evaluate the overall experimental results and to verify whether the network designed in this invention improves the enhancement and recovery of SAR video streams compared with other networks.

[0110] Table 1: Comparison of Experimental Results

[0111]

[0112] In this experiment, to address the phase misalignment and image blurring issues caused by non-uniform platform motion in SAR imaging, the image restoration performance of three types of models was designed and evaluated: the basic U-Net, the U-Net combined with Generative Adversarial Network (U-Net+GAN), and the novel network structure proposed in this invention. The restoration performance of the three methods was compared and analyzed using L1 loss, MSE, structural similarity index (SSIM), and peak signal-to-noise ratio (PSNR).

[0113] First, compared with the traditional U-Net, the network of this invention exhibits significant advantages in all evaluation metrics. Regarding pixel error, L1 and MSE are reduced from 0.01995 and 0.0007641 to 0.01449 and 0.0004459, respectively, representing reductions of 27.4% and 41.6%, indicating that the method of this invention can more accurately restore the original image. In terms of structure preservation, SSIM increases from 0.8726 to 0.9152, an increase of approximately 4.9 percentage points; simultaneously, PSNR increases from 32.13 dB to 34.83 dB, an increase of over 2.7 dB. These results demonstrate that the network of this invention outperforms traditional methods in suppressing nonlinear phase errors, maintaining image structure consistency, and restoring details.

[0114] Furthermore, while introducing adversarial generative mechanisms into U-Net improved some metrics, the overall performance still fell short of the network of this invention. Specifically, U-Net+GAN improved PSNR to 32.53dB (0.4dB higher than U-Net) and SSIM to 0.8790, but the improvements in L1 and MSE were limited, at 0.01966 and 0.0007531 respectively, still higher than the results of the network of this invention. This indicates that although adversarial training can bring some structural and texture enhancement to the base network, it is insufficient to compensate for the lack of pixel reconstruction accuracy, and the network still suffers from instability during the optimization process.

[0115] In comparison, the network of this invention, without relying on additional adversarial mechanisms, has comprehensively outperformed U-Net and U-Net+GAN in four dimensions: L1, MSE, SSIM, and PSNR, fully demonstrating the effectiveness of its structural improvements in addressing motion blur in SAR imaging. Its advantages are not only reflected in the improvement of pixel-level accuracy, but also in the coordinated unity of overall structural preservation and detail enhancement.

[0116] In summary, the network presented in this invention demonstrates a comprehensive and stable performance improvement compared to traditional U-Net and its adversarial enhancement versions, representing an effective structural improvement direction for solving the non-uniform motion blur problem in SAR. Future development could further incorporate perceptual loss or multi-scale discriminators to enhance both subjective image quality and error metrics.

[0117] like Figure 5 As shown, the image comparison demonstrates the effect of the model before and after deblurring. Looking at the horizontal axis, the three images on the left are the three blurred input frames, showing a significant blurring effect and severe loss of image details. The fourth image is the output image after deblurring using the method of this invention. It is clear that the details and structure in the image have been effectively restored, and the blurred areas have been significantly repaired.

[0118] The last image is the target image. Compared to the output image, the deblurred result is highly similar to the target image. The vertical axis represents different batches. This verification output demonstrates the efficiency of the proposed method in restoring details and achieving accurate alignment. Through experiments, the deblurring effect of this scheme exhibits excellent performance in multiple scenarios, especially under complex backgrounds and low-quality input, where image structure and details are effectively preserved.

[0119] The technical solution of this invention can be widely applied in the following fields:

[0120] 1. Unmanned Aerial Vehicle (UAV) Remote Sensing and SAR Imaging System: Improve the image clarity of small SAR devices under flight jitter conditions.

[0121] With the development of the low-altitude economy, unmanned aerial vehicles (UAVs) have become an important platform for remote sensing missions. Synthetic aperture radar (SAR) technology, due to its all-weather and all-time imaging capabilities, has been widely deployed in fixed-wing and multi-rotor UAVs. However, due to the poor stability, wind disturbance, and flight control errors of small UAV platforms, the acquired SAR images often exhibit significant motion blur, severely affecting the resolution of surface textures and the accuracy of target recognition.

[0122] This invention utilizes temporal redundancy and motion information between multiple image frames, selectively preserving sharp regions through an attention mechanism, and compensating for pixel misalignment caused by trajectory offset through an optical flow alignment mechanism, significantly improving image sharpness and structural integrity. Therefore, this method can be used as an image preprocessing module in UAV SAR systems, suitable for tasks such as disaster monitoring, terrain modeling, resource surveys, and battlefield reconnaissance, enhancing the practical value of SAR images and the overall system performance.

[0123] 2. Fusion of vehicle radar and vision system: Improve the clarity and stability of fused image pairs in autonomous driving systems.

[0124] In autonomous driving systems, image sensors work in conjunction with sensors such as millimeter-wave radar and lidar to perceive the environment. Among them, cameras are responsible for providing rich texture and semantic information, but in complex scenarios such as high-speed vehicle travel, road vibration, and changes in lighting, images are prone to dynamic blurring, which affects subsequent target detection, path planning, and decision-making.

[0125] The video deblurring method of this invention can be deployed on an in-vehicle edge computing platform to perform real-time deblurring of consecutive image frames, enhancing image clarity and edge information quality. Simultaneously, this method can utilize adjacent frames for inter-frame structural compensation and sharp region extraction, maintaining stable recovery capabilities even under drastic changes in image frame content, thereby improving the robustness of the perception system and the safety of the autonomous driving system.

[0126] 3. Satellite image processing and orbit synthesis system: Improves image availability in situations of imaging instability or trajectory drift.

[0127] High-resolution Earth observation satellites often face problems such as attitude adjustment lag, orbital disturbances, and non-ideal motion of the imaging platform during the imaging process, which lead to phenomena such as displacement blurring and stripe distortion in the images. This is especially serious in pushbroom imaging and multi-track synthesis scenarios, affecting the accuracy of surface target recognition, change detection, and map matching.

[0128] The method proposed in this invention can be embedded into the satellite ground image processing chain to accurately correct image blur caused by trajectory drift, camera vibration, etc. Through temporal modeling and motion alignment compensation between multiple frames, it restores surface texture details and enhances the geometric accuracy and radiometric consistency of satellite images. This technology is particularly suitable for high-frequency imaging satellite constellation systems, improving the accuracy and quality of multi-source image stitching, consistency analysis, and long-term temporal monitoring.

[0129] like Figure 6 As shown, this embodiment of the invention also provides a SAR video stream enhancement device, comprising:

[0130] Image acquisition module 10 is used to acquire a continuous SAR image sequence containing the current blurred frame, the previous frame and the next frame;

[0131] Feature extraction module 20 is used to input the continuous SAR image sequence into the feature extraction network, extract multi-scale spatial features of each frame of the continuous SAR image sequence, and obtain multiple sets of temporal feature groups;

[0132] Attention fusion module 30 is used to input the multiple sets of temporal feature groups into the attention fusion module, generate a spatial attention weight map based on the local sharpness of each frame image in the continuous SAR image sequence, weight the input features position by position to enhance the features of sharp regions and suppress the features of blurry regions, and output multiple sets of fused attention features.

[0133] The image reconstruction module 40 is used to input the multiple sets of fused attention features into the image reconstruction network, perform upsampling and feature fusion processing, and reconstruct the preliminary deblurred image of the current frame.

[0134] The optical flow estimation module 50 is used to estimate the backward optical flow between the current frame and the previous frame, and the forward optical flow between the current frame and the next frame, respectively, based on the current blurred frame, the previous frame, and the next frame.

[0135] The image alignment module 60 is used to perform reverse sampling transformation on the previous frame and the next frame respectively according to the forward optical flow and the backward optical flow to obtain a forward-aligned image and a backward-aligned image that are aligned with the viewpoint of the current frame.

[0136] The image fusion module 70 is used to perform channel stitching on the preliminary deblurred image, the forward aligned image, and the backward aligned image, and input them into the image fusion network for feature fusion and residual reconstruction, and output the final deblurred image.

[0137] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned SAR video stream enhancement device and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0138] The aforementioned SAR video stream enhancement device can be implemented as a computer program, which can, for example... Figure 7 It runs on the computer device shown.

[0139] Please see Figure 7 , Figure 7 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0140] See Figure 7 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0141] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a SAR video stream enhancement method.

[0142] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0143] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a SAR video stream enhancement method.

[0144] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0145] The processor 502 is used to run a computer program 5032 stored in a memory to implement the SAR video stream enhancement method described above.

[0146] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0147] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0148] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the SAR video stream enhancement method described above.

[0149] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0150] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0151] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0152] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0153] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0154] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A SAR video stream enhancement method, characterized in that, Includes the following steps: Obtain a continuous SAR image sequence containing the current blurred frame, the previous frame, and the next frame; The continuous SAR image sequence is input into a feature extraction network to extract multi-scale spatial features of each frame of the continuous SAR image sequence, resulting in multiple sets of temporal feature groups. The multiple sets of temporal feature groups are input into the attention fusion module. A spatial attention weight map is generated based on the local sharpness of each frame in the continuous SAR image sequence. The input features are weighted position by position to enhance the features of sharp regions and suppress the features of blurry regions. Multiple sets of fused attention features are output. The multiple sets of fused attention features are input into the image reconstruction network for upsampling and feature fusion processing to reconstruct the preliminary deblurred image of the current frame. Based on the current blurred frame, the previous frame, and the next frame, the backward optical flow between the current frame and the previous frame, and the forward optical flow between the current frame and the next frame are estimated respectively. Based on the forward optical flow and the backward optical flow, the previous frame and the next frame are subjected to reverse sampling transformation respectively to obtain a forward-aligned image and a backward-aligned image that are aligned with the current frame's viewpoint. The preliminary deblurred image, the forward-aligned image, and the backward-aligned image are concatenated by channels and input into an image fusion network for feature fusion and residual reconstruction, outputting the final deblurred image.

2. The SAR video stream enhancement method according to claim 1, characterized in that, The extraction of multi-scale spatial features from each frame of the continuous SAR image sequence specifically includes: An encoder structure is used to perform multiple downsampling processes on each frame of the image to extract spatial features at different resolution levels. Features of three frames in the continuous SAR image sequence at the same resolution level are stacked to form a feature group containing temporal information.

3. The SAR video stream enhancement method according to claim 1 or 2, characterized in that, The generation of a spatial attention weight map based on the local sharpness of each frame in the continuous SAR image sequence specifically includes: The input features are subjected to average pooling and max pooling in the channel dimension to obtain two single-channel spatial feature maps. Two single-channel spatial feature maps are stitched together along the channel dimension and then input into a convolutional layer to extract the fused spatial features; The fused spatial features are normalized using the Sigmoid function to obtain the spatial attention weight map α_spatial; The spatial attention weight map α_spatial is multiplied element-wise with the original features in spatial location to highlight the feature contribution of clear regions.

4. The SAR video stream enhancement method according to claim 1, characterized in that, The process of reconstructing the initial deblurred image of the current frame specifically includes: A decoder structure is used to perform multiple upsampling processes on the fused attention features; Skip connections are used to fuse features of the same scale in the encoder with features in the decoder to restore image details and spatial structure.

5. The SAR video stream enhancement method according to claim 1, characterized in that, The estimation of the backward optical flow between the current frame and the previous frame, and the forward optical flow between the current frame and the next frame, specifically includes: An optical flow estimation network is used to receive the current frame and the previous frame, and the current frame and the next frame as inputs, respectively. The output represents the optical flow field of pixel displacement, which is used to characterize motion information between adjacent frames.

6. The SAR video stream enhancement method according to claim 1 or 5, characterized in that, The step of performing reverse sampling transformation on the previous frame and the next frame respectively specifically includes: Pixel-level resampling is performed on the previous or next frame image based on the displacement vector of each pixel in the optical flow field. Transform the image of the previous or next frame to the coordinate system of the current frame to achieve inter-frame alignment.

7. The SAR video stream enhancement method according to claim 1, characterized in that, The image fusion network performs feature fusion and residual reconstruction, specifically including: Feature extraction and fusion of the stitched multi-channel image are performed using a multi-layer convolutional neural network. A residual learning mechanism is used to reconstruct image details and output high-quality, clear images.

8. A SAR video stream enhancement device, characterized in that, include: The image acquisition module is used to acquire a continuous SAR image sequence containing the current blurred frame, the previous frame, and the next frame. The feature extraction module is used to input the continuous SAR image sequence into the feature extraction network, extract the multi-scale spatial features of each frame of the continuous SAR image sequence, and obtain multiple sets of temporal feature groups; The attention fusion module is used to input the multiple sets of temporal feature groups into the attention fusion module, generate a spatial attention weight map based on the local sharpness of each frame image in the continuous SAR image sequence, weight the input features position by position to enhance the features of sharp regions and suppress the features of blurry regions, and output multiple sets of fused attention features. The image reconstruction module is used to input the multiple sets of fused attention features into the image reconstruction network, perform upsampling and feature fusion processing, and reconstruct the preliminary deblurred image of the current frame. The optical flow estimation module is used to estimate the backward optical flow between the current frame and the previous frame, and the forward optical flow between the current frame and the next frame, respectively, based on the current blurred frame, the previous frame, and the next frame. The image alignment module is used to perform reverse sampling transformation on the previous frame and the next frame respectively according to the forward optical flow and the backward optical flow to obtain a forward-aligned image and a backward-aligned image that are aligned with the viewpoint of the current frame. The image fusion module is used to perform channel stitching on the preliminary deblurred image, the forward aligned image, and the backward aligned image, and input them into the image fusion network for feature fusion and residual reconstruction, and output the final deblurred image.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the SAR video stream enhancement method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the SAR video stream enhancement method as described in any one of claims 1 to 7.

Images