Low-snr image reconstruction method, device, equipment and storage medium

Abstract

This invention relates to the field of image processing, specifically to a method, apparatus, device, and storage medium for low signal-to-noise ratio (SNR) image reconstruction. It includes: S1. acquiring a low SNR image sequence containing a reference frame and at least one non-reference frame; S2. performing enhanced correlation coefficient registration on the non-reference frames, aligning the non-reference frames to the reference frames based on the optimal geometric transformation matrix obtained through iteration; S3. calculating the correlation and difference between each frame and the reference frame using an attention mechanism, generating dynamic weights, and weighted fusing features; S4. enhancing the weighted fused feature map using a multi-scale hierarchical structure; S5. inputting the enhanced feature map into a generative adversarial network (GAN) for interactive modulation to generate a reconstructed image. This invention, under multi-frame image input, generates high-quality images through motion compensation, feature extraction and fusion, and interactive reconstruction processing based on a GAN, achieving effective recovery of structural and detail information, thereby improving image reconstruction accuracy under low SNR conditions.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method, apparatus, device, and storage medium for low signal-to-noise ratio image reconstruction. Background Technology

[0002] Image reconstruction under low signal-to-noise ratio (Low SNR) conditions has always been a core challenge in low-level vision. Traditional single-frame-based reconstruction methods often struggle to effectively recover structural details in images when faced with severe noise contamination, resulting in over-smoothing or texture loss. Therefore, multi-frame image fusion has gradually become an important direction for improving reconstruction performance. By extracting complementary information from multiple temporal frames, robustness can be effectively improved and the detail recovery problem that single frames struggle to address can be alleviated. However, effective fusion of multi-frame information still faces two challenges: firstly, complex non-rigid motion or parallax variations may exist between frames, making traditional alignment methods based on optical flow or pixel-level alignment prone to error propagation in low-quality images; secondly, most existing fusion strategies focus on reference frame consistency, making it difficult to fully extract unique but structurally significant details from other frames, leading to information redundancy or fusion failure.

[0003] Furthermore, subjective requirements for image quality vary significantly in real-world applications: some tasks prioritize the preservation of structural details, while others prefer a smooth and natural overall appearance. Therefore, the lack of flexible adjustment of reconstruction intensity severely limits the applicability of existing methods in practical scenarios. Among current mainstream methods, training mechanisms that optimize PSNR are prone to "imbalanced learning" when handling different degrees of degradation; GAN-based control methods are prone to misjudging highly degraded images, treating them as "fake" images, leading to vanishing gradients in the generator and unstable training. Summary of the Invention

[0004] This application aims to address the problems of large alignment errors, insufficient feature fusion, and poor controllability of reconstruction results in existing low signal-to-noise ratio (SNR) image reconstruction. It provides a low SNR image reconstruction method, apparatus, device, and storage medium that can generate high-quality images through motion compensation, feature extraction and fusion, and interactive reconstruction processing based on generative adversarial networks, under multi-frame image input. This enables effective recovery of structural and detail information, thereby improving the accuracy and flexibility of image reconstruction under low SNR conditions.

[0005] In a first aspect, embodiments of this application provide a low signal-to-noise ratio image reconstruction method, which may include:

[0006] S1. Obtain a low signal-to-noise ratio image sequence containing a reference frame and at least one non-reference frame;

[0007] S2. Perform enhanced correlation coefficient registration on the non-reference frame, and align the non-reference frame to the reference frame based on the optimal geometric transformation matrix obtained iteratively;

[0008] S3. Calculate the correlation and difference between each frame and the reference frame through an attention mechanism, generate dynamic weights, and fuse features in a weighted manner;

[0009] S4. Enhance the weighted fused feature map using a multi-scale hierarchical structure;

[0010] S5. Input the enhanced feature map into the generative adversarial network for interactive modulation to generate the reconstructed image.

[0011] The low signal-to-noise ratio image reconstruction method according to the embodiments of this application has at least the following beneficial effects:

[0012] The low signal-to-noise ratio image reconstruction apparatus of this application first acquires reference frame and non-reference frame image sequences through an image acquisition module, and aligns a frame sequence using ECC registration technology; then, it uses an attention fusion method to perform weighted fusion on the aligned frame sequence; subsequently, it performs multi-scale feature enhancement through an Hourglass structure (a Unet); finally, it inputs the enhanced feature map into an interactive modulation network (a GAN network) for interactive modulation to generate an image that conforms to user preferences.

[0013] According to some embodiments of this application, S2 includes:

[0014] S21. Perform grayscale conversion and brightness normalization preprocessing on the reference frame and non-reference frame;

[0015] S22. Adaptively select the motion model and initialize the transformation matrix, set the number of iterations and the convergence threshold;

[0016] S23. Use enhanced correlation coefficient optimization to solve for the optimal transformation matrix;

[0017] S24. Perform reverse resampling alignment according to the motion model and output the aligned frame.

[0018] According to some embodiments of this application, the motion model is a translation model, an affine model, or a homography transformation model.

[0019] According to some embodiments of this application, S3 includes:

[0020] S31. After embedding each non-reference frame through a 3x3 convolution, calculate the correlation map with the reference frame pixel by pixel;

[0021] S32. Input the absolute difference between the correlation map of each frame and the reference frame into the Sigmoid function to generate difference-aware attention weights;

[0022] S33. The reference frame features are output directly, and the non-reference frame features are weighted pixel by pixel and then concatenated along the channel dimension. They are then aggregated into fused features through a 1x1 convolution.

[0023] According to some embodiments of this application, S4 includes:

[0024] S41. Enhance the feature map using residual blocks and extract multi-scale context through two-stage downsampling;

[0025] S42. Stack residual blocks at the lowest resolution layer to integrate global information and suppress noise and misalignment of the global information;

[0026] S43. Upsample step by step and connect to the coding layer features in a skip connection. After a 3x3 convolution, add the output to the input feature residual.

[0027] According to some embodiments of this application, the generative adversarial network in S5 includes:

[0028] The generator modulates the network layer weights according to the input control vector to adjust the reconstruction result;

[0029] The discriminator classifies the reconstructed image and adaptively modulates intermediate features based on the degree of degradation.

[0030] Secondly, embodiments of this application provide a low signal-to-noise ratio image reconstruction apparatus, the apparatus comprising:

[0031] An image acquisition module is used to acquire a low signal-to-noise ratio image sequence containing a reference frame and at least one non-reference frame;

[0032] The registration module is used to perform enhanced correlation coefficient registration on the non-reference frame and align the non-reference frame to the reference frame based on the optimal geometric transformation matrix obtained by iteration.

[0033] The attention fusion module is used to calculate the correlation and difference between each frame and the reference frame through the attention mechanism, generate dynamic weights, and weighted fuse features.

[0034] The feature enhancement module is used to enhance the weighted fused feature map using a multi-scale hierarchical structure;

[0035] The modulation module is used to interactively modulate the enhanced feature map into the generative adversarial network to generate a reconstructed image.

[0036] The low signal-to-noise ratio image reconstruction apparatus according to the embodiments of this application has at least the following beneficial effects:

[0037] The low signal-to-noise ratio (SNR) image reconstruction apparatus of this application first acquires a sequence of low SNR images of reference frames and non-reference frames through an image acquisition module. Then, a registration module uses enhanced correlation coefficient registration technology to align the non-reference frames, achieving spatial consistency across multiple frames. Next, an attention fusion module calculates the correlation and differences between each frame and the reference frame, generates dynamic weights, and performs weighted fusion of features from multiple frames. Following this, a feature enhancement module enhances the fused features using a multi-scale hierarchical structure, integrating global and detailed information. Finally, a modulation module inputs the enhanced feature map into a generative adversarial network for interactive modulation, outputting a reconstructed image that meets user preferences. This modular approach to multi-frame alignment, feature fusion, multi-scale enhancement, and interactive modulation enables high-quality, controllable image reconstruction under low SNR conditions.

[0038] Thirdly, embodiments of this application provide a computer device, the computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the methods described above.

[0039] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for performing the low signal-to-noise ratio image reconstruction method as described in the first aspect embodiment. Since the computer-readable storage medium employs all the technical solutions of the low signal-to-noise ratio image reconstruction method of the above embodiments, it possesses at least all the beneficial effects brought about by the technical solutions of the above embodiments.

[0040] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing this application. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the steps of the low signal-to-noise ratio image reconstruction method in Embodiment 1 of this application;

[0042] Figure 2 This is a schematic diagram of the overall architecture of MFIRe for the low signal-to-noise ratio image reconstruction method of Embodiment 2 of this application;

[0043] Figure 3 This is a schematic diagram of the IMGAN structure in Embodiment 2 of this application;

[0044] Figure labels: (a) Schematic diagram of the encoder structure of the generator, (b) Global feature modulation module of the discriminator;

[0045] Figure 4 This is a schematic diagram of the model training process in Embodiment 3 of this application;

[0046] Figure 5 This is a schematic diagram illustrating the effects of different parameter adjustments in Embodiment 3 of this application;

[0047] Figure 6 This is a schematic diagram of the low signal-to-noise ratio image reconstruction device according to Embodiment 4 of this application. Detailed Implementation

[0048] The present application will now be described in further detail with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the subject matter of the present application to the following embodiments. All technologies implemented based on the content of the present application fall within the scope of protection of the present application.

[0049] Unless otherwise specified, the terms "upper," "lower," "left," "right," "center," "inner," "outer," and "side" used in the description of specific embodiments of this application to indicate orientation or positional relationships are based on the orientation or positional relationships shown in the accompanying drawings, or the orientation or positional relationship in which the product / equipment / device is usually placed. These terms are merely for the purpose of facilitating the description of the solution in this application or simplifying the description in specific embodiments, and for enabling those skilled in the art to quickly understand the solution, and do not indicate or imply that a particular device / component / element must have a specific orientation, or be constructed and operated in a specific positional relationship, and therefore should not be construed as limiting this application.

[0050] In the description of the embodiments of this application, technical terms such as "first" and "second" only distinguish one entity or operation from another, and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary or secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0051] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0052] Example 1

[0053] During the research process, the applicant discovered that when using existing single-frame image reconstruction methods to process low signal-to-noise ratio images, recovering structural details and texture information under severe noise interference requires complex preprocessing, manual parameter adjustment, and multiple iterative filtering steps, achieving only limited reconstruction results at specific noise levels. In solving practical engineering problems, to achieve the technical goal of obtaining stable and high-fidelity reconstructed images under different degradation levels, existing techniques struggle to simultaneously balance inter-frame motion compensation accuracy, detail recovery capability, and flexible adjustment of fusion weights. This leads to problems such as over-smoothing, loss of detail, or unstable training in the image reconstruction results.

[0054] Therefore, after in-depth research on this problem, the applicant proposed a low signal-to-noise ratio (SNR) image reconstruction method. This method addresses the specific technical challenges of reconstructing multi-frame images under low SNR conditions. It acquires image sequences of reference and non-reference frames, aligns the non-reference frames using a motion compensation module, extracts global structural features and local detail features separately, and then uses a fusion network to calculate fusion weights to weight and fuse the two types of features, ultimately outputting a reconstructed image. This achieves synergistic enhancement of image structural and detail information, improving the image reconstruction quality under low SNR conditions.

[0055] The step of generating the reconstructed image further includes: reconstructing the fused feature map based on a generative adversarial network, wherein the generative adversarial network includes a generator and a discriminator; the generator modulates the weights of multiple layers in the network according to the input control vector to achieve interactive adjustment of the image reconstruction result; the discriminator discriminates the reconstructed image output by the generator and adaptively modulates the intermediate features according to the degree of image degradation, thereby improving the stability of the training process and enhancing the image reconstruction performance.

[0056] To address the aforementioned issues, this application proposes a low signal-to-noise ratio (SNR) image reconstruction method (MFIRe), which enables high-precision, interactive reconstruction of low SNR images through multi-frame feature extraction, weighted fusion, and adversarial generation without complex manual operations.

[0057] Please refer to Figure 1 , Figure 1 This diagram illustrates the steps of a low signal-to-noise ratio (SNR) image reconstruction method provided in an embodiment of this application. The low SNR image reconstruction method may include:

[0058] S1. Obtain a low signal-to-noise ratio image sequence containing a reference frame and at least one non-reference frame;

[0059] S2. Perform enhanced correlation coefficient registration on the non-reference frame, and align the non-reference frame to the reference frame based on the optimal geometric transformation matrix obtained iteratively;

[0060] S3. Calculate the correlation and difference between each frame and the reference frame through an attention mechanism, generate dynamic weights, and fuse features in a weighted manner;

[0061] S4. Enhance the weighted fused feature map using a multi-scale hierarchical structure;

[0062] S5. Input the enhanced feature map into the generative adversarial network for interactive modulation to generate the reconstructed image.

[0063] The low signal-to-noise ratio image reconstruction method according to the embodiments of this application has at least the following beneficial effects:

[0064] The low signal-to-noise ratio image reconstruction apparatus of this application first acquires reference frame and non-reference frame image sequences through an image acquisition module, and aligns a frame sequence using ECC registration technology; then, it uses an attention fusion method to perform weighted fusion on the aligned frame sequence; subsequently, it performs multi-scale feature enhancement through an Hourglass structure (a Unet); finally, it inputs the enhanced feature map into an interactive modulation network (a GAN network) for interactive modulation to generate an image that conforms to user preferences.

[0065] According to some embodiments of this application, S2 includes:

[0066] S21. Perform grayscale conversion and brightness normalization preprocessing on the reference frame and non-reference frame;

[0067] S22. Adaptively select the motion model and initialize the transformation matrix, set the number of iterations and the convergence threshold;

[0068] S23. Use enhanced correlation coefficient optimization to solve for the optimal transformation matrix;

[0069] S24. Perform reverse resampling alignment according to the motion model and output the aligned frame.

[0070] According to some embodiments of this application, the motion model is a translation model, an affine model, or a homography transformation model.

[0071] According to some embodiments of this application, S3 includes:

[0072] S31. After embedding each non-reference frame through a 3x3 convolution, calculate the correlation map with the reference frame pixel by pixel;

[0073] S32. Input the absolute difference between the correlation map of each frame and the reference frame into the Sigmoid function to generate difference-aware attention weights;

[0074] S33. The reference frame features are output directly, and the non-reference frame features are weighted pixel by pixel and then concatenated along the channel dimension. They are then aggregated into fused features through a 1x1 convolution.

[0075] According to some embodiments of this application, S4 includes:

[0076] S41. Enhance the feature map using residual blocks and extract multi-scale context through two-stage downsampling;

[0077] S42. Stack residual blocks at the lowest resolution layer to integrate global information and suppress noise and misalignment of the global information;

[0078] S43. Upsample step by step and connect to the coding layer features in a skip connection. After a 3x3 convolution, add the output to the input feature residual.

[0079] According to some embodiments of this application, the generative adversarial network in S5 includes:

[0080] The generator modulates the network layer weights according to the input control vector to adjust the reconstruction result;

[0081] The discriminator classifies the reconstructed image and adaptively modulates intermediate features based on the degree of degradation.

[0082] The low signal-to-noise ratio (SNR) image reconstruction method provided in this application can be applied to many technical fields, such as industrial inspection, medical image enhancement, remote sensing imaging, and nighttime video surveillance. In the above implementation, when reconstructing a low SNR image sequence, a motion compensation module can spatially align multiple frames, extracting global structural features and local detail features separately. A feature fusion module calculates fusion weights and performs weighted fusion of the feature maps, achieving high-quality image reconstruction and restoring image details and structure even in noisy environments.

[0083] After inter-frame motion estimation and compensation, multiple iterative optimizations are performed based on the calculation results of the fusion network. Upon completion of the iterations, the target fusion weights are obtained to accurately characterize the contribution of different frame images to structural and detail information. Based on these fusion weights, a high-quality reconstructed image can be generated, achieving accurate restoration of low signal-to-noise ratio (SNR) images. Specific implementation methods for low SNR image reconstruction can be found in the descriptions in the following embodiments, which will not be elaborated upon here.

[0084] Example 2

[0085] As a further optimization of the preceding embodiments, this application proposes a low signal-to-noise ratio image reconstruction method (MFIRe) based on multi-frame fusion and interactive modulation to meet the personalized needs of image reconstruction in low signal-to-noise ratio scenarios.

[0086] To avoid alignment errors caused by optical flow estimation in real low signal-to-noise ratio images, an image registration method based on enhanced correlation coefficient (ECC) is proposed. Assuming that there is only a translation relationship between images, the transformation matrix is ​​estimated by maximizing the correlation of grayscale images, thereby achieving accurate alignment between image frames.

[0087] To address the issue of reference-frame-dominated fusion strategies neglecting inter-frame complementarity, an attention-based fusion module is proposed. This module calculates the correlation and difference between each frame and the reference frame (the first frame) using an attention mechanism, generating weights to highlight the unique contribution of each frame. This approach preserves common information between frames while emphasizing differential information (such as unique details of a particular frame), thereby improving the quality of the fused features.

[0088] To address the personalized needs of image reconstruction in low signal-to-noise ratio scenarios, an interactive modulation GAN model (IMGAN) is proposed. This model adjusts the weights of each layer of the Unet network using an input control vector, enabling interactive control of the reconstruction results. Furthermore, to address the vanishing gradient problem in the generator, a Degradation Adaptive Modulation (DAM) module is introduced into the discriminator to modulate intermediate features based on the degree of degradation.

[0089] S101. Overall Model Architecture

[0090] Figure 1 The overall structure and process of MFIRe are demonstrated. First, the method takes multiple frames of images and a degradation condition vector as input. The multiple frames are a set of composite degradation images (14 frames, RGB format); the degradation condition vector is a vector that controls the restoration intensity, with each sample having a two-dimensional vector taking values ​​[0,1], determining the restoration effect. By performing ECC registration and attention fusion on the multiple frames, a single-frame feature map is obtained, which is then input into the trained IMGAN for inference. By modifying the degradation vector, images with controllable detail generation are obtained.

[0091] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the overall architecture of MFIRe, a low signal-to-noise ratio image reconstruction method.

[0092] S102. Introduction to Model Components

[0093] S102.1 ECC Registration

[0094] This algorithm uses the center frame as a reference, measures its similarity with other neighboring frames in grayscale space, and iteratively optimizes the optimal geometric transformation matrix between images by maximizing the correlation coefficient between the two images. Considering that the main relationship between multiple frames is translation, this method uses a two-dimensional translation model (MOTION_TRANSLATION) to avoid the instability introduced by higher-order transformations such as rotation and scaling. In the specific implementation, the reference frame and the frame to be registered are first converted into grayscale images, and the 2×3 translation transformation matrix is ​​initialized as the identity matrix. Then, by setting the maximum number of iterations and the optimization convergence threshold, the transformation matrix is ​​estimated under the findTransformECC interface provided by OpenCV, and the warpAffine function is used to register the neighboring frames to the spatial position of the reference frame. To ensure registration robustness, when the algorithm fails to converge, the original unregistered image is retained by default to prevent information loss due to transformation failure. This registration strategy can effectively alleviate the spatial misalignment problem caused by camera shake or slight movement in image sequences, and provide a stable and consistent feature base for subsequent multi-frame fusion and image reconstruction.

[0095] S102.2 Attention Fusion

[0096] The method calculates the correlation and difference between each frame and the reference frame using an attention mechanism, generates dynamic weights to weight and fuse features, and then utilizes a multi-scale hierarchical structure (Hourglass) to enhance the fusion effect. This approach preserves common information between frames while highlighting unique details (such as displacement or texture differences), thus improving the expressive power of the fused features.

[0097] First, the first frame is extracted from the input features as the reference frame, with a shape of [B, C, H, W], where B is the batch number, C is the number of channels, and H and W are the length and width of the feature map. A 3×3 convolution is then applied to the reference frame features to generate an embedding representation. The shape is [B, C, H, W]. Then, for all frames (flattened to [B, T-1, C, H, W]), where T represents the frame number, a 3×3 convolution is used to generate the embedding representation. The shape is [B, T-1, C, H, W].

[0098] Then, the correlation between each frame and the reference frame is calculated:

[0099] in, Let [i] be the correlation map between the i-th frame and the reference frame, with shape [B,1,H,W]. Let H be the feature map of the i-th frame on the c-th channel, with a size of H×W; This represents the feature map of the reference frame (first frame) on the c-th channel, and its size is also H×W.

[0100] Then, the difference between the non-reference frame and the reference frame is calculated:

[0101]

[0102] in, Indicates the i-th frame as a non-reference frame. With reference frame The difference is [B,1,H,W].

[0103] Then, attention weights are generated using the sigmoid function:

[0104]

[0105] in, The attention weights have a shape of [B,1,H,W].

[0106] Attention weights are applied to the features of the non-reference frames to generate weighted features:

[0107]

[0108] in, The weighted features have dimensions [B,C,H,W]. The embedding representation of the i-th frame is given by equation (4). Equation (4) is used to obtain the weighted features of frames other than the reference frame. After concatenation with the reference frame, the features of [B,T*C,H,W] are formed. Then, a single-frame feature map of shape [B,C,H,W] is obtained by 1×1 convolution, which is the fused feature map.

[0109] Finally, the Hourglass structure is input for multi-scale hierarchical enhancement, as shown in the following process:

[0110] Downsampling:

[0111] (1) First 4x4 convolution (stride 2), from [B,C,H,W] to [B,2C,H / 2,W / 2].

[0112] (2) Second 4x4 convolution (stride 2), to [B, 4C, H / 4, W / 4].

[0113] (3) Residual processing: Apply residual blocks (containing 3x3 convolution and ReLU) at each scale to enhance features.

[0114] Upsampling;

[0115] (1) Transposed convolution (2x2, stride 2) from [B,4C,H / 4,W / 4] to [B,2C,H / 2,W / 2].

[0116] (2) Transpose the convolution (2x2, stride 2) to [B,C,H,W].

[0117] (3) Skip connections: Downsampled features are concatenated with upsampled features (doubling the number of channels), and the final fused features are generated through 3x3 convolution and residual connections.

[0118]

[0119] in, For the output of a jump connection, the output is still [B,C,H,W]; This represents the [B,C,H,W] feature map obtained from the previous fusion step. Features during the downsampling process These are features from the upsampling process.

[0120] S102.3 IMGAN (Generator)

[0121] Please refer to Figure 3 , Figure 3 In the diagram, (a) shows the encoder structure of the generator in IMGAN, and (b) shows the global feature modulation module in the discriminator. The fused single-frame feature map is input into IMGAN, where the generator is a Unet-type encoder-decoder structure that also performs interactive modulation by combining degradation information vectors.

[0122] The specific modulation method is as follows:

[0123] Within a certain scale, control is achieved using an encoder:

[0124]

[0125] Where f represents the processing of two convolutions, and These represent the original and modulated feature maps, respectively. These are the corresponding tuning weights.

[0126] The detailed Unet structure is shown below:

[0127] Stage 1 Left (Encoding):

[0128] The input fusion features are passed through a 3x3 convolutional layer (keeping the number of channels at 64).

[0129] Multiple conditional residual blocks are applied, each block containing:

[0130] Two layers of 3x3 convolutions and ReLU activation are used to generate features.

[0131] A linear layer maps the degradation information vector cond([B, 2]) to 64-dimensional weights, modulating the intensity of the features.

[0132] Residual connection: Add the input features to the modulated output.

[0133] The output shape remains [B, 64, H, W], but the features are conditionally modulated to reflect the user-specified recovery strength.

[0134] Stage 2 Left (Encoding):

[0135] By downsampling with 4x4 convolution (stride 2), the resolution is reduced to 64x64, and the number of channels is increased to 128.

[0136] Apply conditional residual blocks, similar to Stage 1, but with 128 channels, and map the degenerate vector to 128-dimensional weights.

[0137] The output shape is [B, 128, H / 2, W / 2].

[0138] Stage 3 Left (Encoding):

[0139] By downsampling again using 4x4 convolution (stride 2), the number of channels increases to 256.

[0140] The application conditional residual block has 256 channels, and the degradation vector is mapped to 256-dimensional weights.

[0141] The output shape is [B, 256, H / 4, W / 4], capturing deep features.

[0142] Stage 3 Right (Decoding):

[0143] The features of Stage 3 Left are processed by applying conditional residual blocks, and degenerate vectors are mapped to 256-dimensional weights to modulate the features while preserving their shape [B, 256, H / 4, W / 4].

[0144] By using 2x2 transposed convolution upsampling, the resolution is doubled, and the number of channels is reduced to 128.

[0145] Stage 2 Right (Decoding):

[0146] The application uses conditional residual blocks, employs degenerate vector mapping to 128-dimensional weights, and modulates the features output by Stage 3 Right with 128 channels.

[0147] Adding the features from Stage 1 Left, and upsampling through 2x2 transposed convolution, the resolution is doubled again, and the number of channels is reduced to 64.

[0148] The output shape is [B, 64, H / 2, W / 2].

[0149] Stage 1 Right (Decoding):

[0150] Apply conditional residual blocks to process the characteristics of Stage 2 Right output, with 64 channels.

[0151] The number of channels is reduced to 32 (RGB) by 3x3 convolution, and a degenerate vector is used to map to 32-dimensional weights. The modulated features are then added to the initial fused features (or input), and the output shape is [B, 32, H, W].

[0152] Finally, the number of channels is adjusted by a convolution, and the output shape is [B, 3, H, W].

[0153] S102.4 IMGAN (Discriminator)

[0154] The discriminator receives a high-resolution image of shape [B, 3, H, W] and a degradation vector cond.

[0155] It contains multiple 3x3 convolutional layers, gradually increasing the number of channels (from 3 channels in the input to higher dimensions, such as 64, 128, 256, 512), and reducing the resolution through stride (stride=2) or pooling layers (such as 2x2 max pooling).

[0156] Each convolutional layer is typically followed by a LeakyReLU activation function (with a negative slope of approximately 0.2) to enhance nonlinearity.

[0157] Finally, the features are compressed into a single-valued or small-dimensional vector ([B, 1], representing the image authenticity score (0 for false, 1 for true) through global average pooling or fully connected layers.

[0158] For the j-th modulation module in the discriminator, two independent fully connected layers are used to generate parameters for scaling and shifting operations, respectively, for global feature modulation. Specifically, we have:

[0159]

[0160] in, and They represent the first One for generating scaling parameters and offset parameters The fully connected layer and These are the feature maps before and after modulation, respectively, where z represents the degradation vector. Note that... and The dimension is equal to the first The number of channels in the intermediate feature map of each modulation module.

[0161] S103 Loss Function

[0162] In this embodiment, multiple loss functions are used for joint optimization, including L1 loss, perceptual loss, and adversarial loss, as shown in the following formula:

[0163] For L1 loss:

[0164]

[0165] in, Represents the loss function. Indicates L1 loss, This represents the reconstructed image. This represents a true, clear image (ground truth).

[0166] Regarding the loss of resistance:

[0167] in, This represents the adversarial loss of the generator. Indicates a degraded image. This represents the degradation condition vector. This indicates the generator output image (reconstructed image). The discriminant's adversarial loss represents the probability that the image is real.

[0168]

[0169] in, This represents the adversarial loss of the discriminator.

[0170] Regarding perceived loss:

[0171] Perceptual loss refers to the calculation of the difference between the reconstructed image and the real image in the high-level feature space of the VGG network (used to enhance perceptual quality).

[0172]

[0173] in, Indicates perceived loss. This indicates feature extraction at a specific layer of VGG19 (usually ReLU5_4). This represents the L1 norm (absolute difference).

[0174] The total loss is:

[0175] in, Indicates the total loss. Indicates adoption and The parameters are optimized using both types of losses.

[0176] After calculating the various loss functions mentioned above, this method trains the generative adversarial network using a joint optimization strategy. During training, the generator based on multi-frame fusion features can fully utilize inter-frame complementary information to recover details and structural features in images under low signal-to-noise ratio conditions. The attention fusion mechanism effectively reduces redundant information interference during the fusion process and enhances the ability to preserve texture and edge information. Simultaneously, through an interactive modulation module, the generator's feature mapping is dynamically adjusted according to the degree of degradation, achieving flexible control over denoising and deblurring effects.

[0177] Through this training process, this method can generate high-fidelity reconstructed images under various complex degradation conditions such as low light, noise, and blur. Compared with traditional single-frame reconstruction techniques, this method shows significant improvements in maintaining image structural integrity, restoring texture details, and suppressing noise. Furthermore, by introducing a user-controllable degradation condition vector, the reconstruction results can not only automatically optimize the overall quality but also flexibly adjust the degree of denoising and deblurring according to actual needs, thereby achieving more personalized image reconstruction results.

[0178] Example 3

[0179] To further verify the effectiveness of the low signal-to-noise ratio image reconstruction method described in this application, training and testing experiments were conducted based on a real image dataset. By employing an inter-frame alignment method with enhanced correlation coefficient (ECC) and a multi-frame feature fusion strategy, the unique detailed information (such as texture and edges) of different frames in multiple images was fully captured, effectively compensating for the lack of information in a single frame. Simultaneously, the experiment introduced a user-controllable degradation condition vector and performed progressive modulation within the multi-layer structure of the generative adversarial network, thereby enabling interactive control of the degree of image denoising and deblurring, providing technical support for personalized adjustments to the subsequent reconstruction results. The following description, in conjunction with the experimental setup, training details, and model performance, provides further clarification.

[0180] S201. Experiment Setup

[0181] The S201.1 dataset uses multi-frame data from RealBSR-RGB, with 14 low-quality images and 1 high-quality image for each scene. A script was then used to generate datasets containing different levels of blur and noise. The covariance of the Gaussian noise ranges from σ∈[0, 50]. For a 21×21 Gaussian blur, the kernel width ranges from r∈[0, 4]. For both blur and noise, the degradation level is uniformly sampled with steps of 0.1 and 1, respectively, and the degradation information is encoded into the conditional vector z.

[0182] S201.2 Training Details

[0183] Before training, we used OpenCV to align frames in the frame sequence to estimate the transformation matrix of non-reference frames. During training, the input images were flipped and rotated for enhancement, with a batch size of 24 and an image patch size of 128×128. A linear warm-up cosine annealing scheduler was used, with a total training duration of 160 epochs and a maximum of 180 epochs. The learning rate was gradually increased from 0 to 2e-4 in the first 15 epochs, and then gradually decreased according to the cosine annealing strategy in the remaining epochs. All experiments were performed on 2×NVIDIA RTX 3090 GPUs. Our training pipeline is as follows: Figure 4 As shown.

[0184] S202. Model Effect

[0185] Please refer to Figure 5 , Figure 5 By dragging the denoising and deblurring sliders, different degradation condition parameters can be generated, producing effect images under different parameter adjustments, which can be used to produce effects with controllable details.

[0186] For images in low signal-to-noise ratio scenarios (generally considered to be low light, noisy, and blurry; here, we're referring to images with combined noise and blur), by dragging the denoising level parameter (0~1, with a step size of 0.1) while keeping the deblurring slider unchanged, we can determine that the denoising effect is best when the denoising level is 0.6. Then, keeping the denoising slider unchanged, we drag the deblurring slider, and we can determine that the effect is best when the deblurring level is 0.2. Finally, we determine the degradation parameters as [0.2, 0.6]. The quality of the reconstruction is subject to personal subjective judgment.

[0187] This application's embodiments capture unique details (such as texture and edges) in each frame using ECC alignment and multi-frame fusion methods, compensating for the deficiencies of information in a single frame and providing a solid foundation for interactive modulation. Subsequently, a user-controllable degradation condition vector is used to progressively modulate multi-level information to achieve interactive modulation.

[0188] Example 4

[0189] As a further optimization of the preceding embodiments, this application provides a low signal-to-noise ratio image reconstruction apparatus, the apparatus including:

[0190] An image acquisition module is used to acquire a low signal-to-noise ratio image sequence containing a reference frame and at least one non-reference frame;

[0191] The registration module is used to perform enhanced correlation coefficient registration on the non-reference frame and align the non-reference frame to the reference frame based on the optimal geometric transformation matrix obtained by iteration.

[0192] The attention fusion module is used to calculate the correlation and difference between each frame and the reference frame through the attention mechanism, generate dynamic weights, and weighted fuse features.

[0193] The feature enhancement module is used to enhance the weighted fused feature map using a multi-scale hierarchical structure;

[0194] The modulation module is used to interactively modulate the enhanced feature map into the generative adversarial network to generate a reconstructed image.

[0195] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the low signal-to-noise ratio (SNR) image reconstruction device provided in this application embodiment. First, the image acquisition module acquires low SNR image sequences of reference frames and non-reference frames. The registration module then uses enhanced correlation coefficient registration technology to align the non-reference frames, achieving spatial consistency across multiple frames. Next, the attention fusion module calculates the correlation and difference between each frame and the reference frame, generates dynamic weights, and performs weighted fusion of features from multiple frames. Then, the feature enhancement module enhances the fused features using a multi-scale hierarchical structure, integrating global and detailed information. Finally, the modulation module inputs the enhanced feature map into a generative adversarial network for interactive modulation, outputting a reconstructed image that meets user preferences. The modular implementation of multi-frame alignment, feature fusion, multi-scale enhancement, and interactive modulation processes enables high-quality and controllable image reconstruction under low SNR conditions.

[0196] It should be understood that the various modules of the low signal-to-noise ratio image reconstruction device provided in the above embodiments are only illustrated by the division of functional modules in the above description. In practical applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0197] The functional modules in the above embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of this application.

[0198] Based on the same concept, embodiments of this application also provide a computer device, which may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the low signal-to-noise ratio image reconstruction method as described above.

[0199] Based on the same concept, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the low signal-to-noise ratio image reconstruction method as described above.

[0200] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0201] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0202] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for reconstructing low signal-to-noise ratio images, characterized in that, Includes the following steps: S1. Obtain a low signal-to-noise ratio image sequence containing a reference frame and at least one non-reference frame; S2. Perform enhanced correlation coefficient registration on the non-reference frame, and align the non-reference frame to the reference frame based on the optimal geometric transformation matrix obtained iteratively; S3. Calculate the correlation and difference between each frame and the reference frame through an attention mechanism, generate dynamic weights, and fuse features in a weighted manner; S31. After embedding each non-reference frame through a 3x3 convolution, calculate the correlation map with the reference frame pixel by pixel; S32. Input the absolute difference between the correlation map of each frame and the reference frame into the Sigmoid function to generate difference-aware attention weights; S33. The reference frame features are output directly, and the non-reference frame features are weighted pixel by pixel and then concatenated along the channel dimension. They are then aggregated into fused features through a 1x1 convolution. S4. Enhance the weighted fused feature map using a multi-scale hierarchical structure; S41. Enhance the feature map using residual blocks and extract multi-scale context through two-stage downsampling; S42. Stack residual blocks at the lowest resolution layer to integrate global information and suppress noise and misalignment of the global information; S43. Upsample step by step and skip connections with the coding layer features, then add the output to the input feature residuals after a 3x3 convolution; S5. Input the enhanced feature map into the generative adversarial network for interactive modulation to generate the reconstructed image.

2. The method according to claim 1, characterized in that, S2 includes: S21. Perform grayscale conversion and brightness normalization preprocessing on the reference frame and non-reference frame; S22. Adaptively select the motion model and initialize the transformation matrix, set the number of iterations and the convergence threshold; S23. Use enhanced correlation coefficient optimization to solve for the optimal transformation matrix; S24. Perform reverse resampling alignment according to the motion model and output the aligned frame.

3. The method according to claim 2, characterized in that, The motion model is a translation model, an affine model, or a homography transformation model.

4. The method according to claim 1, characterized in that, The generative adversarial network in S5 includes: The generator modulates the network layer weights according to the input control vector to adjust the reconstruction result; The discriminator classifies the reconstructed image and adaptively modulates intermediate features based on the degree of degradation.

5. A low signal-to-noise ratio image reconstruction device, characterized in that, include: An image acquisition module is used to acquire a low signal-to-noise ratio image sequence containing a reference frame and at least one non-reference frame; The registration module is used to perform enhanced correlation coefficient registration on the non-reference frame and align the non-reference frame to the reference frame based on the optimal geometric transformation matrix obtained by iteration. The attention fusion module is used to calculate the correlation and difference between each frame and the reference frame through an attention mechanism, generate dynamic weights, and weighted fuse features. It includes: embedding each non-reference frame into a 3x3 convolution, and calculating a correlation map with the reference frame pixel by pixel; inputting the absolute difference between the correlation maps of each frame and the reference frame into a Sigmoid function to generate difference-aware attention weights; the reference frame features are output directly, and the non-reference frame features are weighted pixel by pixel and then concatenated along the channel dimension, and then aggregated into fused features through a 1x1 convolution. The feature enhancement module is used to enhance the weighted fused feature map using a multi-scale hierarchical structure; including: enhancing the feature map through residual blocks, extracting multi-scale context through two-level downsampling; stacking residual blocks in the lowest resolution layer to integrate global information and suppressing noise and misalignment of the global information; upsampling step by step and skipping connections with the coding layer features, and adding the output with the input feature residual after 3x3 convolution; The modulation module is used to interactively modulate the enhanced feature map into the generative adversarial network to generate a reconstructed image.

6. The apparatus according to claim 5, characterized in that, The registration module includes: The preprocessing unit is used to perform grayscale conversion and brightness normalization; The model selection unit is used for adaptive initialization of motion model parameters; The optimization unit is used to iteratively solve for the optimal transformation matrix; The resampling unit is used to perform inverse deformation according to the motion model.

7. A computer device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the method as claimed in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 4.

Images