An image super-resolution reconstruction method based on a deep channel spatial attention network

Abstract

The present application relates to image super-resolution reconstruction, and proposes a method based on deep channel spatial attention network. First, the low-resolution input image is normalized and converted to a four-dimensional tensor, which is cut according to the non-overlapping block, and the inference is forced to float32 precision. The model includes shallow feature extraction, deep channel spatial attention module, upsampling and residual reconstruction layer. The output residual is fused with the bicubic interpolation base after hard clipping, and then clipped again. If the pixel is out of range, Min-Max stretching to [0, 1] is started, and finally quantized to 8-bit image is saved. The method combines channel and spatial attention to enhance high-frequency features, combines residual clipping and contrast stretching double protection mechanism to ensure numerical stability, and cooperates with the blocking strategy to avoid memory overflow, which is suitable for any large size SEM image, realizes high magnification, high fidelity and artifact-free reconstruction.

Description

Technical Field

[0001] This invention relates to the fields of image processing and deep learning technology, specifically to an image super-resolution reconstruction method based on a deep channel spatial attention network, which is particularly suitable for high-magnification reconstruction of SEM grayscale images. Background Technology

[0002] Image super-resolution reconstruction aims to recover high-resolution images containing rich high-frequency details from low-resolution observed images, and has significant application value in fields such as medical image analysis, SEM image data processing, and high-definition display. Early traditional interpolation algorithms had low computational complexity, but were limited by the model's expressive power, failing to effectively reconstruct high-frequency details and easily leading to blurred image edges and texture distortion. Currently, deep learning-based super-resolution methods significantly improve image reconstruction quality by learning the mapping relationship from low resolution to high resolution end-to-end, and have become the mainstream research direction in this field.

[0003] However, existing deep learning-based super-resolution techniques suffer from insufficient high-frequency feature modeling capabilities and blurred reconstructed textures in practical engineering applications. Most existing networks directly learn end-to-end pixel-level mappings or only optimize overall pixel loss, lacking structured modeling of high-frequency residual information and effective joint attention to channel and spatial dimensional features. This results in poor image generation quality and insufficient high-frequency characteristics. Summary of the Invention

[0004] The purpose of this invention is to provide an image super-resolution reconstruction method based on a deep channel spatial attention network to solve technical problems in existing technologies, such as insufficient high-frequency detail recovery, overflow of inference values ​​in deep networks leading to completely white / black images, memory overflow of large-size images, and edge stitching artifacts. This invention enhances high-frequency feature modeling through a joint channel and spatial dual attention mechanism, introduces a dual protection mechanism of high-frequency residual safety truncation and emergency contrast stretching to ensure stable inference values, and combines strict non-overlapping block division and high-precision calculation strategies to achieve efficient and stable reconstruction of arbitrarily large-size images.

[0005] To achieve the above objectives, this invention provides an image super-resolution reconstruction method based on a deep channel spatial attention network, and the specific technical solution adopted is as follows:

[0006] S1: Obtain low-resolution input image and parameter configuration

[0007] It receives a low-resolution 8-bit grayscale image to be reconstructed, supports formats such as PNG, JPEG, and BMP, and determines the target magnification factor. Currently, the achievable magnification factors are 2, 4, or 8 times.

[0008] S2: Image Preprocessing and Tensor Normalization

[0009] The formula maps the pixel values ​​of the input image from [0, 255] to the floating-point range of [0, 1]: Reshape the data into the four-dimensional tensor format required by the network. , in For batches, It is a single channel. Defines the image's height and width.

[0010] S3: Constructing a Deep Channel Spatial Attention Network Model (DCSAN)

[0011] The network model comprises, in sequence, a shallow feature extraction (SFE) module, a deep channel spatial attention (CSAM) module, an upsampling module, and a residual reconstruction layer, as designed below:

[0012] (1) Shallow Feature Extraction Module (SFE): Employing... Convolution maps a single-channel input to a high-dimensional feature space (default number of channels). Extracting basic feature maps

[0013] (2) Deep Channel Spatial Attention Module (CSAM): Contains multiple residual attention groups for in-depth extraction of high-frequency structural features. Its computational logic includes the following sub-modules:

[0014] 1. Channel Attention Block (CAB): For input features Simultaneously, global average pooling (GAP) and global max pooling (GMP) are performed, and the channel weights are generated after fusion via a shared multilayer perceptron (MLP).

[0015]

[0016] in, For the Sigmoid activation function, This is element-wise multiplication.

[0017] 2. Spatial Attention Block (SAB): Along the channel dimension The average and maximum values ​​are taken and concatenated, and then used to generate spatial weights through a convolutional layer.

[0018]

[0019] Multiple CSABs are stacked to form a Channel Spatial Attention Group (CSAG), and then the outputs of multiple CSAGs are concatenated and fused with the initial input features, combined with the DropPath regularization strategy:

[0020]

[0021] (3) Long jump connection and fusion: integrating shallow features with deep features Add, through Convolutional fusion: .

[0022] (4) Upsampling module: Based on the magnification factor s, cascaded Each upsampled block. Each block is processed through... Convolution expands the channels to The spatial resolution is doubled by pixel rearrangement, and then the number of channels is restored by convolution:

[0023]

[0024] (5) Residual Reconstruction Layer: The upsampled features are processed... The convolution mapping is a single channel, outputting a high-frequency residual map res.

[0025] S4: Adaptive Blocking and High-Precision Inference Strategy

[0026] To address the issue of limited video memory for large images, the normalized image tensor is divided into blocks of a preset size (default). Perform strict, non-overlapping segmentation; execute inference commands independently for each image block, and enforce the use of float32 precision calculations to avoid half-precision truncation errors. Blocks with an edge size of less than 16 are skipped without further processing.

[0027] S5: High-frequency residual safety truncation and reconstruction fusion

[0028] During the inference phase, to prevent numerical explosion in deep networks, the residuals output by the reconstruction layer are hard-truncated and then fused with the bicubic interpolation-amplified basis graph.

[0029]

[0030] This mechanism fundamentally eliminates artifacts of pure white or pure black caused by abnormal extreme values ​​of disability.

[0031] S6: Boundary Detection and Emergency Contrast Stretching

[0032] Get the output tensor minimum value With the maximum value ; like or If a slight out-of-bounds condition is detected, an emergency Min-Max stretching algorithm is activated to proportionally compress the tensor to [the specified value]. The interval, the formula is: If the boundary is not crossed, then it will be retained. This step serves as a double safeguard, forcibly preserving the grayscale gradient of the microstructure texture.

[0033] S7: Quantization and Independent Preservation

[0034] Multiply the processed floating-point tensor by 255 and round down to convert it into an 8-bit unsigned integer image array: Each image patch is saved independently as a PNG file according to the original image coordinates, without edge blending or stitching, to avoid stitching artifacts and facilitate local feature analysis.

[0035] Preferably, in step S3, all convolutional layer weights are initialized using the Kaiming normal distribution; for the up-dimensional convolutional layers inside CSAB, the tail convolutional layers of CSAG, and the residual reconstruction layers, their weights are all initialized to zero to ensure that the network is an identity mapping in the initial stage.

[0036] The present invention has the following excellent effects:

[0037] 1. This invention constructs a deep channel spatial attention module (CSAM). By deeply combining channel attention blocks (CABs) and spatial attention blocks (SABs), the network can adaptively and accurately locate the feature channels and key spatial texture positions that contribute most to image reconstruction during deep feature extraction. This mechanism effectively overcomes the edge blurring and loss of subtle textures caused by the lack of structured high-frequency modeling in traditional super-resolution networks, resulting in a qualitative leap in edge sharpness and microstructural integrity of the reconstructed image, greatly satisfying the stringent requirements of high-precision imaging.

[0038] 2. To address the frequent numerical explosion and memory overflow issues in the practical deployment of deep learning super-resolution models, this invention provides a full-link numerical safety protection mechanism from the feature layer to the output layer. At the feature reconstruction end, a high-frequency residual safety truncation strategy is introduced to strictly limit the residuals within a reasonable range, fundamentally eliminating image artifacts of pure white or pure black caused by abnormal amplification of deep features. At the output end, an emergency Min-Max contrast stretching strategy is superimposed as a double insurance, forcibly preserving the grayscale gradient of microstructure textures to ensure zero information loss. Simultaneously, combined with a strict non-overlapping block strategy and float32 high-precision calculation, this not only perfectly resolves the memory bottleneck and half-precision truncation error in large-size image processing but also completely eliminates the unavoidable edge seam artifacts of traditional block stitching by saving independent slices by coordinates, significantly improving the reliability and robustness of super-resolution technology in complex real-world scenarios. Attached Figure Description

[0039] To more clearly illustrate the implementation of the present invention or the solutions in the prior art, the drawings used in the description of the implementation examples or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram illustrating the process principle of the present invention. Detailed Implementation

[0041] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of an image super-resolution reconstruction method based on a deep channel spatial attention network proposed in this invention for rapid SEM scanning image processing. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments may be combined in any suitable form. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0042] Please see Figure 1 This document illustrates a specific implementation of an image super-resolution reconstruction method based on a deep channel spatial attention network (DCSAN) according to an embodiment of the present invention. This embodiment uses a low-resolution microscopic tissue image acquired by a scanning electron microscope (SEM) in fast scanning mode as an example for detailed explanation. In actual SEM operation, fast scanning mode is often used to reduce electron beam damage to the sample and improve detection efficiency, but this leads to low signal-to-noise ratio, blurred grain boundaries, and severe loss of high-frequency details in the acquired image. The present invention aims to utilize a deep channel spatial attention network (DCSAN) for high-magnification super-resolution reconstruction of this type of SEM image. The specific steps are as follows:

[0043] Step S1: Obtain low-resolution SEM input image and parameter configuration

[0044] In the context of materials microstructure analysis, an 8-bit grayscale SEM image to be reconstructed is first acquired using the rapid scanning mode of a scanning electron microscope. Image formats supported include PNG, JPEG, BMP, and TIFF. Since subsequent high-frequency detail recovery and magnification are required, the system receives this low-resolution SEM image and determines the target magnification. Preferably, in this embodiment, the target magnification is configured to 4x to observe nanoscale precipitates or grain boundary features.

[0045] Step S2: Image preprocessing and tensor normalization

[0046] The original pixel value distribution of the SEM image is as follows: To adapt to the numerical computation distribution of deep learning networks and accelerate convergence, the pixel values ​​of the input image are mapped to... The formula for the floating-point range is: Subsequently, the data is reshaped into the standard four-dimensional tensor format required by the network. , in This is the batch size (default is 1 during inference). Represents the grayscale characteristics of a single channel in SEM. These represent the height and width of the SEM image, respectively.

[0047] Step S3: Build and run the Deep Channel Spatial Attention Network (DCSAN) model.

[0048] The preprocessed SEM image tensor is input into a pre-trained DCSAN model. The network model sequentially includes a shallow feature extraction (SFE) module, a deep channel spatial attention (CSAM) module, an upsampling module, and a residual reconstruction layer. Specifically:

[0049] (1) Shallow Feature Extraction Module (SFE): Employing... Convolution maps a single-channel SEM image to a 128-dimensional high-dimensional feature space, extracting the basic feature map. .

[0050] (2) Deep Channel Spatial Attention Module (CSAM): Grain boundaries, pores, and precipitates in SEM images have extremely strong local structural characteristics. This module extracts high-frequency structural features in depth through a combined channel and spatial dual attention mechanism:

[0051] 1. Channel Attention Block (CAB): Simultaneously performs Global Average Pooling (GAP) and Global Max Pooling (GMP) on the input features to capture global statistical information and local extremum features of different microphases in the SEM image. Channel weights are then generated after fusion via a shared multilayer perceptron (MLP).

[0052]

[0053] 2. Spatial Attention Block (SAB): The average and maximum values ​​along the channel dimension are taken and concatenated, and then passed through a convolutional layer to generate spatial weights.

[0054]

[0055] This allows for precise location of edges and textures in SEM images.

[0056] 3. Encapsulate the above modules into Channel Spatial Attention Blocks (CSABs) and stack them to form Channel Spatial Attention Groups (CSAGs). The outputs of multiple CSAGs are concatenated and fused with the initial input features, and combined with the DropPath regularization strategy to form a Channel Spatial Attention Amplifier (CSAM), outputting deep features. .

[0057] (3) Long jump connection and fusion: integrating shallow features with deep features Added, through Accumulation and Fusion: This is to avoid the loss of shallow contour information in SEM.

[0058] (4) Upsampling module: based on magnification Cascade Each upsampled block. Each block is processed through... The accumulation will expand the channel to The spatial resolution is doubled by using pixel rearrangement, and then the number of channels is restored by convolution.

[0059] (5) Residual Reconstruction Layer: The upsampled features are processed... The convolution mapping is converted to a single channel, outputting the high-frequency residual map res of the SEM image.

[0060] Step S4: Adaptive Block Segmentation and High-Precision Inference Strategy

[0061] Since SEM images are typically large (e.g., 2000×2000 or larger), direct full-image inference can easily lead to memory overflow. The system will normalize the SEM image tensor according to a preset block size. Perform strict, non-overlapping segmentation. Inference is performed independently for each image block, and float32 precision computation is enforced to avoid truncation errors caused by half-precision (AMP) computation in deep attention networks, which is crucial for preserving subtle grayscale gradations in SEM images. Blocks with image edge sizes smaller than 16 are skipped to prevent errors in convolution operations.

[0062] Step S5: High-frequency residual safety truncation and reconstruction fusion

[0063] When inferring SEM images using deep neural networks, the residual map `res` is prone to numerical explosion due to the interplay of complex backgrounds and high-frequency noise. To prevent all-white / all-black artifacts in the final image, a hard truncation constraint is applied to the residual output of the reconstruction layer. Simultaneously, the original low-resolution SEM input image is upsampled to the target size using bicubic interpolation to serve as the base image. The final fusion output is: This mechanism fundamentally eliminates the damage to the display effect of SEM images caused by extreme pixel values.

[0064] Step S6: Boundary Detection and Emergency Contrast Stretching

[0065] Get the output tensor minimum value With the maximum value Because the atomic number contrast of different phases in SEM images varies significantly, if or If a slight out-of-bounds error is detected, the emergency Min-Max stretching algorithm is activated: This step, as a double safeguard, proportionally compresses the tensor to... The specified range forcibly preserves the grayscale gradient levels of microstructures (such as grain boundaries and precipitates) to ensure no information loss. If the range is not exceeded, the data is directly retained. .

[0066] Step S7: Quantization and Independent Storage

[0067] The processed floating-point tensor is multiplied by 255 and rounded down to convert it into an 8-bit unsigned integer image array. Each SEM image block is saved independently as a PNG file according to the original image coordinates (such as row number and column number). This embodiment does not perform edge blending and stitching, thus completely avoiding the seam artifacts generated at SEM grain boundaries in traditional block stitching. It also facilitates materials scientists to independently extract and quantitatively analyze specific local areas (such as specific grains or defects).

[0068] Preferably, in this embodiment, all convolutional layer weights in step S3 are initialized using the Kaiming normal distribution; for the weights of the up-dimensional convolutional layers within CSAB, the tail convolutional layers of CSAG, and the residual reconstruction layers, zero initialization is performed. This ensures that the network exhibits an identity mapping in the initial stage, greatly accelerating the convergence speed of the network on complex SEM datasets and stabilizing the initial gradient flow. Thus, the high-magnification, high-fidelity, artifact-free super-resolution reconstruction process for rapidly scanning low-resolution SEM images is completed.

[0069] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0070] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. An image super-resolution reconstruction method based on a deep channel spatial attention network, characterized in that, Includes the following steps: S1: Obtain the low-resolution input image and target magnification; S2: Preprocess the input image by normalizing the pixel values ​​to the [0,1] range and converting them into a four-dimensional tensor format; S3: Construct a deep channel spatial attention network model, which includes a shallow feature extraction module, a deep channel spatial attention module, an upsampling module, and a residual reconstruction layer connected in sequence. S4: Divide the normalized image tensor into non-overlapping segments according to the preset block size, perform inference independently for each image block, and force the use of float32 precision for calculation; S5: Hard-truncate the high-frequency residual map output by the residual reconstruction layer to obtain a safe residual, and add the safe residual to the base map upsampled to the target size by bicubic interpolation, fuse them and truncate them again to the [0,1] interval to obtain the reconstructed image tensor; S6: Detect the pixel extreme values ​​of the reconstructed image tensor. If there are out-of-bounds values, enable Min-Max stretching to compress them proportionally to the [0,1] interval. S7: Quantize the processed tensor to an 8-bit integer and save it as an image file independently according to the image block coordinates.

2. The image super-resolution reconstruction method based on a deep channel spatial attention network according to claim 1, characterized in that, The depth channel spatial attention module (CSAM) includes multiple cascaded channel spatial attention groups (CSAG), and each CSAG contains at least one channel spatial attention block (CSAB). The CSAB includes a channel attention block (CAB) and a spatial attention block (SAB). The channel attention block is used to simultaneously perform global average pooling and global max pooling on the input features, and after fusion by a shared multilayer perceptron, channel weights are generated, and the channel weights are multiplied element-wise with the input features. The spatial attention block is used to take the average and maximum values ​​of the input features along the channel dimension and concatenate them. The spatial weights are generated by the convolutional layer and then multiplied element-wise with the input features. The output of the deep channel spatial attention module is obtained by concatenating and fusing the initial input features with the outputs of each CSAG, and introducing the DropPath regularization strategy.

3. The image super-resolution reconstruction method based on a deep channel spatial attention network according to claim 1, characterized in that, Steps S5 and S6 together constitute a dual numerical security protection mechanism: In step S5, the residuals are hard-truncated, limiting their numerical range to between [-1.0, 1.0], and then truncated again after being fused with the upsampled base map, in order to prevent all-white or all-black artifacts caused by numerical explosion in deep networks. In step S6, when the fused image tensor has pixel values ​​that exceed the [0,1] range, the Min-Max stretching algorithm is used to linearly map the entire tensor back to the [0,1] range to forcibly preserve the gray-scale gradient levels of the microstructure and ensure zero information loss. The image is a grayscale image obtained by scanning electron microscopy (SEM), and the magnification is 2, 4, or 8 times.

Images