Image reconstruction method

Abstract

The image reconstruction method is used for super-resolution image reconstruction, and at least comprises an original image, a generator, a discriminator and a loss function. The method at least comprises the following steps: S1, inputting the original image into the generator, and generating a sample by the generator under the action of the loss function; s2, the discriminator discriminates an error between the sample and the original image; s3, if the error does not meet the set target, the S1 is repeated; and if the error meets the set target, outputting a reconstructed image. The invention provides an image super-resolution reconstruction method to solve the problems of poor image perception quality, edge detail loss and material feature blurring in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and particularly to a method for image reconstruction. Background Art

[0002] Due to the limitations of the physical characteristics of microscopic devices and imaging technologies, the resolution of material microscopic images is often limited, making it difficult to accurately characterize the fine features of their edges. Existing materials, especially nanomaterials, with their excellent properties, have shown great application potential in many fields. This is more attributed to the great progress of characterization tools such as scanning electron microscopy (SEM) and TEM. They can not only help us intuitively reveal the surface morphology of materials but also open the door to the world of microscopic molecular structures for us. However, despite the significant progress of characterization tools such as SEM, there are still some challenges in practical applications. Among them, the limitation of image resolution is a particularly prominent problem. Due to the constraints of the physical characteristics of scanners and imaging protocols, fine-scale features of materials, such as the morphology of catalyst particles and the wall structure of CNTs, are often difficult to be accurately captured. This not only affects the in-depth understanding of material properties but also limits the exploration of unknown fields and the capture of important information. Therefore, using super-resolution reconstruction technology, important information and features of materials can be accurately captured and comprehensive material information can be obtained. These efforts can open up new paths for the research and application of nanomaterials and promote the further development of nanoscience and technology.

[0003] Existing super-resolution reconstruction technologies are mostly based on traditional numerical calculations, interpolation calculations, and traditional convolutional neural networks. They not only have high computational difficulty and long algorithm time but also have the following defects in complex scenarios: 1. Edge blurring: When traditional methods magnify images, it is difficult to maintain the edge sharpness, resulting in the loss of details of substances in the images; 2. Poor coordination of multiple loss functions: Existing technologies often use a single loss function and cannot optimize image content, edges, and perceptual quality simultaneously; 3. Limitations of the attention mechanism: The conventional attention module has insufficient ability to dynamically adjust channel and spatial features, affecting the reconstruction quality.

[0004] Therefore, it is urgent and necessary to develop a super-resolution reconstruction algorithm that can accurately capture detail features, optimize image content, edges, and perceptual quality simultaneously, and dynamically adjust image features. Summary of the Invention

[0005] The present invention provides a method for image super-resolution reconstruction to solve the problems of poor image perceptual quality, loss of edge details, and blurred material features existing in the prior art.

[0006] To solve the above technical problems, the present invention is implemented as follows:

[0007] The present invention provides an image super-resolution reconstruction method for improving the image perception quality, accurately capturing edge details, and clarifying material features, which at least includes an original image, a generator, a discriminator, and a loss function; the method at least includes the following steps:

[0008] S1: Input the original image into the generator, and the generator generates samples under the action of the loss function;

[0009] S2: The discriminator discriminates the error between the sample and the original image;

[0010] S3: If the error does not meet the set target, repeat S1; if the error meets the set target, output the reconstructed image.

[0011] Optionally, the generator at least includes a convolutional block attention mechanism, a residual network, and an upsampling module.

[0012] Optionally, the loss function is composed of the sum of at least one weight sub-function; the weight sub-function is the product of a weight factor and a sub-function; the sub-function at least includes at least one of an edge-preserving loss function, a contrast loss function, a perceptual loss function, an image loss function, an adversarial loss function, or a total variation loss function.

[0013] Optionally, the loss function formula is or where, represents the i-th sub-function, σ represents a learnable uncertainty parameter, the σ represents the uncertainty of the i-th sub-function, and N represents the total number of sub-functions.

[0014] Optionally, the sub-function at least includes the edge-preserving loss function, and the edge-preserving loss function is where α is a loss factor; M is the total number of pixels; G represents an edge operator; G(I real ) i,j and G(I gen ) i,j respectively represent the edge feature values of the real image and the generated image at the coordinate (i,j); p is an exponent for adjusting the loss sensitivity.

[0015] Optionally, the convolutional block attention mechanism at least includes at least one of a channel attention mechanism and a spatial attention mechanism; the channel attention mechanism is adjusted by dynamic parameters; the dynamic parameters at least include at least one of global average pooling and global maximum pooling; the spatial attention mechanism includes deformable convolution.

[0016] Optionally, the channel attention mechanism is X channel =σ(W2·Xmax ·(W1·X avg +b1)+b2), where σ is the activation function; W1, W2, b1, and b2 are the dynamic parameters; X avg is the global average pooling; X max is the global max pooling.

[0017] Optionally, the spatial attention mechanism is where σ is the activation function; ω i is the weight of the convolutional kernel; p i is the original position of the convolutional kernel; Δp i is the dynamically generated offset; m i is the dynamically generated modulation factor.

[0018] Optionally, the convolutional block attention mechanism is X out = g·X channel +(1 - g)·X spatial , where g is the gating weight.

[0019] Optionally, the original image is preferably a microscopic image of a nanomaterial, and the nanomaterial at least includes a carbon nanomaterial, and the carbon nanomaterial at least includes at least one of a carbon nanotube and graphene.

[0020] The present invention solves the problem of mismatched contribution degrees of sub - functions in the loss function by setting a loss function and adding a weight factor; solves the problem of loss of image edge details by setting an edge - preserving loss function and adding a loss factor and an exponent for adjusting the loss sensitivity; solves the problem of mismatched contribution degrees of the channel attention mechanism and the spatial attention mechanism in the convolutional block attention mechanism by setting a gating weight; solves the problem of blurring of material features by setting dynamic parameters and the offset of the deformable convolution; jointly solves the problem of poor image perception quality through the loss function and the convolutional block attention mechanism; the present invention solves the problems of poor image perception quality, loss of edge details, and blurring of material features existing in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments of the present invention. Obviously, the following - described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 Represents a flowchart of an image reconstruction provided by an embodiment of the present invention;

[0023] Figure 2It shows a composition diagram of image reconstruction provided by the embodiments of the present invention;

[0024] Figure 3 It shows a raw image of carbon nanotubes and sample diagrams of image reconstruction by different methods provided by the embodiments of the present invention;

[0025] Figure 4 It shows a raw image of a catalyst and sample diagrams of image reconstruction by different methods provided by the embodiments of the present invention;

[0026] Explanation of reference numerals:

[0027] 10, raw image; 20, generator; 30, discriminator; 40, loss function; 50, sample. Specific embodiments

[0028] Next, the technical solutions in the present invention will be clearly and completely described in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative inventions belong to the scope of protection of the present invention.

[0029] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, the "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0030] See Figures 1-4 , the present invention provides an image super-resolution reconstruction method for improving image perception quality, accurately capturing edge details and clear material features, at least including a raw image (10), a generator (20), a discriminator (30) and a loss function (40); the method at least includes the following steps:

[0031] S1: Input the raw image (10) into the generator (20), and the generator (20) generates a sample (50) under the action of the loss function (40);

[0032] S2: The discriminator (30) discriminates the error (60) between the sample (50) and the raw image (10);

[0033] S3: If the error (60) does not meet the set target, repeat S1; if the error (60) meets the set target, output the reconstructed image.

[0034] In the present invention, the following method is specifically adopted: First, microscopic images such as low-resolution scanning electron microscope (SEM) images of the material are collected and input into the generator (20) as the original image (10) for training. During the training process, the loss function (40) plays a key role in optimizing the samples (50) generated by the generator (20) to reduce the loss. At the same time, the discriminator (30) discriminates the generated samples (50), and based on the discrimination results, promotes the continuous optimization of the generator to generate more realistic details. When the discriminator (30) determines that the error between the generated sample (50) and the original image (10) reaches the pre-set target, the training process stops, and at this time, the generator (20) outputs the reconstructed image.

[0035] The beneficial effects of the present invention are as follows: By setting the loss function (40) and adding a weight factor, the problem of mismatched contribution degrees of sub-functions in the loss function (40) is solved; by setting an edge-preserving loss function and adding a loss factor and an exponent for adjusting the loss sensitivity, the problem of loss of image edge details is solved; by setting gating weights, the problem of mismatched contribution degrees of the channel attention mechanism and the spatial attention mechanism in the convolutional block attention mechanism (21) is solved; by setting dynamic parameters and the offset of the deformable convolution, the problem of blurred material features is solved; through the loss function (40) and the convolutional block attention mechanism (21), the problem of poor image perception quality is jointly solved; the present invention solves the problems of poor image perception quality, loss of edge details, and blurred material features existing in the prior art.

[0036] It should be noted that the material involved in the present invention is preferably a nanomaterial. Due to the limitation and influence of physical equipment on the characterization of nanomaterials, super-resolution reconstruction can make nanomaterials clearer and information capture more complete.

[0037] Preferably, the nanomaterial at least includes a carbon material, and the carbon material at least includes carbon nanotubes and graphene.

[0038] Optionally, the generator (20) at least includes a convolutional block attention mechanism (21), a residual network (22), and an upsampling module (23).

[0039] In the present invention, the core structure of the generator (20) consists of a convolutional block attention mechanism (21), a residual network (22), and an upsampling module (23). Taking a low-resolution image as input data and feeding it into the generator (20), when the generator (20) generates a sample (50), the convolutional block attention mechanism (21) intelligently identifies which channels and spatial regions are more critical for generating high-quality samples by dynamically adjusting the weights of channel and spatial features, and then increases the weights of these important features accordingly while reducing the weights of unimportant features to enhance the attention to key information. For example, in the SEM image of a nanomaterial, the features of the edges and the tube wall structure need to be preferentially enhanced, and the convolutional block attention mechanism (21) can automatically increase the weights of the relevant channels and suppress background noise. The residual network (22) uses a stack of multiple residual blocks to solve the problem of gradient disappearance in the training of deep networks. The residual connection allows low-frequency information (such as basic textures) to be directly transmitted to the deep layers, avoiding information loss, and at the same time extracting multi-scale features (such as the microscopic morphology of catalyst particles). These multi-level features complement each other and work together to provide a rich and comprehensive information basis for the generator (20) to generate high-quality samples. Finally, the upsampling module (23) gradually increases the resolution using sub-pixel convolution or transposed convolution. Sub-pixel convolution generates a high-resolution image by channel rearrangement, reducing checkerboard artifacts; transposed convolution restores details through a learnable deconvolution kernel to ensure the continuity of the material surface morphology.

[0040] Optionally, the loss function (40) consists of the sum of at least one weight sub-function; the weight sub-function is the product of a weight factor and a sub-function; the sub-function includes at least one of an edge-preserving loss function, a contrast loss function, a perceptual loss function, an image loss function, an adversarial loss function, or a total variation loss function.

[0041] In the present invention, the loss function (40) solves the problem that a single function cannot take into account image content, edges, and perceptual quality through multi-task collaborative optimization. For the weight sub-functions, each sub-function is assigned a weight factor, and the contributions of each sub-function are automatically balanced through learnable uncertainty parameters. For example, when the priority of edge detail restoration is high, the weight factor of the edge-preserving loss function in the sub-function increases to strengthen edge optimization. The types of sub-functions are: an edge-preserving loss function, which is used to ensure the sharpness of the material edges and prevent detail blurring; a contrast loss function, which enhances the local contrast of the image and improves texture details and visual perception; a perceptual loss function, which is based on feature matching of a pre-trained network to improve the overall perceptual quality of the image; an image loss function, which measures the pixel-level difference between the generated image and the real image to ensure the consistency of basic content; an adversarial loss function, which drives the generator to generate more realistic details; a total variation loss function, which suppresses image noise and maintains local smoothness.

[0042] Optionally, the formula of the loss function (40) is or where represents the i-th sub-function, and σ i represents a learnable uncertainty parameter, and the σ i represents the uncertainty of the i-th sub-function, and N represents the total number of sub-functions.

[0043] In the present invention, the total formula of the loss function (40) realizes dynamic weight allocation through Bayesian uncertainty modeling. represents the weighting of the sub-function, and logσ i prevents σ i from being too large and causing the weight to fail. The i-th sub-function represented by represents different sub-functions. During the training process, the model automatically learns the uncertainty (σ i ) of each sub-function. For example, if the σ i of the edge-preserving loss function is small, the contribution of the edge-preserving loss function to the loss function is significantly enhanced, thereby accurately restoring the material edge structure in the microscopic image.

[0044] Optionally, the sub-function at least includes the edge-preserving loss function, and the edge-preserving loss function is where α is a loss factor; M is the total number of pixels; G represents an edge operator; G(I real ) i,j and G(I gen ) i,j respectively represent the edge feature values of the real image and the generated image at the coordinate (i, j); p is an exponent for adjusting the loss sensitivity.

[0045] In the present invention, the edge-preserving loss function is specifically designed for the edge blurring problem of the material microscopic image. The edge features of the real image and the generated image are extracted using an edge operator, and the difference norm between the two is calculated. α is a loss factor for controlling the overall intensity of the edge loss function, and p is an exponent for adjusting the loss sensitivity, which can adjust the sensitivity to the edge error. For example, when p = 1, it is more robust to small errors, and when p = 2, it is more sensitive to significant errors. In the reconstruction of granular material images, this function can significantly improve the clarity of the granular boundaries and avoid edge diffusion caused by traditional methods.

[0046] Optionally, the convolutional block attention mechanism (21) at least includes at least one of a channel attention mechanism and a spatial attention mechanism; the channel attention mechanism is adjusted by dynamic parameters; the dynamic parameters at least include at least one of global average pooling and global maximum pooling; the spatial attention mechanism includes deformable convolution.

[0047] In the present invention, the convolution block attention mechanism (21) dynamically optimizes feature expression through a channel attention mechanism or a spatial attention mechanism. For the channel attention mechanism, the dynamic parameters use global average pooling and global maximum pooling to capture channel set statistics, and generate channel weights through a fully connected layer. The channel attention mechanism can enhance the response of important channels (such as channels containing material defects) and suppress redundant channels. For the deformable convolution included in the spatial attention mechanism, the position of the receptive field of the convolution kernel is adaptively adjusted through dynamically generated offsets and modulation factors. In material images, deformable convolution can more accurately align irregular structures (such as curved parts of nanowires) and avoid feature misalignment caused by traditional fixed convolution kernels.

[0048] Optionally, the channel attention mechanism is X channel =σ(W2·X max ·(W1·X avg +b1)+b2), where σ is the activation function; W1, W2, b1, b2 are the dynamic parameters; X avg is the global average pooling; X max is the global maximum pooling.

[0049] In the present invention, the channel attention mechanism generates channel weights by fusing the features of global average pooling and global maximum pooling. avg and X max Respectively reflect the mean and peak characteristics of the channel. W1, W2, b1, and b2 are dynamic parameters used to fuse the two pooling results. At the same time, the global average pooling and maximum pooling of the dynamic parameters are consistent with X avg and X max There is no difference. σ is the activation function, which maps the output to a weight value of [0,1]. Any activation function that can map the output to a weight value of [0,1] can be used.

[0050] Optionally, the spatial attention mechanism is Among them, σ is the activation function; ω i is the weight of the convolution kernel; p i is the original position of the convolution kernel; Δp i is the dynamically generated offset; m i is a dynamically generated modulation factor.

[0051] In the present invention, the spatial attention mechanism dynamically adjusts the sampling position and weight of the convolution kernel through deformable convolution. i The offset predicted according to the input feature map content makes the convolution kernel focus on the key area; the modulation factor m i , controls the weight of the sampling position and further improves the feature response of important areas.

[0052] Optionally, the convolutional block attention mechanism (21) is X out = g·X channel + (1 - g)·X spatial , where g is the gating weight.

[0053] In the present invention, the fusion of the channel attention mechanism and the spatial attention mechanism achieves dynamic balance through the gating weight (g). For the gating weight g, it is a parameter that can be automatically learned, and its value range is [0, 1]. It can be used to automatically allocate the contribution ratio of the channel attention mechanism and the spatial attention mechanism, enabling the model to adaptively select the optimal strategy.

[0054] Optionally, the original image (10) is preferably a microscopic image of a nanomaterial. The nanomaterial at least includes a carbon nanomaterial, and the carbon nanomaterial at least includes at least one of carbon nanotubes and graphene.

[0055] It should be specifically noted that the microscopic image of the nanomaterial is a microscopic image taken by SEM, TEM, etc.

[0056] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or device including that element.

[0057] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that makes a contribution to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0058] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit and scope protected by the claims of the present invention, and all of them fall within the protection scope of the present invention.

[0059] Embodiment 1

[0060] First, data preparation and preprocessing are carried out. Low-resolution scanning electron microscope images of carbon nanotubes are collected as the original input data, with a total of 3,687 images. The dataset is divided into a training set (2,946 images) and a validation set (741 images) according to a ratio of 0.8:0.2, and random shuffling is performed to enhance the generalization ability. The original images are centrally cropped to remove redundant background noise, and a region of interest of 800×800 pixels is generated.

[0061] Secondly, the model is constructed. The generator is based on a residual neural network and contains 16 residual blocks. Each residual block consists of two convolutional layers, a batch normalization layer, and a PReLU activation function. A convolutional block attention mechanism is embedded after each residual block to dynamically optimize the feature weights through channel attention and spatial attention. The pixel shuffling technique is used to gradually magnify the low-resolution feature map to a high-resolution image through sub-pixel convolution technology to avoid checkerboard artifacts. The discriminator adopts a Markov discriminator structure, which consists of 8 convolutional layers, a batch normalization layer, and a LeakyReLU activation function, and outputs the authenticity probability of the image. The final layer outputs the discriminant result through a Sigmoid activation function.

[0062] Thirdly, the loss function is designed. The edge-preserving loss function uses the Sobel operator to extract the edge features of the real image and the generated image, and its calculation formula is: where α = 1.5, p = 1, to enhance the edge sharpness; the adversarial loss function is where D is the discriminator, I gen is the generated image, and N is the batch size; the total variation loss function can suppress image noise, and its formula is: Subsequently, the final loss function is designed as whose weight factors are 1, 0.01, 0.006, 2e -8 , 0.01, 0.01.

[0063] After the neural network and the loss function are designed, the neural network is trained under the action of the loss function. The Adam optimizer is used during the training process, and the initial learning rate is 1×10 -4, dynamically adjust the learning rate; the batch size is 16, the gradient clipping threshold is 0.1, and the maximum number of training epochs is 200; and an alternating training strategy is adopted. In each round of training, first update the discriminator network parameters, and then update the generator network parameters. The validation set is used to monitor the model performance, and the model checkpoint with the lowest validation loss is retained.

[0064] After the training is completed, quantitative analysis and comparison are carried out between Example 1 of the present invention and nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, convolutional neural network, and generative adversarial neural network. The image quality indicators PSNR (31.492 dB) and SSIM (0.932) of Example 1 are significantly better than those of the nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, convolutional neural network, and generative adversarial neural network methods. Specifically, the PSNR value of Example 1 is 16.91% and 4.77% higher than that of the best-performing traditional method (bicubic interpolation) and deep learning method (generative adversarial neural network), respectively, which fully demonstrates that Example 1 has the best super-resolution reconstruction performance.

[0065] Qualitative analysis is also carried out in Example 1. The edge of the carbon nanotube tube wall in the reconstructed image is clear, and the morphological details of the catalyst particles are complete.

[0066]

Claims

1. A method for image reconstruction, characterized in that, For super-resolution reconstructed images, it at least includes an original image (10), a generator (20), a discriminator (30), and a loss function (40); the method at least includes the following steps: S1: Input the original image (10) into the generator (20), and the generator (20) generates a sample (50) under the action of the loss function (40); S2: The discriminator (30) discriminates the error (60) between the sample (50) and the original image (10); S3: If the error (60) does not meet the set target, repeat S1; if the error (60) meets the set target, output the reconstructed image.

2. The method for image reconstruction according to claim 1, wherein The generator (20) at least includes a convolutional block attention mechanism (21), a residual network (22), and an upsampling module (23).

3. The method for image reconstruction according to claim 1, wherein The loss function (40) consists of the sum of at least one weight sub-function; the weight sub-function is the product of a weight factor and a sub-function; the sub-function at least includes at least one of an edge-preserving loss function, a contrast loss function, a perceptual loss function, an image loss function, an adversarial loss function, or a total variation loss function.

4. The method for image reconstruction according to claim 3, wherein The formula of the loss function (40) is or where represents the i-th sub-function, and σ i represents the learnable uncertainty parameter, and the σ i represents the uncertainty of the i-th sub-function, and N represents the total number of the sub-functions.

5. The method for image reconstruction according to claim 3, wherein The sub-function at least includes the edge-preserving loss function, and the edge-preserving loss function is where α is a loss factor; M is the total number of pixels; G represents an edge operator; G(I real ) i,j and G(I gen ) i,j represent the edge feature values of the real image and the generated image at the coordinates (i, j) respectively; p is an exponent for adjusting the loss sensitivity.

6. The method for image reconstruction according to claim 2, wherein, The convolutional block attention mechanism (21) at least includes at least one of a channel attention mechanism and a spatial attention mechanism; the channel attention mechanism is adjusted by dynamic parameters; the dynamic parameters at least include at least one of global average pooling and global max pooling; the spatial attention mechanism includes deformable convolution.

7. The method for image reconstruction according to claim 6, wherein The channel attention mechanism is X channel = σ(W2·X max ·(W1·X avg + b1)+ b2), where σ is the activation function; W1, W2, b1, b2 are the dynamic parameters; X avg is the global average pooling; X max is the global max pooling.

8. The method for image reconstruction according to claim 6, wherein The spatial attention mechanism is where σ is the activation function; ω i is the weight of the convolution kernel; p i is the original position of the convolution kernel; Δp i is the dynamically generated offset; m i is the dynamically generated modulation factor.

9. The method for image reconstruction according to claim 6, wherein The convolutional block attention mechanism (21) is X out = g·X channel +(1 - g)·X spatial , where g is the gating weight.

10. The method for image reconstruction according to claim 1, characterized in that, The original image (10) is preferably a microscopic image of a nanomaterial, the nanomaterial at least includes a carbon nanomaterial, and the carbon nanomaterial at least includes at least one of a carbon nanotube and graphene.

Images