A low-quality image super-resolution reconstruction method based on a generative adversarial network

Abstract

The application discloses a kind of low-quality image super-resolution reconstruction methods based on adversarial generative network, including, with the generation dynamic point diffusion function of differentiable optical diffraction calculation layer, and based on the function, wavefront aberration dynamic compensation mechanism is established, wavefront aberration is predicted and compensated, and the low-resolution input feature of optimization is obtained;According to the optimization feature, generator and discriminator are trained using physical constraint adversarial training strategy, and high-resolution image conforming to optical physical law is generated;The diffraction characteristics of high-resolution image are optimized through multi-scale diffraction consistency verification module, to ensure the physical authenticity and consistency of image under different resolutions;The application can effectively handle the complex degradation characteristics of low-quality image, improve the detail quality and optical rationality of generated image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of low-quality image super-resolution processing, and particularly relates to a low-quality image super-resolution reconstruction method based on a generative adversarial network. BACKGROUND

[0002] With the rapid development of computer vision, remote sensing imaging, medical image analysis, and security monitoring, image super-resolution reconstruction technology as an important image quality improvement method has received extensive attention in recent years. In practical application scenarios, due to the performance limitations of imaging devices, environmental interference, transmission compression, and physical defects of optical systems, the acquired images often exhibit low-quality characteristics, such as low resolution, severe noise, blur distortion, insufficient contrast, and missing details. These low-quality images not only severely affect the accuracy and robustness of subsequent image processing tasks, but also pose great challenges to traditional super-resolution reconstruction methods.

[0003] In recent years, with the rapid development of deep learning technology, super-resolution methods based on convolutional neural networks (CNN) have made significant progress, especially the introduction of generative adversarial networks (GAN), which makes the generated high-resolution images more visually similar to real images. However, existing super-resolution methods based on generative adversarial networks are mostly designed for ideal or mildly degraded images, ignoring the complex degradation characteristics of low-quality images in real scenarios, lacking effective modeling and joint processing mechanisms for noise, blur, compression distortion, and other degradation factors, resulting in poor reconstruction results when processing low-quality images. The generated high-resolution images often have obvious artifacts, missing details, and texture distortion. In addition, existing methods lack effective physical constraint mechanisms and fail to fully consider the optical system characteristics and imaging mechanisms in the formation process of low-quality images, resulting in deficiencies in physical reality and cross-scale consistency. Therefore, there is an urgent need for a low-quality image super-resolution reconstruction method to address the technical problems in the current field. SUMMARY

[0004] This section aims to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification to avoid obscuring the purpose of this section, abstract, and title, and such simplifications or omissions cannot be used to limit the scope of the present application.

[0005] In view of the above existing problems, the present application is proposed. Therefore, the present application provides a low-quality image super-resolution reconstruction method based on a generative adversarial network to solve the problems proposed in the background art.

[0006] To solve the above technical problems, the application provides the following technical scheme: a low-quality image super-resolution reconstruction method based on an adversarial generative network, comprising:

[0007] An optical parameter is acquired to generate a dynamic point spread function, and a wavefront aberration dynamic compensation mechanism is established based on the dynamic point spread function to predict and compensate wavefront aberration, thereby obtaining optimized low-resolution input features;

[0008] According to the optimized low-resolution input features, a physical constraint adversarial training strategy is used to train a generator and a discriminator, thereby obtaining a high-resolution image that conforms to the laws of optics and physics;

[0009] The diffraction characteristics of the high-resolution image are optimized to ensure the physical authenticity and consistency of the image at different resolutions.

[0010] As a preferred scheme of the low-quality image super-resolution reconstruction method based on the adversarial generative network, wherein: the optical parameter is acquired to generate a dynamic point spread function, comprising:

[0011] The aperture shape, focal length and wavelength in the optical system are encoded into a three-dimensional learnable tensor, and a dynamic point spread function is generated based on the improved Fraunhofer diffraction principle.

[0012] As a preferred scheme of the low-quality image super-resolution reconstruction method based on the adversarial generative network, wherein: the improved Fraunhofer diffraction principle comprises:

[0013] A dynamic aperture mask is defined on the aperture plane, the dynamic aperture mask is subjected to Fourier transform, and the complex light field is obtained by modulating the wavefront aberration function;

[0014] The inverse Fourier transform is performed on the complex light field to calculate the square of its amplitude.

[0015] As a preferred scheme of the low-quality image super-resolution reconstruction method based on the adversarial generative network, wherein: based on the dynamic point spread function, a wavefront aberration dynamic compensation mechanism is established to predict and compensate wavefront aberration, thereby obtaining optimized low-resolution input features, comprising:

[0016] A low-resolution image is acquired, a three-dimensional Zernike coefficient matrix is predicted by a lightweight neural network, the Zernike coefficient matrix is linearly combined with a Zernike basis function in normalized polar coordinates, and a wavefront aberration function is calculated;

[0017] The input feature map generated based on the low-resolution image and the phase modulation term of the wavefront aberration function are subjected to complex domain convolution by the generator to generate a compensated input feature map.

[0018] As a preferred scheme of the low-quality image super-resolution reconstruction method based on the generative adversarial network of the application, wherein:

[0019] Design a shift variable convolution kernel, each convolution kernel corresponds to a local point spread function of an image block, and each convolution kernel is decomposed into a product of three matrices by matrix decomposition.

[0020] As a preferred scheme of the low-quality image super-resolution reconstruction method based on the generative adversarial network of the application, wherein: the generator and the discriminator are trained by adopting a physical constraint adversarial training strategy, comprising:

[0021] The optimized low-resolution input features are input into the generator, and the low-resolution features are mapped into high-resolution images through multi-layer convolution and up-sampling operations;

[0022] A multi-scale discriminator is constructed, and the high-resolution images output by the generator are subjected to Fourier transform in each scale branch, and the ratio of the amplitude to the Fourier transform amplitude of the low-resolution image is calculated as the modulation transfer function of the generated image;

[0023] The absolute value sum of the difference between the modulation transfer function of the image and the pre-calculated modulation transfer function of the real image is calculated as the physical consistency adversarial loss value.

[0024] As a preferred scheme of the low-quality image super-resolution reconstruction method based on the generative adversarial network of the application, wherein:

[0025] During the training process of the generator and the discriminator, the gradients of the optical parameters and the network parameters are subjected to orthogonalization constraint, and the loss of the network parameters is minimized and the loss of the optical parameters is maximized at the same time, and the trace of the gradient product of the two is added as a regularization term.

[0026] As a preferred scheme of the low-quality image super-resolution reconstruction method based on the generative adversarial network of the application, wherein: the high-resolution image is subjected to diffraction characteristic optimization, comprising:

[0027] Different scale diffraction ring features are extracted by ring Gabor filters, wherein each filter is composed of a Gaussian attenuation term and a phase oscillation term based on ring spacing;

[0028] The KL divergence between the generated image and the real high-resolution image in different scale diffraction ring feature distribution is calculated, and the KL divergence values of all scales are added as the cross-scale consistency loss value.

[0029] As a preferred scheme of the low-quality image super-resolution reconstruction method based on the generative adversarial network of the application, wherein:

[0030] If the difference between the mean value of the diffraction ring features of the generated image and the mean value of the real image features exceeds a set threshold, it is determined that a non-physical diffraction ring mode is detected.

[0031] The absolute value of the difference between the generated image and the mean value of the real image features is calculated, and the weight of each scale feature is adjusted according to the normalized ratio of the difference and the mean value of the real image features.

[0032] Compared with the prior art, the present application has the following beneficial effects:

[0033] 1. By obtaining optical parameters to generate a dynamic point spread function, and combining a wavefront aberration dynamic compensation mechanism, the degradation factors such as blur, noise and distortion introduced by optical system defects in low-quality images are modeled, compared with the limitations of traditional super-resolution methods that are difficult to handle multiple degradations in real scenes, the present application can jointly optimize these degradation effects to generate high-resolution images with rich details and visual reality, thereby improving the reconstruction effect of low-quality images.

[0034] 2. A physical constraint adversarial training strategy is adopted, the optical transmission characteristics of the generated image are evaluated by modulating the transfer function, and the generator is optimized by the physical consistency adversarial loss value, so as to ensure that the generated high-resolution image conforms to the optical physical law, compared with the existing methods based on the generative adversarial network, the present application can eliminate the artifacts of the generated image and ensure the physical reality of the generated image. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor. Among them:

[0036] Figure 1 The overall flowchart of the low-quality image super-resolution reconstruction method based on the adversarial generative network described in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the above objectives, characteristics and advantages of the present application more obvious and comprehensible, the specific embodiments of the present application will be described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all the other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present application.

[0038] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. The present application, however, can be practiced in a variety of ways beyond the specific details set forth herein without departing from the scope of the present application. It can be appreciated by those skilled in the art that the present application can be practiced without such specific details.

[0039] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. The "in one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an embodiment that is independent of or selected from other embodiments.

[0040] The present application is described in detail in conjunction with the schematic drawings. In the detailed description of the embodiments of the present application, the sectional view of the device structure is partially enlarged without the general proportion for the convenience of description, and the schematic drawings are only examples, which should not limit the scope of protection of the present application. In addition, the three-dimensional spatial dimensions of length, width and depth should be included in actual manufacturing.

[0041] Meanwhile, in the description of the present application, it should be noted that the terms "upper, lower, inner and outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first, second or third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0042] In the present application, unless otherwise explicitly specified and limited, the terms "mounting, connection, connection" should be understood broadly, for example: it can be fixed connection, detachable connection or integral connection; it can also be mechanical connection, electrical connection or direct connection, it can also be indirectly connected through intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0043] Example 1

[0044] Reference Figure 1For the first embodiment of the present application, the embodiment provides a low-quality image super-resolution reconstruction method based on an adversarial generative network, comprising:

[0045] S1, acquiring an optical parameter to generate a dynamic point spread function, and establishing a wavefront aberration dynamic compensation mechanism based on the dynamic point spread function to predict and compensate wavefront aberration, and obtaining an optimized low-resolution input feature;

[0046] It should be noted that, since the traditional optical system usually assumes that the optical parameter is a fixed quantity, and the present application scheme encodes the aperture shape D, focal length f and wavelength λ in the optical system into a three-dimensional learnable tensor, so that it can be dynamically adjusted in the model training process, thereby allowing the model to adaptively optimize the optical characteristics according to the input image content;

[0047] Further, the aperture shape D, focal length f and wavelength λ in the optical system are encoded into a three-dimensional learnable tensor And based on the improved Fraunhofer diffraction principle, a dynamic point spread function is generated;

[0048] Wherein, HxW represents the spatial resolution of the image, and each spatial position corresponds to the local optical characteristics;

[0049] Specifically, each dimension in the three-dimensional learnable tensor corresponds to the aperture shape D, focal length f and wavelength λ respectively;

[0050] It should be explained that the traditional Fraunhofer diffraction is a mathematical description of far-field diffraction, and its core assumption is that the observation plane is far enough from the aperture, and the incident light is a monochromatic plane wave; If applied to the super-resolution reconstruction of the image, there are two defects: first, since the traditional optical system assumes that the optical system parameters are globally fixed, it cannot model the spatially variant aberration in actual imaging, and second, it ignores some dynamic degradation scenarios (such as deformation of mobile phone lenses with temperature / pressure, multi-spectral mixed lighting, etc.);

[0051] Specifically, the traditional Fraunhofer diffraction formula is expressed as follows:

[0052]

[0053] Wherein, I(x,y) represents the light intensity distribution of the image plane (output result), which is the final imaging result after the light wave passes through the optical system; is expressed as a Fourier transform, which represents the transformation from the aperture plane to the image plane; P(u,v) is expressed as an aperture mask; is expressed as an amplitude scaling factor of far-field diffraction, which describes the energy attenuation law when the light wave propagates to the far field; A is a normalized amplitude coefficient; z is the propagation distance; x and y represent the coordinates of the x-axis and y-axis respectively;

[0054] It should be noted that u and v are two-dimensional spatial coordinates (unit: meter) of the aperture plane in the optical system; in the dynamic aperture mask, it represents the light-transmitting area of the aperture, while in the wavefront aberration function, it represents the distribution position of the aberration on the aperture plane;

[0055] Specifically, the process based on the improved Fraunhofer diffraction principle is as follows: first, a dynamic aperture mask is defined on the aperture plane, which is dynamically generated by a three-dimensional learnable tensor and used to modulate the light field distribution; then, the mask is combined with a wavefront aberration function, where the wavefront aberration function describes the phase distortion of the light wave and is predicted by the subsequent steps; then, the Fourier transform is performed on the modulated light field distribution to calculate the complex light field on the image plane; finally, the square of the amplitude of the complex light field is taken to obtain the dynamic point spread function representing the light intensity distribution, which is expressed as follows:

[0056]

[0057] where PSF dynamic (x,y) is the generated dynamic point spread function, and P θ (i,v) represents the dynamic aperture mask, W(u,v) represents the wavefront aberration function, and represents element-wise multiplication, which combines the frequency domain response of the aperture and the phase modulation term caused by aberration to form a complex light field, simulating the propagation process of the light wave through the actual optical system, and j is the imaginary unit; represents the inverse Fourier transform, which is used to convert the dynamic aperture mask and the phase modulation caused by aberration from the spatial domain to the frequency domain, and then through the inverse transform, it is converted into a dynamic point spread function; |·| 2 represents the light intensity distribution;

[0058] Specifically, P θ (u,v) is represented as:

[0059]

[0060] where Sigmoid(·) is an activation function that constrains the dynamic aperture mask to [0, 1], simulating the light transmittance of the aperture (0 is completely blocked, and 1 is completely transparent); B k (u,v) represents the B-spline basis function, which is used to construct a continuous aperture shape; θ k is a learnable parameter that controls the weight of each B-spline basis; K represents the number of B-spline basis functions;

[0061] It should be noted that, since the aperture in the optical parameter is assumed to be a fixed shape (such as a circle or a square) in the traditional optical system, and the aperture of the actual optical system (such as a mobile phone lens) may dynamically change due to mechanical deformation or diaphragm adjustment, it is necessary to simulate the irregular and dynamically changing aperture shape by linear combination of B-spline bases;

[0062] Further, a low-resolution image is acquired, a three-dimensional Zernike coefficient matrix is predicted by a lightweight neural network, and the Zernike coefficient matrix is linearly combined with a Zernike basis function in normalized polar coordinates to calculate a wavefront aberration function;

[0063] It should be explained that the lightweight neural network is an efficient model in the field of deep learning, which aims to reduce the model complexity and parameter quantity while ensuring the model performance by simplifying the structure, using low-rank approximation and depth separable convolution technology, and the commonly used lightweight neural networks include: SqueezeNet series, ShuffleNet series, MnasNet, etc.

[0064] Specifically, the predicted three-dimensional Zernike coefficient matrix is expressed as:

[0065] U=Ψ φ (I LR ;θ φ )

[0066] Wherein, Ψ φ is a lightweight neural network, I LR is an input low-resolution image, H′×W′ is an image space resolution output by the lightweight neural network; θ φ represents a set of learnable parameters, including the weights and biases of all layers in the lightweight neural network.

[0067] It should be noted that 36 represents the channel of the three-dimensional Zernike coefficient matrix, which is defined in ISO 10110-5, wherein ISO 10110-5 adopts 36 Zernike polynomials by default, and each term of the Zernike polynomial corresponds to a specific aberration mode.

[0068] Specifically, the wavefront aberration function is expressed as:

[0069]

[0070] Wherein, Z k is the k-th Zernike basis function, which is used to describe the aberration of a specific mode; (ρ,θ) is the normalized polar coordinate.

[0071] Further, the generator is used to perform complex domain convolution on the input feature map generated based on the low-resolution image and the phase modulation term of the wavefront aberration function, to generate a compensated input feature map;

[0072] Specifically, in the residual block of the generator, the input feature map is compensated:

[0073]

[0074] wherein F in is the input feature map, * represents complex domain convolution, F comp is the compensated input feature map.

[0075] It should be noted that by considering the dynamic coupling of the aberration mode and the image content, the limitation of the global uniform compensation in the traditional method is overcome, and different aberration compensation strategies for different regions of the image are completed.

[0076] S2, according to the optimized low-resolution input feature, a physical constraint counter-training strategy is used to train the generator and the discriminator, to obtain a high-resolution image conforming to the optical physical law.

[0077] Further, a shift variable convolution kernel is designed, each convolution kernel corresponds to a local point spread function of an image block, and each convolution kernel is decomposed into the product of three matrices through matrix decomposition.

[0078] Specifically, the shift variable convolution kernel {E m,n} is represented as:

[0079]

[0080] wherein E m,n represents a local convolution kernel (sub-kernel) of the image block (m, n) position, corresponding to the point spread function of the position; L m,n is a left matrix in matrix decomposition, usually called left singular vector matrix, used to represent the row space basis of E m,n ; r is the rank of the matrix; ∑ m,n is a diagonal matrix in matrix decomposition, called singular value matrix, whose diagonal elements are non-negative singular values, used to control the accuracy and complexity of matrix decomposition; is a right matrix transpose in matrix decomposition, usually called transpose of right singular vector matrix, used to represent the column space basis of E m,n ; rank represents a constraint condition.

[0081] For example, assuming that the parameter amount of the original shift variable convolution kernel E m,n is k x k, after decomposition, the parameter amount is: L m,n : k x r, ∑ m,n : r x r, The total is expressed as: r x (2k+r), so it can be seen that when r = 3, it is much smaller than k 2 ;

[0082] It should be noted that the low-rank constraint (which can vary depending on the value of r) reduces the number of parameters, thereby reducing the computational and storage costs, while preserving the main information of the convolution kernel;

[0083] Further, the optimized low-resolution input feature is input into the generator, and the low-resolution feature is mapped to a high-resolution image through multi-layer convolution and up-sampling operation;

[0084] Specifically, the up-sampling operation includes transpose convolution or pixel rearrangement operation, which enlarges the spatial size of the low-resolution input feature to the target high-resolution feature to generate a high-resolution image;

[0085] Further, by constructing a multi-scale discriminator, the high-resolution image output by the generator is subjected to Fourier transform in each scale branch, and the ratio of its amplitude to the Fourier transform amplitude of the low-resolution image is calculated as the modulation transfer function of the generated image;

[0086] Specifically, the modulation transfer function is expressed as:

[0087]

[0088] Where s represents the down-sampling scale, and the image is evaluated at different resolutions (such as original size, 1 / 2 size, 1 / 4 size, …) to ensure that the generated image has physical consistency at each scale; l represents spatial frequency, G θ is the generator; The modulation transfer function of the generated image at the down-sampling scale and the spatial frequency l is used to measure the performance of the generated image in optical characteristics;

[0089] Further, the absolute value sum of the difference between the modulation transfer function of the image and the pre-calculated modulation transfer function of the real image is calculated as the physical consistency adversarial loss value;

[0090] Specifically, the physical consistency adversarial loss value L phys is calculated as follows:

[0091]

[0092] Where S represents the number of down-sampling scales, and ‖·‖1 represents the L1 norm (sum of absolute differences); represents the modulation transfer function of the real high-resolution image at the down-sampling scale;

[0093] It should be noted that by comparing the modulation transfer functions of the generated image and the real image, the optical characteristics of the generator output in the frequency domain can be constrained to be consistent with the real image, thereby reducing non-physical artifacts;

[0094] Furthermore, in the training process of the generator and the discriminator, the gradients of the optical parameters and the network parameters are subjected to an orthogonalization constraint, and by minimizing the loss of the network parameters and maximizing the loss of the optical parameters, while adding the trace of the gradient product of the two as a regularization term, the entanglement of the parameter solution can be realized;

[0095] Specifically, the formula for realizing parameter solution entanglement is as follows:

[0096]

[0097] Where θ is the set of the generator and the lightweight neural network; Indicates the gradient of the loss function L with respect to θ; Indicates the gradient of the loss function L with respect to Θ, where Θ refers to a three-dimensional learnable tensor;

[0098] Indicates the gradient orthogonalization term, which calculates the trace of the matrix product of And L adv Indicates the adversarial loss, which is a standard loss form based on the generative adversarial network;

[0099] It should be noted that minimizing θ is to make the generator generate more realistic images, and maximizing Θ is to enhance the exploratory nature of the optical parameters to ensure that the three-dimensional learnable tensor constructed can cover a variety of optical scenarios;

[0100] S3, by optimizing the diffraction characteristics of the high-resolution image, ensuring the physical reality and consistency of the image under different resolutions;

[0101] It needs to be explained that the generative adversarial network generates visually realistic images in the super-resolution task, but these images may not conform to the laws of optical physics; for example, if the generated image contains non-physical textures or artifacts, it is unacceptable in scenarios that require high-precision calculation; in addition, low-quality images are usually strongly affected by complex optical degradation (such as diffraction and scattering), and even if they are processed by technology, the high-resolution images obtained still need to be consistent with these degradation mechanisms; therefore, the obtained high-resolution image needs to be optimized for diffraction characteristics;

[0102] Further, different scale diffraction ring features are extracted by a ring Gabor filter, where each filter is composed of a Gaussian attenuation term and a phase oscillation term based on ring spacing;

[0103] Specifically, the design of the annular Gabor filter is as follows:

[0104]

[0105] wherein r is the control ring spacing, represents the frequency of annular oscillation, also known as the scale index, used to control the density of the diffraction ring; σ represents the control bandwidth; is represented as a Gaussian decay term, representing the amplitude distribution of the filter in the spatial domain, which presents exponential decay with the increase of the distance (x 2 +y 2 ); is represented as a phase oscillation term based on the ring spacing;

[0106] Further, the KL divergence between the generated image and the real high-resolution image in the distribution of diffraction ring features at different scales is calculated, and the sum of the KL divergence values at all scales is taken as the cross-scale consistency loss value;

[0107] Specifically, the cross-scale consistency loss value L diff is represented as:

[0108]

[0109] wherein R represents a scale set containing all Gabor filter scales used for feature extraction; is the probability distribution of the generated image at the rth scale under the diffraction ring feature; is the probability distribution of the real high-resolution image at the rth scale under the diffraction ring feature; is represented as the KL divergence, used to measure the difference between the generated image and the real image in the rth scale feature distribution, which can be extended as:

[0110]

[0111] Further, if the difference between the mean value of the diffraction ring features of the generated image and the mean value of the features of the real image exceeds the set threshold, it is determined that the non-physical diffraction ring mode is detected;

[0112] Specifically, the set threshold is represented as:

[0113]

[0114] wherein, is the mean value of the diffraction ring features of the generated image at the rth scale, is the mean value of the diffraction ring features of the real high-resolution image at the rth scale; τ is the set threshold, which is usually determined by experimental parameter adjustment, or based on the feature distribution results of model training, or the diffraction features of the real image are statistically obtained;

[0115] Further, the absolute value of the difference between the generated image and the mean value of the real image features is calculated, and the weight of each scale feature is adjusted according to the normalized ratio of the difference and the mean value of the real image features;

[0116] Specifically, the weight adjustment process of each scale feature is as follows:

[0117]

[0118] wherein w (r) represents the weight of the diffraction ring feature under the rth scale, represents the absolute difference (subject to L1 norm) between the generated image and the real image in the mean value of the rth scale feature; and δ represents a very small positive number, which approaches 0;

[0119] It should be noted that since the cross-scale consistency loss value is adjusted by optimizing the feature distribution through global (all scales) features, local anomalies (for example, extreme deviation in some scales) cannot be completely eliminated, and then the weight of each scale feature can be adjusted for local correction to enhance the fineness of feature optimization.

[0120] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as object-oriented programming language Java and interpreted scripting language JavaScript.

[0121] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0122] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0123] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0124] Although preferred embodiments of the application have been described herein, substitutions and alterations are possible in view of the disclosure of this application without departing from the spirit and scope of the present application. Therefore, it is the intent of the appended claims to cover all such variations and modifications as come within the scope of the present application. The group or groups of processes and / or blocks may have been presented in an arrangement

[0125] which is quite different from the arrangement of the method, although more simple arrangements can also be employed. As such, interchanging of the blocks and order of the blocks can occur depending upon the implementation, in addition to being altered, objects can be grouped differently. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A low-quality image super-resolution reconstruction method based on an adversarial generative network, characterized in that, include: Optical parameters are obtained to generate a dynamic point spread function, and a dynamic wavefront aberration compensation mechanism is established based on the dynamic point spread function to predict and compensate for wavefront aberrations, thereby obtaining optimized low-resolution input features. The process of obtaining optical parameters to generate a dynamic point spread function includes: The aperture shape, focal length, and wavelength in the optical system are encoded as three-dimensional learnable tensors, and a dynamic point spread function is generated based on the improved Fraunhofer diffraction principle. The improved Fraunhofer diffraction principle includes: A dynamic aperture mask is defined on the aperture plane, a Fourier transform is performed on the dynamic aperture mask, and modulation is performed in combination with the wavefront aberration function to obtain a complex optical field. Perform an inverse Fourier transform on the complex light field and calculate the square of its amplitude; Based on the optimized low-resolution input features, a physical constraint adversarial training strategy is used to train the generator and discriminator to obtain a high-resolution image that conforms to the laws of optical physics. The training of the generator and discriminator using a physical constraint adversarial training strategy includes: The optimized low-resolution input features are fed into the generator, and through multi-layer convolution and upsampling operations, the low-resolution features are mapped to a high-resolution image. A multi-scale discriminator is constructed. In each scale branch, the high-resolution image output by the generator is subjected to Fourier transform, and the ratio of its amplitude to the Fourier transform amplitude of the low-resolution image is calculated as the modulation transfer function of the generated image. The sum of the absolute values ​​of the differences between the modulation transfer function of the image and the pre-calculated modulation transfer function of the real image is used as the physical consistency adversarial loss value. By optimizing the diffraction characteristics of the high-resolution image, the physical authenticity and consistency of the image at different resolutions are ensured. Optimizing the diffraction characteristics of the high-resolution image includes: Diffraction ring features at different scales are extracted using a ring Gabor filter, where each filter consists of a Gaussian attenuation term and a phase oscillation term based on the ring spacing. Calculate the KL divergence between the generated image and the real high-resolution image at different scales of diffraction ring feature distribution, and sum the KL divergence values ​​at all scales as the cross-scale consistency loss value. 2.The low-quality image super-resolution reconstruction method based on the generative adversarial network of claim 1, wherein, Based on the dynamic point spread function, a dynamic compensation mechanism for wavefront aberrations is established to predict and compensate for wavefront aberrations, resulting in optimized low-resolution input features, including: A low-resolution image is acquired, and a three-dimensional Zernike coefficient matrix is ​​predicted using a lightweight neural network. The Zernike coefficient matrix is ​​then linearly combined with the Zernike basis functions in normalized polar coordinates to calculate the wavefront aberration function. The generator performs complex-domain convolution between the input feature map generated from the low-resolution image and the phase modulation term of the wavefront aberration function to generate a compensated input feature map. 3.The low-quality image super-resolution reconstruction method based on the generative adversarial network of claim 2, wherein, Also includes: We design a shiftable variable convolution kernel, where each kernel corresponds to the local point spread function of an image patch, and each kernel is decomposed into the product of three matrices through matrix decomposition. 4.The low-quality image super-resolution reconstruction method based on the generative adversarial network of claim 1, wherein, Also includes: In the training process of the generator and the discriminator, an orthogonalization constraint is imposed on the gradients of the optical parameters and the network parameters, and the trace of the product of the gradients of the two is added as a regularization term while minimizing the loss of the network parameters and maximizing the loss of the optical parameters. 5.The low-quality image super-resolution reconstruction method based on the generative adversarial network of claim 1, wherein, Also included are: If the difference between the mean of the diffraction ring features of the generated image and the mean of the features of the real image exceeds a set threshold, it is determined that the non-physical diffraction ring mode is detected; The absolute value of the difference between the generated image and the mean of the features of the real image is calculated, and the weight of each scale feature is adjusted according to the normalized ratio of the difference to the mean of the features of the real image.

Images