Image restoration method
By constructing an image restoration model and conditional adversarial generation network with multi-scale feature fusion, combined with the prior information of superstructure lenses, the image blur problem caused by dispersion and complex optical phenomena in superstructure lens imaging is solved, and high-quality image restoration is achieved Effect.
Patent Information
- Application Number
- CN202411143546.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-29
AI Technical Summary
Superstructure lenses have image blur problems caused by dispersion and complex optical phenomena during imaging, and the prior art is difficult to deal with effectively, especially in the research of photoelectric computing fusion, which affects image clarity and color accuracy.
Using an image restoration method, an image restoration model including generator and discriminator is constructed, and a network is generated through multi-scale feature fusion and conditional adversariality, combined with the prior information of the superstructure lens, and used Wiener filtering or Richardson-Lucy algorithm to remove it. Blur processing, output high-quality reconstructed images.
It significantly improves the clarity and color reduction of the image, can capture structural details more finely, enhance global feature extraction capabilities, and improves image defuzzing and restoration performance. It is suitable for specific scenes of super lens images.
Smart Images

Figure CN119048377A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of image processing and relates to an image restoration method, in particular to a meta-lens image restoration method. Background Art
[0002] A metalens is a flat lens that uses a metasurface to focus light. It manipulates the optical response of incident light, including its amplitude, phase, and polarization, through a series of artificial antennas (i.e., metasurface structures). Metalens can perform complex optical operations at an extremely thin level, achieving a focusing effect similar to that of a traditional lens, but with a smaller size and lighter weight. Metalens has the following advantages:
[0003] Thin and light with high efficiency: The thickness of the meta-lens is only a few hundred nanometers to micrometers, much smaller than traditional lenses, and can achieve efficient optical focusing;
[0004] Low-cost mass production: Produced through semiconductor chip technology, which greatly reduces production costs and is suitable for large-scale applications;
[0005] Versatility: Metalens can also realize additional functions such as polarization manipulation and splitting to meet the needs of more complex application scenarios;
[0006] Metalens cameras are widely used in photography, AR / VR, aerospace, smart cars, microscopic imaging, etc. They can significantly reduce the size and weight of camera systems and improve imaging quality and performance.
[0007] However, meta-lens imaging still has the following problems:
[0008] Dispersion and chromatic aberration: The intrinsic characteristics of meta-lenses cause their phase changes to depend on wavelength, which can cause dispersion problems, resulting in chromatic aberration of light of different wavelengths when imaging, which in turn affects the clarity and color accuracy of the image. For example, in the research of optoelectronic computing fusion meta-lenses, chromatic aberration is one of the key factors limiting its widespread application.
[0009] Complex optical process: The imaging process of lens-free cameras or imaging systems based on meta-lenses often involves complex optical phenomena such as diffraction and interference, which cause the scene information to form complex patterns or stripes on the sensor rather than a direct and clear image.
[0010] Image restoration technology can correct image blur caused by dispersion, diffraction and other effects, and restore clearer image details. For example, in the study of optoelectronic computing fusion, the processed image showed a clearer outline in the edge extraction comparison experiment. Image restoration technology can learn the mapping relationship of color difference, thereby restoring the original color of the color image and improving the color restoration degree.
[0011] With the development of deep learning, deep learning-based methods such as U-Net provide a new way to restore meta-lens images. However, the current research on this type of algorithm is mainly aimed at scenes such as motion deblurring and defocusing, and cannot be directly applied to meta-lenses. Therefore, it is worth further research and exploration to combine the prior information of meta-lenses with the characteristics of meta-lens images and effectively integrate image feature information. Summary of the invention
[0012] In view of this, an object of the present invention is to provide an image restoration method.
[0013] In order to achieve the above object, the present invention provides the following technical solutions:
[0014] An image restoration method comprises the following steps:
[0015] S1. Construct a data set, collect true value images and meta-lens imaging images to establish data pairs, and form a data set;
[0016] S2. Construct an image restoration model including at least a generator and a discriminator, wherein the generator includes at least an image feature extraction module, a feature propagation module, and a feature decoding module, which outputs a reconstructed image after multi-scale feature fusion of the meta-lens imaging image; the discriminator uses the reconstructed image and the corresponding true value image as inputs of the discriminator, and guides the image restoration model to generate a higher quality restored image by performing block discrimination on the reconstructed image and the true value image;
[0017] S3, constructing a loss function for the constructed image restoration model, which includes a loss function based on a conditional adversarial generation network and an image restoration loss function;
[0018] S4, training the image restoration model based on the constructed data set and loss function;
[0019] S5. Use the generator in the trained image restoration model to restore the metalens imaging image collected in real time by the metalens imaging module, and output the restored image.
[0020] Furthermore, in the generator of the image restoration model, the image feature extraction module generates input features of several scales through several downsampling convolutions, extracts features through corresponding channel information, including multiple plane convolutions, and fuses information of different feature layers of the same scale through jump connections and upsampling to fuse feature information of other scales, and finally outputs the fused feature information of multiple scales;
[0021] The feature information extracted by the image feature extraction module is input into the prior information feature propagation layer of the feature propagation module according to the corresponding scale channel. The feature information of each size is processed by the deblurring algorithm based on the prior information of the meta-lens point spread function, and then transmitted to the feature decoding module according to their respective channels.
[0022] The feature decoding module uses a fully convolutional network model for feature inputs of different scales. By fusing feature information of different scales, it realizes the conversion of multi-scale characteristic information from feature space to image space and outputs a reconstructed image.
[0023] Furthermore, in the feature extraction module, the image is first input into the image feature extraction module according to the original size, and the image feature extraction module establishes information channels of different scales through downsampling operations of different scales;
[0024] The first sampling scale channel is obtained by downsampling the first sampling scale, and the second sampling scale channel is obtained by downsampling the second sampling scale, thereby obtaining the original scale channel, the first sampling scale channel, and the second sampling scale channel;
[0025] In the original scale channel, the first sampling scale channel and the second sampling scale channel, a plurality of planar convolution layers corresponding to the respective channels are synchronously used to extract features;
[0026] After passing through multiple planar convolutional layers, each scale channel of the image feature extraction module is provided with a jump connection layer, through which the feature information of different layers in the same scale is fused;
[0027] The fused features are then subjected to several planar convolutional layers to further extract feature information;
[0028] After further extracting features through several planar convolutional layers, the features in the second sampling scale channel are directly input into the prior information feature propagation layer under the second sampling scale channel corresponding to the image feature extraction module in the feature propagation module; and the features in the second sampling scale channel are also adjusted to the first sampling scale through an upsampling operation;
[0029] The features in the first sampling scale channel and the features in the second sampling scale channel are fused through a planar convolution layer, and after fusion, they are directly input into the prior information feature propagation layer under the first sampling scale channel corresponding to the image feature extraction module in the feature propagation module; and the fused features in the first sampling scale channel after the fusion of the features in the second sampling scale channel are also adjusted to the original size through an upsampling operation, and are fused with the features in the original size channel in the original size channel through a planar convolution layer, and the features in the fused original size channel are also directly input into the prior information feature propagation layer in the feature propagation module.
[0030] Further, in the feature propagation module, a priori information feature propagation layer under the original size, a priori information feature propagation layer under the first sampling size, and a priori information feature propagation layer under the second sampling size are correspondingly provided for the original size, the first sampling size, and the second sampling scale channels in the image feature extraction module;
[0031] The prior information feature propagation layer corresponding to each channel inputs the feature information of each scale into the restoration model based on the prior information of the meta-lens point spread function for deblurring;
[0032] The point spread function information is downsampled to match the feature size requirements of different scales. It adopts an explicit model-based deblurring algorithm, including the Wiener filter algorithm or the Richardson-Lucy algorithm. The parameters of the Wiener filter or the Richardson-Lucy algorithm are designed to be optimizable weight information for training.
[0033] Furthermore, if the feature propagation module is implemented using Wiener filtering, its calculation expression is:
[0034]
[0035] Among them, K is the signal-to-noise ratio information of the image, H(u,v) is the frequency domain transform of the meta-lens point spread function, and G(u,v) is the frequency domain transform of the characteristic information output by the image feature extraction module. Finally, An inverse Fourier transform is performed as the output of the feature propagation module.
[0036] If the feature propagation module is implemented using the Richardson-Lucy algorithm, its calculation expression is:
[0037]
[0038] Where: the initial f(x, y) represents the feature information of the true value image output by the feature extraction module, * represents the convolution operation, h(x, y) and g(x, y) represent the point spread function information and the feature information output by the blurred image feature extraction module respectively, and n represents the number of iterations of the image.
[0039] Furthermore, in the feature decoding module, the feature decoding module fuses the features in each scale channel to output a reconstructed image, where:
[0040] The feature decoding module also includes the original size channel, the first sampling size channel and the second sampling size channel. After each channel receives feature information of different scales, it first uses an SE attention network layer to process the features based on the channel attention mechanism;
[0041] Then, in the original scale channel, several planar convolution layers are used to process to obtain the original size feature image, the original size feature image is downsampled once and then fused with the features in the first sampling size channel, and processed by several planar convolution layers to obtain the first sampling size feature image;
[0042] Then, the first sampling size feature image is downsampled once, and it is combined with the feature image in the second sampling size channel through several planar convolution layers to obtain the second sampling size feature image;
[0043] Next, the second sampling size feature image is subjected to an upsampling operation and then fused with the first sampling size feature through a skip connection layer. The fused feature image is processed through several planar convolution layers and then adjusted to an image of the same size as the original through an upsampling operation.
[0044] Then, it is fused with the original size feature image through a jump connection layer in the original size channel, and then processed by several plane layers. After the processing is completed, the reconstructed image is output.
[0045] Furthermore, in the discriminator of the image restoration model, based on the architecture of patchGAN, the input reconstructed image and the true value image are divided into blocks for discrimination. By dividing the image into blocks, N*N blocks are established, and each block is discriminated and classified separately. Finally, the calculation results of all blocks are averaged to output the final discriminator result.
[0046] Furthermore, the loss function is expressed as:
[0047]
[0048] Among them, G is the image output by the generator, and D is the image output by the discriminator;
[0049] The loss function consists of two parts: the loss function based on the conditional adversarial generation network And the image restoration loss function;
[0050] Loss function based on conditional adversarial generation network It is expressed as:
[0051]
[0052] Image restoration loss function includes L2 loss function And the loss function based on gradient information in:
[0053]
[0054] The gradient calculation kernel function is defined as:
[0055] k 1 =[1-1],
[0056] The beneficial effects of the present invention are:
[0057] In the process of image processing, the present invention fully adopts and applies the prior information of the meta-lens image. In order to capture finer structural details, multi-scale convolution operations are incorporated into the technology. This operation can extract features across different spatial scales, thereby ensuring that structures of all sizes in the image can be effectively identified and processed.
[0058] In addition, the features extracted from different convolutional layers are effectively integrated through channel fusion technology. This fusion not only enriches the feature representation, but also significantly enhances the ability to extract global features, allowing the algorithm to understand the image content more comprehensively.
[0059] In order to further improve the quality of image restoration, the conditional generative adversarial network (cGAN) structure is adopted. In this structure, the task of the generator is to generate clear images that are as realistic as possible, while the discriminator strives to distinguish between the generated images and the real clear images. Through this adversarial training, the image restoration ability learned by the generator has been significantly improved.
[0060] Finally, this method shows superior performance in image deblurring and restoration. The application of the present invention is not limited to this. In fact, it can also be flexibly applied to various other public blurred image dataset tasks, showing wide applicability and practicality.
[0061] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0063] Figure 1 It is a schematic diagram of the overall structure of the image restoration model of the present invention;
[0064] Figure 2 It is a structural schematic diagram of a generator in the image restoration model of the present invention;
[0065] Figure 3 The prior information of the point spread function images of the red, green and blue spectral segments in the generator of the present invention;
[0066] Figure 4 It is a schematic diagram of the structure of the discriminator in the image restoration model of the present invention;
[0067] Figure 5 A simplified schematic diagram of the meta-lens imaging image restoration system of the present invention;
[0068] Figure 6 It is a schematic diagram of the comparison between the image before and after restoration and the true value image of the meta-lens imaging image restoration system of the present invention. DETAILED DESCRIPTION
[0069] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.
[0070] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.
[0071] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0072] See also Figure 1 to Figure 6 , is an image restoration method, which at least includes the following steps:
[0073] S1. Construct a data set, collect true value images and meta-lens imaging images to establish data pairs, and form a data set;
[0074] S2. Construct an image restoration model including at least a generator and a discriminator, wherein the generator includes at least an image feature extraction module, a feature propagation module, and a feature decoding module, which outputs a reconstructed image after multi-scale feature fusion of the meta-lens imaging image; the discriminator uses the reconstructed image and the corresponding true value image as inputs of the discriminator, and guides the image restoration model to generate a higher quality restored image by performing block discrimination on the reconstructed image and the true value image;
[0075] S3, constructing a loss function for the constructed image restoration model, which includes a loss function based on a conditional adversarial generation network and an image restoration loss function;
[0076] S4, training the image restoration model based on the constructed data set and loss function;
[0077] S5. Use the generator in the trained image restoration model to restore the metalens imaging image collected in real time by the metalens imaging module, and output the restored image.
[0078] Example
[0079] In step S1, this embodiment establishes data pairs by respectively collecting the true value image and the meta-lens imaging image of the same object in the same form, the same angle, and the same time, and collects a large number of data pairs according to the above method, and then expands them to obtain a data set.
[0080] In step S2, the overall structure diagram of the image restoration model constructed in this embodiment is as follows: Figure 1 As shown in , it at least includes a generator and a discriminator. Specifically, the structural diagram of the generator is as follows Figure 2 As shown, it at least includes an image feature extraction module, a feature propagation module based on prior information, and a feature decoding module.
[0081] The image is first input into the image feature extraction module according to the original size. The image feature extraction module establishes information channels of different scales through downsampling operations of different scales.
[0082] In this embodiment, a first sampling scale channel is obtained by a downsampling operation with a first sampling scale of "2x", and a second sampling scale channel is obtained by a downsampling operation with a second sampling scale of "4x", thereby obtaining an original scale channel, a first sampling scale channel, and a second sampling scale channel. In the original scale channel, the first sampling scale channel, and the second sampling scale channel, several planar convolution layers corresponding to the respective channels are synchronously used to extract features. Preferably, in this embodiment, four planar convolution layers are used in each scale channel. These planar convolution operations can capture key information such as edges and textures of the image.
[0083] Then, after passing through multiple plane convolution layers, each scale channel of the image feature extraction module is provided with a jump connection layer, through which the feature information of different layers in the same scale is fused to enhance the feature expression capability. In this embodiment, the feature information output by the second plane convolution layer in each scale channel is fused with the feature information output by the fourth plane convolution layer. The fused features are further extracted by using several plane convolution layers. In this embodiment, the number of plane convolution layers for further extracting feature information in each scale channel is set to 2.
[0084] After further extracting features through several planar convolution layers, the features in the second sampling scale channel are directly input into the prior information feature propagation layer under the second sampling scale channel corresponding to the image feature extraction module in the feature propagation module; and the features in the second sampling scale channel are also adjusted to the first sampling scale through a "2x" upsampling operation, so as to be fused with the features in the first sampling scale channel, and the features in the first sampling scale channel and the features in the second sampling scale channel are fused through a planar convolution layer, and after fusion, they are directly input into the prior information feature propagation layer under the first sampling scale channel corresponding to the image feature extraction module in the feature propagation module; and the fused features in the first sampling scale channel after fusion of the features in the second sampling scale channel are also adjusted to the original size through a "2x" upsampling operation, so as to be fused with the features in the original size channel through a planar convolution layer in the original size channel, and the features in the fused original size channel are also directly input into the prior information feature propagation layer in the feature propagation module.
[0085] In summary, in this embodiment, the image feature extraction module first generates input features of three scales through two downsampling convolutions, and then Figure 2 The channel information extraction shown includes multiple planar convolutions, fusion of information from different feature layers of the same scale through jump connections, and upsampling to fuse feature information of other scales, and finally outputs fused feature information of three scales.
[0086] In the feature propagation module based on prior information, the prior information feature propagation layer under the original size, the prior information feature propagation layer under the first sampling size, and the prior information feature propagation layer under the second sampling size are correspondingly set for the original size, the first sampling size, and the second sampling size channels in the image feature extraction module. The prior information feature propagation layer corresponding to each channel inputs the restoration model based on the prior information of the meta-lens point spread function for deblurring the feature information of each scale, such as Figure 3 The prior information of the point spread function image of the red, green and blue spectral segments is shown. The point spread function information adopts a downsampling method to match the feature size requirements of different scales. The model can adopt an explicit model-based deblurring algorithm, including a physical model-based algorithm such as a Wiener filter algorithm or a Richardson-Lucy algorithm. The parameters of the Wiener filter or the Richardson-Lucy algorithm are designed as optimizable weight information to participate in the training.
[0087] The expression of Wiener filtering is:
[0088]
[0089] Among them, K is the signal-to-noise ratio information of the image, H(u,v) is the frequency domain transform of the meta-lens point spread function, and G(u,v) is the frequency domain transform of the characteristic information output by the image feature extraction module. Finally, An inverse Fourier transform is performed as the output of the feature propagation module.
[0090] The expression of Richardson-Lucy algorithm is:
[0091]
[0092] Where: the initial f(x, y) represents the feature information of the true value image output by the feature extraction module, * represents the convolution operation, h(x, y) and g(x, y) represent the point spread function information and the feature information output by the blurred image feature extraction module respectively, and n represents the number of iterations of the image.
[0093] The feature propagation module deblurs the feature information corresponding to each scale and propagates it to the feature decoding module for feature fusion to output the reconstructed image. Similarly, in the feature decoding module, there are original size channels, first sampling size channels and second sampling size channels. After receiving feature information of different scales in each channel, an SE attention network layer is first used to process the features based on the channel attention mechanism. Then, in the original scale channel, several plane convolution layers are first used to process to obtain the original size feature image. The original size feature image is subjected to a "2x" downsampling operation and then fused with the features in the first sampling size channel. After being processed by several plane convolution layers, the first sampling size feature image is obtained. In this embodiment, the number of plane convolution layers for both operations is set to 2. Then, the first sampling size feature image is subjected to a "2x" downsampling operation, and it is combined with the feature image in the second sampling size channel through several plane convolution layers to obtain the second sampling size feature image. In this embodiment, the number of plane convolution layers in the second sampling size channel is set to 4. Next, the second sampling size feature image is subjected to a "2x" upsampling operation, and then fused with the first sampling size feature through a jump connection layer. The fused feature image is processed through several planar convolution layers and then adjusted to an image of the same size as the original size through a "2x" upsampling operation, and then fused with the original size feature image through a jump connection layer in the original size channel. After fusion, it is processed through several planar layers, and the reconstructed image is output after the processing is completed.
[0094] like Figure 4 The network structure diagram of the discriminator shown in the figure is based on the patchGAN architecture. The input reconstructed image and the true value image are discriminated by blocks. By dividing the image into blocks, N*N blocks are established, and each block is discriminated and classified respectively. Finally, the calculation results of all blocks are averaged to output the final discriminator result. The block method can effectively capture high-frequency information.
[0095] In step S3, this embodiment provides a loss function for training the aforementioned image restoration model. A loss function is constructed, and the loss function includes two parts: a loss function based on a conditional adversarial generation network and an image restoration loss function, wherein the image restoration loss function includes an L2 loss function and a loss function based on gradient information, and the image restoration loss function is used to reduce blur and improve clarity.
[0096]
[0097] Among them, G is the image output by the generator, and D is the image output by the discriminator;
[0098]
[0099] The gradient calculation kernel function is defined as k 1 =[1-1],
[0100] In step S4, the given image restoration model is trained according to the constructed data set and loss function, which mainly includes setting the hyperparameters of the image restoration model, and continuously adjusting the hyperparameters according to the training results, and finally confirming the weight of the image restoration model.
[0101] Specifically, in this embodiment, the aforementioned clear true value image and meta-lens imaging image data set are divided into a training set and a validation set, and the training set data is input into the image restoration network according to a batch_size of 4. The generator model inputs the restored image through the image feature extraction module, the feature propagation module and the feature decoding module, wherein three-scale input features are realized by 2x and 4x downsampling, and the Wiener filter parameter K in the feature propagation module is initialized to 0.1.
[0102] The loss function used in this implementation is
[0103]
[0104] The generator and the discriminator are optimized alternately, and the Adam solver optimization algorithm is used, where the momentum parameter β 1 =0.4,β 2 = 0.999. By evaluating the quality of the validation set images, the weight parameters of the final generator are determined.
[0105] In step S5, the above-mentioned image restoration model is compiled in the form of software, and then connected to the meta-lens imaging module to obtain the image of the meta-lens imaging module in real time for image restoration and output a clear picture.
[0106] This embodiment provides a meta-lens image restoration system based on the above restoration method. Figure 5 As shown, the system includes a meta-lens imaging module, a control computer and a display device, wherein the control computer carries the generator model of the trained image restoration model in Example 1 in the form of software. Figure 6 A schematic diagram comparing a metalens imaging image restored by a metalens image restoration system and a true image is shown. It can be seen that before restoration, the metalens imaging image is blurrier than the true image. After restoration, there is almost no difference between the metalens imaging image and the true image.
[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. An image restoration method, characterized in that: It includes the following steps: S1. Construct a data set, collect true value images and meta-lens imaging images to establish data pairs, and form a data set; S2. Construct an image restoration model including at least a generator and a discriminator, wherein the generator includes at least an image feature extraction module, a feature propagation module, and a feature decoding module, which outputs a reconstructed image after multi-scale feature fusion of the meta-lens imaging image; the discriminator uses the reconstructed image and the corresponding true value image as inputs of the discriminator, and guides the image restoration model to generate a higher quality restored image by performing block discrimination on the reconstructed image and the true value image; S3, constructing a loss function for the constructed image restoration model, which includes a loss function based on a conditional adversarial generation network and an image restoration loss function; S4, training the image restoration model based on the constructed data set and loss function; S5. Use the generator in the trained image restoration model to restore the metalens imaging image collected in real time by the metalens imaging module, and output the restored image.
2. The image restoration method according to claim 1, characterized in that: In the generator of the image restoration model, the image feature extraction module generates input features of several scales through several downsampling convolutions, extracts features through corresponding channel information, including multiple plane convolutions, and fuses information of different feature layers of the same scale through jump connections and upsampling to fuse feature information of other scales, and finally outputs the fused feature information of multiple scales; The feature information extracted by the image feature extraction module is input into the prior information feature propagation layer of the feature propagation module according to the corresponding scale channel. The feature information of each size is processed by the deblurring algorithm based on the prior information of the meta-lens point spread function, and then transmitted to the feature decoding module according to their respective channels. The feature decoding module uses a fully convolutional network model for feature inputs of different scales. By fusing feature information of different scales, it realizes the conversion of multi-scale characteristic information from feature space to image space and outputs a reconstructed image.
3. The image restoration method according to claim 2, characterized in that: In the feature extraction module, the image is first input into the image feature extraction module according to the original size. The image feature extraction module establishes information channels of different scales through downsampling operations of different scales. The first sampling scale channel is obtained by downsampling the first sampling scale, and the second sampling scale channel is obtained by downsampling the second sampling scale, thereby obtaining the original scale channel, the first sampling scale channel, and the second sampling scale channel; In the original scale channel, the first sampling scale channel and the second sampling scale channel, a plurality of planar convolution layers corresponding to the respective channels are synchronously used to extract features; After passing through multiple planar convolutional layers, each scale channel of the image feature extraction module is provided with a jump connection layer, through which the feature information of different layers in the same scale is fused; The fused features are then subjected to several planar convolutional layers to further extract feature information; After further extracting features through several planar convolutional layers, the features in the second sampling scale channel are directly input into the prior information feature propagation layer under the second sampling scale channel corresponding to the image feature extraction module in the feature propagation module; and the features in the second sampling scale channel are also adjusted to the first sampling scale through an upsampling operation; The features in the first sampling scale channel and the features in the second sampling scale channel are fused through a planar convolution layer, and after fusion, they are directly input into the prior information feature propagation layer under the first sampling scale channel corresponding to the image feature extraction module in the feature propagation module; and the fused features in the first sampling scale channel after the fusion of the features in the second sampling scale channel are also adjusted to the original size through an upsampling operation, and are fused with the features in the original size channel in the original size channel through a planar convolution layer, and the features in the fused original size channel are also directly input into the prior information feature propagation layer in the feature propagation module.
4. The image restoration method according to claim 3, characterized in that: In the feature propagation module, a priori information feature propagation layer under the original size, a priori information feature propagation layer under the first sampling size, and a priori information feature propagation layer under the second sampling size are correspondingly provided for the original size, the first sampling size, and the second sampling scale channels in the image feature extraction module; The prior information feature propagation layer corresponding to each channel inputs the feature information of each scale into the restoration model based on the prior information of the meta-lens point spread function for deblurring; The point spread function information is downsampled to match the feature size requirements of different scales. It adopts an explicit model-based deblurring algorithm, including the Wiener filter algorithm or the Richardson-Lucy algorithm. The parameters of the Wiener filter or the Richardson-Lucy algorithm are designed to be optimizable weight information for training.
5. The image restoration method according to claim 4, characterized in that: If the feature propagation module is implemented using Wiener filtering, its calculation expression is: Among them, K is the signal-to-noise ratio information of the image, H(u,v) is the frequency domain transform of the meta-lens point spread function, and G(u,v) is the frequency domain transform of the characteristic information output by the image feature extraction module. Finally, An inverse Fourier transform is performed as the output of the feature propagation module.
6. The image restoration method according to claim 4, characterized in that: If the feature propagation module is implemented using the Richardson-Lucy algorithm, its calculation expression is: Where: the initial f(x, y) represents the feature information of the true value image output by the feature extraction module, * represents the convolution operation, h(x, y) and g(x, y) represent the point spread function information and the feature information output by the blurred image feature extraction module respectively, and n represents the number of iterations of the image.
7. The image restoration method according to claim 4, characterized in that: In the feature decoding module, the feature decoding module fuses the features in each scale channel to output a reconstructed image, where: The feature decoding module also includes the original size channel, the first sampling size channel and the second sampling size channel. After each channel receives feature information of different scales, it first uses an SE attention network layer to perform channel attention feature processing; Then, in the original scale channel, several planar convolution layers are used to process to obtain the original size feature image, the original size feature image is downsampled once and then fused with the features in the first sampling size channel, and processed by several planar convolution layers to obtain the first sampling size feature image; Then, the first sampling size feature image is downsampled once, and it is combined with the feature image in the second sampling size channel through several planar convolution layers to obtain the second sampling size feature image; Next, the second sampling size feature image is subjected to an upsampling operation and then fused with the first sampling size feature through a skip connection layer. The fused feature image is processed through several planar convolution layers and then adjusted to an image of the same size as the original through an upsampling operation. Then, it is fused with the original size feature image through a jump connection layer in the original size channel, and then processed by several plane layers. After the processing is completed, the reconstructed image is output.
8. The image restoration method according to claim 1, characterized in that: In the discriminator of the image restoration model, based on the architecture of patchGAN, the input reconstructed image and the true value image are divided into blocks for discrimination. By dividing the image into blocks, N*N blocks are established, and each block is discriminated and classified separately. Finally, the calculation results of all blocks are averaged to output the final discriminator result.
9. The image restoration method according to claim 1, characterized in that: The loss function is expressed as: Among them, G is the image output by the generator, and D is the image output by the discriminator; The loss function consists of two parts: the loss function based on the conditional adversarial generation network And the image restoration loss function; Loss function based on conditional adversarial generation network It is expressed as: Image restoration loss function includes L2 loss function And the loss function based on gradient information in: The gradient calculation kernel function is defined as:
Citation Information
Patent Citations
Super-lens chromatic aberration recovery method based on deep learning
CN114331891A
High-quality imaging method based on integrated super-structure lens system
CN118295131A
Bi-skip-net-based image deblurring method
WO2020087607A1
Cited By
Hyperspectral imaging method of super-structure lens
CN119063846A
A hyperspectral imaging method using a meta-lens
CN119063846B
