Super-lens image restoration method based on fuzzy prior and semantic segmentation

Abstract

The invention discloses a super-lens image restoration method based on fuzzy prior and semantic segmentation. The method comprises the following steps: eliminating an image visual angle displacement error by adopting a feature point automatic registration algorithm; the method comprises the following steps: designing a parallel multi-scale convolution adapter on the basis of an encoder optimized by a pre-trained visual Transform model, and generating encoding features adaptive to the degradation characteristics of a super lens; splicing the degraded image and the semantic segmentation mask into a joint tensor along a channel dimension, and obtaining a prior feature based on a fuzzy prior extraction network; performing spatial feature enhancement on the semantic segmentation mask based on a semantic segmentation graph convolutional network to obtain graph features aligned with the coding features; splicing the coding features, the prior features and the image features along channel dimensions to generate fusion features, and finally outputting a preliminary recovery image; and optimizing the whole model by adopting a staged training strategy, and finally outputting a high-quality recovery graph. According to the method, the physical rationality and detail fidelity of the recovery quality are improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically a superlens image restoration method based on fuzzy prior and semantic segmentation. Background Technology

[0002] Superlenses, as an emerging metasurface optical element, enable ultrathin and lightweight imaging systems by modulating the wavefront through nanostructures, demonstrating great potential in fields such as medical endoscopy and micro-imaging. However, superlenses have inherent physical limitations in the imaging process: First, due to the dispersive properties of nanostructures, superlenses exhibit severe axial and lateral chromatic aberration under broadband light sources, causing a shift in the focusing position of light of different wavelengths, resulting in image blurring and color distortion; second, after light waves are modulated by superlenses, some high-frequency information is lost due to diffraction limiting, causing detail degradation. The essence of these problems is information loss during optical modulation, and how to recover high-quality content from degraded images becomes a key challenge.

[0003] In recent years, deep learning-based methods have been widely applied to image restoration tasks. For example, the MetaScope work proposed an optically driven neural network that compensates for chromatic aberration by modeling the point spread function of a superlens. However, this method relies on accurate optical simulation and lacks generalization ability in real-world scenarios. Furthermore, it does not fully exploit prior knowledge of the image, resulting in limited restoration effects on complex textures and edges. Another type of research focuses on utilizing prior knowledge, proposing a fuzzy prior extraction module based on generative adversarial networks. This module learns global structural priors through the transformation from edge maps to color images, effectively improving the robustness of dehazing tasks. However, this method faces modal differences when directly applied to superlens images: the fuzzy priors in dehazing tasks mainly target atmospheric scattering models, while the degradation of superlenses stems from chromatic aberration and diffraction, requiring adaptation to optical characteristics.

[0004] In summary, existing methods have three major limitations: 1. Insufficient optical modeling: The physical degradation mechanism of the superlens is not fully integrated into the neural network, resulting in a lack of physical consistency in the reconstruction results. 2. Limited prior knowledge: Existing prior extraction networks are mostly designed for specific tasks (such as dehazing) and do not consider the chromatic aberration characteristics of the superlens. 3. Inefficient use of spatial information: Convolutional networks struggle to model long-range dependencies, while graph convolutions can compensate for this deficiency, but have not yet been integrated with the superlens reconstruction task.

[0005] Therefore, a superlens image restoration method based on fuzzy prior and semantic segmentation is proposed. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention provides a superlens image restoration method based on fuzzy priors and semantic segmentation. It learns the degradation priors of the superlens through a GAN-style prior network, enhances spatial features by combining a segmentation-driven graph convolutional network, and achieves efficient feature extraction by referencing the encoder adaptation mechanism of the base model. Ultimately, it improves the physical plausibility and detail fidelity of the restored image.

[0007] The technical solution to achieve the above objectives is: A hyperlens image restoration method based on fuzzy prior and semantic segmentation includes: Step S1: Obtain the degraded image and the ternary dataset using the superlens, and use an automatic feature point registration algorithm to eliminate image viewpoint displacement error. The ternary dataset consists of the degraded image... Reference image and semantic segmentation mask composition; Step S2: Based on the encoder optimized by the pre-trained visual Transformer model, design a parallel multi-scale convolution adapter to generate encoded features that adapt to the degradation characteristics of the superlens, which will serve as the subsequent feature carrier. Step S3: Concatenate the degraded image and the semantic segmentation mask along the channel dimension into a joint tensor, and extract prior features based on the fuzzy prior extraction network; Step S4: The semantic segmentation mask is spatially enhanced based on the semantic segmentation graph convolutional network to obtain graph features aligned with the encoded features; Step S5: The encoded features, prior features, and graph features are concatenated along the channel dimension to generate fused features. The decoder then sequentially processes these features through transposed convolution, layer normalization, and ReLU activation to gradually restore the feature map resolution and finally output the preliminary restored image. Step S6: The overall model is optimized using a phased training strategy, and the trained overall model is used to accurately restore the superlens degraded image, finally outputting a high-quality restored image.

[0008] Preferably, in step S1, imaging is achieved by controlling the superlens in front of the light wave using a nanostructure, and axial / lateral chromatic aberration is generated under a broadband light source based on the dispersive properties of the superlens nanostructure to obtain a superlens-degraded image containing image blur, color distortion, and detail degradation. ; Synchronously acquire corresponding high-quality reference images Manually labeled semantic segmentation mask Construct a ternary dataset ; in, , , Image height, This represents the image width.

[0009] Preferably, in step S2, the encoder backbone extracts basic image features through a pre-trained visual Transformer model, and a parallel multi-scale convolutional adapter is connected to the pre-trained visual Transformer model in parallel to gradually refine intermediate features and generate encoded features adapted to the degradation characteristics of the superlens. This provides a basic feature carrier for subsequent fuzzy prior fusion and graph convolution enhancement.

[0010] Preferably, in step S3, the fuzzy prior extraction network includes an encoder and a generator. With discriminator ,in, The encoder passes the joint tensor through three convolutional layers with a stride of 2. Size downsampling to Simultaneously, the number of channels is expanded to 256 dimensions to extract shallow degradation features. ; generator Employing an encoder-decoder structure to convert shallow degradation features Gradually restore the image to its original resolution and output the prior features. ; Discriminator Adversarial training drives generator optimization, combined with content loss: ; And the losses inflicted by the fight: ; Set the total loss: ; Supervised network training, the final output dimension is Prior features ; In the above formula, To balance the hyperparameters.

[0011] Preferably, in step S4, the semantic segmentation graph convolutional network includes two parts: graph construction and graph attention convolution, wherein... Graph construction: The image is divided into superpixel regions, with each superpixel serving as a graph node. The channel and spatial attributes of the image patch corresponding to each node are extracted to form a node feature vector. The edge weights between nodes are calculated based on the space-channel joint matrix, and the graph structure is constructed. Graph Attention Convolution: This method uses a graph attention network to aggregate node information in a graph structure, with an output dimension of [dimensionality missing]. graph features ,in, For the number of nodes, Upsampled to the original image size using bilinear interpolation, and then compared with the encoded features. Alignment enhances spatial context features.

[0012] Preferably, in step S4, the edge weights between nodes are calculated based on the space-channel joint matrix to capture the correlation between spatial layout and channel features; The formula for calculating the edge weights between nodes is as follows: ; In the formula, For nodes and Channel feature similarity, , They are nodes and Spatial location coordinates, denoted as Euclidean distance between the node spatial coordinates. Distance threshold This is an indicator function.

[0013] Preferably, in step S5, the encoded features are... Prior features Graph features By stitching along the channel dimension, a fused feature is generated. and will integrate features The input consists of four upsampled concatenated decoders. Each upsampled image is sequentially processed by transposed convolution, layer normalization, and ReLU activation to gradually restore the feature map resolution. The decoder is trained using a joint loss method consisting of multi-scale structural similarity loss and L1 loss, and the final output is a preliminary restored image.

[0014] Preferably, in step S6, a phased training strategy is used to optimize the overall model: Phase 1: Using datasets related to the segmentation task, the fuzzy prior extraction network and the semantic segmentation graph convolutional network are pre-trained independently. The second stage involves treating the encoder, fuzzy prior extraction network, semantic segmentation map convolutional network, and decoder as a whole, and performing end-to-end joint fine-tuning using the Adam optimizer for 100 training epochs. During training, data augmentation techniques such as image rotation, horizontal / vertical flipping, and color dithering are introduced. The final output is a high-quality restored image. .

[0015] Compared with the prior art, the beneficial effects of the present invention are: The fuzzy prior extraction network provided by this invention constructs a degradation modeling model based on the concept of conditional GAN ​​and semantic mask guidance. It inputs the superlens degradation image and semantic segmentation mask together, and accurately learns the chromatic aberration prior distribution caused by dispersion through adversarial training and content loss constraints. This not only improves the physical rationality of the prior features, but also avoids the prior from deviating from the image structure by using semantic information, and provides a targeted fuzz correction basis for subsequent restoration. The semantic segmentation graph convolutional network provided by this invention introduces superpixel graph construction and graph attention convolution strategies. Through superpixel node partitioning and spatial-channel joint weight calculation, it efficiently captures long-range dependencies between pixels. Compared with traditional convolution, it is more suitable for spatial correlation modeling of degraded regions in superlens images. It achieves enhanced spatial semantic consistency in complex degradation scenarios and outputs enhanced feature maps with complete structure and clear details. The multimodal feature fusion decoder provided by this invention integrates encoder basic features, fuzzy prior features, and graph convolution enhancement features. It employs multi-scale structural similarity and L1 joint loss for supervised training, which enhances the color consistency and edge integrity of the restored results while preserving the fidelity of image details. This design avoids overfitting the model to local noise and improves the model's generalization ability through a staged training strategy (pre-training sub-modules first and then fine-tuning end-to-end), effectively ensuring the stability of the restored image under different degrees of superlens degradation. In summary, the overall network of this invention, with fuzzy prior modeling and semantic graph convolutional enhancement as its core, and based on a pre-trained ViT encoder and decoder architecture, achieves integrated processing of "degradation analysis - feature optimization - accurate restoration" for superlens-degraded images, taking into account restoration accuracy, physical rationality, and multi-scene adaptability. By modeling the chromatic aberration blurring law, long-range spatial dependence, and multimodal feature complementarity in the degradation modeling, feature enhancement, and image reconstruction stages respectively, high-fidelity restoration and physical consistency optimization of superlens-degraded images are achieved, effectively solving the problems of incomplete restoration of superlens-specific chromatic aberration blurring and insufficient long-range feature correlation modeling in traditional methods. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a superlens image restoration method based on fuzzy prior and semantic segmentation according to the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, a hyperlens image restoration method based on fuzzy prior and semantic segmentation includes: Step S1: Obtain the degraded image and the ternary dataset using the superlens, and use an automatic feature point registration algorithm to eliminate image viewpoint displacement error. The ternary dataset consists of the degraded image... Reference image and semantic segmentation mask composition.

[0019] In this embodiment, imaging is achieved by manipulating a superlens in front of the light wave using a nanostructure. Based on the dispersive properties of the superlens nanostructure, axial / lateral chromatic aberration is generated under a broadband light source to obtain a degraded superlens image containing image blur, color distortion, and detail degradation. ; Synchronously acquire corresponding high-quality reference images Manually labeled semantic segmentation mask Construct a ternary dataset ; in, , , Image height, This represents the image width.

[0020] In this embodiment, the dataset covers a wide range of scenarios and is representative, with particular attention to practical application needs. In the data preprocessing stage, an automatic registration algorithm based on feature points is used to finely align image pairs and eliminate displacement errors caused by differences in viewing angles. Subsequently, radiometric calibration and response normalization are performed to ensure brightness consistency between images. The dataset contains detailed metadata annotations, such as optical parameters, environmental conditions, and equipment configuration information at the time of image capture, providing complete data support for the study of superlens imaging characteristics. This rich and rigorous dataset lays a solid foundation for the development of advanced superlens image restoration algorithms and is expected to promote the technological development of the field of computational imaging.

[0021] Step S2: Based on the encoder optimized by the pre-trained visual Transformer model, design a parallel multi-scale convolution adapter to generate encoded features adapted to the degradation characteristics of the superlens, which will serve as the subsequent feature carrier.

[0022] In this embodiment, the encoder backbone extracts basic image features through a pre-trained visual Transformer model, and a parallel multi-scale convolutional adapter is connected to the pre-trained visual Transformer model in parallel to gradually refine intermediate features and generate encoded features adapted to the degradation characteristics of the superlens. This provides a basic feature carrier for subsequent fuzzy prior fusion and graph convolution enhancement.

[0023] Step S3: Concatenate the degraded image and the semantic segmentation mask along the channel dimension into a joint tensor, and obtain the prior features based on the fuzzy prior extraction network. This network, through an adversarial training mechanism, can effectively model the degradation patterns unique to superlenses, providing important prior knowledge guidance for the subsequent image restoration process.

[0024] In this embodiment, the fuzzy prior extraction network includes an encoder and a generator. With discriminator ,in, The encoder passes the joint tensor through three convolutional layers with a stride of 2. Size downsampling to Simultaneously, the number of channels is expanded to 256 dimensions to extract shallow degradation features. ; Specifically, multi-scale convolutional layers are used to extract degraded images. shallow features ,in It consists of 3 convolutional layers with a stride of 2, and the output feature dimension is... That is, by progressively downsampling the input degraded image From the original resolution Convert to The feature map is expanded to 256 dimensions to obtain a shallow feature representation. This multi-scale design enables the network to capture degenerate features from local texture to global structure. generator Employing an encoder-decoder structure to convert shallow degradation features Gradually restore the image to its original resolution and output the prior features. ; Specifically, the input is a degraded image. With semantic segmentation mask The generator is the joint tensor formed by splicing along the channel dimension. Typically, an encoder-decoder structure is used, which gradually restores the low-resolution feature map to the same high spatial resolution as the original reference image through a series of stacked modules containing upsampling (such as transposed convolution or interpolation operations) and standard convolutional layers, and finally outputs the predicted prior features or reconstructed image. Discriminator As an adversarial network in adversarial training, it is used to determine whether the input sample comes from the generator. The synthesized result is still a real reference image. Through this authenticity discrimination mechanism, the discriminator... Continuously drive the generator Optimize its parameters so that the generated results continuously approach the distribution of real images in terms of visual quality and structural realism, thereby ensuring that the prior features have stronger discriminative and structural preservation capabilities. Discriminator Adversarial training drives generator optimization, combined with content loss: ; And the losses inflicted by the fight: ; Set the total loss: ; Supervised network training, the final output dimension is Prior features ; In the above formula, To balance the hyperparameters.

[0025] Step S4: The semantic segmentation mask is spatially enhanced based on the semantic segmentation graph convolutional network to obtain graph features aligned with the encoded features.

[0026] In this embodiment, the semantic segmentation graph convolutional network includes two parts: graph construction and graph attention convolution. Graph construction: The image is divided into superpixel regions, with each superpixel serving as a graph node. The channel and spatial attributes of the image patch corresponding to each node are extracted to form a node feature vector. The edge weights between nodes are calculated based on the space-channel joint matrix, and the graph structure is constructed. Graph Attention Convolution: This method uses a graph attention network to aggregate node information in a graph structure, with an output dimension of [dimensionality missing]. graph features ,in, For the number of nodes, Upsampled to the original image size using bilinear interpolation, and then compared with the encoded features. Alignment enhances spatial context features.

[0027] In this embodiment, the edge weights between nodes are calculated based on the space-channel joint matrix to capture the correlation between spatial layout and channel features; The formula for calculating the edge weights between nodes is as follows: ; In the formula, For nodes and Channel feature similarity, , They are nodes and Spatial location coordinates, denoted as Euclidean distance between the node spatial coordinates. Distance threshold This is an indicator function.

[0028] Step S5: The encoded features, prior features, and graph features are concatenated along the channel dimension to generate fused features. These features are then processed sequentially by the decoder through transposed convolution, layer normalization, and ReLU activation to gradually restore the feature map resolution, and finally output the preliminary restored image.

[0029] In the embodiment, the encoded features Prior features Graph features By stitching along the channel dimension, a fused feature is generated. and will integrate features The input consists of four upsampled concatenated decoders. Each upsampled image is sequentially processed by transposed convolution, layer normalization, and ReLU activation to gradually restore the feature map resolution. The decoder is trained using a joint loss method consisting of multi-scale structural similarity loss and L1 loss, and the final output is a preliminary restored image.

[0030] Step S6: The overall model is optimized using a phased training strategy, and the trained overall model is used to accurately restore the superlens degraded image, finally outputting a high-quality restored image.

[0031] In this embodiment, a phased training strategy is used to optimize the overall model: Phase 1: Using datasets related to the segmentation task, the fuzzy prior extraction network and the semantic segmentation graph convolutional network are pre-trained independently. The second stage involves treating the encoder, fuzzy prior extraction network, semantic segmentation graph convolutional network, and decoder as a whole, and using the Adam optimizer for end-to-end joint fine-tuning to optimize overall performance. The initial learning rate is set to 0.0001, the batch size is set to 16 samples per training batch, and the training cycle is 100 epochs. To enhance the model's generalization ability and robustness, data augmentation techniques such as image rotation, horizontal / vertical flipping, and color jitter are introduced during training, effectively improving the model's performance on unknown data. The final output is a high-quality restored image. .

[0032] This method achieves end-to-end processing of superlens images from "degradation modeling to feature enhancement to precise restoration", significantly improving the detail fidelity, color consistency and physical rationality of the restored images, and is suitable for medical endoscopic diagnosis and industrial inspection scenarios.

[0033] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for superlens image restoration based on fuzzy prior and semantic segmentation, characterized in that, include: Step S1: Obtain the degraded image and the ternary dataset using the superlens, and use an automatic feature point registration algorithm to eliminate image viewpoint displacement error. The ternary dataset consists of the degraded image... Reference image and semantic segmentation mask composition; Step S2: Based on the encoder optimized by the pre-trained visual Transformer model, design a parallel multi-scale convolution adapter to generate encoded features that adapt to the degradation characteristics of the superlens, which will serve as the subsequent feature carrier. Step S3: Concatenate the degraded image and the semantic segmentation mask along the channel dimension into a joint tensor, and extract prior features based on the fuzzy prior extraction network; Step S4: The semantic segmentation mask is spatially enhanced based on the semantic segmentation graph convolutional network to obtain graph features aligned with the encoded features; Step S5: The encoded features, prior features, and graph features are concatenated along the channel dimension to generate fused features. The decoder then sequentially processes these features through transposed convolution, layer normalization, and ReLU activation to gradually restore the feature map resolution and finally output the preliminary restored image. Step S6: The overall model is optimized using a phased training strategy, and the trained overall model is used to accurately restore the superlens degraded image, finally outputting a high-quality restored image.

2. The superlens image restoration method based on fuzzy prior and semantic segmentation according to claim 1, characterized in that, In step S1, imaging is achieved by controlling the superlens in front of the light wave using a nanostructure. Based on the dispersive properties of the superlens nanostructure, axial / lateral chromatic aberration is generated under a broadband light source to obtain a superlens-degraded image containing image blur, color distortion, and detail degradation. ; Synchronously acquire corresponding high-quality reference images Manually labeled semantic segmentation mask Construct a ternary dataset ; in, , , Image height, This represents the image width.

3. The method for superlens image restoration based on fuzzy prior and semantic segmentation according to claim 2, characterized in that, In step S2, the encoder backbone extracts basic image features through a pre-trained visual Transformer model, and a parallel multi-scale convolutional adapter is connected to the pre-trained visual Transformer model in parallel to gradually refine intermediate features and generate encoded features adapted to the degradation characteristics of the superlens. This provides a basic feature carrier for subsequent fuzzy prior fusion and graph convolution enhancement.

4. The superlens image restoration method based on fuzzy prior and semantic segmentation according to claim 3, characterized in that, In step S3, the fuzzy prior extraction network includes an encoder and a generator. With discriminator ,in, The encoder passes the joint tensor through three convolutional layers with a stride of 2. Size downsampling to Simultaneously, the number of channels is expanded to 256 dimensions to extract shallow degradation features. ; generator Employing an encoder-decoder structure to convert shallow degradation features Gradually restore the image to its original resolution and output the prior features. ; Discriminator Adversarial training drives generator optimization, combined with content loss: ； And the losses inflicted by the fight: ； Set the total loss: ； Supervised network training, the final output dimension is Prior features ; In the above formula, To balance the hyperparameters.

5. The superlens image restoration method based on fuzzy prior and semantic segmentation according to claim 4, characterized in that, In step S4, the semantic segmentation graph convolutional network includes two parts: graph construction and graph attention convolution. Graph construction: The image is divided into superpixel regions, with each superpixel serving as a graph node. The channel and spatial attributes of the image patch corresponding to each node are extracted to form a node feature vector. The edge weights between nodes are calculated based on the space-channel joint matrix, and the graph structure is constructed. Graph Attention Convolution: This method uses a graph attention network to aggregate node information in a graph structure, with an output dimension of [dimensionality missing]. graph features ,in, For the number of nodes, Upsampled to the original image size using bilinear interpolation, and then compared with the encoded features. Alignment enhances spatial context features.

6. The superlens image restoration method based on fuzzy prior and semantic segmentation according to claim 5, characterized in that, In step S4, the edge weights between nodes are calculated based on the space-channel joint matrix to capture the correlation between spatial layout and channel features; The formula for calculating the edge weights between nodes is as follows: ； In the formula, For nodes and Channel feature similarity, , They are nodes and Spatial location coordinates, denoted as Euclidean distance between the node spatial coordinates. Distance threshold This is an indicator function.

7. The method for superlens image restoration based on fuzzy prior and semantic segmentation according to claim 5, characterized in that, In step S5, the encoded features are... Prior features Graph features By stitching along the channel dimension, a fused feature is generated. and will integrate features The input consists of four upsampled layers connected in series to form a decoder. Each upsampled layer is sequentially processed by transposed convolution, layer normalization, and ReLU activation to gradually restore the feature map resolution. The decoder is trained by using a joint loss consisting of multi-scale structural similarity loss and L1 loss, and the final output is a preliminary restored image.

8. The method for superlens image restoration based on fuzzy prior and semantic segmentation according to claim 7, characterized in that, In step S6, a phased training strategy is used to optimize the overall model: Phase 1: Using datasets related to the segmentation task, the fuzzy prior extraction network and the semantic segmentation graph convolutional network are pre-trained independently. The second stage: The encoder, fuzzy prior extraction network, semantic segmentation graph convolutional network and decoder are treated as a whole, and the Adam optimizer is used for end-to-end joint fine-tuning. The training cycle is 100 rounds. Data augmentation techniques such as image rotation, horizontal / vertical flipping, and color dithering are introduced during training; the final output is a high-quality reconstructed image. .

Images