Super-lens endoscope image restoration and segmentation method and electronic equipment

Abstract

The invention discloses a super-lens endoscope image restoration and segmentation method and electronic equipment, and belongs to the field of image processing. The method comprises the following steps: firstly, acquiring a super-lens distortion image with chromatic aberration and extracting initial features; thirdly, correcting light intensity unevenness and enhancing feature semantics through a cross-modal interaction module by utilizing scene and detail text priori generated by a large model; then, through a task-guided multi-expert selection and characteristic decomposition integration strategy, adaptively repairing chromatic aberration; meanwhile, a real undistorted image is introduced to generate guide features through a multi-scale contour extraction module, so that the authenticity of feature learning is ensured; and finally, fusing the optimized features, synchronously inputting a segmentation decoder and a recovery decoder, and outputting a segmentation mask and a high-fidelity recovery image. According to the method and the device, high-quality recovery and accurate segmentation of the super-lens endoscope image are realized, and the practicability is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and electronic device for image restoration and segmentation of a superlens endoscope. Background Technology

[0002] In recent years, endoscopes based on metasurface lenses have shown great potential in the field of medical microscopy due to their miniaturization, high integration, and flexible wavefront modulation capabilities. Unlike traditional complex lens groups, metasurface lenses achieve modulation of the phase and amplitude of incident light by arranging nanoscale metaatomic arrays on a planar substrate, thereby achieving focused imaging at the millimeter scale, making them very suitable for scenarios such as minimally invasive surgery.

[0003] However, superlenses have an inherent drawback: their phase modulation characteristics are highly dependent on the incident light wavelength, resulting in a focal length-wavelength dependence. Without specialized achromatic design, light of different wavelengths cannot be perfectly focused on the same image plane, leading to chromatic aberration problems such as edge color shift and detail blurring during imaging, severely affecting image quality and subsequent diagnostic analysis.

[0004] In existing technologies, some studies have attempted to use deep learning models (such as convolutional neural networks) to correct aberrations in superlenses, or to optimize superatomic structures to match specific imaging requirements. However, these methods have significant limitations: 1. The model has poor adaptability to dynamic scenarios such as organ peristalsis in clinical practice; 2. Structural optimization cannot simultaneously achieve a large field of view, high resolution, and wide spectral response; 3. In multimodal imaging research, images from different modalities (such as optical and ultrasound) are often simply superimposed, lacking in-depth cross-modal feature interaction and fusion, which can easily lead to information redundancy or loss of key features. 4. The lack of effective use of clinical scenario information (such as lesion characteristics and operating habits) leads to insufficient system robustness and clinical adaptability.

[0005] Therefore, there is an urgent need for a collaborative processing method that can effectively correct chromatic aberration of the superlens and simultaneously achieve high-quality image restoration and accurate semantic segmentation, so as to improve the imaging quality and auxiliary diagnostic capabilities of the superlens endoscope in real clinical environments. Summary of the Invention

[0006] To address the aforementioned issues, this application provides a method and electronic device for image restoration and segmentation of a superlens endoscope, which solves the image quality degradation problem caused by chromatic aberration in superlenses and achieves end-to-end collaborative optimization of image restoration and semantic segmentation, thereby improving image clarity, detail fidelity, and lesion area segmentation accuracy in complex clinical scenarios.

[0007] The first technical solution adopted in this application is: providing a method for image restoration and segmentation of a superlens endoscope, including the following steps: S1. Obtain the distorted image generated by the superlens; S2. Input the distorted image into the encoder to extract basic features and generate an initial feature map; S3. Input the initial feature map into the optical prior information fusion and adjustment module, perform cross-modal interaction based on the text prior generated by the large language-visual model, and output semantically enhanced features; the cross-modal interaction is achieved through two rounds of multimodal iterative interactive attention: the first round uses visual features as the query and text features as the value to generate semantically weighted text features, and the second round uses the semantically weighted text features as the query and visual features as the value to generate semantically enhanced features. S4. Input the semantic enhancement features into the multi-level optical information-guided adaptive processing module. After concatenating the task-specific prompts with local features, calculate the similarity with the prompts of the learnable processing unit and select the processing unit to process the pixel. Then, perform high- and low-frequency decomposition on the semantic enhancement features and perform differential processing by the processing unit to output the preliminary repair features. The high- and low-frequency decomposition decomposes the features into low-frequency features and high-frequency features through a dynamic learnable low-pass filter. S5. Input the real, distortion-free image into the contour-guided deep embedding module, extract contextual features and generate prior feature maps through multi-scale deep convolution to obtain the guiding features; S6. The preliminary repair features and the guiding features are fused to obtain the distortion-free features, and these features are simultaneously input into the segmentation decoder and the recovery decoder; the segmentation decoder outputs the segmentation mask, and the recovery decoder outputs the recovered image.

[0008] In an optional embodiment, in step S1, the phase distribution of the superlens is determined based on the radial position and the incident light wavelength.

[0009] In an optional embodiment, in step S3, the optical prior information fusion adjustment module includes: The environment prior perception submodule is used to output semantically enhanced scene features based on textual priors describing the overall scene and background of the image. The detail prior perception submodule is used to output semantically enhanced detail features based on textual priors describing image details and textures.

[0010] In an optional embodiment, in step S4, the multi-layered optical information-guided adaptive processing module includes: The task prompt-guided multi-processing unit selection and integration submodule is used to generate task-specific prompts and select a preset number of processing units to process pixels based on the task-specific prompts and local features, and combine global attention mechanism with the processing unit using equalization loss for constraint. The feature decomposition and multi-processing unit integration submodule is used to decompose the input features into low-frequency features and high-frequency features, and select a preset number of processing units for processing and integration.

[0011] In an optional embodiment, the processing unit uses the balance loss as the coefficient of variation squared loss.

[0012] In an optional embodiment, in step S5, the contour-guided depth embedding module includes: The multi-scale deep convolution submodule is used to extract and aggregate surrounding context features of multiple scales through deep convolutions of different scales; The dual-path prior knowledge extraction submodule consists of two paths: the first path generates a weighted prior feature map and fuses it with the original image features, and the second path extracts a prior feature map for cross-level attention guidance.

[0013] In an optional embodiment, the kernel size used in the multi-scale depth convolution submodule includes at least three different sizes.

[0014] In an optional embodiment, in step S6, the segmentation decoder adopts a structure that includes downsampling and upsampling, and restores the feature map resolution through a transposed convolutional layer, while combining the features of the encoder's intermediate layer to output a segmentation mask.

[0015] In an optional embodiment, in step S6, the restoration decoder processes the distortion-reduced features through a structure including convolutional layers, residual connections, and attention mechanisms, and outputs the restored image.

[0016] The second technical solution adopted in this application is: providing an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the superlens endoscope image restoration and segmentation method as described in any of the preceding claims.

[0017] Due to the adoption of the above technical solution, this application has at least one of the following beneficial effects compared with the prior art:

[0018] 1. By deeply integrating the text priors generated by the large model with optical imaging features, semantic information is injected from both scene and detail dimensions, improving the semantic consistency and detail recognition of the features, and providing richer feature representations for subsequent processing.

[0019] 2. By introducing task prompts and a multi-processing unit selection mechanism, it is possible to perform adaptive and differentiated repair processing for different degradation types and regions in the image, which significantly improves the correction accuracy and generalization ability of complex degradation problems such as chromatic aberration.

[0020] 3. By using a contour-guided deep embedding module with real, distortion-free images as priors, the network is guided to learn the essential features of the target domain, effectively preventing the model from overfitting to the distortion pattern and ensuring the structural authenticity and naturalness of the restored image.

[0021] 4. An end-to-end framework for parallel restoration and segmentation tasks was constructed, sharing optimized distortion removal features, which enables the image restoration process to directly support high-precision segmentation, realizing correction as analysis, and improving the overall system's accuracy in identifying lesion areas and imaging quality in clinical scenarios. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 A schematic flowchart of a method for image restoration and segmentation using a superlens endoscope provided in an embodiment of this application; Figure 2 A schematic flowchart of a method for image restoration and segmentation using a superlens endoscope provided in another embodiment of this application; Figure 3 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0024] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0025] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0026] Existing superlens endoscopic imaging techniques generally suffer from severe chromatic aberration (such as edge color shift and detail blur) caused by wavelength-dependent focal length due to the lack of achromatic optimization. Existing image post-processing methods typically rely on only single visual modality information, lacking effective integration of optical physical priors and clinical semantic context. Furthermore, they do not introduce structural guidance from real, distortion-free images during the restoration process, resulting in low image restoration quality and weak generalization ability in dynamic clinical scenarios, and difficulty in simultaneously achieving high-precision semantic segmentation. In addition, traditional models use a uniform processing strategy for high- and low-frequency distortion features, failing to specifically correct different types of degradation, further limiting performance.

[0027] In view of this, this application improves the semantic consistency and detail recognition of features by deeply fusing the text prior generated by the large model with optical imaging features and injecting semantic information from both scene and detail dimensions; for example Figure 1 , 2 As shown, Figure 1 This is a schematic flowchart of a method for image restoration and segmentation using a superlens endoscope according to an embodiment of this application. Figure 2 This is a flowchart illustrating a method for image restoration and segmentation using a superlens endoscope according to another embodiment of this application. The method includes the following steps:

[0028] S1. Obtain the distorted image generated by the superlens, that is, obtain the distorted image of the superlens endoscope imaging. This image is generated by the superlens composed of a nanoscale superatom array. The superlens achieves light focusing by controlling the incident light wavefront through periodically or non-periodically arranged superatoms. The phase compensation amount of the superatoms is determined based on the phase distribution formula. Due to the lack of achromatic optimization, the focal length of the superlens is wavelength dependent, resulting in chromatic aberration problems such as edge color shift and blurred details in the generated image.

[0029] S2. Input the distorted image into the encoder for basic feature extraction to generate an initial feature map; input the distorted image obtained in S1 into the encoder with superlens optimization design, and perform basic feature extraction on the distorted image through the encoder's initial layer to generate an initial feature map X, which serves as the feature carrier for subsequent modules.

[0030] S3. Input the initial feature map into the optical prior information fusion and adjustment module. Based on the text prior generated by the large language-visual model, cross-modal interaction is performed to output semantically enhanced features. Cross-modal interaction is achieved through two rounds of multimodal iterative interactive attention: the first round uses visual features as the query and text features as the value to generate semantically weighted text features, and the second round uses semantically weighted text features as the query and visual features as the value to generate semantically enhanced features. Input the initial feature map X into the optical prior information fusion and adjustment module. This module includes an environment prior perception submodule and a detail prior perception submodule. The environment prior perception submodule generates scene-dimensional text priors of the overall scene and background of the image based on LLaVA. The text priors are encoded using a parameter-frozen CLIP text encoder to obtain text features. After being mapped to the same dimension as the visual scene features through a fully connected layer, semantically weighted text features and semantically enhanced scene features are generated sequentially through two rounds of multimodal iterative interactive attention.

[0031] The detail prior perception submodule generates detail dimension text priors for the "details and textures" of the image based on LLaVA. It adopts the same encoding, dimension matching and multimodal interaction process as the environment prior perception submodule, and outputs semantically enhanced detail features to achieve adaptive correction of uneven light intensity attenuation in the image.

[0032] S4. The semantic enhancement features are input into the multi-level optical information-guided adaptive processing module. After concatenating task-specific prompts and local features, the similarity with the prompts from the learnable processing unit is calculated, and the processing unit is selected to process the pixels. The semantic enhancement features are then decomposed into high and low frequencies and processed by the processing unit for differentiation, outputting preliminary repair features. The high and low frequency decomposition uses a dynamic learnable low-pass filter to decompose the features into low-frequency and high-frequency features. The semantic enhancement features output from S3 are input into the multi-level optical information-guided expert selection fine-tuning module. This module includes a task prompt-guided multi-expert selection and integration submodule and a feature decomposition and multi-expert integration submodule.

[0033] The task-prompt-guided multi-expert selection and integration submodule performs convolution and global average pooling on the input features, generates a degradation information vector through Softmax, and merges it with the preset task-related basic prompts to generate task-specific prompts. After concatenating the task prompts and local features, it calculates the similarity with learnable expert prompts, selects Top-K experts to process pixels, combines Transformer layers to capture global correlations, and uses the coefficient of variation squared loss to constrain fair use of experts.

[0034] The feature decomposition and multi-expert integration submodule decomposes the output features of the above submodules into low-frequency features (corresponding to the global structure) and high-frequency features (corresponding to detail edges) through a dynamically learnable low-pass filter. After the two types of features are processed by LayerNorm, depthwise convolution and global average pooling respectively, the expert importance weights are predicted and Top-K experts are selected for processing. The features are enhanced by DConv and point convolution and the results are integrated. The original image is then superimposed on the decoder after the image is reconstructed and repaired to obtain the preliminary repair features.

[0035] S5. Inputting a real, lossless image into a contour-guided deep embedding module, extracting contextual features and generating prior feature maps through multi-scale deep convolution, and obtaining guiding features; inputting a real, lossless image into a contour-guided deep embedding module, which includes a multi-scale deep convolution submodule and a dual-path prior knowledge extraction submodule.

[0036] The multi-scale deep convolution submodule extracts multi-scale surrounding context features through three scales of deep convolution modules (n=3, n=5, and n=7, including n×n deep convolution, 90° rotation, element-wise multiplication, and element-wise addition), and aggregates them through channel concatenation.

[0037] In the dual-path prior knowledge extraction submodule, one path aggregates features and generates a weighted prior feature map through point convolution, square root, and sigmoid activation. This weighted prior feature map is then fused with the original image features and input into the encoder. The other path aggregates features and generates a prior feature map for cross-level attention guidance through the prior knowledge extraction submodule with the same structure. This guides the encoder to learn features that fit the image restoration task.

[0038] S6. Fuse the preliminary repair features and guiding features to obtain the dedistorted features, and simultaneously input them into the segmentation decoder and the restoration decoder; the segmentation decoder outputs a segmentation mask, and the restoration decoder outputs the restored image. Fuse the preliminary repair features output from S4 with the guiding features generated in S5 to obtain the dedistorted features, and simultaneously input them into the segmentation decoder and the restoration decoder. The segmentation decoder adopts a downsampling + upsampling structure. It gradually restores the feature map resolution through transposed convolutional layers, and combines the intermediate layer features of the encoder to supplement details. It outputs a segmentation mask with the same size as the input image, and is optimized under the supervision of the segmentation loss (compared with the real segmentation label Y). The restoration decoder eliminates subtle distortions through multi-layer convolution, residual blocks, and attention mechanisms, restoring image texture, color, and details, and outputting a restored realistic image. It is optimized under the supervision of restoration loss (compared to a real, distortion-free image); ultimately achieving the restoration and semantic segmentation of the superlens endoscope image.

[0039] In step S1, the phase distribution of the superlens is determined based on the radial position and the incident light wavelength; the phase distribution formula is... This is the core of the superlens design. Among them, This represents the radial distance from the superatomic point to the center of the lens. Indicates the wavelength of the incident light. To design the focal length, this formula shows that superatoms at different positions need to provide corresponding phase compensation to converge the incident light to the focal plane. Because the optical response of superatoms is wavelength-sensitive, the focal length of a superlens without achromatic optimization is... Follow Change, that is This causes light of different wavelengths to fail to converge on the same image plane, introducing chromatic aberration into the image, manifesting as color blurring and detail obscuration. This means that for the same spatial point (a fixed r), red, green, and blue light require different amounts of ideal phase compensation, but the superatomic array can only provide a fixed phase, thus making it impossible to achieve perfect focusing on all wavelengths simultaneously.

[0040] In step S3, the optical prior information fusion and adjustment module includes: The environment prior perception submodule is used to output semantically enhanced scene features based on textual priors describing the overall scene and background of the image. The detail prior perception submodule is used to output semantically enhanced detail features based on textual priors describing image details and textures.

[0041] The environment prior perception submodule generates text priors based on LLaVA's description of the "overall scene and background" of the image. Text features are encoded using the CLIP text encoder with parameter freezing. The features are then mapped to the same dimension as the visual scene features via a fully connected layer. Deep fusion is then achieved through two rounds of multimodal iterative interactive attention (MIIA): the first round uses visual features as the query and text features as the value to generate semantically weighted text features. The second round was Using the query and visual features as values, output semantically enhanced scene features. .

[0042] The detail prior perception submodule follows a similar process, generating text priors based on LLaVA's description of image "details and textures". After being processed through the same process, semantically enhanced detailed features are output. The two sub-modules work together to enhance feature representations at both the macro-scene and micro-detail levels, providing semantic guidance for subsequent repair.

[0043] In step S4, the multi-level optical information-guided adaptive processing module includes: The task-specific prompt-guided multi-processing unit selection and integration submodule is used to generate task-specific prompts and select a preset number of processing units to process pixels based on task-specific prompts and local features, and combines global attention mechanism with processing unit equalization loss for constraint. The feature decomposition and multi-processing unit integration submodule is used to decompose the input features into low-frequency features and high-frequency features, and select a preset number of processing units for processing and integration.

[0044] The task-driven multi-processing unit selection and integration submodule first generates task-specific prompts: it performs convolution and global average pooling on the input image, obtains a degradation information vector through Softmax, and fuses it with preset task-related basic prompts to generate the task-specific prompts. Next, The model concatenates local features and calculates their similarity to the prompts from learnable processing units (experts). The top-K processing units with the highest similarity are selected to process each pixel. Simultaneously, a Transformer layer is introduced to capture global correlations between pixels, and a balance loss (such as the squared coefficient of variation loss) is used to constrain the processing units, preventing the model from over-relying on any particular type of processing unit.

[0045] The feature decomposition and multi-processing unit integration submodule decomposes features into low-frequency features (corresponding to the global structure of the image) and high-frequency features (corresponding to detail edges) using a dynamically learnable low-pass filter. After the two types of features are processed by LayerNorm, depthwise convolution, etc., the importance weight of each processing unit is predicted. The Top-K processing units are selected to process the high-frequency and low-frequency features respectively. Finally, the features are enhanced by depthwise convolution and point convolution and integrated into the output.

[0046] The processing units use a balance loss equal to the squared coefficient of variation loss. To ensure that each processing unit is used evenly during training and to avoid the model becoming overly reliant on one class, which could lead to a decline in generalization ability, the squared coefficient of variation loss is introduced as a constraint. Its calculation formula is as follows: ;in, Indicates the first The total number of times each processing unit was selected. For average number of uses, Standard deviation, This loss function prevents division by zero for extremely small positive numbers; it encourages all processing units to be uniformly activated during training, thereby improving the model's adaptability to different types of degradation.

[0047] In step S5, the contour-guided depth embedding module includes: The multi-scale deep convolution submodule is used to extract and aggregate surrounding context features of multiple scales through deep convolutions of different scales; The dual-path prior knowledge extraction submodule consists of two paths: the first path generates a weighted prior feature map and fuses it with the original image features, and the second path extracts a prior feature map for cross-level attention guidance.

[0048] The contour-guided deep embedding module aims to leverage the structural priors of real, lossless images to guide the encoder in learning feature representations that better fit the image restoration task, thus avoiding overfitting of the model to distortion noise.

[0049] The multi-scale deep convolutional submodule employs three different scales (e.g. The depthwise convolutional modules, each containing Depthwise convolution, 90° rotation, element-wise multiplication, and element-wise addition operations are used to extract multi-scale contextual features. The extracted features are then aggregated via channel concatenation and fed into a dual-path prior knowledge extraction submodule.

[0050] In the dual-path prior knowledge extraction submodule, the first path generates a weighted prior feature map through point convolution, square root, and sigmoid activation, which is then fused with the original image features and input into the encoder; the second path extracts a prior feature map for cross-level attention guidance through the same structure, further enhancing the encoder's ability to perceive image structure.

[0051] The multi-scale depthwise convolutional submodule uses at least three different kernel sizes; the multi-scale depthwise convolutional submodule employs multiple kernel sizes (such as... , , This involves deep convolution operations to capture contextual information across different receptive fields. Small-scale convolutions focus on local details, while large-scale convolutions capture global structure. The aggregation of multi-scale features can more comprehensively represent image content, improving the effectiveness of subsequent prior extraction and feature guidance.

[0052] In step S6, the segmentation decoder employs a structure incorporating downsampling and upsampling, and restores the feature map resolution through transposed convolutional layers. Simultaneously, it combines features from intermediate encoder layers to output a segmentation mask. The segmentation decoder uses a typical encoder-decoder architecture, progressively upsampling through multiple transposed convolutional layers to restore the feature map to the original input resolution. During decoding, multi-scale features from intermediate encoder layers are introduced (through skip connections) to supplement detailed information and improve the accuracy of segmentation boundaries. The final output is a segmentation mask with the same size as the input image, and it is optimized under supervised supervision using segmentation loss (such as cross-entropy loss) and the ground truth labels.

[0053] In step S6, the restoration decoder processes the distortion-removed features through a structure including convolutional layers, residual connections, and an attention mechanism, outputting the restored image. The restoration decoder consists of multiple convolutional layers, residual blocks, and self-attention modules, progressively eliminating subtle noise, blur, and color distortion remaining in the distortion-removed features. Residual connections facilitate gradient flow and feature reuse, while the attention mechanism enhances the model's ability to focus on important regions. The final output is a high-quality, realistic restored image, achieved through restoration loss (such as...). (or perceptual loss) and supervised optimization with real, lossless images.

[0054] Regarding the above embodiments, this application provides an electronic device; please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The computer device includes a memory and a processor, wherein the memory and the processor are coupled to each other, the memory stores program data, and the processor executes the program data to implement the steps of any embodiment of the above-described superlens endoscope image restoration and segmentation method.

[0055] In this embodiment, the processor may also be referred to as a CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0056] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0057] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0058] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0059] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for image restoration and segmentation in a superlens endoscope, characterized in that, Includes the following steps: S1. Obtain the distorted image generated by the superlens; S2. Input the distorted image into the encoder to extract basic features and generate an initial feature map; S3. Input the initial feature map into the optical prior information fusion and adjustment module, perform cross-modal interaction based on the text prior generated by the large language-visual model, and output semantically enhanced features; the cross-modal interaction is achieved through two rounds of multimodal iterative interactive attention: the first round uses visual features as the query and text features as the value to generate semantically weighted text features, and the second round uses the semantically weighted text features as the query and visual features as the value to generate the semantically enhanced features. S4. Input the semantic enhancement features into the multi-level optical information-guided adaptive processing module. After concatenating the task-specific prompts with local features, calculate the similarity with the prompts of the learnable processing unit and select the processing unit to process the pixel. Then, perform high- and low-frequency decomposition on the semantic enhancement features and perform differential processing by the processing unit to output the preliminary repair features. The high- and low-frequency decomposition decomposes the features into low-frequency features and high-frequency features through a dynamic learnable low-pass filter. S5. Input the real, distortion-free image into the contour-guided deep embedding module, extract contextual features and generate prior feature maps through multi-scale deep convolution to obtain the guiding features; S6. The preliminary repair features and the guiding features are fused to obtain the distortion-free features, and these features are simultaneously input into the segmentation decoder and the recovery decoder; the segmentation decoder outputs the segmentation mask, and the recovery decoder outputs the recovered image.

2. The method for image restoration and segmentation of a superlens endoscope according to claim 1, characterized in that, In step S1, the phase distribution of the superlens is determined based on the radial position and the incident light wavelength.

3. The method for image restoration and segmentation of a superlens endoscope according to claim 1, characterized in that, In step S3, the optical prior information fusion and adjustment module includes: The environment prior perception submodule is used to output semantically enhanced scene features based on textual priors describing the overall scene and background of the image. The detail prior perception submodule is used to output semantically enhanced detail features based on textual priors describing image details and textures.

4. The method for image restoration and segmentation of a superlens endoscope according to claim 1, characterized in that, In step S4, the multi-level optical information-guided adaptive processing module includes: The task prompt-guided multi-processing unit selection and integration submodule is used to generate task-specific prompts and select a preset number of processing units to process pixels based on the task-specific prompts and local features, and combine global attention mechanism with the processing unit using equalization loss for constraint. The feature decomposition and multi-processing unit integration submodule is used to decompose the input features into low-frequency features and high-frequency features, and select a preset number of processing units for processing and integration.

5. The method for image restoration and segmentation of a superlens endoscope according to claim 4, characterized in that, The processing unit uses the equalization loss as the squared loss of the coefficient of variation.

6. The method for image restoration and segmentation of a superlens endoscope according to claim 1, characterized in that, In step S5, the contour-guided depth embedding module includes: The multi-scale deep convolution submodule is used to extract and aggregate surrounding context features of multiple scales through deep convolutions of different scales; The dual-path prior knowledge extraction submodule consists of two paths: the first path generates a weighted prior feature map and fuses it with the original image features, and the second path extracts a prior feature map for cross-level attention guidance.

7. The method for image restoration and segmentation of a superlens endoscope according to claim 6, characterized in that, The kernel sizes used in the multi-scale depth convolution submodule include at least three different sizes.

8. The method for image restoration and segmentation of a superlens endoscope according to claim 1, characterized in that, In step S6, the segmentation decoder adopts a structure that includes downsampling and upsampling, and restores the feature map resolution through a transposed convolutional layer, while combining the features of the encoder's intermediate layer to output a segmentation mask.

9. The method for image restoration and segmentation of a superlens endoscope according to claim 1, characterized in that, In step S6, the restoration decoder processes the distortion-reduced features through a structure including convolutional layers, residual connections, and attention mechanisms, and outputs the restored image.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the superlens endoscope image restoration and segmentation method as described in any one of claims 1 to 9.

Images