Endoscope imaging optimization method

Abstract

The invention discloses an endoscope imaging optimization method, and belongs to the technical field of medical optical imaging and image processing. The method comprises the following steps: firstly, obtaining a super-structure lens distortion image, a high-definition truth value image and a semantic segmentation label; extracting multi-scale initial features through a multi-branch encoder; carrying out feature enhancement and self-adaptive downsampling by utilizing a dynamic super-surface feature extraction module; constructing robust features through a dynamic parameter generation module in combination with the high-halal truth value image; then, extracting contour guide features from the high-halal truth value image and fusing the contour guide features with robust features; and finally, image segmentation and recovery tasks are executed in parallel, and a segmentation mask and a distorted image are output. By fusing physical prior and data driving, the problems of chromatic aberration, information dispersion and the like of super-structure lens imaging are effectively solved, and the image definition, the small focus recognition precision and the model generalization ability are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of medical optical imaging and image processing technology, and in particular to an endoscopic imaging optimization method. Background Technology

[0002] In recent years, with the development of metalens technology, it has shown great potential in miniature imaging systems such as medical endoscopes. However, because metalenses are based on nanoscale metaatomic arrays for wavefront modulation, their phase distribution is wavelength-dependent, which often leads to optical defects such as uneven intensity, chromatic aberration, and information dispersion during imaging, seriously affecting image quality and the accuracy of subsequent clinical analysis.

[0003] In existing technologies, some studies have attempted to correct superlens imaging using convolutional neural networks, but these studies generally suffer from the following problems: 1. The receptive field of the feature extraction module is fixed, making it difficult to adapt to the characteristics of metasurface information being dispersed and distributed across multiple scales; 2. The calibration process often relies solely on data-driven approaches and lacks integration with optical physical models, resulting in a lack of physical consistency in the results; 3. Multi-task operations (such as image restoration and semantic segmentation) are usually performed independently, failing to achieve feature sharing and collaborative optimization, resulting in low efficiency and easy loss of related information. Summary of the Invention

[0004] To address the aforementioned issues, this application provides an endoscopic imaging optimization method to resolve problems such as mismatch between feature extraction and optical properties, poor physical consistency of correction results, and insufficient multi-task collaboration in existing technologies, thereby achieving high-quality restoration and accurate semantic segmentation of superlens-distorted images.

[0005] The first technical solution adopted in this application is: providing an endoscopic imaging optimization method, including: Acquire the distorted image generated by the metalens and the corresponding high-resolution reference image; Feature extraction is performed on the distorted image to obtain initial image features; The initial image features are subjected to adaptive downsampling processing to obtain downsampled features; Based on the high-resolution reference image and the downsampling features, robust image features are generated; Extract contour guidance features from the high-resolution reference image and fuse the contour guidance features with the robust image features to obtain correction features; Based on the correction features, image segmentation and image restoration tasks are performed in parallel to output the segmentation result and the distortion-free restored image, respectively.

[0006] In an optional embodiment, the step of feature extraction of the distorted image is implemented by an encoder; the initial layer of the encoder includes a multi-branch convolutional structure, which includes at least two branches with convolutional kernels of different sizes for extracting image features at different scales.

[0007] In an optional embodiment, the multi-branch convolutional structure includes: The first branch, with a convolutional kernel of the first size, is used to extract detailed features; A second branch with a second-sized convolutional kernel larger than the first size is used to extract contour features; A third branch with a third-size convolutional kernel larger than the second size is used to extract contextual features.

[0008] In an optional embodiment, the step of adaptively downsampling the initial image features is implemented by a dynamic feature extraction and downsampling module; the dynamic feature extraction and downsampling module includes: The dynamic feature extraction submodule is used to enhance the features of the input image; and The dynamic downsampling submodule is used to downsample the enhanced image features based on importance weights.

[0009] In an optional embodiment, the dynamic feature extraction submodule includes: The feature splitting unit is used to split the input features into first path features and second path features; The feature enhancement unit is used to perform enhancement transformations on the second path features, including dilated convolution and channel attention processing; and The feature fusion unit is used to fuse the enhanced transformed second path features with the first path features.

[0010] In an optional embodiment, the dilated convolution processing in the feature enhancement unit employs a reparameterizable structure; during the training phase, multiple convolutional branches with different dilation rates operate in parallel; during the inference phase, the parameters of the multiple convolutional branches are merged into a single convolution.

[0011] In an optional embodiment, the dynamic downsampling submodule includes: The weight prediction unit is used to predict the importance weights of pixels in each region of the input feature map. A downsampling unit is used to perform downsampling operations on the input feature map; and The weighted fusion unit is used to weight the downsampled features according to the importance weights and output the adaptive downsampled features.

[0012] In an optional embodiment, the step of generating robust image features based on the high-resolution reference image and the downsampling features includes: A pixel-level importance mask is constructed based on the similarity between the high-resolution reference image and the downsampled features; The downsampled features are weighted and aggregated using the pixel-level importance mask; and The aggregated features are mapped to dynamic convolution parameters and applied to the downsampled features to generate the robust image features.

[0013] In an optional embodiment, the step of extracting contour-guided features from the high-resolution reference image includes: Extracting multi-scale contextual features from the high-resolution reference image; and The contour guidance feature is generated based on the multi-scale contextual features.

[0014] In an optional embodiment, the steps of performing the image segmentation task and the image restoration task in parallel include: The corrected features are input into a segmentation decoder, which upsamples and combines them with intermediate features from the encoder to output the segmentation result; and The corrected features are input into the restoration decoder, which performs optimization processing through a network structure containing convolutional layers, residual blocks, and attention mechanisms, and outputs the restored image.

[0015] Due to the adoption of the above technical solution, this application has at least one of the following beneficial effects compared with the prior art:

[0016] 1. By using a multi-branch initial encoder and a dynamic metasurface feature extraction module, especially the cavity reparameterization block and adaptive downsampling, the problems of scattered metalens information and weak features of small targets are effectively solved, and the accurate capture of microscopic features is achieved.

[0017] 2. High-resolution ground truth images are introduced as priors for optical perception correction, and the structural information of the real images is incorporated into the feature optimization process through the contour guidance module, which avoids model fitting distortion noise and ensures the physical rationality of the correction results.

[0018] 3. Adopting an end-to-end architecture of correction-analysis, it simultaneously completes the two major tasks of image restoration and semantic segmentation, achieving a seamless connection from low-level repair to high-level clinical analysis, which is more in line with actual diagnosis and treatment needs.

[0019] 4. By integrating the dynamic parameter generation module of the pre-trained model with input-specific information, the generalization ability and robustness of the model on different batches of hyperlens samples are significantly improved, solving the problem of large generalization error in existing data-driven methods. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a schematic flowchart of an endoscopic imaging optimization method provided in an embodiment of this application; Figure 2 This is a schematic flowchart of an endoscopic imaging optimization method provided in another embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0022] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] Existing metalenticule endoscopic imaging techniques generally rely on pure data-driven deep learning models for distortion correction. However, this approach has several drawbacks: First, traditional convolutional networks employ fixed receptive fields and uniform downsampling strategies, which cannot adapt to the dispersed encoding of metalenticule information, leading to significant loss of crucial details such as small lesions during feature extraction and dimensionality reduction. Second, the image restoration process lacks prior optical and physical constraints, optimizing only at the pixel-level, which easily produces artifacts that violate the true anatomical structure and exhibits poor generalization ability across different batches of metalenticules. Furthermore, image restoration and semantic segmentation tasks are typically handled in isolation, failing to achieve synergistic gains between low-order imaging optimization and high-order clinical analysis.

[0025] In view of this, the endoscopic imaging optimization method provided in this application improves image fidelity while enhancing the accuracy of small target recognition and the robustness of the model by adaptively downsampling to preserve key details, introducing contour-guided features from high-resolution reference images to inject physical priors, and performing segmentation and reconstruction tasks in parallel based on the fused correction features; such as Figure 1 As shown, Figure 1 This is a schematic flowchart of an endoscopic imaging optimization method provided in an embodiment of this application. Figure 2 This is a flowchart illustrating an endoscopic imaging optimization method according to another embodiment of this application. The endoscopic imaging optimization method includes: The process involves acquiring distorted images generated by metalensing and their corresponding high-resolution reference images. To establish a foundation for supervised learning, a dataset aligned with the distortion-ground value-label alignment needs to be constructed. Specifically, high-quality medical endoscopic baseline images are selected as the high-resolution ground value images. To simulate the inherent defects of metalenses, images are projected onto a display device and captured using an experimental metalens camera. This metalens is composed of a nanoscale array of metaatoms, and its phase modulation is wavelength-dependent, resulting in different images. It exhibits distortions such as uneven intensity, color misalignment, and information dispersion. Meanwhile, it is being examined by professionals. Perform pixel-level annotation to generate semantic segmentation labels. This identifies specific anatomical structures or lesion areas, thereby forming training sample triplets. .

[0026] Feature extraction is performed on the distorted image to obtain initial image features; distorted image As input, the data is first fed into a dedicated encoder. The initial layer of this encoder employs a multi-branch convolutional structure, designed to initially extract and fuse feature information at different spatial scales from the distorted input, outputting an initial feature map. .

[0027] Adaptive downsampling is performed on the initial image features to obtain downsampled features; initial feature map The data is input into the dynamic metasurface feature extraction module. This module first enhances the features through a dynamic feature extraction submodule, which is designed to address the dispersed nature of metasurface imaging information by employing a hierarchical adaptation and dual-path processing mechanism. Subsequently, the dynamic downsampling submodule adaptively downsamples the enhanced features. By predicting spatial importance weights, it reduces the feature map resolution while preserving key details, outputting the downsampled features. .

[0028] Robust image features are generated based on high-resolution reference images and downsampling features. To incorporate prior knowledge of ideal imaging into the feature extraction process, a dynamic parameter generation module is introduced in this step. This module uses high-resolution ground truth images... For reference, combined with downsampling features The network computes pixel-level importance associations using learnable cue tokens and leverages a shared supernetwork to generate dynamic convolutional kernel parameters that depend on the input. These parameters tune feature extraction behavior in real time, enabling the network to learn more robust feature representations in distorted regions. .

[0029] Contour-guided features are extracted from the high-resolution reference image and fused with robust image features to obtain correction features. To further utilize the clear structural information in the ground truth image, a contour-guided module is used to extract... Extract multi-scale contour features; combine the contour guiding features with robust features. By fusing these elements, clear spatial structural constraints are provided for subsequent tasks, and corrective features are obtained. .

[0030] Based on correction features, image segmentation and image restoration tasks are performed in parallel to output segmentation results and distortion-free restored images, respectively; correction features It is fed in parallel to the segment decoder and the recovery decoder.

[0031] The segmentation decoder employs an upsampling architecture and combines features from the encoder's intermediate layers to progressively restore resolution, ultimately outputting a semantic segmentation mask. This process is determined by the segmentation loss function. Supervision, and Authentic Labels Compare and optimize.

[0032] The restoration decoder uses a network structure that includes convolutional layers, residual blocks, and attention mechanisms to progressively eliminate distortion and reconstruct texture and color, outputting a restored image. This process is defined by the recovery loss function. Supervision, and high-definition truth Perform comparison and optimization; the total loss function is ,in This is the balance coefficient.

[0033] In summary, the endoscopic imaging optimization method of this embodiment first acquires metalens distortion images, high-resolution ground truth images, and semantic segmentation labels; then, it extracts multi-scale initial features through a multi-branch encoder; next, it uses a dynamic metasurface feature extraction module for feature enhancement and adaptive downsampling; then, it combines the high-resolution ground truth image with a dynamic parameter generation module to construct robust features; subsequently, it extracts contour-guided features from the high-resolution ground truth image and fuses them with the robust features; finally, it performs image segmentation and restoration tasks in parallel, outputting a segmentation mask and a dedistorted image. This invention, by fusing physical priors and data-driven approaches, effectively solves problems such as chromatic aberration and information dispersion in metalens imaging, significantly improving image clarity, small lesion recognition accuracy, and model generalization ability.

[0034] In another alternative embodiment, the step of feature extraction of the distorted image is implemented by an encoder; the initial layer of the encoder includes a multi-branch convolutional structure, which includes at least two branches with convolutional kernels of different sizes for extracting image features at different scales.

[0035] Multi-branch convolutional structures include: The first branch, with a convolutional kernel of the first size, is used to extract detailed features; A second branch with a second-size convolutional kernel larger than the first size is used to extract contour features; A third branch with a third-size convolutional kernel larger than the second size is used to extract contextual features.

[0036] The multi-branch structure of the encoder's initial layer is designed to address the multi-scale limitations of superlens imaging, and it contains three parallel convolutional branches:

[0037] The first branch uses a 3×3 convolution kernel with a stride of 1. This branch focuses on capturing high-frequency detail features in the image, such as the subtle texture of tissue surfaces or the edges of instruments.

[0038] The second branch uses a 5×5 convolution kernel with a stride of 1. This branch is used to extract medium-scale contour features, such as the overall shape of the lesion region.

[0039] The third branch uses a 7×7 convolution kernel with a stride of 2. This branch aims to obtain global context information and achieve preliminary spatial downsampling.

[0040] Each branch processes the input image independently. The output of the second branch is upsampled to the same spatial size as the first branch, and then concatenated with the outputs of the first and third branches along the channel dimension. The concatenated features are then processed sequentially through a batch normalization (BatchNorm) layer and a ReLU activation function, ultimately generating a feature with dimension [missing information]. initial feature map ,in This represents the total number of channels. and For height and width.

[0041] In another optional embodiment, the step of adaptively downsampling the initial image features is implemented by a dynamic feature extraction and downsampling module; the dynamic feature extraction and downsampling module includes: The dynamic feature extraction submodule is used to enhance the features of the input image; and The dynamic downsampling submodule is used to downsample the enhanced image features based on importance weights.

[0042] The dynamic feature extraction submodule includes: The feature splitting unit is used to split the input features into first path features and second path features; Feature enhancement units are used to perform enhancement transformations on the second path features, including dilated convolution and channel attention processing; and The feature fusion unit is used to fuse the enhanced transformation of the second path features with the first path features.

[0043] The dilated convolution processing in the feature enhancement unit adopts a reparameterizable structure; during the training phase, multiple convolutional branches with different dilation rates work in parallel; during the inference phase, the parameters of multiple convolutional branches are merged into a single convolution.

[0044] The working principle of the dynamic feature extraction submodule is described in detail below: The input to the dynamic feature extraction submodule is the initial image features. First, a 1×1 convolutional layer is used to... Lightweight processing is performed to adapt to the characteristics of features at different depths (e.g., shallow layers preserve details, deep layers add semantics). Subsequently, the processed feature map is evenly split into two parts along the channel dimension, denoted as the first path feature and the second path feature, respectively.

[0045] The second path features are fed into an enhancement network consisting of multiple identical sub-modules connected in series. Each sub-module performs the following transformations on the input features in sequence:

[0046] Dilated convolution processing employs a reparameterizable structure. During training, multiple parallel convolutional branches with different dilation rates (e.g., 3 branches for shallow features and 5 branches for deep features) are used simultaneously to process input features to capture multi-scale contextual information. Each branch contains convolutional layers and batch normalization layers.

[0047] Channel attention processing performs global average pooling on the features after dilated convolution to obtain channel description vectors. These vectors are then passed through a small network containing fully connected layers, ReLU activation, and a sigmoid function to generate importance weights for each channel. These weights are then multiplied with the original features to achieve adaptive recalibration of the channel dimensions.

[0048] Subsequently, the features are typically transformed nonlinearly through a simple feedforward network (such as two 1×1 convolutional layers with ReLU activation) and added to the input of the submodule through residual connections to stabilize the training.

[0049] The second path features, enhanced by several sub-modules, are concatenated with the first path features passed through identity mapping (or simple projection) along the channel dimension to output the enhanced feature map. This dual-path design ensures that detailed information is not lost, while the contextual information is fully enhanced.

[0050] The dynamic downsampling submodule includes: The weight prediction unit is used to predict the importance weights of pixels in each region of the input feature map. A downsampling unit is used to perform downsampling operations on the input feature map; and The weighted fusion unit is used to weight the downsampled features according to their importance and output adaptive downsampled features.

[0051] The working principle of the dynamic downsampling submodule is described in detail below: The input to the dynamic downsampling submodule is the enhanced feature map. Its goal is to downsample the feature map by a factor of 2. The space is divided into non-overlapping 2×2 pixel regions. Average pooling is performed on the four pixel feature vectors within each region to obtain a region representative vector. This vector is then input into a lightweight network (such as a miniature multilayer perceptron) and normalized using the Softmax function to generate importance weights for the four pixels within that region. The sum of the weights is 1.

[0052] Simultaneously, a standard downsampling operation is performed on each 2×2 region, such as taking the average or maximum value of all pixel features within the region to obtain a preliminary downsampled feature value. .

[0053] The importance weights generated by the weight prediction unit are combined with the preliminary features obtained by the downsampling unit. Specifically, the final output feature map is located at... eigenvalues The feature vectors of the original 4 pixels within the 2×2 region According to their corresponding weights Weighted summation yields: .

[0054] This operation allows the downsampling process to be dynamically adjusted based on content importance, compressing spatial information while preserving pixel information crucial to small targets or key structures, thus outputting adaptive downsampling features. .

[0055] In another optional embodiment, the step of generating robust image features based on the high-resolution reference image and downsampling features includes: Pixel-level importance masks are constructed based on the similarity between high-resolution reference images and downsampled features; Weighted aggregation of downsampled features using pixel-level importance masks; and The aggregated features are mapped to dynamic convolution parameters and applied to downsampled features to generate robust image features.

[0056] The module initializes a set of learnable vectors, called cue tokens. Each token represents a potential semantic concept. Each hint token is calculated separately. Downsampling characteristics Each spatial location The similarity between feature vectors (e.g., using dot product or cosine similarity). For each token The similarity between this response map and the entire feature map forms a two-dimensional response map. This response map is then normalized in spatial dimension using the Softmax function, resulting in a pixel-level importance mask. It clearly indicates which regions in the feature map are semantically relevant to the token. This process can be formally represented as calculating attention weights for each location.

[0057] For each importance mask Use this as a weight for downsampling features Perform weighted global average pooling. Specifically, calculate... This yields a feature vector that summarizes the information of the region emphasized by the mask. .

[0058] All The vector corresponding to each token The vectors are concatenated to form a comprehensive layer embedding vector. This vector encodes the current input image (via...). ) and specific association information between prior knowledge sources (guided by tokens).

[0059] Embedded vectors Input a lightweight shared supernetwork (typically a multilayer perceptron, MLP). The supernetwork outputs a set of dynamic parameters, such as the weight offsets for a specific convolutional layer in the network. and offset .

[0060] During this forward propagation, the actual parameters of the target convolutional layer were adjusted to... , ;in These are the original static parameters of this layer. These dynamically generated parameters enable the network to adaptively adjust its feature extraction behavior based on the content of the current input image and the prior information from the corresponding high-resolution reference image. The adjusted convolutional layer is then applied to the downsampled features. Or its subsequent derived features can generate more robust image features that are more robust to specific distortion patterns. .

[0061] In another optional embodiment, the step of extracting contour-guided features from the high-resolution reference image includes: Extracting multi-scale contextual features from high-resolution reference images; and Contour-guided features are generated based on multi-scale contextual features.

[0062] High-resolution reference images Input a multi-scale feature extraction submodule, which deploys multiple processing branches in parallel, each branch using convolutional kernels of different scales. The processing proceeds. In a preferred embodiment, three branches are employed, using 3×3, 5×5, and 7×7 convolutional kernels respectively (depth-separable convolutions can be used to reduce the number of parameters). These convolutional operations at different scales can simultaneously capture multi-scale contextual features, ranging from subtle edges and local contours to regional semantics. Each branch may contain convolution, normalization, activation, and some feature enhancement operations (such as rotation, feature interaction, etc.). Finally, the feature maps output by each branch retain their respective spatial resolution and semantic hierarchy.

[0063] The extracted multi-scale contextual features are then fused. Typically, the feature maps output from different branches are concatenated along the channel dimension to form a fused feature containing rich scale information. This fused feature is then input into a prior knowledge extraction submodule. This submodule usually consists of a 1×1 convolutional layer (for cross-channel information integration and dimensionality reduction) and a sigmoid activation function. The 1×1 convolutional layer learns the contribution weights of different scale features to the final contour prediction; the sigmoid function maps the output values ​​to the (0, 1) interval, generating the final contour-guided feature map. In this feature map In the text, the pixel position with the higher value corresponds to... This feature map represents regions with high confidence levels at structural boundaries (such as organ margins and lesion outlines). It will be used to guide the correction of distorted image features.

[0064] In another optional embodiment, the steps of performing the image segmentation task and the image restoration task in parallel include: The corrected features are input into the segmentation decoder, which upsamples and combines them with the intermediate features from the encoder to output the segmentation result; and The corrected features are input into the restoration decoder, which performs optimization processing through a network structure containing convolutional layers, residual blocks, and attention mechanisms, and outputs the restored image.

[0065] The corrected features, as shared, high-quality feature representations, are simultaneously input into two independently designed decoders to perform different advanced vision tasks in parallel.

[0066] 1. The segmentation decoder starts with corrected features. This decoder typically consists of multiple cascaded upsampling stages. The core operation of each stage is upsampling (which can be achieved through transposed convolution, pixel shuffling, or bilinear interpolation), aiming to gradually restore the spatial resolution of the feature map to the size of the original input image.

[0067] After each upsampling stage, the decoder's current feature map is fused with intermediate layer features from the encoder network that have the same or similar spatial resolution. This process is called skip connections. For example, when the decoder feature map is upsampled to... At that time, it will be similar to the signal generated by the encoder during a certain downsampling stage. The intermediate feature maps are stitched together or added together. These intermediate features from the encoder are rich in spatial details and positional information of the original image, which can effectively compensate for the detail blurring and inaccurate positioning problems that may be caused by the decoder due to multiple upsampling.

[0068] After several cycles of upsampling, feature fusion, and convolutional thinning, the feature map is finally restored to... The resolution is then determined. Finally, a 1×1 convolutional layer maps the number of channels to the number of target classes. Then, the Softmax function is used to obtain the probability distribution map of each pixel belonging to each category, which is the final segmentation result (segmentation mask). .

[0069] 2. The recovery decoder also takes the corrected features as input. Its main structure consists of a series of stacked residual blocks. Each residual block contains two or more 3×3 convolutional layers, interspersed with batch normalization layers and activation functions such as ReLU, and uses residual connections (adding the block input and output) to facilitate gradient flow and deep network training.

[0070] Attention modules are embedded between or within residual blocks to enhance model capabilities. Optional attention modules include:

[0071] Channel attention mechanisms (such as the SE module) reweight the channel dimensions of the feature map, allowing the network to focus on color channels or feature channels with richer information, which helps to correct color differences and enhance important features.

[0072] Spatial attention mechanism: The spatial dimensions of the feature map are reweighted, enabling the network to focus on areas of severe distortion or requiring key repair in the image.

[0073] After progressive refinement through multiple residual blocks and attention modules, the semantic information of the feature map is gradually converted into pixel-level texture and color information. Finally, one or more convolutional layers map the number of feature channels to 3 (corresponding to RGB three channels), and a suitable activation function (such as Tanh) is used to constrain the pixel values ​​to the target range (such as [-1,1] or [0,1]), outputting the final distortion-free restored image. .

[0074] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0075] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0076] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0077] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An endoscopic imaging optimization method, characterized in that, include: Acquire the distorted image generated by the metalens and the corresponding high-resolution reference image; Feature extraction is performed on the distorted image to obtain initial image features; The initial image features are subjected to adaptive downsampling processing to obtain downsampled features; Based on the high-resolution reference image and the downsampling features, robust image features are generated; Extract contour guidance features from the high-resolution reference image and fuse the contour guidance features with the robust image features to obtain correction features; Based on the correction features, image segmentation and image restoration tasks are performed in parallel to output the segmentation result and the distortion-free restored image, respectively.

2. The method according to claim 1, characterized in that, The feature extraction step of the distorted image is implemented by an encoder; the initial layer of the encoder includes a multi-branch convolutional structure, which includes at least two branches with convolutional kernels of different sizes for extracting image features at different scales.

3. The method according to claim 2, characterized in that, The multi-branch convolutional structure includes: The first branch, with a convolutional kernel of the first size, is used to extract detailed features; A second branch with a second-sized convolutional kernel larger than the first size is used to extract contour features; A third branch with a third-size convolutional kernel larger than the second size is used to extract contextual features.

4. The method according to claim 1, characterized in that, The step of adaptively downsampling the initial image features is implemented through a dynamic feature extraction and downsampling module; the dynamic feature extraction and downsampling module includes: The dynamic feature extraction submodule is used to enhance the features of the input image; and The dynamic downsampling submodule is used to downsample the enhanced image features based on importance weights.

5. The method according to claim 4, characterized in that, The dynamic feature extraction submodule includes: The feature splitting unit is used to split the input features into first path features and second path features; The feature enhancement unit is used to perform enhancement transformations on the second path features, including dilated convolution and channel attention processing; and The feature fusion unit is used to fuse the enhanced transformed second path features with the first path features.

6. The method according to claim 5, characterized in that, The dilated convolution processing in the feature enhancement unit adopts a reparameterizable structure; during the training phase, multiple convolutional branches with different dilation rates work in parallel; during the inference phase, the parameters of the multiple convolutional branches are merged into a single convolution.

7. The method according to claim 4, characterized in that, The dynamic downsampling submodule includes: The weight prediction unit is used to predict the importance weights of pixels in each region of the input feature map. A downsampling unit is used to perform downsampling operations on the input feature map; and The weighted fusion unit is used to weight the downsampled features according to the importance weights and output the adaptive downsampled features.

8. The method according to claim 1, characterized in that, The steps for generating robust image features based on the high-resolution reference image and the downsampling features include: A pixel-level importance mask is constructed based on the similarity between the high-resolution reference image and the downsampled features; The downsampled features are weighted and aggregated using the pixel-level importance mask; and The aggregated features are mapped to dynamic convolution parameters and applied to the downsampled features to generate the robust image features.

9. The method according to claim 1, characterized in that, The steps for extracting contour-guided features from the high-resolution reference image include: Extracting multi-scale contextual features from the high-resolution reference image; and The contour guidance feature is generated based on the multi-scale contextual features.

10. The method according to claim 1, characterized in that, The steps for performing image segmentation and image restoration tasks in parallel include: The corrected features are input into a segmentation decoder, which upsamples and combines them with intermediate features from the encoder to output the segmentation result; and The corrected features are input into the restoration decoder, which performs optimization processing through a network structure containing convolutional layers, residual blocks, and attention mechanisms, and outputs the restored image.

Images