A semi-supervised method for enhancing laser microscopy imaging

Abstract

The present invention discloses a semi-supervised laser microscopy enhancement method, comprising the following steps: constructing a laser microscopy image dataset and normalizing the images; constructing a two-stage deep learning network structure, comprising a first-stage unsupervised degradation representation learning network and a second-stage supervised super-resolution image reconstruction network; training the two-stage deep learning network; and performing network inference, inputting a low-resolution laser microscopy image, and outputting a high-resolution image, thereby achieving laser microscopy super-resolution. The present invention combines an unsupervised contrastive learning strategy with a supervised image reconstruction method, fully utilizing unlabeled data for feature extraction while simultaneously improving reconstruction quality through supervised learning, effectively enhancing the model's generalization capability and reconstruction accuracy. The present invention also introduces an encoder with a transformer (a deep learning model) architecture to capture complex degradation information from a global perspective, ensuring that the model can adapt to various complex degradations.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a semi-supervised laser microscopy imaging enhancement method. Background Art

[0002] Laser microscopy, an advanced microscopic imaging method, uses lasers as a light source and scans samples through an optical system to obtain high-resolution images. Compared to traditional light source imaging systems, laser microscopy can significantly improve image resolution. However, due to factors such as laser energy limitations, sample scattering, detector resolution, and environmental noise, the images obtained still fail to fully reveal the sample's microstructure and details. Therefore, image enhancement processing of laser microscopy images to further improve their resolution is of great significance for the detailed study and analysis of samples.

[0003] Currently, image super-resolution technologies are relatively mature and can be divided into two main categories. One is traditional image interpolation algorithms, such as bilinear interpolation and bicubic interpolation, which increase the resolution of low-resolution images by interpolating pixels. The other is deep learning-based methods, which use neural network training models to learn the mapping relationship between low-resolution and high-resolution images, achieving super-resolution reconstruction.

[0004] Traditional interpolation algorithms simply interpolate based on adjacent pixels and cannot effectively restore the lost high-frequency and global information of the image, resulting in blurred image edges and unclear textures. In addition, traditional interpolation methods have weak generalization capabilities and are difficult to adapt to complex images, especially in scenes such as laser microscopy that require high detail resolution, where their performance is particularly limited. Although the fully supervised method based on deep learning is superior to the traditional interpolation algorithm in generalization ability, it requires a large amount of labeled image data for training, the data acquisition cost is high, and the training process is time-consuming. In addition, these methods often do not take into account the optical laws of laser imaging, which makes it difficult for conventional super-resolution technology to accurately adapt to microscopy tasks, affecting the quality of the reconstructed image and the actual application effect.

[0005] In view of the above shortcomings, the present invention proposes a semi-supervised laser microscopy imaging enhancement method. Summary of the Invention

[0006] To overcome the shortcomings of the existing technology, the present invention aims to provide a semi-supervised laser microscopy enhancement method, which utilizes some labeled data and a large amount of unlabeled data for training, thereby reducing data dependence and improving the generalization ability of the model. At the same time, based on the characteristics of laser imaging, the optical transfer function is introduced to further constrain the network, thereby accelerating model convergence and further improving image quality.

[0007] The technical effects achieved by the present invention can be achieved through the following specific technical solutions:

[0008] The semi-supervised laser microscopy imaging enhancement method comprises the following steps:

[0009] Step 1: Construct a laser microscopy image dataset and normalize the images;

[0010] Step 2: Construct a two-stage deep learning network structure, including an unsupervised degradation representation learning network in the first stage and a supervised super-resolution image reconstruction network in the second stage;

[0011] Step 3: Train the two-stage deep learning network;

[0012] Step 4: Perform network inference, input a low-resolution laser microscopy image, and output a high-resolution image to achieve laser microscopy super-resolution.

[0013] Furthermore, in step 1, low-resolution images obtained by laser microscopy are collected as a data set and divided into labeled data and unlabeled data; for the labeled data, the corresponding high-resolution images are used as labels; and then all images are normalized so that their pixel values are distributed in the range of [0, 1].

[0014] Furthermore, in step 2, the unsupervised degradation representation learning network is trained on unlabeled data using contrastive learning. In terms of dataset composition, positive samples are generated by transforming different patches within the same image, while negative samples are generated by transforming patches within different images. Through contrastive learning, representations of multiple degradation patterns are learned to guide the subsequent reconstruction process. The supervised super-resolution image reconstruction network is composed of several convolutional, pooling, and activation function layers. Its input is a low-resolution image and the degradation representation vector learned through unsupervised learning, and its output is a high-resolution reconstructed image.

[0015] Furthermore, in the unsupervised degradation representation learning network, an encoder based on the transformer architecture (a deep learning model) is employed. Compared to traditional convolutional neural network encoders, this architecture can capture complex degradations from a global perspective and exhibits stronger generalization capabilities. First, the encoder extracts features from the input image to generate a degradation representation vector. This representation vector then passes through a two-layer multi-layer perceptron network to form positive and negative sample vectors used for contrastive learning loss calculation. During unsupervised model training, positive sample pairs are generated by transforming different patches in the same image, while negative sample pairs are generated by transforming patches in different images. Through contrastive learning, the unsupervised network can effectively learn representations for various degradation patterns, providing guidance for the subsequent reconstruction process.

[0016] Furthermore, in the supervised super-resolution image reconstruction network, the degraded representation vector forms a dynamic convolution kernel after passing through two fully connected layers and one reshaping layer; then, the dynamic volume is modulated by the optical transfer function and optical features are added to obtain a convolution kernel that conforms to physical constraints; finally, the low-resolution image is convolved with the convolution kernel after physical constraints and input into the subsequent layers to generate a super-resolution reconstructed image.

[0017] Furthermore, in step 3, the specific content of training the two-stage deep learning network is as follows:

[0018] Training the unsupervised contrastive learning network: Initialize the unsupervised contrastive learning network parameters, select two different regions of the same image as positive sample pairs, and select regions of other low-resolution images as negative sample pairs. Use the Adam optimizer (an optimization algorithm) and the contrastive loss function to train the unsupervised contrastive learning network for 100 rounds, so that the representations of samples with the same degradation are closer, and the representations of samples with different degradation are farther apart.

[0019] Train the supervised super-resolution image reconstruction network: Initialize the parameters of the supervised super-resolution image reconstruction network, and convert the low-resolution images in the annotated dataset into degraded representation vectors using the trained encoder; the degraded representation vectors are transformed in dimension and modulated by the optical transfer function to form a dynamic convolution kernel, and the low-resolution image is convolved with the convolution kernel and input into the subsequent super-resolution image reconstruction network for training. The supervised super-resolution image reconstruction network is trained for a total of 500 rounds.

[0020] Furthermore, the specific process of optical transfer function modulating the dynamic convolution kernel is as follows:

[0021] ① Physical modeling: For a diffraction-limited system, the optical transfer function (OTF) can be analytically expressed as:

[0022] (1)

[0023] Where, f x 、f y is the sampling point position, NA is the numerical aperture, λ is the laser wavelength, and circ is the circular domain function, indicating that the response frequency outside the frequency is zero;

[0024] ② Discretize the continuous optical transfer function into a matrix that matches the size of the dynamic convolution kernel and normalize it to the range of [0,1];

[0025] ③ Dynamic convolution kernel W raw Perform Fourier transform to obtain its frequency domain representation W raw (f x ,f y ), in the frequency domain, the dynamic convolution kernel W rawMultiply element-wise with the discretized optical transfer function matrix to suppress components exceeding the system cutoff frequency:

[0026] (2)

[0027] Where W constrained represents the constrained convolution kernel in the frequency domain, W raw (f x ,f y ) is the frequency domain representation of the dynamic convolution kernel, OTF(f x ,f y ) means the discretized optical transfer function;

[0028] ④ Yes W constrained Perform inverse Fourier transform to obtain the convolution kernel after physical constraints W final .

[0029] Furthermore, the loss function adopted by the unsupervised contrastive learning network is for:

[0030] (3)

[0031] Where B is the number of low-resolution images, exp represents exponential operation, represents the jth patch from the i-th image, represents the number of samples in the queue, represents the jth negative sample, is the scale factor;

[0032] The loss function expression of the supervised super-resolution image reconstruction network is as follows:

[0033] (4)

[0034] (5)

[0035] Where, L total Represents the total loss of the reconstruction network, L L1 is the mean square error loss, OTF is the discretized optical transfer function, L OTF is the physical loss introduced by the optical transfer function, F represents the Fourier transform, λ is the proportional coefficient, and y SR and y HR represent low-resolution images and high-resolution images respectively, and M is the total number of frequency domain components.

[0036] Furthermore, in step 4, after network training is completed, a low-resolution image x is input, and a high-resolution image y is output, and the formula is as follows:

[0037]

[0038] In the formula, G(x) represents the entire network reasoning process;

[0039] Through the above process, super-resolution reconstruction of low-resolution images can be achieved.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] (1) Reduce the dependence on large-scale labeled data: The present invention adopts a semi-supervised learning method. In the first stage of unsupervised contrastive learning, the model can obtain a large amount of degradation representation information. Subsequently, in the second stage, only a small amount of labeled data is needed for network training to obtain better generalization ability.

[0042] (2) Adaptability to various types of image degradation: Traditional super-resolution algorithms usually rely on artificially set degradation methods such as bilinear interpolation or bicubic interpolation, and have poor reconstruction effects on complex degradation. The present invention introduces an encoder based on the transformer architecture (a deep learning model) to capture complex degradation information from a global perspective, and combines contrastive learning to effectively distinguish different degradation types, ensuring that the model can adapt to various complex degradations.

[0043] (3) Introducing physical constraints to improve image quality: This invention innovatively uses the optical transfer function as prior information to impose physical constraints on the dynamic convolution kernel. This strategy not only ensures that the restoration results conform to optical laws, but also accelerates model convergence, while significantly improving the quality of super-resolution images, making them clearer and more realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Flow chart of the method of the present invention;

[0045] Figure 2 This is a schematic diagram of the degradation representation learning network structure of the present invention;

[0046] Figure 3 Schematic diagram of the super-resolution image reconstruction network structure of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the specific application scenario of wafer microscopic images. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0048] Limited by imaging principles and hardware conditions, the resolution of laser microscopy images is difficult to meet the growing research and development needs. The application background of this invention is to break through the limitation of laser microscopy resolution through image enhancement technology.

[0049] like Figure 1 As shown in the figure, a semi-supervised laser microscopy enhancement method includes dataset construction, construction of a two-stage deep learning network structure, training of the two-stage deep learning network, and network inference. The specific contents are as follows:

[0050] (1) Hardware environment

[0051] The hardware configuration used for deep learning network training in the example of the present invention is: NVIDIA 4070SUPER GPU (12GB video memory), AMD 9700X CPU (6 cores and 12 threads), 32GB of memory; the deep learning framework used is PyTorch 2.6.0 and CUDA 12.6.

[0052] (2) Construct an image dataset and normalize the images.

[0053] A dataset of 10,000 low-resolution wafer images (256×256 resolution) was collected using laser microscopy. Of these, 2,000 (20%) had corresponding high-resolution labeled images, serving as labeled data, and the remaining 8,000 (80%) served as unlabeled data. All images were then normalized so that their pixel values fell within the range [0, 1] to accelerate network training.

[0054] (3) Construct a two-stage deep learning network structure, including an unsupervised degradation representation learning network in the first stage and a supervised super-resolution image reconstruction network in the second stage.

[0055] The unsupervised degradation representation learning network in the first stage uses the contrastive learning method to train on unlabeled data, extracting discriminative degradation representation vectors from low-resolution images to accurately characterize different degradation conditions; the supervised super-resolution image reconstruction network in the second stage uses the degradation representation vectors learned in the first stage, combined with labeled high- and low-resolution image data, to achieve adaptive super-resolution for different degraded images, effectively improving image restoration quality and model generalization ability.

[0056] like Figure 2The figure shows an unsupervised degradation representation learning network. The specific structural process involves: first, the encoder extracts features from the input image to generate a degradation representation vector; then, the degradation representation vector passes through a two-layer multi-layer perceptron network to form positive and negative sample vectors for calculating the learning loss. This application uses a Transformer-based encoder (consisting of 8 layers of Transformer blocks, each with a multi-head self-attention mechanism and a feedforward neural network). The input is a 256×256 low-resolution wafer image, and the output is a 128-dimensional degradation representation vector. Positive sample pairs are formed by randomly cropping two 256×256 regions from the same wafer image to simulate local differences under the same degradation conditions. Negative sample pairs are formed by randomly cropping 256×256 regions from different wafer images, covering different defect types or noise levels, representing different degradation patterns.

[0057] like Figure 3 Figure 2 shows a supervised super-resolution image reconstruction network. The detailed structural process involves: the degraded representation vector generated by the unsupervised degradation representation learning network passes through two fully connected layers and a reshaping layer to form a dynamic convolution kernel. Next, the dynamic convolution is modulated by the optical transfer function (OTF) to incorporate optical features, thereby obtaining a convolution kernel that conforms to physical constraints. Finally, the low-resolution image is convolved with the physically constrained convolution kernel and fed into subsequent layers to generate a super-resolution reconstructed image. The supervised super-resolution image reconstruction network consists of a dynamic convolution module, a residual convolution block, and an upsampling layer. The input is a low-resolution image (256×256) and a degraded representation vector (128 dimensions), and the output is a high-resolution reconstructed image (1024×1024). The degraded representation vector passes through two fully connected layers (with dimensions increasing from 128 to 512 to 1024) and a reshaping layer to form a dynamic convolution kernel. This is then modulated by the optical transfer function (OTF) to ensure compliance with the physical constraints of laser microscopy.

[0058] (4) Training a two-stage deep learning network.

[0059] First, the unsupervised degradation representation learning network is trained for 100 rounds to enable the degradation encoder to learn the representation that distinguishes degradation; then the super-resolution reconstruction network is trained for 500 rounds to achieve adaptive super-resolution for different degradation modes. The detailed training steps are as follows:

[0060] (1) Training an unsupervised contrastive learning network

[0061] First, initialize the network parameters, select two different regions of the same image as positive sample pairs (with the same degradation mode), and select regions of other low-resolution images as negative sample pairs (possibly with different degradation modes); then, use the Adam optimizer (an optimization algorithm) and set the learning rate , batch size B = 64. Number of queue samples N in the contrast loss function queue = 4096, with a temperature parameter τ = 0.1. Training was performed for 100 epochs, with each epoch covering all 8,000 unlabeled images. The distance between the representations of positive pairs of samples from the same wafer image was kept below 0.2, while the distance between the representations of negative pairs of samples from different images was kept above 0.8.

[0062] (2) Training a supervised super-resolution image reconstruction network

[0063] First, initialize the network parameters, select the Adam optimizer, and set the learning rate to 5x10 -5 The low-resolution images in the labeled dataset are converted into degraded representation vectors using a trained encoder. The degraded representation vectors are then transformed and modulated by the optical transfer function to form a convolution kernel. The low-resolution images are then convolved with the convolution kernel and fed into the subsequent super-resolution image reconstruction network for training. The supervised super-resolution reconstruction network is trained for 500 rounds.

[0064] The specific process of optical transfer function modulating the dynamic convolution kernel is as follows:

[0065] ① Physical modeling: For a diffraction-limited system, the optical transfer function (OTF) can be analytically expressed as:

[0066] (1)

[0067] Where, f x 、f y is the sampling point position, NA is the numerical aperture, λ is the laser wavelength, circ It is a circular function, indicating that the response frequency outside the frequency is zero.

[0068] ② Discretize the continuous optical transfer function into a matrix that matches the size of the dynamic convolution kernel and normalize it to the range of [0,1].

[0069] ③ Dynamic convolution kernel W raw Perform Fourier transform to obtain its frequency domain representation W raw (f x ,f y ). In the frequency domain, W raw Multiply element-wise with the discretized optical transfer function matrix to suppress components exceeding the system cutoff frequency:

[0070] (2)

[0071] Where W constrained represents the constrained convolution kernel in the frequency domain, W raw (f x ,f y) is the frequency domain representation of the dynamic convolution kernel, OTF(f x ,f y ) refers to the discretized optical transfer function.

[0072] ④ To W constrained Perform inverse Fourier transform to obtain the convolution kernel W after physical constraints final .

[0073] The loss function is a key factor in network training. The loss function used by the unsupervised contrastive learning network of the present invention is for:

[0074] (3)

[0075] Where B is the number of low-resolution images, exp represents the exponential operation, represents the jth patch from the i-th image, represents the number of samples in the queue, represents the jth negative sample, is the scale factor.

[0076] The loss function expression of the supervised super-resolution reconstruction network is as follows:

[0077] (4)

[0078] (5)

[0079] Among them, L total Represents the total loss of the reconstruction network, L L1 is the mean square error loss, OTF is the discretized optical transfer function, L OTF is the physical loss introduced by the optical transfer function, F represents the Fourier transform, λ is the proportional coefficient, and y SR and y HR represent low-resolution images and high-resolution images respectively, and M is the total number of frequency domain components.

[0080] In the process of dynamic convolution kernel generation, the present invention introduces the frequency domain response of the optical transfer function as a priori constraint to ensure that the generated convolution kernel meets the physical limits of the actual optical system within the frequency domain, effectively avoiding the introduction of false high-frequency information or details that violate the diffraction law during the reconstruction process, thereby improving the authenticity and reliability of super-resolution imaging.

[0081] (5) Perform network inference, input low-resolution laser microscopy images, and output high-resolution images to achieve laser microscopy super-resolution.

[0082] After completing the training of the two-stage deep learning network, the low-resolution image is input x, then the output high-resolution image y is:

[0083] (6)

[0084] Where G(x) represents the entire network reasoning process.

[0085] Through the above process, super-resolution reconstruction of low-resolution images can be achieved.

[0086] (6) Experimental results and analysis

[0087] The performance of the present invention was compared with that of the traditional method on the test samples, and the results are as follows:

[0088]

[0089] The proposed method significantly outperforms traditional methods in terms of PSNR and SSIM, demonstrating that the reconstructed images are closer to true high-resolution wafer images, with sharper edge and texture details. Furthermore, the proposed method achieves optimal performance using only 20% of the labeled data (2,000 pairs), while traditional fully supervised methods require over 80% labeled data to achieve similar results, demonstrating the low dependence of semi-supervised learning on labeled data.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semi-supervised laser microscopy imaging enhancement method, characterized in that: The following steps are involved: Step 1: Construct a laser microscopy image dataset and normalize the images; Step 2: Construct a two-stage deep learning network structure, including an unsupervised degradation representation learning network in the first stage and a supervised super-resolution image reconstruction network in the second stage; In the supervised super-resolution image reconstruction network, the degraded representation vector is passed through two fully connected layers and a reshaping layer to form a dynamic convolution kernel. Next, the dynamic convolution kernel is modulated by the optical transfer function and optical features are added to obtain a convolution kernel that conforms to physical constraints. Finally, the low-resolution image is convolved with the convolution kernel after the physical constraints and input into the subsequent layers to generate a super-resolution reconstructed image. Step 3: Train the two-stage deep learning network; Step 4: Perform network inference, input a low-resolution laser microscopy image, and output a high-resolution image to achieve laser microscopy super-resolution. The specific process of optical transfer function modulating the dynamic convolution kernel is as follows: ① Physical modeling: For a diffraction-limited system, the optical transfer function (OTF) can be analytically expressed as: Where, f x 、f y is the sampling point position, NA is the numerical aperture, λ is the laser wavelength, and circ is the circular domain function, indicating that the response frequency outside the frequency is zero; ② Discretize the continuous optical transfer function into a matrix that matches the size of the dynamic convolution kernel and normalize it to the range of [0,1]; ③ Dynamic convolution kernel W raw Perform Fourier transform to obtain its frequency domain representation W raw (f x ,f y ), in the frequency domain, the dynamic convolution kernel W raw Multiply element-wise with the discretized optical transfer function matrix to suppress components exceeding the system cutoff frequency: W constrained (f x ,f y )=W raw (f x ,f y )·OTF(f x ,f y ) (2) Among them, W constrained represents the constrained convolution kernel in the frequency domain, W raw (f x ,f y ) is the frequency domain representation of the dynamic convolution kernel, OTF(f x ,f y ) means the discretized optical transfer function; ④ To W constrained Perform inverse Fourier transform to obtain the convolution kernel W after physical constraints final .

2. A semi-supervised laser microscopy enhancement method according to claim 1, characterized in that: In step 1, low-resolution images obtained by laser microscopy are collected as a data set and divided into labeled data and unlabeled data; for the labeled data, the corresponding high-resolution images are used as labels; and then all images are normalized so that their pixel values are distributed in the range of [0, 1].

3. The method for enhancing semi-supervised laser microscopy according to claim 1, wherein: In step 2, the unsupervised degradation representation learning network is trained on unlabeled data using a contrastive learning method. In terms of data set composition, positive samples are generated by changing different blocks in the same image, and negative samples are obtained by transforming blocks in different images. Through contrastive learning, characterization methods for multiple degradation modes are learned to provide guidance for the subsequent reconstruction process. The supervised super-resolution image reconstruction network is composed of several convolution, pooling and activation function layers, with input being a low-resolution image and a degradation representation vector learned in unsupervised learning, and output being a high-resolution reconstructed image.

4. The method for enhancing semi-supervised laser microscopy according to claim 3, wherein: In the unsupervised degradation representation learning network, a transformer-based encoder is used. First, the encoder extracts features from the input image to generate a degradation representation vector. This degradation representation vector then passes through a two-layer multi-layer perceptron network to form positive and negative sample vectors for calculating contrastive learning loss. During the unsupervised model training process, positive sample pairs are generated by varying different patches in the same image, while negative sample pairs are obtained by transforming patches in different images. Through contrastive learning, the unsupervised network can effectively learn representation methods for multiple degradation patterns, providing guidance for the subsequent reconstruction process.

5. The method for enhancing semi-supervised laser microscopy according to claim 1, wherein: In step 3, the specific content of training the two-stage deep learning network is as follows: Train the unsupervised contrastive learning network: Initialize the unsupervised contrastive learning network parameters, select two different regions of the same image as positive sample pairs, and select regions of other low-resolution images as negative sample pairs. Use the Adam optimizer and contrastive loss function to train the unsupervised contrastive learning network for 100 rounds, so that the representations of samples with the same degradation are closer, and the representations of samples with different degradation are farther apart. Train the supervised super-resolution image reconstruction network: Initialize the parameters of the supervised super-resolution image reconstruction network, and convert the low-resolution images in the annotated dataset into degraded representation vectors using the trained encoder; the degraded representation vectors are transformed in dimension and modulated by the optical transfer function to form a dynamic convolution kernel, and the low-resolution image is convolved with the convolution kernel and input into the subsequent super-resolution image reconstruction network for training. The supervised super-resolution image reconstruction network is trained for a total of 500 rounds.

6. A semi-supervised laser microscopy enhancement method according to claim 5, characterized in that: The loss function L used by the unsupervised contrastive learning network is degrad for: Where B is the number of low-resolution images, exp represents the exponential operation, represents the jth patch from the i-th image, N queue represents the number of samples in the queue, represents the jth negative sample, τ is the scaling factor; The loss function expression of the supervised super-resolution image reconstruction network is as follows: THE total =L L1 +λL OTF (4) Among them, L total Represents the total loss of the reconstruction network, L L1 is the mean square error loss, OTF is the discretized optical transfer function, L OTF is the physical loss introduced by the optical transfer function, F represents the Fourier transform, λ is the proportional coefficient, and y SR and y HR represent low-resolution images and high-resolution images respectively, and M is the total number of frequency domain components.

7. The method for enhancing semi-supervised laser microscopy according to claim 5, wherein: In step 4, after network training is completed, a low-resolution image x is input, and a high-resolution image y is output. The formula is as follows: y=G(x) Where G(x) represents the entire network inference process, through which super-resolution reconstruction of low-resolution images can be achieved.

Images