Semi-supervised laser microscopic imaging enhancement method

Abstract

The invention discloses a semi-supervised laser microscopic imaging enhancement method, which comprises the following steps of: constructing a laser microscopic image data set, and carrying out normalization processing on an image; constructing a two-stage deep learning network structure, wherein the two-stage deep learning network structure comprises a first-stage unsupervised degradation representation learning network and a second-stage supervised super-resolution image reconstruction network; training the two-stage deep learning network; and network reasoning is carried out, a low-resolution laser microscopic image is input, a high-resolution image is output, and the super-resolution of laser microscopic imaging is realized. According to the method, an unsupervised comparative learning strategy and a supervised image reconstruction method are combined, feature extraction is performed by fully utilizing unlabeled data, the reconstruction quality is improved through supervised learning, and the generalization ability and the reconstruction precision of the model are effectively enhanced; according to the method, an encoder of a transform (a deep learning model) architecture is introduced to capture complex degradation information from a global perspective, and it is ensured that the model can adapt to various complex degradation.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a semi-supervised laser microscopy imaging enhancement method. Background Art

[0002] As an advanced microscopy imaging method, laser microscopy imaging uses a laser as a light source and scans a sample through an optical system to obtain high-resolution images. Compared with traditional light source imaging systems, laser microscopy imaging can significantly improve image resolution. However, due to factors such as laser energy limitations, sample scattering, detector resolution, and environmental noise, the obtained images are still difficult to fully display the microscopic structure and details of the sample. Therefore, image enhancement processing of laser microscopy imaging images to further improve their resolution is of great significance for the refined research and analysis of samples.

[0003] Currently, the technologies for image super-resolution are relatively mature and can be mainly divided into two categories. One is traditional image interpolation algorithms, such as bilinear interpolation, bicubic interpolation, etc. These methods enhance the resolution by performing pixel interpolation on low-resolution images. The other is deep learning-based methods that use neural network training models to learn the mapping relationship between low-resolution images and high-resolution images to achieve super-resolution reconstruction.

[0004] Traditional interpolation algorithms simply perform interpolation based on adjacent pixels and cannot effectively restore the lost high-frequency information and global information of the image, resulting in blurred image edges and unclear textures. In addition, traditional interpolation methods have weak generalization ability and are difficult to adapt to complex images, especially in scenarios such as laser microscopy imaging that require high detail resolution, where their performance is particularly limited. Although the fully supervised methods based on deep learning are superior to traditional interpolation algorithms in terms of generalization ability, they require a large amount of labeled image data for training, with high data acquisition costs and time-consuming training processes. In addition, these methods often do not consider the optical laws of laser imaging, which makes it difficult for conventional super-resolution technologies to accurately adapt to microscopy imaging tasks and affects the quality and practical application effect of the reconstructed images.

[0005] In view of the above drawbacks, the present invention proposes a semi-supervised laser microscopy imaging enhancement method. Summary of the Invention

[0006] In order to make up for the deficiencies of the prior art, the present invention aims to provide a semi-supervised laser microscopy imaging enhancement method that uses partial labeled data and a large amount of unlabeled data for training, reduces data dependence, and improves the generalization ability of the model; at the same time, in view of the characteristics of laser imaging, an optical transfer function is introduced to further constrain the network, thereby accelerating model convergence and further improving image quality.

[0007] The technical effects achieved by the present invention can be realized through the following specific technical solutions: A semi-supervised laser microscopy imaging enhancement method includes the following steps: Step 1: Construct a laser microscopy image dataset and normalize the images. Step 2: Construct a two-stage deep learning network structure, including an unsupervised degradation representation learning network in the first stage and a supervised super-resolution image reconstruction network in the second stage. Step 3: Train the two-stage deep learning network. Step 4: Perform network inference, input a low-resolution laser microscopy image, and output a high-resolution image to achieve super-resolution of laser microscopy imaging.

[0008] Further, in Step 1, collect the low-resolution images obtained by laser microscopy as the dataset, and divide it into labeled data and unlabeled data; for the labeled data, use its corresponding high-resolution image as the label; then normalize all the images so that their pixel values are distributed in the range of [0, 1].

[0009] Further, in Step 2, for the unsupervised degradation representation learning network, use the method of contrastive learning to train on unlabeled data; in terms of the dataset composition, positive samples are generated by changing different patches in the same picture, and negative samples are obtained by transforming patches in different images. Through contrastive learning, learn the representation methods of various degradation modes to provide guidance for the subsequent reconstruction process; the supervised super-resolution image reconstruction network is composed of several convolutional, pooling, and activation function layers, and its input is the low-resolution image and the degradation representation vector learned in unsupervised learning, and the output is the high-resolution reconstructed image.

[0010] Further, in the unsupervised degradation representation learning network, use an encoder based on the transformer (a deep learning model) architecture. Compared with the traditional convolutional neural network encoder, this structure can capture complex degradations from a global perspective and has stronger generalization ability. First, the encoder extracts features from the input image to generate a degradation representation vector; then, this degradation representation vector passes through two layers of multi-layer perceptron networks to form positive and negative sample vectors for calculating the contrastive learning loss; during the training process of the unsupervised model, positive sample pairs are generated by changing different patches in the same picture, and negative sample pairs are obtained by transforming patches in different images. Through contrastive learning, the unsupervised network can effectively learn the representation methods of various degradation modes to provide guidance for the subsequent reconstruction process.

[0011] Furthermore, in the supervised super-resolution image reconstruction network, the degradation representation vector forms a dynamic convolution kernel after passing through two fully connected layers and a reshaping layer. Then, the dynamic convolution is modulated by the optical transfer function to add optical features, thereby obtaining a convolution kernel that conforms to physical constraints. Finally, the low-resolution image is convolved with the physically constrained convolution kernel and input into the subsequent layer to generate a super-resolution reconstructed image.

[0012] Furthermore, in step 3, the specific content of training the two-stage deep learning network is as follows: Train the unsupervised contrast learning network: Initialize the parameters of the unsupervised contrast learning network. Select two different regions of the same image as the positive sample pair, and select the regions of other low-resolution images as the negative sample pair. Use the Adam (an optimization algorithm) optimizer and the contrast loss function to perform 100 rounds of training on the unsupervised contrast learning network, so that the sample representations with the same degradation are closer, while the representations with different degradations are farther apart. Train the supervised super-resolution image reconstruction network: Initialize the parameters of the supervised super-resolution image reconstruction network. Convert the low-resolution images in the labeled dataset into degradation representation vectors using the trained encoder. The degradation representation vector forms a dynamic convolution kernel through dimensional transformation and optical transfer function modulation. The low-resolution image is convolved with the convolution kernel and then input into the subsequent super-resolution image reconstruction network for training. The supervised super-resolution image reconstruction network is trained for a total of 500 rounds.

[0013] Furthermore, the specific process of modulating the dynamic convolution kernel by the optical transfer function is as follows: ① Physical modeling. For a diffraction-limited system, the optical transfer function OTF can be analytically expressed as: (1) In the formula, f x , f y are the sampling point positions, NA is the numerical aperture, λ is the laser wavelength, circ is the circular domain function, indicating that the response frequency outside the frequency is zero; ② Discretize the continuous optical transfer function into a matrix that matches the size of the dynamic convolution kernel and normalize it to the range [0,1]; ③ Perform Fourier transform on the dynamic convolution kernel W raw to obtain its frequency domain representation W raw (f x , f y ). In the frequency domain, multiply the dynamic convolution kernel W raw element-wise with the discretized optical transfer function matrix to suppress the components beyond the system cut-off frequency: (2) In the formula, W constrained represents the constrained convolution kernel in the frequency domain, Wraw (f x , f y ) is the frequency domain representation of the dynamic convolution kernel, and OTF(f x , f y ) refers to the discretized optical transfer function; ④ Perform inverse Fourier transform on W constrained to obtain the convolution kernel after physical constraint W final .

[0014] Further, the loss function adopted by the unsupervised contrast learning network is: (3) where B is the number of low-resolution images, exp represents the exponential operation, represents the j-th patch from the i-th image, represents the number of samples in the queue, represents the j-th negative sample, is the scaling factor; The expression of the loss function of the supervised super-resolution image reconstruction network is as follows: (4) (5) where L total represents the total loss of the reconstruction network, L L1 is the mean square error loss, OTF is the discretized optical transfer function, L OTF is the physical loss introduced by the optical transfer function, F represents the Fourier transform, λ is the proportionality coefficient, y SR and y HR represent the low-resolution image and the high-resolution image respectively, and M is the total number of frequency domain components.

[0015] Further, in the step 4, after the network training is completed, input the low-resolution image x, then the high-resolution image y is output, and the formula is as follows: where G(x) represents the entire network inference process; Through the above process, super-resolution reconstruction of low-resolution images can be achieved.

[0016] Compared with the prior art, the present invention has the following advantages: (1) Reducing the dependence on large-scale labeled data: The present invention adopts a semi-supervised learning method. During the unsupervised contrast learning process in the first stage, the model can obtain a large amount of degradation representation information. Subsequently, in the second stage, only a small amount of labeled data is required for network training to obtain good generalization ability.

[0017] (2) Adapting to various image degradation types: Traditional super-resolution algorithms usually rely on artificially set degradation methods such as bilinear interpolation or bicubic interpolation, and have poor reconstruction effects on complex degradations. The present invention introduces the encoder of the transformer architecture (a deep learning model) to capture complex degradation information from a global perspective, and combines contrast learning to effectively distinguish different degradation types, ensuring that the model can adapt to various complex degradations.

[0018] (3) Introducing physical constraints to improve image quality: The present invention innovatively uses the optical transfer function as prior information to physically constrain the dynamic convolution kernel. This strategy not only ensures that the restoration result conforms to optical laws, but also accelerates model convergence, and at the same time significantly improves the quality of super-resolution images, making them clearer and more realistic. Description of the Drawings

[0019] Figure 1 is the flowchart of the method of the present invention; Figure 2 is the schematic diagram of the degradation representation learning network structure of the present invention; Figure 3 is the schematic diagram of the super-resolution image reconstruction network structure of the present invention. Specific Embodiments

[0020] In order to make the objectives, technical solutions and advantages of the present invention clearer, the following further details the present invention in combination with the specific application scenario of the wafer microscopic image. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0021] Limited by the imaging principle and hardware conditions, the resolution of laser microscopic imaging images is difficult to meet the growing R & D needs. The application background of the present invention is to break through the limitation of the resolution of laser microscopic imaging through image enhancement technology.

[0022] As Figure 1 shown, a semi-supervised laser microscopic imaging enhancement method includes dataset construction, constructing a two-stage deep learning network structure, training the two-stage deep learning network, and network inference. The specific content is as follows: (I) Hardware Environment The hardware configuration for the deep learning network training in the embodiments of the present invention is as follows: NVIDIA 4070SUPER GPU (12GB video memory), AMD 9700X CPU (6 cores and 12 threads), 32GB of memory; the deep learning framework used is pytorch 2.6.0 and CUDA 12.6.

[0023] (2) Construct an image dataset and normalize the images.

[0024] Collect the low-resolution wafer images obtained by laser microscopy imaging as the dataset. A total of 10,000 low-resolution images (resolution 256×256) are collected, among which 2,000 (20%) have corresponding high-resolution labeled images as the labeled data, and the remaining 8,000 (80%) are used as unlabeled data. Then, all the images are normalized so that their pixel values are distributed within the range of [0,1] to accelerate the network training speed.

[0025] (3) Construct a two-stage deep learning network structure, including an unsupervised degradation representation learning network in the first stage and a supervised super-resolution image reconstruction network in the second stage.

[0026] The unsupervised degradation representation learning network in the first stage is trained on the unlabeled data using the contrastive learning method to extract discriminative degradation representation vectors from the low-resolution images to accurately represent different degradation conditions; the supervised super-resolution image reconstruction network in the second stage uses the degradation representation vectors learned in the first stage and combines the labeled high- and low-resolution image data to achieve adaptive super-resolution of different degraded images, effectively improving the image restoration quality and the model generalization ability.

[0027] As Figure 2 shown in the unsupervised degradation representation learning network, the specific structural process includes: First, the encoder extracts features from the input image to generate a degradation representation vector; subsequently, the degradation representation vector passes through two layers of multi-layer perceptron networks to form positive and negative sample vectors for calculating the contrastive learning loss. This application uses an encoder based on Transformer (including 8 layers of Transformer blocks, each layer containing a multi-head self-attention mechanism and a feed-forward neural network). The input is a 256×256 low-resolution wafer image, and the output is a 128-dimensional degradation representation vector. Two 256×256 regions are randomly cropped from the same wafer image to form a positive sample pair to simulate local differences under the same degradation conditions. A 256×256 region is randomly cropped from different wafer images to form a negative sample pair, covering different defect types or noise levels, representing different degradation modes.

[0028] As Figure 3The supervised super-resolution image reconstruction network is shown as follows. The specific structural process includes: the degradation representation vector generated by the unsupervised degradation representation learning network forms a dynamic convolution kernel after passing through two fully connected layers and a reshaping layer. Then, the dynamic convolution is modulated by the optical transfer function to add optical features, thereby obtaining a convolution kernel that conforms to physical constraints. Finally, the low-resolution image is convolved with the physically constrained convolution kernel and input into the subsequent layer to generate a super-resolution reconstructed image. The supervised super-resolution image reconstruction network includes a dynamic convolution module, a residual convolution block, and an upsampling layer. The input is a low-resolution image (256×256) and a degradation representation vector (128-dimensional), and the output is a high-resolution reconstructed image (1024×1024). The degradation representation vector forms a dynamic convolution kernel through two fully connected layers (with dimensions changing from 128→512→1024) and a reshaping layer, and is combined with the modulation of the optical transfer function (OTF) to ensure compliance with the physical constraints of laser microscopy imaging.

[0029] (4) Train a two-stage deep learning network.

[0030] First, train the unsupervised degradation representation learning network for 100 rounds so that the degradation encoder can learn to distinguish degradation representations. Subsequently, train the super-resolution reconstruction network for 500 rounds to achieve adaptive super-resolution for different degradation methods. The detailed training steps are as follows: (1) Train the unsupervised contrastive learning network First, initialize the network parameters. Select two different regions of the same image as positive sample pairs (with the same degradation method), and select regions of other low-resolution images as negative sample pairs (which may have different degradation methods). Then, use the Adam (an optimization algorithm) optimizer and set the learning rate , and the batch size B = 64. The number of queue samples N queue in the contrastive loss function is 4096, and the temperature parameter τ = 0.1. Train for 100 rounds, and traverse all 8,000 unlabeled data in each round. Make the representation distance of the positive sample pairs of the same wafer image less than 0.2, and the representation distance of the negative sample pairs of different images greater than 0.8.

[0031] (2) Train the supervised super-resolution image reconstruction network First, initialize the network parameters. Select the Adam optimizer and set the learning rate to 5x10 -5 . The batch size B = 32. Convert the low-resolution images in the labeled dataset into degradation representation vectors using the trained encoder. Then, the degradation representation vector forms a convolution kernel through dimensional transformation and optical transfer function modulation. The low-resolution image is convolved with the convolution kernel and input into the subsequent super-resolution image reconstruction network for training. The supervised super-resolution reconstruction network is trained for 500 rounds in total.

[0032] Among them, the specific process of modulating the dynamic convolution kernel by the optical transfer function is as follows: ① Physical modeling. For a diffraction-limited system, the optical transfer function OTF can be analytically expressed as: (1) In the formula, f x , f y are the sampling point positions, NA is the numerical aperture, λ is the laser wavelength, circ is the circular domain function, indicating that the response frequency outside the frequency is zero.

[0033] ② Discretize the continuous optical transfer function into a matrix matching the size of the dynamic convolution kernel and normalize it to the range of [0, 1].

[0034] ③ Perform Fourier transform on the dynamic convolution kernel W raw to obtain its frequency domain representation W raw (f x , f y ). In the frequency domain, multiply W raw element by element with the discretized optical transfer function matrix to suppress the components exceeding the cut-off frequency of the system: (2) In the formula, W constrained represents the constrained convolution kernel in the frequency domain, W raw (f x , f y ) is the frequency domain representation of the dynamic convolution kernel, and OTF(f x , f y ) refers to the discretized optical transfer function.

[0035] ④ Perform inverse Fourier transform on W constrained to obtain the physically constrained convolution kernel W final .

[0036] The loss function is a key factor in network training. The loss function adopted by the unsupervised contrastive learning network of the present invention is: (3) Among them, B is the number of low-resolution images, exp represents the exponential operation, represents the j-th patch from the i-th image, represents the number of samples in the queue, represents the j-th negative sample, is the scaling factor.

[0037] The expression of the loss function of the supervised super-resolution reconstruction network is as follows: (4) (5) Among them, L total represents the total loss of the reconstruction network, L L1 is the mean squared error loss, OTF is the discretized optical transfer function, and L OTF is the physical loss introduced by the optical transfer function, F represents the Fourier transform, λ is the proportionality coefficient, y SR and y HR represent the low-resolution image and the high-resolution image respectively, and M is the total number of frequency domain components.

[0038] During the generation process of the dynamic convolution kernel, the present invention introduces the frequency domain response of the optical transfer function as a prior constraint to ensure that the generated convolution kernel conforms to the physical limit of the actual optical system within the frequency domain range, effectively avoiding the introduction of false high-frequency information or details that violate the diffraction law during the reconstruction process, thereby improving the authenticity and reliability of super-resolution imaging.

[0039] (V) Perform network inference, input the low-resolution laser microscopy image, and output the high-resolution image to achieve super-resolution of laser microscopy imaging.

[0040] After completing the training of the two-stage deep learning network, input the low-resolution image x , then the high-resolution image y is output, that is: (6) In the formula, G(x) represents the entire network inference process.

[0041] Through the above process, super-resolution reconstruction of the low-resolution image can be achieved.

[0042] (VI) Experimental results and analysis Compare the performance of the present invention with traditional methods on test samples, and the results are as follows:

[0043] The method of the present invention is significantly superior to traditional methods in terms of PSNR and SSIM, indicating that the reconstructed image is closer to the real high-resolution wafer image, and the edge and texture details are clearer. Moreover, the present invention can achieve the optimal performance by only using 20% labeled data (2,000 pairs), while traditional fully supervised methods require more than 80% labeled data to approach this effect, verifying the low dependence of semi-supervised learning on labeled data.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semi-supervised laser microscopy imaging enhancement method, characterized in that: The following steps are involved: Step 1: construct a laser microscopy image dataset and normalize the images; Step 2: Construct a two-stage deep learning network structure, including an unsupervised degradation representation learning network in the first stage and a supervised super-resolution image reconstruction network in the second stage; Step 3: Train a two-stage deep learning network; Step 4: Perform network inference, input low-resolution laser microscopy images, output high-resolution images, and achieve laser microscopy super-resolution.

2. A semi-supervised laser microscopy imaging enhancement method according to claim 1, characterized in that: In step 1, low-resolution images obtained by laser microscopy are collected as a data set, which are divided into labeled data and unlabeled data; for the labeled data, the corresponding high-resolution images are used as labels; and then all images are normalized so that their pixel values ​​are distributed in the range of [0, 1].

3. A semi-supervised laser microscopy imaging enhancement method according to claim 1, characterized in that: In step 2, the unsupervised degradation representation learning network is trained on unlabeled data using a contrastive learning method; in terms of data set composition, positive samples are generated by changes in different blocks in the same image, and negative samples are obtained by transforming blocks in different images. Through contrastive learning, characterization methods of multiple degradation modes are learned to provide guidance for subsequent reconstruction processes; the supervised super-resolution image reconstruction network is composed of a number of convolution, pooling and activation function layers, with inputs being low-resolution images and degradation representation vectors learned in unsupervised learning, and outputting high-resolution reconstructed images.

4. A semi-supervised laser microscopy imaging enhancement method according to claim 3, characterized in that: In the unsupervised degradation representation learning network, an encoder based on the transformer architecture is used. First, the encoder extracts features from the input image to generate a degradation representation vector. Then, the degradation representation vector passes through a two-layer multi-layer perceptron network to form positive and negative sample vectors for contrastive learning loss calculation. In the unsupervised model training process, positive sample pairs are generated by changing different blocks in the same image, and negative sample pairs are obtained by transforming blocks in different images. Through contrastive learning, the unsupervised network can effectively learn the representation methods of various degradation patterns and provide guidance for the subsequent reconstruction process.

5. A semi-supervised laser microscopy imaging enhancement method according to claim 4, characterized in that: In the supervised super-resolution image reconstruction network, the degraded representation vector forms a dynamic convolution kernel after passing through two fully connected layers and one reshaping layer; then, the dynamic volume is modulated by the optical transfer function and optical features are added to obtain a convolution kernel that conforms to physical constraints; finally, the low-resolution image is convolved with the convolution kernel after physical constraints and input into the subsequent layers to generate a super-resolution reconstructed image.

6. A semi-supervised laser microscopy imaging enhancement method according to claim 1, characterized in that: In step 3, the specific content of training the two-stage deep learning network is as follows: Train the unsupervised contrastive learning network: Initialize the parameters of the unsupervised contrastive learning network, select two different regions of the same image as positive sample pairs, and select regions of other low-resolution images as negative sample pairs; use the Adam optimizer and contrastive loss function to perform 100 rounds of unsupervised contrastive learning network training, so that the representations of samples with the same degradation are closer, while the representations of samples with different degradation are farther apart; Train the supervised super-resolution image reconstruction network: initialize the parameters of the supervised super-resolution image reconstruction network, convert the low-resolution images in the annotated dataset into degraded representation vectors using the trained encoder; the degraded representation vectors are transformed in dimension and modulated by the optical transfer function to form a dynamic convolution kernel, and the low-resolution image is convolved with the convolution kernel and input into the subsequent super-resolution image reconstruction network for training. The supervised super-resolution image reconstruction network is trained for 500 rounds in total.

7. A method for enhancing semi-supervised laser microscopy imaging according to claim 6, characterized in that: The specific process of optical transfer function modulating the dynamic convolution kernel is as follows: ① Physical modeling: For a diffraction-limited system, the optical transfer function OTF can be analytically expressed as: (1) In the formula, f x 、f y is the sampling point position, NA is the numerical aperture, λ is the laser wavelength, circ is the circular domain function, indicating that the response frequency outside the frequency is zero; ② Discretize the continuous optical transfer function into a matrix that matches the size of the dynamic convolution kernel and normalize it to the range of [0,1]; ③ Dynamic convolution kernel W raw Perform Fourier transform to obtain its frequency domain representation W raw (f x ,f y ), in the frequency domain, the dynamic convolution kernel W raw Multiply element-wise with the discretized optical transfer function matrix to suppress components exceeding the system cutoff frequency: (2) Among them, W constrained represents the constrained convolution kernel in the frequency domain, W raw (f x ,f y ) is the frequency domain representation of the dynamic convolution kernel, OTF(f x ,f y ) means the discretized optical transfer function; ④ For W constrained Perform inverse Fourier transform to obtain the physically constrained convolution kernel W final .

8. A method for enhancing semi-supervised laser microscopy imaging according to claim 6, characterized in that: The loss function adopted by the unsupervised contrastive learning network is for: (3) Where B is the number of low-resolution images, exp represents exponential operation, represents the jth patch from the i-th image, represents the number of samples in the queue, represents the jth negative sample, is the scale factor; The loss function expression of the supervised super-resolution image reconstruction network is as follows: (4) (5) Among them, L total Represents the total loss of the reconstruction network, L L1 is the mean square error loss, OTF is the discretized optical transfer function, L OTF is the physical loss introduced by the optical transfer function, F represents the Fourier transform, λ is the proportional coefficient, and y SR and HR represent low-resolution images and high-resolution images respectively, and M is the total number of frequency domain components.

9. A method for enhancing semi-supervised laser microscopy imaging according to claim 6, characterized in that: In step 4, after the network training is completed, a low-resolution image x is input, and a high-resolution image y is output, and the formula is as follows: In the formula, G(x) represents the entire network reasoning process. Through the above process, super-resolution reconstruction of low-resolution images can be achieved.

Images