Laser confocal ocular surface image splicing method based on deep learning

Abstract

The invention provides a laser confocal ocular surface image stitching method based on deep learning. Efficient and accurate image stitching is realized through unsupervised learning. Firstly, an ocular surface image is preprocessed, and a data set is constructed; then, a multi-scale feature pyramid is constructed by using a convolutional neural network, image features are automatically extracted and pre-aligned, and image alignment is optimized through global motion aggregation and local motion correction technologies. Thirdly, constructing a content fusion generative adversarial network which comprises a generator and a discriminator to smooth the fused image and generate a spliced image conforming to human eye perception; a staged training strategy is adopted, image alignment is optimized firstly, and then image smooth fusion is achieved. And finally, automatically finishing image alignment and fusion by using the trained network to obtain a seamless large-view confocal ocular surface image. According to the method, the splicing precision and robustness are improved, the method is suitable for image splicing in different environments, and the splicing success probability is remarkably improved.

Description

Technical Field

[0001] The present invention relates to an image stitching method, and in particular to a laser confocal ocular surface image stitching method based on deep learning. Background Art

[0002] Demodex is a common parasite in the human body. In the eyes, Demodex mainly parasitizes on eyelashes, eyelash follicles and meibomian glands. In recent years, more and more studies have found that the parasitism of Demodex in the eyes can lead to diseases such as blepharitis and meibomian gland dysfunction. In clinical practice, real-time, non-invasive laser scanning confocal microscopy is often used to examine patients' Demodex infection. However, due to the limitations of the imaging principle, the image field of confocal microscopes is limited, which is not conducive to doctors' timely detection of mites. Image stitching can stitch and fuse multiple small-field images with overlapping areas under similar perspectives into an image with a large field of view. Image stitching technology can increase the field of view of the image without changing the hardware facilities of the confocal microscope, which is conducive to doctors' rapid diagnosis.

[0003] Image stitching technology can be divided into traditional methods and deep learning-based methods. Traditional image stitching methods mainly consist of four parts: feature detection, feature matching, image registration, and image fusion. The SIFT algorithm is one of the most classic feature extraction algorithms, including scale space extreme value detection, key point location, direction assignment, and key point description. These extracted key points maintain stability to image transformations such as rotation, scale scaling, and brightness changes. The AutoStitch algorithm proposed by Brown is a pioneering work in the field of automatic image stitching, which uses the SIFT algorithm for feature detection. In recent years, more and more methods not only use the detected feature points for stitching, but also use other feature elements to improve the quality of stitching. SURF is used in LPC to detect feature points in the image, and LSD is used to detect line segments in the image. By introducing coplanar local sub-region matching and global collinearity structure constraints, stitching artifacts and deformation are effectively reduced. Traditional image stitching methods are simple to implement, but their features are manually designed based on specific assumptions and rules, and are only applicable to specific situations. In addition, the accuracy of image registration is heavily dependent on the quality of feature points or lines detected by the feature detector. When the number of detected feature points is small or the distribution of feature points is uneven, it will cause more matching errors and lead to image stitching failure. In laser confocal ocular surface images, there are many low-light, weak textures, and repeated textures, which poses a great challenge to traditional image stitching methods.

[0004] In recent years, deep learning-based methods have gradually been applied in the field of image stitching. Deep learning-based image stitching methods mainly include supervised and unsupervised methods. The former, such as VFIS-Net and LP-Net, use image pairs and synthetic pseudo-labels to complete network training, but due to the limited ability to process parallax, their performance on real data sets is poor. Unsupervised image stitching networks such as UDIS and REwarp perform well in the stitching of natural images, but use cost volumes in the image registration stage, which is slow in calculation and can only focus on the correlation of local features while ignoring the correlation of global features, limiting the image registration capability. UDIS++ is the current SOTA model on the natural image stitching dataset UDIS-D. It completes image fusion by comparing the brightness difference between the reference image and the target image to find the seam, but this is difficult for confocal ocular surface images because the brightness difference in the overlapping area is very small and the seam cannot be found effectively.

[0005] The examination of ocular Demodex mites using a laser confocal scanning microscope has many advantages such as being real-time and non-invasive. However, it has the problem of a small imaging field of view due to the limitation of the imaging principle. Traditional image stitching methods are likely to fail in stitching because they fail to detect enough feature points on the confocal ocular surface image or because the detected feature points are unevenly distributed.

[0006] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0007] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and provide a laser confocal ocular surface image stitching method based on deep learning.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A laser confocal ocular surface image stitching method based on deep learning, comprising the following steps:

[0010] S1. Preprocess the ocular surface images and construct a dataset for stitching;

[0011] S2. construct an unsupervised image alignment network, wherein the unsupervised image alignment network automatically extracts image features by constructing a multi-scale feature pyramid through a convolutional neural network to achieve pre-alignment of images; calculates a homography matrix based on a deep feature map through global motion aggregation technology to estimate the global transformation between images; further, through local motion correction technology, TPS interpolation is used to locally correct the image to optimize the alignment between images;

[0012] S3. construct an unsupervised image fusion network, which is designed as a content fusion generative adversarial network, and includes a generator and a discriminator; the generator is responsible for learning to generate a fusion mask to smooth the fused image, and the discriminator is used to guide the generator to obtain a fusion probability map to learn to smoothly fuse the images to be spliced;

[0013] S4. A phased training strategy is adopted, where the unsupervised image alignment network is first trained to optimize image alignment, and then the unsupervised image fusion network is trained to achieve smooth image fusion.

[0014] S5. Use the trained unsupervised image alignment network and unsupervised image fusion network to automatically complete the alignment and fusion of the images to be stitched, and obtain a seamless large-field confocal ocular surface image.

[0015] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the laser confocal ocular surface image stitching method based on deep learning is implemented.

[0016] The present invention has the following beneficial effects:

[0017] The present invention provides a laser confocal ocular surface image stitching method based on deep learning, which applies deep learning technology to the stitching of laser confocal ocular surface images, and realizes efficient and accurate image stitching by constructing an unsupervised image alignment network and a content fusion network. The image alignment network uses a convolutional neural network to automatically extract image features, and uses a multi-scale feature pyramid for pre-alignment. Through global motion aggregation and local motion correction technology, the alignment between images is optimized, especially accurate alignment is achieved in overlapping areas, while maintaining small distortion in non-overlapping areas. In addition, the content fusion network adopts the design of a generative adversarial network, and realizes smooth fusion of images by learning to generate fusion masks, avoiding obvious boundaries and artifacts, thereby obtaining a seamless and smooth stitching image that conforms to human eye perception. The model of the present invention not only improves the quality of the stitching image, but also enhances the robustness of the model, so that it can adapt to images collected by devices in different environments, and significantly improves the probability of successful stitching. Compared with the traditional image stitching method, the present invention shows stronger adaptability and superiority when processing laser confocal ocular surface images with low light, weak texture and repeated texture.

[0018] The unsupervised eye confocal image stitching model designed by the present invention can realize the automatic stitching of a set of laser confocal ocular surface images, automatically extract robust features from the images, and improve the stitching effect and success rate. Among them, the registered image is obtained through the image alignment network, which realizes precise alignment in the overlapping area and maintains small distortion in the non-overlapping area. The content fusion network realizes efficient and smooth fusion, and obtains the laser confocal ocular surface stitching image that conforms to the perception of the human eye.

[0019] The stitched image obtained by the present invention has higher accuracy in the overlapping area and smaller distortion in the non-overlapping area, and can obtain a seamless and smooth stitched image that is more in line with human eye perception; the present invention has strong robustness and can achieve stitching for laser confocal ocular surface images collected by equipment in different environments, which greatly improves the probability of successful stitching compared to traditional image stitching methods.

[0020] The image pre-alignment network mainly consists of three parts: multi-scale feature extraction, global motion aggregation, and local motion correction. The multi-scale feature extraction part uses a convolutional neural network to construct a multi-scale feature pyramid, and follows the coarse to fine strategy to pre-align images at different scales. Since the deep layer of the convolutional neural network extracts high-level semantic information in the image, the global motion aggregation takes the deep feature map of the feature pyramid as input, and estimates the global change by regressing the four-point offset to calculate the homography matrix. Based on the global motion, the local motion correction uses the shallow features in the feature pyramid as input to regress the information of the grid motion, and further uses TPS interpolation for local correction to achieve better alignment in the overlapping area. Using TPS interpolation for alignment may cause excessive deformation of the image, and a grid loss is further designed to constrain it. The pre-aligned image and its content mask and boundary mask can be obtained through the image alignment network, and it is input into the content fusion network to obtain the fusion mask to achieve smooth fusion and avoid obvious boundaries and artifacts.

[0021] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the method for automatically stitching laser confocal ocular surface images according to an embodiment of the present invention.

[0023] Figure 2 It is a process of constructing a laser confocal image stitching model data set and effect diagrams of different image contrast enhancement algorithms according to an embodiment of the present invention.

[0024] Figure 3 is a schematic diagram of a stitching model according to an embodiment of the present invention, which is mainly composed of an image alignment network and a content fusion network.

[0025] Figure 4 Schematic diagram of the structure of the image alignment network according to an embodiment of the present invention.

[0026] Figure 5 It is a schematic diagram of the structure of the content fusion network according to an embodiment of the present invention.

[0027] Figure 6 It is the core function structure of the space transformation network STN of the embodiment of the present invention.

[0028] Figure 7 This is an example diagram of laser confocal ocular surface image stitching according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0030] The present invention proposes a laser confocal ocular surface image stitching method based on deep learning, including the steps of laser confocal ocular surface image preprocessing, data set construction, image stitching model training and the like, so as to input the small-field confocal ocular surface image to be stitched into the pre-trained image stitching model and return the stitched large-field confocal ocular surface image.

[0031] See also Figure 1 The embodiment of the present invention provides a laser confocal ocular surface image stitching method based on deep learning, comprising the following steps:

[0032] Step S1. Preprocess the ocular surface image and construct a dataset for stitching.

[0033] In some embodiments, in step S1, the preprocessing of the ocular surface image specifically includes: cutting useless information from the ocular surface image, and using the Laplacian contrast enhancement algorithm to enhance contrast. The construction of the data set for stitching specifically includes: selecting the image with the current frame number i and the subsequent multiple frames in the continuous shooting process for screening; using the ORB operator to detect feature points for the images with frame numbers i and j, j=i+1; using the RANSAC algorithm to filter out outliers and perform feature point matching; adding image pairs with feature point matching numbers higher than a preset threshold to the data set, and filtering out image pairs below the threshold; and manually checking the screened image pairs to filter out low-quality image pairs.

[0034] Step S2. Construct an unsupervised image alignment network, which automatically extracts image features by constructing a multi-scale feature pyramid through a convolutional neural network to achieve pre-alignment of images. Through global motion aggregation technology, the homography matrix is ​​calculated based on the deep feature map to estimate the global transformation between images; further, through local motion correction technology, TPS interpolation is used to locally correct the image and optimize the alignment between images.

[0035] In some embodiments, in step S2, the method of automatically extracting image features by constructing a multi-scale feature pyramid through a convolutional neural network to achieve image pre-alignment specifically includes: using a convolutional neural network with the same structure and shared weights as a multi-scale feature extractor to automatically extract robust features in the confocal image; using a VGG16 network as a feature extractor, and replacing the maximum pooling layer in VGG16 with a BlurPool layer, wherein the BlurPool layer includes two parts: maximum dense sampling with a step size of 1 and low-pass filtering downsampling with a step size of 2, and low-pass filtering is performed through a filter with a Gaussian kernel; using a coarse to fine strategy to capture the correspondence between image features at different scales, and extracting feature maps of 1 / 8 size and 1 / 16 size for subsequent local motion correction and global motion aggregation calculations, wherein the feature map of 1 / 16 size is used to extract global motion information, and the feature map of 1 / 8 size is used to extract local motion information.

[0036] In some embodiments, in step S2, the global motion aggregation technology is used to calculate the homography matrix based on the deep feature map to estimate the global transformation between images, specifically including: using a homography matrix with eight degrees of freedom to establish a global mapping relationship between two images, and realizing the geometric correspondence between images through the homography matrix; using a deep feature map, that is, a 1 / 16 size feature map, to extract global motion information, and using the rich semantic information contained in the deep feature map of the convolutional neural network to estimate the global transformation between images; inputting the 1 / 16 size feature map obtained by the feature extractor of the reference image and the target image into the correlation calculation module for processing to obtain a global displacement stream; inputting the obtained global displacement stream into the global motion prediction network, completing the extraction of global motion through the network, thereby calculating the homography matrix and realizing global alignment between images.

[0037] In some embodiments, in step S2, the local motion correction technology is used to locally correct the image using TPS interpolation to optimize the alignment between images, specifically including: based on global motion aggregation, using a 1 / 8 size feature map to extract local motion information to achieve more precise alignment; using the TPS flexible interpolation method to find a smooth surface passing through the control point grid by minimizing the energy function to achieve deformation correction in the local area; wherein, a deformation grid is defined using TPS interpolation, which covers the image alignment area and is used to control the details of the image deformation; defining a TPS interpolation function, including a first-order polynomial and a radial basis function for capturing local trends, and calculating the Euclidean distance from the interpolation point to the control point; in order to ensure the uniqueness of the TPS interpolation solution, setting constraints to ensure that the control points on the reference image and the target image coincide, and adding additional constraints to avoid ambiguity in the solution; designing a local motion corrector, using STN to distort the target image, and then comparing it with the feature map of the reference image to calculate the local displacement flow; inputting the local displacement flow into the TPS motion prediction network to complete the extraction of local motion information and further optimize the alignment between images.

[0038] Step S3. Construct an unsupervised image fusion network, which is designed as a content fusion generative adversarial network, including a generator and a discriminator; the generator is responsible for learning to generate a fusion mask to smooth the fused image, and the discriminator is used to guide the generator to obtain a fusion probability map to learn to smoothly fuse the images to be spliced.

[0039] In some embodiments, in step S3, the construction of the unsupervised image fusion network specifically includes: designing a generator, which uses a twin network with the same structure and shared weights to perform multi-scale downsampling in the encoding stage to capture the difference between the reference image and the target image; performing a difference operation on the last layer of feature maps at each scale to obtain residual feature maps at different scales to enhance the difference in feature representation; in the decoding stage, using the residual feature maps of adjacent scales to input into the upsampling layer, and introducing jump connections to generate decoded feature maps at different scales; through upsampling and splicing operations, using the decoded multi-scale feature maps, generating a fusion probability map of the distorted reference image in the overlapping area; using the fusion probability map as a fusion mask, multiplying the distorted versions of the reference image and the target image with the corresponding fusion mask, and then adding them to form the final splicing result; designing a discriminator, which performs quality judgment on the generated fused image through a combination of convolution, normalization and activation functions, as well as adaptive average pooling and linear layers, to guide the training and optimization of the generator.

[0040] Step S4. A phased training strategy is adopted to first train the unsupervised image alignment network to optimize image alignment, and then train the unsupervised image fusion network to achieve smooth image fusion.

[0041] In some embodiments, in step S4, the staged training strategy specifically includes: performing two-stage training on the unsupervised image alignment network: in the first stage, only multi-scale feature extraction and global motion aggregation are trained, and the loss function based on the homography matrix is ​​used for optimization; in the second stage, combined with local motion correction, the loss function based on TPS interpolation is used for optimization, and the grid loss is introduced to constrain excessive deformation; in the training of the unsupervised image fusion network, when training the generator, a combination of boundary loss, adversarial loss and balance loss is used to generate a smooth and natural fused image; when training the discriminator, a binary cross entropy loss function is used to distinguish the generated fused image from the real fused image.

[0042] In a preferred embodiment, the method for calculating the grid loss specifically includes: calculating the horizontal and vertical edge vectors of the deformed grid generated by the network; cross-multiplying the edge vector of each grid with the corresponding unit vector to obtain a sine value to evaluate the alignment of the deformed grid; by comparing the sine values ​​of adjacent grids, constraining the deformation differences between grids to avoid discontinuities in the splicing area; comprehensively considering the deformation of the grid in the horizontal and vertical directions, and incorporating the grid loss into the total loss function by means of a weighted sum to optimize the effect of local motion correction.

[0043] Step S5. Using the trained unsupervised image alignment network and unsupervised image fusion network, the alignment and fusion of the images to be stitched are automatically completed to obtain a seamless large-field confocal ocular surface image.

[0044] The present invention innovatively applies deep learning technology to laser confocal ocular surface image stitching, and significantly improves the accuracy and success probability of stitching by constructing an unsupervised image alignment network and content fusion network. The image alignment network uses multi-scale feature extraction and convolutional neural network to automatically extract image features, realize image pre-alignment, and calculate the homography matrix through global motion aggregation technology, estimate the global transformation between images, and use local motion correction technology to perform local correction on the image using TPS interpolation to optimize the alignment between images, especially to achieve accurate alignment in overlapping areas, while maintaining small distortion in non-overlapping areas. The content fusion network adopts the design of generative adversarial network, and generates fusion masks and smooth fusion images through generator learning. The discriminator guides the generator to obtain a fusion probability map, learns to smoothly fuse the images to be stitched, avoids obvious boundaries and artifacts, and obtains seamless and smooth stitched images that meet human eye perception. The model of the present invention not only improves the quality of the stitched image, but also enhances the robustness of the model, so that it can adapt to images collected by devices in different environments, and significantly improves the probability of successful stitching. Compared with traditional image stitching methods, the present invention shows stronger adaptability and superiority when processing laser confocal ocular surface images with low light, weak texture and repeated texture, especially in automatic stitching, accuracy maintenance and robustness.

[0045] The following further describes specific embodiments of the present invention and examples of algorithm implementation thereof.

[0046] Figure 1 The present invention is a flowchart of the method for automatically stitching laser confocal ocular surface images, which mainly includes the steps of shooting laser confocal ocular surface images, image preprocessing, data set construction, image stitching model training, and automatic image stitching.

[0047] A laser confocal ocular surface image stitching method based on deep learning, the designed laser confocal ocular surface image stitching model based on deep learning is mainly composed of two parts: an image alignment network and a content fusion network. The image pre-alignment network is mainly composed of three parts: multi-scale feature extraction, global motion aggregation, and local motion correction. The multi-scale feature extraction part uses a convolutional neural network to construct a multi-scale feature pyramid, and follows the coarse to fine strategy to pre-align images at different scales. Since the deep layer of the convolutional neural network extracts high-level semantic information in the image, the global motion aggregation uses the deep feature map of the feature pyramid as input, and calculates the homography matrix by regressing the four-point offset to estimate the global change. Local motion correction regresses the shallow features in the feature pyramid as input on the basis of global motion to obtain the information of grid motion, and further uses TPS interpolation for local correction to achieve better alignment in overlapping areas. Using TPS interpolation for alignment may cause excessive deformation of the image, so the preferred embodiment designs a grid loss for constraint. The pre-aligned image and its content mask and boundary mask can be obtained through the image alignment network, which are input into the content fusion network to obtain the fusion mask and then achieve smooth fusion to avoid obvious boundaries and artifacts. The content fusion network of the present invention is essentially a generative adversarial network.

[0048] Confocal ocular surface image data preprocessing and dataset construction

[0049] The specific construction process of constructing the confocal ocular surface image mosaic dataset is as follows:

[0050] a. Source of confocal ocular surface images. Continuous shooting was performed using the Sequence Scan method of a laser confocal microscope, and a total of 2863 confocal ocular surface images were obtained, where the sequence number of the image represents the time frame position of the image during the continuous shooting process;

[0051] b. Cropping and contrast enhancement. After cropping the useless information of the image, its size is changed to 384×384. Since the confocal image obtained has poor detail texture, it is contrast enhanced. It is found that the Laplacian contrast enhancement algorithm can better highlight the details and retain the original structure compared with Gamma transformation, histogram equalization and logarithmic transformation. Compared with limited contrast adaptive histogram equalization (CLAHE), it avoids redundant detail texture information. Therefore, the Laplacian contrast enhancement algorithm is used as the final contrast enhancement algorithm;

[0052] c. Image screening between adjacent frames. Since images with frame numbers far apart may be located in different focal planes during the shooting process and cannot be used for stitching, the images of the current frame number and the next 7 frames are screened. The frame number of the current image is recorded as i, and the frame number of the next frame image is recorded as j, that is, j=i+1. Then, the ORB operator is used to detect feature points for images with frame numbers i and j, and RANSAC is used to filter out outliers and match them. Then, image pairs with feature point matching numbers higher than the threshold are added to the data set, and image pairs with matching numbers lower than the threshold are filtered out. Finally, the filtered image pairs are manually inspected to filter out low-quality image pairs.

[0053] Figure 2 The process of constructing the laser confocal image stitching model data set of the present invention and the effect diagram of different image contrast enhancement algorithms are shown.

[0054] Figure 3 The stitching model of the embodiment of the present invention is shown, which is mainly composed of an image alignment network and a content fusion network, wherein the registered reference image and target image are obtained through the image alignment network, and the fusion mask is obtained through the content fusion network to smoothly fuse the images.

[0055] Unsupervised image alignment network construction

[0056] Figure 4 The structure of the image alignment network according to an embodiment of the present invention is shown.

[0057] The image alignment network mainly consists of three parts: multi-scale feature extraction, global motion aggregation, and local motion correction. The multi-scale feature extractor uses a convolutional neural network with the same structure and shared weights to automatically extract robust features in confocal images, avoiding the limitations of traditional manual features. The classic VGG16 was selected as the feature extractor. After contrast enhancement of confocal images, the high-frequency information in the image increases, and the maximum pooling may cause aliasing effect during downsampling due to the sampling frequency being lower than twice the Nyquist frequency. Therefore, the maximum pooling in VGG16 is replaced by BlurPool. Different from the maximum pooling downsampling with a step size of 2, BlurPool contains two parts: maximum dense sampling with a step size of 1 and low-pass filtering downsampling with a step size of 2. Low-pass filtering with a filter with a Gaussian kernel can alleviate the aliasing effect caused by the maximum pooling downsampling to a certain extent.

[0058] In order to improve the alignment accuracy of the network, the correspondence between image features at different scales is captured, and the image alignment is performed at different scales following the coarse to fine strategy. The multi-scale feature extractor extracts feature maps of 1 / 8 and 1 / 16 sizes for subsequent local motion correction and global motion aggregation calculations. Specifically, since the deep feature maps of the convolutional neural network contain relatively rich semantic information but have low resolution, while the shallow feature maps contain relatively rich edge, texture and other information and have high resolution, the 1 / 16 feature map is used to extract global motion information, and the 1 / 8 feature map is used to extract local motion information, and the global motion information is further corrected.

[0059] First, a homography matrix with eight degrees of freedom is used to obtain the global mapping relationship between the two images, namely:

[0060]

[0061] Since the eight parameters h1 to h8 in the homography matrix are coupled by translation and shear, it is difficult for the network to learn these eight parameters directly. Therefore, they are decoupled. Instead of learning these eight parameters directly, the offsets of the four vertices of the image are learned, and then the homography matrix is ​​solved by the direct linear transformation algorithm. The two 1 / 16 feature maps obtained by the feature extractor of the reference image and the target image are input into NMS-FDL for correlation calculation to obtain the global displacement flow, and the obtained global displacement flow is input into the Homo Motion Prediction Network to complete the extraction of global motion.

[0062] Since the homography matrix is ​​a rigid transformation, we hope that the network can further align the reference image and the target image, so TPS flexible interpolation is used to further improve the alignment accuracy of the network. TPS interpolation finds a smooth surface through the control point grid by minimizing the energy function. The TPS interpolation function in two-dimensional space is:

[0063]

[0064] Where P(x R ,y R ) is a first-order polynomial used to capture the global trend, n is the number of control points, ω i is the unknown coefficient, U(r) is the radial basis function, usually U(r) = r 2 logr,||P i -P|| is the Euclidean distance from the interpolation point to the control point.

[0065] In order to ensure the uniqueness of the solution, in addition to making the control points on the reference image and the target image coincide, three constraints need to be added, namely:

[0066]

[0067] Therefore, the matrix representation of the TPS interpolation solution process is:

[0068]

[0069] Where K n×n is the matrix determined by the control points, P n×3 is the matrix composed of the coordinates of the control points on the reference image, that is is the matrix composed of the coordinates of the control points on the target image, that is

[0070] According to the principle of TPS interpolation, the displacement of the control point needs to be obtained, so a local motion corrector is designed. First, the 1 / 8 feature map of the target image is distorted using STN according to the output of global motion aggregation, and then it and the feature map of the reference image are input into NMS-FDL to calculate the local displacement stream, and then the local displacement stream is input into TPS Motion Prediction Network to complete the extraction of local motion information.

[0071] Figure 6 The core function structure of the spatial transformation network STN in the embodiment of the present invention is shown, on the basis of which the original Localization Network is cancelled, and the image is deformed according to the homography matrix or TPS interpolation matrix obtained by regression.

[0072] Unsupervised image fusion network construction

[0073] Figure 5 The structure of the content fusion network according to the embodiment of the present invention is shown.

[0074] After completing the initial alignment of the images, if the two images are directly fused in the overlapping area, a relatively obvious fusion boundary will appear, and due to the performance limitations of the image alignment network, artifacts will also appear in the overlapping area. A content fusion generative adversarial network is designed, and the discriminator is used to guide the generator to obtain the fusion probability map, and learn to fuse the images to be spliced ​​as smoothly as possible.

[0075] The generator uses a twin network with the same structure and shared weights for multi-scale downsampling in the encoding stage. In order to better capture the difference between the reference image and the target image, the last layer of feature maps at each scale is subtracted to obtain residual feature maps at different scales. Then decoding is performed. In order to avoid the coarse granularity of the generated content fusion mask due to the large scale span, the feature map obtained by decoding is not input into the upsampling layer like U-Net in the decoding stage. Instead, the residual feature map of the adjacent scale is input into the upsampling layer, and a jump connection is introduced. The decoded feature maps at different scales are output through a single-channel 3×3 convolution. After obtaining the decoded feature maps at different scales, they are upsampled and spliced. Finally, a single-channel 1×1 convolution and sigmoid activation function are used to obtain the fusion probability map of the distorted reference image in the overlapping area. The fusion probability map of the distorted fused image in the overlapping area can be obtained through simple post-processing. The final spliced ​​image is:

[0076]

[0077] The backbone of the discriminator is composed of N stacked blocks of Conv+BN+RELU, and then outputs the discrimination result through adaptive average pooling and two linear layers.

[0078] Unsupervised image alignment network and unsupervised image fusion network training

[0079] Different loss functions are used for the output of the image alignment network at different scales. For global motion aggregation, the homography matrix H is obtained based on the output of the network and the direct linear transformation algorithm is used, and then the target image is deformed using STN. The loss function is:

[0080] Loss H =|Image R ×Warp(Ones,H)-Warp(Image T ,H)|

[0081] Ones represents the same R Matrices of the same size and all 1. For the extraction of local motion information, the TPS interpolation weights are obtained according to the displacement of the control points, and then transformed using STN. The loss function is:

[0082]

[0083] However, TPS interpolation may produce large deformations, so a grid loss is designed to constrain this deformation. The sine value of the network horizontal edge vector and the horizontal unit vector is calculated according to the following formula:

[0084]

[0085] Where <,> represents the cross product, represents the horizontal edge vector of the i-th grid, represents the horizontal unit vector. In order to constrain the deformation inside the grid, we hope The value of is small. In order to prevent large deformation between adjacent grids, we hope The deformation constraint in the vertical direction of the grid is similar to that in the horizontal direction, and the total loss of local motion correction is:

[0086]

[0087] For the image alignment network, we initially trained global motion aggregation and local motion correction together, but the effect was not very good. Therefore, we adopted a two-stage training. In the first stage, we freeze the local motion correction and only train multi-scale feature extraction and global motion aggregation. H As the loss function, the second stage unfreezes the local motion correction, which is trained together with the feature extraction and global motion aggregation, with Loss TPS As the loss function, the model training was implemented on a Nvidia 2080Ti.

[0088] The content mask ContentMask is obtained by using the same distortion method as the image for the white mask with all 1s, and then the boundary mask Boundary Mask is obtained by dilation operation. When the generator obtains the fusion mask, it is hoped that the distorted reference image and the target image will not change significantly at the fusion edge, so the illumination loss of the spliced ​​image at the boundary and the distorted image at the boundary is constrained. Here, the 1 norm is used, and the boundary loss is defined as:

[0089]

[0090] In order to encourage the generator to generate smooth spliced ​​images as much as possible to confuse the discriminator, a binary cross entropy loss is used to quantify the difference between the predicted value and the true label, so the adversarial loss of the generator is:

[0091]

[0092] in Denotes the binary cross entropy loss function. In order to avoid the generated fusion mask being completely dominated by the reference image or the target image in the overlapping area, a balanced loss is designed:

[0093]

[0094] Therefore, the total loss of the generator is:

[0095]

[0096] The adversarial loss of the discriminator is symmetric to that of the generator. The loss of the discriminator is:

[0097] Loss D

[0098] The content fusion generative adversarial network is trained on images processed by the image alignment network. To avoid inconsistent image sizes within each batch due to distorted registration, the batch size is set to 1. Model training is implemented on an Nvidia 2080Ti.

[0099] Laser confocal ocular surface image stitching

[0100] After completing the training of the image alignment network and the image fusion network, the Image_Alignment_Network.pth file and the Content_Composition_GAN.pth file are obtained. Then the trained model is used for inference. First, Image_Alignment_Network.pth is loaded as the pre-trained weight of the image alignment network, and then the image to be spliced ​​is input into the image alignment network to obtain the displacement of the TPS interpolation control point, and the image is warped and registered according to the displacement of the control point and the interpolation equation. Then the warped reference image and target image as well as their content masks and boundary masks are input into the generator that has loaded Content_Composition_GAN.pth as the pre-trained weight to obtain the fusion mask of the reference image and the target image. The reference image and the target image are weighted by the fusion mask to obtain the final spliced ​​image. The above process can be automatically implemented in a Python script.

[0101] Figure 7 This is an example of the stitching of laser confocal ocular surface images by the present invention, which shows the stitching results of laser confocal images of eye eyelashes and eyelash follicles. The upper row is the input image pair to be stitched, and the lower row is the stitching effect diagram of different methods. It can be seen that the SIFT algorithm failed to align well in the image registration stage, so obvious fusion artifacts appeared at the red arrow in the figure. The REW algorithm is slightly misaligned at the position of the red arrow in the figure, and the final stitched image has a clear fusion boundary. UDIS++ has obvious misalignment during alignment, and due to the limited ability to capture the difference in features between the reference image and the target image, obvious fusion boundaries can be seen in the red box area and the green box area. The method of the present invention does not have obvious misalignment and fusion artifacts in the red box area, and also maintains a high degree of clarity in the green box, achieving smooth fusion and avoiding obvious fusion boundaries.

[0102] The present invention has the following advantages and outstanding effects:

[0103] ① The present invention can realize automatic stitching of a group of laser confocal ocular surface images; ② The stitched image obtained by the present invention has higher accuracy in the overlapping area and smaller distortion in the non-overlapping area, and can obtain a seamless and smooth stitched image that is more in line with human eye perception; ③ The present invention has strong robustness, and can realize stitching of laser confocal ocular surface images collected by equipment in different environments, which greatly improves the probability of successful stitching compared with traditional image stitching methods.

[0104] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0105] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0106] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0107] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0108] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0109] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0110] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0111] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0112] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0113] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0114] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0115] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0116] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A laser confocal ocular surface image stitching method based on deep learning, characterized in that: The following steps are involved: S1. Preprocess the ocular surface images and construct a dataset for stitching; S2. construct an unsupervised image alignment network, wherein the unsupervised image alignment network automatically extracts image features by constructing a multi-scale feature pyramid through a convolutional neural network to achieve pre-alignment of images; calculates a homography matrix based on a deep feature map through global motion aggregation technology to estimate the global transformation between images; further, through local motion correction technology, TPS interpolation is used to locally correct the image to optimize the alignment between images; S3. construct an unsupervised image fusion network, which is designed as a content fusion generative adversarial network, and includes a generator and a discriminator; the generator is responsible for learning to generate a fusion mask to smooth the fused image, and the discriminator is used to guide the generator to obtain a fusion probability map to learn to smoothly fuse the images to be spliced; S4. A phased training strategy is adopted, where the unsupervised image alignment network is first trained to optimize image alignment, and then the unsupervised image fusion network is trained to achieve smooth image fusion. S5. Use the trained unsupervised image alignment network and unsupervised image fusion network to automatically complete the alignment and fusion of the images to be stitched, and obtain a seamless large-field confocal ocular surface image.

2. The laser confocal ocular surface image stitching method based on deep learning according to claim 1, characterized in that: In step S1, the preprocessing of the ocular surface image specifically includes: cutting out useless information from the ocular surface image, and performing contrast enhancement using a Laplacian contrast enhancement algorithm.

3. The laser confocal ocular surface image stitching method based on deep learning according to claim 1 or 2, characterized in that: In step S1, the step of constructing a data set for splicing specifically includes: Select the image with the current frame number i and the subsequent multiple frames of images in the continuous shooting process for screening; The ORB operator is used to detect feature points for images with frame numbers i and j, j = i + 1; Use the RANSAC algorithm to filter out outliers and perform feature point matching; Image pairs with feature point matching numbers above a preset threshold are added to the data set, and image pairs below the threshold are filtered out; The screened image pairs are manually inspected to filter out low-quality ones.

4. The laser confocal ocular surface image stitching method based on deep learning according to any one of claims 1 to 3, characterized in that: In step S2, the method of automatically extracting image features by constructing a multi-scale feature pyramid through a convolutional neural network to achieve image pre-alignment specifically includes: A convolutional neural network with the same structure and shared weights is used as a multi-scale feature extractor to automatically extract robust features from confocal images. Use the VGG16 network as the feature extractor, and replace the maximum pooling layer in VGG16 with a BlurPool layer, which contains two parts: maximum dense sampling with a step size of 1 and low-pass filtering downsampling with a step size of 2, and low-pass filtering is performed through a filter with a Gaussian kernel; Using the coarse to fine strategy, the correspondence between image features at different scales is captured, and feature maps of 1 / 8 and 1 / 16 sizes are extracted for subsequent calculation of local motion correction and global motion aggregation. The feature map of 1 / 16 size is used to extract global motion information, and the feature map of 1 / 8 size is used to extract local motion information.

5. The laser confocal ocular surface image stitching method based on deep learning according to any one of claims 1 to 4, characterized in that: In step S2, the global motion aggregation technique is used to calculate the homography matrix based on the deep feature map to estimate the global transformation between images, specifically including: A homography matrix with eight degrees of freedom is used to establish a global mapping relationship between two images, and the geometric correspondence between the images is achieved through the homography matrix; A 1 / 16 size deep feature map is used to extract global motion information, and the rich semantic information contained in the deep feature map of the convolutional neural network is used to estimate the global transformation between images. The 1 / 16 size feature map obtained by the feature extractor of the reference image and the target image is input into the correlation calculation module for processing to obtain the global displacement flow; The obtained global displacement flow is input into the global motion prediction network, through which the global motion is extracted, and the homography matrix is ​​calculated to achieve global alignment between images.

6. The laser confocal ocular surface image stitching method based on deep learning according to any one of claims 1 to 5, characterized in that: In step S2, the local motion correction technology is used to locally correct the image using TPS interpolation to optimize the alignment between images, specifically including: Based on global motion aggregation, a 1 / 8 size feature map is used to extract local motion information to achieve finer alignment. The TPS flexible interpolation method is used to find a smooth surface passing through the control point grid by minimizing the energy function to achieve deformation correction in the local area. A deformation grid is defined by TPS interpolation, which covers the image alignment area and is used to control the details of image deformation. Define TPS interpolation functions, including first-order polynomials and radial basis functions for capturing local trends, and calculate the Euclidean distance from interpolation points to control points; To ensure the uniqueness of the TPS interpolation solution, constraints are set to ensure that the control points on the reference image and the target image coincide, and additional constraints are added to avoid ambiguity in the solution; Design a local motion corrector, use STN to distort the target image, then compare it with the feature map of the reference image to calculate the local displacement flow; The local displacement stream is input into the TPS motion prediction network to complete the extraction of local motion information and further optimize the alignment between images.

7. The laser confocal ocular surface image stitching method based on deep learning according to any one of claims 1 to 6, characterized in that: In step S3, the construction of the unsupervised image fusion network specifically includes: Design a generator that uses a Siamese network with the same structure and shared weights to perform multi-scale downsampling in the encoding phase to capture the differences between the reference image and the target image; Perform a difference operation on the last layer feature map at each scale to obtain residual feature maps at different scales to enhance the difference of feature representation; In the decoding stage, the residual feature maps of adjacent scales are input into the upsampling layer, and skip connections are introduced to generate feature maps decoded at different scales; Through upsampling and splicing operations, the decoded multi-scale feature map is used to generate the fusion probability map of the distorted reference image in the overlapping area; Using the fusion probability map as a fusion mask, the distorted versions of the reference image and the target image are multiplied with the corresponding fusion masks and then added to form the final stitching result; A discriminator is designed to judge the quality of the generated fused image through a combination of convolution, normalization and activation functions, as well as adaptive average pooling and linear layers, to guide the training and optimization of the generator.

8. The laser confocal ocular surface image stitching method based on deep learning according to any one of claims 1 to 7, characterized in that: In step S4, the phased training strategy specifically includes: The unsupervised image alignment network is trained in two stages: in the first stage, only multi-scale feature extraction and global motion aggregation are trained, and the loss function based on the homography matrix is ​​used for optimization; in the second stage, local motion correction is combined, and the loss function based on TPS interpolation is used for optimization, and the grid loss is introduced to constrain excessive deformation; In the training of the unsupervised image fusion network, a combination of boundary loss, adversarial loss and balanced loss is used when training the generator to generate smooth and natural fused images; when training the discriminator, a binary cross entropy loss function is used to distinguish the generated fused images from the real fused images.

9. The laser confocal ocular surface image stitching method based on deep learning as claimed in claim 8, characterized in that: The calculation method of the grid loss specifically includes: Compute the horizontal and vertical edge vectors of the deformed mesh generated by the network; The edge vector of each mesh is cross-multiplied with the corresponding unit vector to obtain the sine value to evaluate the alignment of the deformed mesh; By comparing the sine values ​​of adjacent meshes, the deformation differences between meshes are constrained to avoid discontinuities in the splicing area; The grid loss is incorporated into the total loss function by comprehensively considering the horizontal and vertical deformations of the grid through a weighted sum method to optimize the effect of local motion correction.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the deep learning-based laser confocal ocular surface image stitching method as described in any one of claims 1 to 9 is implemented.

Images