A deep learning-based laser confocal ocular surface image splicing method

Abstract

The application provides a kind of laser confocal ocular surface image splicing method based on deep learning, realizes efficient, accurate image splicing by unsupervised learning. First, the ocular surface image is preprocessed and a data set is constructed. Then, a multi-scale feature pyramid is constructed using a convolutional neural network, image features are automatically extracted and pre-aligned, and image alignment is optimized through global motion aggregation and local motion correction techniques. Next, a content fusion generative adversarial network is constructed, including a generator and a discriminator, to smoothly fuse images and generate spliced images that conform to human eye perception. A phased training strategy is adopted, first optimizing image alignment, then achieving image smooth fusion. Finally, the trained network is used to automatically complete image alignment and fusion, resulting in a seamless large field of view confocal ocular surface image. This method improves splicing accuracy and robustness, is suitable for image splicing in different environments, and significantly improves the probability of successful splicing.

Description

TECHNICAL FIELD

[0001] The application relates to an image splicing method, in particular to a laser confocal ocular surface image splicing method based on deep learning. BACKGROUND

[0002] Demodex is a common parasite in human body, and in the eye, demodex mainly parasitizes on the eyelashes, eyelash follicles and meibomian glands. In recent years, more and more studies have found that the parasitization of demodex in the eye can cause diseases such as blepharitis and meibomian gland dysfunction. Clinically, real-time and non-invasive laser scanning confocal microscopy is often used to check the demodex infection of patients. However, due to the limitation of imaging principle, the confocal microscope has a limited field of view of the obtained image, which is not conducive to the timely discovery of mites by doctors. Image splicing can splice and fuse multiple small field of view images with overlapping areas under similar viewing angles into one large field of view image. Using the image splicing technology can increase the field of view of the image without changing the hardware facilities of the confocal microscope, which is conducive to the rapid diagnosis of doctors.

[0003] Image splicing technology can be divided into traditional methods and deep learning-based methods. Traditional image splicing methods mainly consist of four parts: feature detection, feature matching, image registration and image fusion. SIFT algorithm is one of the most classic feature extraction algorithms, which includes scale space extreme value detection, key point positioning, direction assignment and key point description. These extracted key points maintain stability under image transformations such as rotation, scale change and brightness change. AutoStitch algorithm proposed by Brown is a pioneering work in the field of automatic image splicing, which uses SIFT algorithm for feature detection. In recent years, more and more methods not only use detected feature points for splicing, but also use other feature elements to improve the quality of splicing. In LPC, SURF is used to detect feature points in the image, and LSD is used to detect line segments in the image. By introducing coplanar local sub-region matching and global collinearity structure constraint, the splicing artifacts and deformation are effectively reduced. Traditional image splicing methods are simple to implement, but their features are based on specific assumptions and rules designed by hand, which are only suitable for specific situations, and the image registration accuracy is seriously dependent on the quality of the feature points or lines detected by the feature detector. When the number of detected feature points is small or the distribution of feature points is uneven, it will cause more matching errors and lead to image splicing failure. In the laser confocal ocular surface image, there are many low light, weak texture and repetitive texture situations, which brings great challenges to traditional image splicing methods.

[0004] In recent years, deep learning-based methods have also been gradually applied in the field of image stitching. Deep learning-based image stitching methods mainly include supervised and unsupervised methods. The former, such as VFIS-Net and LP-Net, uses image pairs and synthetic pseudo-labels to complete network training, but due to the limited ability to handle parallax, its performance on real data sets is poor. Unsupervised image stitching networks such as UDIS and REwarp perform well in natural image stitching, but use cost volume in image registration, which is slow and can only focus on the correlation of local features while ignoring the correlation of global features, limiting the image registration capability. UDIS++ is the current SOTA model on the natural image stitching data set UDIS-D, which finds the seam by comparing the brightness difference between the reference image and the target image to complete image fusion, but this is difficult for confocal ocular surface images because the brightness difference in the overlapping area is small and cannot effectively find the seam.

[0005] Using a laser confocal scanning microscope to check ocular Demodex has many advantages such as real-time and non-invasive, but due to the limitations of imaging principles, it has the problem of small imaging field of view, and traditional image stitching methods cannot detect enough feature points on the confocal ocular surface image or the detected feature points are unevenly distributed, which may cause the stitching to fail.

[0006] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] The main purpose of the present application is to solve the problems in the above background, and to provide a laser confocal ocular surface image stitching method based on deep learning.

[0008] To achieve the above purpose, the present application adopts the following technical solutions:

[0009] A laser confocal ocular surface image stitching method based on deep learning, comprising the following steps:

[0010] S1. Preprocessing the ocular surface image to construct a data set for stitching;

[0011] S2. Constructing an unsupervised image alignment network, the unsupervised image alignment network automatically extracts image features through a multi-scale feature pyramid constructed by a convolutional neural network to realize pre-alignment of the images; through a global motion aggregation technology, a homography matrix is calculated based on deep feature maps to estimate the global transformation between images; further through a local motion correction technology, TPS interpolation is used to correct the images locally to optimize the alignment between images;

[0012] S3. Constructing an unsupervised image fusion network, the unsupervised image fusion network is designed as a content fusion generative adversarial network, the network includes a generator and a discriminator; the generator is responsible for learning to generate a fusion mask to smooth the fusion image, and the discriminator is used to guide the generator to obtain a fusion probability map to learn to smooth the fusion to-be-stitched image;

[0013] S4. A phased training strategy is adopted, first, the unsupervised image alignment network is trained to optimize image alignment, and then the unsupervised image fusion network is trained to realize image smoothing fusion;

[0014] S5. Using the trained unsupervised image alignment network and unsupervised image fusion network, the alignment and fusion of the to-be-stitched image are automatically completed, and a seamless large field-of-view confocal ocular surface image is obtained.

[0015] A computer program product comprising a computer program, which, when executed by a processor, implements the deep learning-based laser confocal ocular surface image stitching method.

[0016] The present application has the following beneficial effects:

[0017] The present application provides a deep learning-based laser confocal ocular surface image stitching method, which applies deep learning technology to the stitching of laser confocal ocular surface images, and realizes efficient and accurate image stitching by constructing an unsupervised image alignment network and a content fusion network. The image alignment network automatically extracts image features using a convolutional neural network and performs pre-alignment using a multi-scale feature pyramid, optimizes the alignment between images through global motion aggregation and local motion correction technology, especially in the overlapping area, and realizes accurate alignment while maintaining small distortion in the non-overlapping area. In addition, the content fusion network adopts the design of a generative adversarial network, generates a fusion mask through learning, realizes the smooth fusion of images, avoids obvious boundaries and artifacts, and thus obtains a seamless and smooth stitched image that meets human eye perception. The model of the present application not only improves the quality of the stitched image, but also enhances the robustness of the model, so that it can adapt to images collected by devices in different environments, significantly improving the probability of successful stitching. Compared with traditional image stitching methods, the present application exhibits stronger adaptability and superiority when processing laser confocal ocular surface images with low light, weak texture and repeated texture.

[0018] The unsupervised laser confocal image splicing model designed by the application can realize automatic splicing of a group of laser confocal ocular surface images, automatically extract robust features from the images, and improve the splicing effect and success rate. Among them, the registered images are obtained through the image alignment network, which realizes accurate alignment in the overlapping area and maintains small distortion in the non-overlapping area, realizes efficient and smooth fusion through the content fusion network, and obtains laser confocal ocular surface splicing images that conform to human eye perception.

[0019] The splicing image obtained by the application has higher accuracy in the overlapping area and smaller distortion in the non-overlapping area, and can obtain a seamless and smooth splicing image that conforms to human eye perception; the application has strong robustness, and can realize splicing of laser confocal ocular surface images collected by the device under different environments, greatly improving the splicing success probability compared with the traditional image splicing method.

[0020] The image pre-alignment network mainly consists of three parts: multi-scale feature extraction, global motion aggregation and local motion correction. The multi-scale feature extraction part adopts a convolutional neural network to construct a multi-scale feature pyramid, and follows the coarse to fine strategy to pre-align the images at different scales. Since the deep layer of the convolutional neural network extracts high-level semantic information in the image, the global motion aggregation takes the deep feature map of the feature pyramid as input, calculates the homography matrix by regressing four-point offset, and estimates the global change. The local motion correction takes the shallow features in the feature pyramid as input to regress the grid motion information based on the global motion, and further uses TPS interpolation for local correction to achieve better alignment in the overlapping area. Using TPS interpolation for alignment may cause excessive deformation of the image, and further a grid loss is designed to constrain it. Through the image alignment network, pre-aligned images, their content masks and boundary masks can be obtained, which are input into the content fusion network to obtain a fusion mask and realize smooth fusion, avoiding obvious boundaries and artifacts.

[0021] Other beneficial effects in the embodiments of the application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of the laser confocal ocular surface image automatic splicing method of the embodiments of the application.

[0023] Figure 2 is a flowchart of the laser confocal image splicing model data set construction of the embodiments of the application and the effect diagram of different image contrast enhancement algorithms.

[0024] Figure 3 is a schematic diagram of the splicing model of the embodiments of the application, mainly consisting of an image alignment network and a content fusion network.

[0025] Figure 4 FIG. 1 is a structural schematic diagram of an image alignment network of an embodiment of the present application.

[0026] Figure 5 FIG. 2 is a structural schematic diagram of a content fusion network of an embodiment of the present application.

[0027] Figure 6 FIG. 3 is a core function structure of a spatial transformation network STN of an embodiment of the present application.

[0028] Figure 7 FIG. 4 is a laser confocal ocular surface image splicing example diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application are described in detail below. It should be emphasized that the following description is only exemplary, and is not intended to limit the scope of the present application and its applications.

[0030] The present application proposes a laser confocal ocular surface image splicing method based on deep learning, including the steps of laser confocal ocular surface image preprocessing, data set construction, image splicing model training, etc., realizing inputting the small field of view confocal ocular surface image to be spliced into the pre-trained image splicing model and returning the spliced large field of view confocal ocular surface image.

[0031] Referring to Figure 1 , the embodiment of the present application provides a laser confocal ocular surface image splicing method based on deep learning, including the following steps:

[0032] Step S1. Preprocess the ocular surface image to construct a data set for splicing.

[0033] In some embodiments, in step S1, the preprocessing of the ocular surface image specifically includes: performing useless information cropping on the ocular surface image, and performing contrast enhancement using a Laplacian contrast enhancement algorithm. The construction of the data set for splicing specifically includes: selecting the image with the current frame number i and the multiple images after it in the continuous shooting process for screening; performing feature point detection on the images with frame numbers i and j using an ORB operator, j = i + 1; screening out outliers and performing feature point matching using a RANSAC algorithm; adding the image pairs with a feature point matching number higher than a preset threshold to the data set, and screening out the image pairs lower than the threshold; and performing manual inspection on the screened image pairs to filter out low-quality image pairs.

[0034] Step S2. Constructing an unsupervised image alignment network, which automatically extracts image features through a convolutional neural network to construct a multi-scale feature pyramid and realize pre-alignment of images. Through global motion aggregation technology, a homography matrix is calculated based on deep feature maps to estimate the global transformation between images; further through local motion correction technology, TPS interpolation is used to correct the image locally, and the alignment between images is optimized.

[0035] In some embodiments, in step S2, the automatic extraction of image features through a convolutional neural network to construct a multi-scale feature pyramid and realize pre-alignment of images specifically includes: using a convolutional neural network with the same structure and shared weights as a multi-scale feature extractor to automatically extract robust features in the confocal image; using a VGG16 network as a feature extractor and replacing the max pooling layer in VGG16 with a BlurPool layer, which includes two parts of maximum dense sampling with a step of 1 and low-pass filtering downsampling with a step of 2, and low-pass filtering is performed through a filter with a Gaussian kernel; using a coarse to fine strategy to capture the correspondence between image features at different scales, extracting 1 / 8 size and 1 / 16 size feature maps for subsequent local motion correction and global motion aggregation calculation, wherein the 1 / 16 size feature map is used for extracting global motion information and the 1 / 8 size feature map is used for extracting local motion information.

[0036] In some embodiments, in step S2, the global motion aggregation technology is used to calculate a homography matrix based on deep feature maps to estimate the global transformation between images, specifically including: using a homography matrix with eight degrees of freedom to establish the global mapping relationship between two images, and realizing the geometric correspondence between images through the homography matrix; using deep feature maps, i.e., 1 / 16 size feature maps, to extract global motion information, and using the rich semantic information contained in the deep feature maps of the convolutional neural network to estimate the global transformation between images; inputting the 1 / 16 size feature maps obtained from the reference image and the target image through the feature extractor into the correlation calculation module for processing to obtain a global displacement flow; inputting the obtained global displacement flow into a global motion prediction network to complete the extraction of global motion through the network, thereby calculating the homography matrix and realizing the global alignment between images.

[0037] In some embodiments, in step S2, the local motion correction technique is adopted to correct the image by TPS interpolation, and the alignment between images is optimized, specifically including: on the basis of global motion aggregation, 1 / 8 size feature map is used to extract local motion information to achieve finer alignment; TPS flexible interpolation method is adopted to find a smooth surface through control point grid by minimizing the energy function to achieve deformation correction in local area; wherein, TPS interpolation is used to define a deformation grid which covers the image alignment area and is used to control the details of image deformation; TPS interpolation function is defined, including a first order polynomial and a radial basis function for capturing local trends, and calculation of the Euclidean distance from the interpolation point to the control point; in order to ensure the uniqueness of the TPS interpolation solution, the constraint condition is set to ensure that the control points on the reference image and the target image coincide, and additional constraints are added to avoid ambiguity of the solution; a local motion corrector is designed to warp the target image by STN, and then compare it with the feature map of the reference image to calculate the local displacement flow; the local displacement flow is input into the TPS motion prediction network to extract the local motion information and further optimize the alignment between images.

[0038] Step S3. An unsupervised image fusion network is constructed, which is designed as a content fusion generative adversarial network, including a generator and a discriminator; the generator is responsible for learning to generate a fusion mask to smooth the fusion image, and the discriminator is used to guide the generator to obtain a fusion probability map to learn to smooth the fusion to-be-stitched images.

[0039] In some embodiments, in step S3, the construction of the unsupervised image fusion network specifically includes: designing a generator which adopts a twin network with the same structure and shared weights for multi-scale down-sampling in the encoding stage to capture the differences between the reference image and the target image; difference operation is performed on the last layer feature map at each scale to obtain residual feature maps at different scales to enhance the difference of feature representation; in the decoding stage, the residual feature maps of adjacent scales are input into the up-sampling layer, and a skip connection is introduced to generate the decoded feature maps at different scales; through up-sampling and splicing operations, the decoded multi-scale feature maps are used to generate the fusion probability map of the warped reference image in the overlapping area; the fusion probability map is used as a fusion mask to multiply the warped versions of the reference image and the target image with the corresponding fusion mask, and then add them to form the final stitching result; a discriminator is designed to discriminate the quality of the generated fusion image by combining convolution, normalization and activation function, as well as adaptive average pooling and linear layer, to guide the training and optimization of the generator.

[0040] Step S4. A phased training strategy is adopted to first train the unsupervised image alignment network to optimize the image alignment, and then train the unsupervised image fusion network to realize the image smooth fusion.

[0041] In some embodiments, in step S4, the stage-by-stage training strategy specifically comprises: two-stage training of the unsupervised image alignment network: in the first stage, only multi-scale feature extraction and global motion aggregation are trained, and optimization is performed using a loss function based on homography matrix; in the second stage, local motion correction is combined, a loss function based on TPS interpolation is used for optimization, and a grid loss is introduced to constrain excessive deformation; in the training of the unsupervised image fusion network, when the generator is trained, a combination of boundary loss, adversarial loss and balance loss is adopted to generate smooth and natural fusion images; when the discriminator is trained, a binary cross-entropy loss function is used to distinguish the generated fusion images from the real fusion images.

[0042] In preferred embodiments, the calculation method of the grid loss specifically comprises: calculating the horizontal and vertical edge vectors of the deformed grid generated by the network; performing cross multiplication of the edge vectors of each grid with the corresponding unit vectors to obtain sine values to evaluate the alignment degree of the deformed grid; by comparing the sine values of adjacent grids, the deformation difference between the grids is constrained to avoid discontinuity in the splicing area; by comprehensively considering the deformation in the horizontal and vertical directions of the grid, the grid loss is included in the total loss function in the form of weighted sum to optimize the effect of local motion correction.

[0043] Step S5. Using the trained unsupervised image alignment network and unsupervised image fusion network, the alignment and fusion of the images to be spliced are automatically completed to obtain a seamless large-field confocal ocular surface image.

[0044] The application innovatively applies deep learning technology to laser confocal ocular surface image stitching, and significantly improves the stitching accuracy and success probability by constructing an unsupervised image alignment network and a content fusion network. The image alignment network uses multi-scale feature extraction and convolutional neural network to automatically extract image features, realizes pre-alignment of the images, calculates the homography matrix through global motion aggregation technology, estimates the global transformation between the images, uses TPS interpolation for local correction of the images, optimizes the alignment between the images, especially realizes accurate alignment in the overlapping area, and maintains small distortion in the non-overlapping area. The content fusion network uses the design of a generative adversarial network, generates a fusion mask through a generator to smooth the fusion image, and uses a discriminator to guide the generator to obtain a fusion probability map and learn to smooth the fusion image to be stitched to avoid obvious boundaries and artifacts, and obtain a seamless and smooth stitched image consistent with human eye perception. The model of the application not only improves the quality of the stitched image, but also enhances the robustness of the model, so that it can adapt to images collected by devices in different environments, significantly improves the success probability of stitching, and has stronger adaptability and superiority in processing laser confocal ocular surface images with low light, weak texture and repeated texture, especially in automatic stitching, accuracy preservation and robustness.

[0045] The specific embodiments of the application and examples of algorithm implementation are further described below.

[0046] Figure 1 is a flowchart of the implementation of the laser confocal ocular surface image automatic stitching method of the application, mainly including laser confocal ocular surface image shooting, image preprocessing, data set construction, image stitching model training, image automatic stitching and the like.

[0047] The application discloses a laser confocal ocular surface image splicing method based on deep learning, and a laser confocal ocular surface image splicing model based on deep learning is designed and mainly comprises an image alignment network and a content fusion network. The image pre-alignment network mainly comprises three parts of multi-scale feature extraction, global motion aggregation and local motion correction. The multi-scale feature extraction part adopts a convolutional neural network to construct a multi-scale feature pyramid, and follows a coarse to fine strategy to pre-align the images at different scales. Since the deep layer of the convolutional neural network extracts high-level semantic information in the image, the global motion aggregation takes the deep layer feature map of the feature pyramid as input, calculates a homography matrix by regressing four-point offset, and estimates the global change. The local motion correction takes the shallow layer feature in the feature pyramid as input to regress the grid motion information on the basis of the global motion, and further performs local correction by using TPS interpolation, so that better alignment in the overlapping area is achieved. The image may be deformed excessively by using the TPS interpolation for alignment, therefore, the preferred embodiment designs a grid loss to constrain. The pre-aligned image, the content mask and the boundary mask thereof are obtained by the image alignment network, and are input into the content fusion network to obtain a fusion mask, so that smooth fusion is achieved, and obvious boundaries and artifacts are avoided. The content fusion network of the application is essentially a generative adversarial network.

[0048] Confocal ocular surface image data preprocessing and dataset construction

[0049] The specific construction process of the confocal ocular surface image splicing dataset is as follows:

[0050] a. Confocal ocular surface image source. The Sequence Scan mode of the laser confocal microscope is used for continuous shooting, and a total of 2863 confocal ocular surface images are obtained, wherein the serial number of the image represents the time frame position in the continuous shooting process;

[0051] b. Cropping and contrast enhancement. After the useless information of the picture is cropped, the size becomes 384x384. Since the confocal images obtained by shooting have poor detail texture, the contrast of the images is enhanced. It is found that the Laplacian contrast enhancement algorithm can better highlight the details and retain the original structure compared with the Gamma transformation, histogram equalization and logarithmic transformation. Compared with the limited contrast adaptive histogram equalization (CLAHE), the Laplacian contrast enhancement algorithm avoids redundant detail texture information, and therefore is adopted as the final contrast enhancement algorithm.

[0052] c. Inter-adjacent frame image screening. Since the images far apart in the frame sequence number may be located in different focal planes and cannot be used for stitching during shooting, the images of the current frame sequence number and the next 7 frames are screened. The frame sequence number of the current image is recorded as i, and the frame sequence number of the next image is recorded as j, that is, j = i + 1. Then, the images with frame sequence numbers i and j are subjected to feature point detection by using the ORB operator, and the outliers are screened out by using RANSAC and matched, and then the image pairs with a feature point matching number higher than a threshold are added to the data set, and the image pairs lower than the threshold are screened out. Finally, the screened image pairs are manually checked to filter out low-quality image pairs.

[0053] Figure 2 The flow of constructing the laser confocal image stitching model data set of the application and the effect diagram of different image contrast enhancement algorithms are shown.

[0054] Figure 3 The stitching model of the embodiment of the application is shown, which mainly consists of an image alignment network and a content fusion network. The registered reference image and target image are obtained by the image alignment network, and the fusion mask is obtained by the content fusion network, and the images are smoothly fused.

[0055] Unsupervised image alignment network construction

[0056] Figure 4 The structure of the image alignment network of the embodiment of the application is shown.

[0057] The image alignment network mainly consists of three parts of multi-scale feature extraction, global motion aggregation and local motion correction. The multi-scale feature extractor automatically extracts robust features in the confocal image by using the convolutional neural network with the same structure and shared weights, avoiding the limitations of traditional manual features. The classic VGG16 is selected as the feature extractor. After the contrast enhancement of the confocal image, the high-frequency information in the image is increased, and the aliasing effect may occur in the down-sampling process of the maximum pooling due to the sampling frequency being lower than twice the Nyquist frequency, so the maximum pooling in VGG16 is replaced by BlurPool. Unlike the maximum pooling down-sampling with a step of 2, BlurPool contains two parts of maximum dense sampling with a step of 1 and low-pass filtering down-sampling with a step of 2. The low-pass filtering by the filter with a Gaussian kernel can alleviate the aliasing effect caused by the maximum pooling down-sampling to a certain extent.

[0058] To improve the alignment accuracy of the network, the correspondence between image features at different scales is captured, and the coarse-to-fine strategy is followed to align the images at different scales. A multi-scale feature extractor extracts 1 / 8 size and 1 / 16 size feature maps for subsequent local motion correction and global motion aggregation calculation. Specifically, since the deep feature map of the convolutional neural network contains more semantic information but has lower resolution, and the shallow feature map contains more edge, texture and other information and has higher resolution, the 1 / 16 size feature map is used to extract the global motion information, and the 1 / 8 size feature map is used to extract the local motion information, and the global motion information is further corrected.

[0059] First, a homography matrix with eight degrees of freedom is used to obtain the global mapping relationship between two images, that is:

[0060]

[0061] Since the eight parameters h1-h8 in the homography matrix are coupled with translation, shearing, etc., it is difficult for the network to learn these eight parameters directly, so they are decoupled, and instead of learning the eight parameters directly, the offsets of the four vertices of the image are learned, and then the homography matrix is solved by direct linear transformation algorithm. The two 1 / 16 size feature maps obtained by the feature extractor from the reference image and the target image are input to NMS-FDL for correlation calculation to obtain the global displacement flow, and the obtained global displacement flow is input into Homo Motion Prediction Network to complete the extraction of global motion.

[0062] Since the homography matrix is a rigid transformation, it is hoped that the network can further align the reference image and the target image, so TPS flexible interpolation is used to further improve the alignment accuracy of the network. TPS interpolation finds a smooth surface passing through the control point grid by minimizing the energy function, and the TPS interpolation function in two-dimensional space is:

[0063]

[0064] where P(x R ,y R ) is a first-order polynomial for capturing global trends, n is the number of control points, ω i is an undetermined coefficient, U(r) is a radial basis function, and U(r) = r 2 logr in the usual case, and ||P i -P|| is the Euclidean distance from the interpolation point to the control point.

[0065] In order to ensure the uniqueness of the solution, in addition to the control points on the reference image and the target image are coincided, three constraint conditions are needed, that is:

[0066]

[0067] Therefore, the matrix representation of the TPS interpolation solving process is:

[0068]

[0069] Wherein K n×n is the matrix determined by the control points, P n×3 is the matrix composed of the control point coordinates on the reference image, that is is the matrix composed of the control point coordinates on the target image, that is

[0070] According to the principle of TPS interpolation, the displacement of the control points needs to be obtained, and therefore a local motion corrector is designed. The 1 / 8 feature map of the target image is deformed by using STN according to the output of the global motion aggregation, and then the feature map is input into the NMS-FDL together with the feature map of the reference image to obtain the local displacement flow, and then the local displacement flow is input into the TPS Motion Prediction Network to extract the local motion information.

[0071] Figure 6 The core function structure of the spatial transformation network STN in the embodiment of the application is shown, and on the basis of the core function structure, the original Localization Network is cancelled, and the image is deformed according to the homography matrix or the TPS interpolation matrix obtained by regression.

[0072] Unsupervised image fusion network building

[0073] Figure 5 The structure of the content fusion network in the embodiment of the application is shown.

[0074] After the preliminary alignment of the images is completed, if the two images are directly fused in the overlapping area, a relatively obvious fusion boundary will appear, and due to the limitation of the performance of the image alignment network, artifacts will also appear in the overlapping area. A content fusion generative adversarial network is designed, a discriminator is used to guide the generator to obtain a fusion probability map, and the generator learns to fuse the images to be spliced as smoothly as possible.

[0075] The generator adopts a twin network with the same structure and shared weights for multi-scale down-sampling in the encoding stage. In order to better capture the differences between the reference image and the target image, the last layer feature map of each scale is subtracted to obtain the residual feature map at different scales. Then decoding is performed. In order to avoid the generated content fusion mask being too coarse due to too large scale span, the decoded feature map is not input into the up-sampling layer as in the U-Net, but the residual feature map of the adjacent scale is input into the up-sampling layer, and the skip connection is introduced. The feature map of different scales is output by a single-channel 3x3 convolution. After obtaining the decoded feature map at different scales, up-sampling and splicing are performed, and finally a single-channel 1x1 convolution and a sigmoid activation function are used to obtain the fusion probability map of the warped reference image in the overlapping area. The fusion probability map of the warped fusion image in the overlapping area can be obtained by simple post-processing, and the final spliced image is:

[0076]

[0077] The backbone of the discriminator is stacked by N blocks of Conv+BN+RELU, and then outputs the discrimination result through adaptive average pooling and two linear layers.

[0078] Unsupervised image alignment network and unsupervised image fusion network training

[0079] For the output of the image alignment network at different scales, different loss functions are used. For global motion aggregation, the homography matrix H is calculated according to the output of the network and using the direct linear transformation algorithm, and then the target image is deformed using STN, and the loss function is:

[0080] Loss H =|Image R ×Warp(Ones,H)-Warp(Image T ,H)|

[0081] Where Ones represents a matrix with the same size as Image R and all 1s. For the extraction of local motion information, the TPS interpolation weight is obtained according to the displacement of the control points, and then the STN is used for transformation, and the loss function is:

[0082]

[0083] However, TPS interpolation may cause large deformation, so a grid loss is designed to constrain this deformation. The sine value of the horizontal edge vector of the network and the horizontal unit vector is calculated according to the following formula:

[0084]

[0085] where <,> denotes the cross product, represents the horizontal edge vector of the i-th mesh, represents the horizontal unit vector. To constrain the deformation inside the mesh, we hope is small, and to prevent a relatively large deformation between adjacent meshes, we hope is small. The deformation constraint in the vertical direction of the mesh is similar to the horizontal direction, and the total loss of the local motion correction is:

[0086]

[0087] For the image alignment network, the global motion aggregation and local motion correction are first trained jointly, but the effect is not very good. Therefore, a two-stage training is adopted, the first stage freezes the local motion correction, only trains the multi-scale feature extraction and global motion aggregation, and takes Loss H as the loss function, and the second stage unfreezes the local motion correction, trains the feature extraction and global motion aggregation together, and takes Loss TPS as the loss function. The model training is implemented on an Nvidia 2080Ti.

[0088] By using the same distortion method as the image on the white mask with all 1s, the content mask ContentMask is obtained, and then the boundary mask Boundary Mask is obtained through the inflation operation. When the generator obtains the fusion mask, it is hoped that the distorted reference image and the target image do not change obviously at the fusion edge, so the illumination loss of the spliced image at the boundary and the distorted image at the boundary is constrained, and the 1 norm is adopted here, and the boundary loss is defined as:

[0089]

[0090] In order to encourage the generator to generate as smooth a spliced image as possible to confuse the discriminator, the binary cross entropy loss is adopted to quantify the difference between the predicted value and the true label, so the adversarial loss of the generator is:

[0091]

[0092] where represents the binary cross entropy loss function. In order to avoid the generated fusion mask being completely dominated by the reference image or the target image in the overlapping area, a balance loss is designed:

[0093]

[0094] Therefore, the total loss of the generator is:

[0095]

[0096] The adversarial loss of the discriminator is symmetrical to that of the generator, and the loss of the discriminator is:

[0097] Loss D

[0098] The content composition GAN is trained on the image processed by the image alignment network, and the batch size is set to 1 to avoid inconsistent image size caused by distortion registration in each batch. The model training is implemented on an Nvidia 2080Ti.

[0099] Laser confocal ocular surface image stitching

[0100] After the training of the image alignment network and the image fusion network is completed, the Image_Alignment_Network.pth file and the Content_Composition_GAN.pth file are obtained. Then the trained model is used for inference, the Image_Alignment_Network.pth is loaded as the pre-training weight of the image alignment network, and then the image to be stitched is input into the image alignment network to obtain the displacement of the TPS interpolation control point, and the image is distorted and registered according to the displacement of the control point and the interpolation equation. Then the distorted reference image and target image, as well as their content mask and boundary mask, are input into the generator with Content_Composition_GAN.pth as the pre-training weight to obtain the fusion mask of the reference image and the target image, and the final stitching image is obtained by weighting the reference image and the target image through the fusion mask. The above process can be automatically implemented in a Python script.

[0101] Figure 7 is an example of the laser confocal ocular surface image stitching of the present application, which shows the laser confocal image stitching result of the eyelashes and the eyelash follicles of the eye. The upper row is the input image pair to be stitched, and the lower row is the stitching effect diagram of different methods. It can be seen that the SIFT algorithm cannot be well aligned in the image registration stage, so there are obvious fusion artifacts at the red arrow position in the figure. The REW algorithm is slightly misaligned at the red arrow position in the figure, and the final stitching image has obvious fusion boundary. UDIS++ has obvious misalignment when aligning, and due to the limited ability to capture the feature difference between the reference image and the target image, obvious fusion boundary can be seen in the red box area and the green box area. The method of the present application has no obvious misalignment and fusion artifact in the red box area, and maintains high clarity in the green box, realizes smooth fusion, and avoids obvious fusion boundary.

[0102] The present application has the following advantages and outstanding effects:

[0103] ①The present application can realize automatic splicing of a group of laser confocal ocular surface images;②The spliced image obtained by the present application has higher precision in the overlapping area and smaller distortion in the non-overlapping area, and can obtain a spliced image that is more in line with human eye perception and seamless and smooth;③The present application has strong robustness, and can realize splicing of laser confocal ocular surface images collected by the equipment under different environments, greatly improving the probability of successful splicing compared with the traditional image splicing method.

[0104] The embodiment of the present application also provides a storage medium for storing a computer program, which is executed to perform at least the method described above.

[0105] The embodiment of the present application also provides a control device, which comprises a processor and a storage medium for storing a computer program;wherein the processor is used to execute the computer program to perform at least the method described above.

[0106] The embodiment of the present application also provides a processor, which executes a computer program to perform at least the method described above.

[0107] The storage medium can be realized by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM). The magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiment of the present application is intended to include but not limited to these and any other suitable types of memory.

[0108] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other manners. The described device embodiments are merely illustrative, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, or direct coupling or communication connection between the components can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0109] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0110] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0111] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program executes the steps including the above-mentioned method embodiments when executed; and the foregoing storage medium includes: mobile storage device, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various storage program codes.

[0112] Alternatively, the integrated unit of the present application, if implemented in the form of a software function module and sold or used as an independent product, can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes: mobile storage device, ROM, RAM, magnetic disk or optical disk, and various storage program codes.

[0113] The methods disclosed in the several method embodiments of the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0114] The features disclosed in the several product embodiments of the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0115] The features disclosed in the several method or device embodiments of the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0116] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application should not be limited to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of equivalent substitutions or obvious modifications can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. A deep learning-based laser confocal ocular surface image stitching method, characterized by, The method comprises the following steps: S1. Preprocessing the confocal ocular surface images obtained by shooting with a laser confocal microscope to construct a data set for splicing; S2. Constructing an unsupervised image alignment network, which automatically extracts image features through a multi-scale feature pyramid constructed by a convolutional neural network to realize pre-alignment of the images; calculates a homography matrix based on deep feature maps to estimate the global transformation between the images through a global motion aggregation technology; further corrects the images locally through TPS interpolation to optimize the alignment between the images through a local motion correction technology; S3. Constructing an unsupervised image fusion network designed as a content fusion generative adversarial network, which comprises a generator and a discriminator; the generator is responsible for learning to generate a fusion mask to smooth the fused image, and the discriminator is used to guide the generator to obtain a fusion probability map to learn to smooth the fused image to be spliced; S4. Using a phased training strategy, first training the unsupervised image alignment network to optimize image alignment, and then training the unsupervised image fusion network to achieve image smoothing fusion; S5. Using the trained unsupervised image alignment network and unsupervised image fusion network, automatically completing the alignment and fusion of the images to be spliced to obtain a seamless large-field confocal ocular surface image. 2.The deep learning-based laser confocal ocular surface image stitching method of claim 1, wherein, In step S1, the preprocessing of the ocular surface images specifically comprises: cropping useless information of the ocular surface images, and performing contrast enhancement using a Laplacian contrast enhancement algorithm. 3.The deep learning-based laser confocal ocular surface image stitching method according to claim 1 or 2, characterized in that, In step S1, the construction of the data set for splicing specifically comprises: Selecting images with a current frame number i and multiple frames of images after the current frame number i for screening; Detecting feature points of the images with frame numbers i and j using an ORB operator, j = i + 1; Screening outliers and matching feature points using a RANSAC algorithm; Adding image pairs with a number of feature point matches higher than a preset threshold to the data set, and screening out image pairs with a number of feature point matches lower than the threshold; Artificially checking the screened image pairs to filter out low-quality image pairs.

4. The deep learning-based laser confocal ocular surface image stitching method according to any one of claims 1 to 2, characterized in that, In step S2, the automatic extraction of image features through a multi-scale feature pyramid constructed by a convolutional neural network to realize pre-alignment of the images specifically comprises: Using a convolutional neural network with the same structure and shared weights as a multi-scale feature extractor to automatically extract robust features in the confocal images; Using a VGG16 network as a feature extractor, and replacing the maximum pooling layer in the VGG16 with a BlurPool layer, which includes two parts of maximum dense sampling with a step of 1 and low-pass filtering downsampling with a step of 2, and performs low-pass filtering through a filter with a Gaussian kernel; Using a coarse to fine strategy to capture the corresponding relationship between image features at different scales, and extracting 1 / 8 size and 1 / 16 size feature maps for subsequent local motion correction and global motion aggregation calculation, wherein the 1 / 16 size feature map is used for extracting global motion information, and the 1 / 8 size feature map is used for extracting local motion information.

5. The deep learning-based laser confocal ocular surface image stitching method according to any one of claims 1 to 2, characterized in that, In step S2, the global motion aggregation technology is used to calculate a homography matrix based on deep feature maps to estimate the global transformation between images, and specifically includes: A homography matrix with eight degrees of freedom is used to establish a global mapping relationship between two images, and the geometric correspondence between images is realized through the homography matrix; A 1 / 16 size deep feature map is used to extract global motion information, and the rich semantic information contained in the deep feature map of the convolutional neural network is used to estimate the global transformation between images; The 1 / 16 size feature maps obtained by the reference image and the target image through the feature extractor are input into the correlation calculation module for processing to obtain a global displacement flow; The obtained global displacement flow is input into the global motion prediction network, and the global motion is extracted through the global motion prediction network to calculate the homography matrix and realize the global alignment between images.

6. The deep learning-based laser confocal ocular surface image stitching method according to any one of claims 1 to 2, characterized in that, In step S2, the local motion correction technology is used to correct the image using TPS interpolation to optimize the alignment between images, and specifically includes: On the basis of global motion aggregation, a 1 / 8 size feature map is used to extract local motion information to achieve more precise alignment; The TPS interpolation method is used to find a smooth surface passing through the control point grid by minimizing the energy function to realize deformation correction in the local area; wherein, the TPS interpolation defines a deformation grid covering the image alignment area, which is used to control the details of image deformation; The TPS interpolation function is defined, including a first order polynomial and a radial basis function for capturing local trends, and a calculation of the Euclidean distance from the interpolation point to the control point; In order to ensure the uniqueness of the TPS interpolation solution, a constraint condition is set to ensure that the control points on the reference image and the target image coincide, and an additional constraint is added to avoid ambiguity of the solution; A local motion corrector is designed, which uses a spatial transformation network STN to warp and deform the target image, and then compares it with the feature map of the reference image to calculate a local displacement flow; The local displacement flow is input into the TPS motion prediction network to complete the extraction of local motion information and further optimize the alignment between images.

7. The deep learning-based laser confocal ocular surface image stitching method according to any one of claims 1 to 2, characterized in that, In step S3, the unsupervised image fusion network is constructed, and specifically includes: A generator is designed, which uses a twin network with the same structure and shared weights for multi-scale downsampling in the encoding stage to capture the differences between the reference image and the target image; Difference operation is performed on the last layer feature map at each scale to obtain residual feature maps at different scales, enhancing the difference of feature representation; In the decoding stage, the residual feature maps of adjacent scales are input into the upsampling layer, and a skip connection is introduced to generate decoded feature maps at different scales; Through upsampling and splicing operations, the decoded multi-scale feature maps are used to generate a fusion probability map of the warped reference image in the overlapping area; The fusion probability map is used as a fusion mask to multiply the warped versions of the reference image and the target image with the corresponding fusion mask, and then add them to form the final splicing result. The discriminator is designed to discriminate the quality of the generated fusion image by a combination of convolution, normalization and activation function, adaptive average pooling and linear layer, so as to guide the training and optimization of the generator.

8. The deep learning-based laser confocal ocular surface image stitching method according to any one of claims 1 to 2, characterized in that, The step S4 specifically includes: The unsupervised image alignment network is trained in two stages: in the first stage, only the multi-scale feature extraction and global motion aggregation are trained, and a loss function based on the homography matrix is used for optimization; in the second stage, the local motion correction is combined, a loss function based on TPS interpolation is used for optimization, and a grid loss is introduced to constrain the excessive deformation; In the training of the unsupervised image fusion network, when the generator is trained, a combination of boundary loss, adversarial loss and balance loss is used to generate a smooth and natural fusion image; when the discriminator is trained, a binary cross-entropy loss function is used to distinguish the generated fusion image from the real fusion image. 9.The deep learning-based laser confocal ocular surface image stitching method of claim 8, wherein, The method for calculating the grid loss specifically includes: Calculate the horizontal and vertical edge vectors of the deformed grid generated by the network; Cross-multiply the edge vector of each grid with the corresponding unit vector to obtain the sine value to evaluate the alignment of the deformed grid; By comparing the sine values of adjacent grids, the deformation difference between the grids is constrained to avoid discontinuity in the splicing area; By considering the deformation of the grid in the horizontal and vertical directions, the grid loss is included in the total loss function by means of weighted sum to optimize the effect of local motion correction.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the deep learning-based laser confocal ocular surface image splicing method according to any one of claims 1-9.

Images