Cornea laser confocal microscope-oriented multi-image automatic rapid splicing method

Abstract

The invention discloses a multi-image automatic rapid splicing method for a cornea laser confocal microscope. A synthetic data set with labels is generated by constructing an initial data set and performing controlled data enhancement, including brightness correction, cutting and displacement disturbance. A neural network model comprising a feature extractor, a feature maximum correlation matching layer and a regression sub-network prediction head is designed, and a synthetic data set is used for training to predict the offset between images. According to the method, a multi-scale prediction strategy is adopted, information of deep and shallow feature maps is combined, and the splicing precision is improved. And finally, through image preprocessing and offset prediction, automatic and rapid splicing of multiple images is realized, the imaging visual field is effectively expanded, and the accuracy and efficiency of image analysis are improved. According to the method, the limitation of a traditional splicing technology is overcome, GPU acceleration can be utilized, the processing speed is remarkably increased, and a brand-new technical platform is provided for research and diagnosis of eye diseases.

Description

Technical Field

[0001] The invention relates to the fields of image processing, biomedical engineering and artificial intelligence, and in particular to a method for automatically and quickly stitching multiple images for a corneal laser confocal microscope. Background Art

[0002] Laser confocal microscopy uses the principle of optical conjugation for imaging. Due to the conjugation of the illumination point, detection point and object point, it has a series of advantages such as high signal-to-noise ratio, strong depth penetration ability, and three-dimensional imaging, and has been widely used in the medical field. As a non-invasive high-resolution imaging tool, it can provide detailed images of the ocular surface structure, greatly enhancing doctors' understanding and diagnosis of eye diseases. More importantly, it also allows doctors and researchers to observe living tissues at the subcellular level, which is crucial for studying diseases such as dry eye, corneal dystrophy, glaucoma, etc.

[0003] However, due to the limitations of the imaging principle of the laser confocal microscope, it brings the problem of small imaging field of view while obtaining high signal-to-noise ratio images. This problem is difficult to solve through hardware optimization and design. However, accurate judgment of diseases requires images with a large field of view to bring more information. Therefore, the present invention can use the image stitching algorithm to merge and stitch multiple small field of view images with overlapping areas under similar viewing angles into an image with a large field of view, and retain as much information as possible from the original multiple small field of view images. When the traditional image stitching method is applied to laser confocal corneal images, there will be an insufficient number of feature point detections or uneven distribution of feature points. This manually designed feature greatly reduces the accuracy of stitching, and it cannot be accelerated using a GPU.

[0004] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0005] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and to provide a method for automatically and quickly stitching multiple images for corneal laser confocal microscopy.

[0006] To achieve the above object, the present invention adopts the following technical solutions: A method for automatically and quickly stitching multiple images for corneal laser confocal microscopy comprises the following steps: S1. Data acquisition: corneal images are collected using a corneal laser confocal microscope to construct an initial data set, wherein the images contain structural information of corneal tissue; S2, controlled data enhancement: the initial data set is screened to remove out-of-focus or invalid images; the remaining images are brightness corrected to make their brightness consistent; a fixed-size image block is cropped in the center area of ​​the image as a reference image, and the image block is randomly displaced to generate a target image and a corresponding offset label, and a synthetic data set with labels is constructed; S3, neural network model construction and training: build a neural network model, the model includes a feature extractor, a feature maximum correlation matching layer and a regression subnetwork prediction head; use the synthetic data set to train the model, extract image features through the feature extractor, the feature maximum correlation matching layer calculates the correlation between features, the regression subnetwork prediction head predicts the offset between image pairs based on the correlation results, and optimizes the model using a hybrid loss function until the model can accurately predict the offset; S4, image stitching: pre-process the multiple corneal images collected, adjust the image size and perform brightness correction; select adjacent image pairs in turn and input them into the trained neural network model to predict the offset between the image pairs; when the predicted offset exceeds the preset threshold, terminate the stitching of the current group of images and use the previous images as a group of images to be stitched; for each group of images to be stitched, calculate the global coordinate system according to the predicted offset, and fuse and stitch the images to generate a corneal image with a large field of view.

[0007] Furthermore, step S2 specifically includes: The initial dataset was screened to remove images that were out of focus or did not contain corneal structural information; Perform brightness correction on the screened images to make the image brightness consistent, so as to reduce boundary artifacts during stitching and highlight the main structure of the cornea; Crop a fixed-size image patch in the center area of ​​the image as a reference image; Performing random displacement perturbation on the cropped image block to generate a target image and a corresponding offset label, wherein the displacement perturbation remains consistent in the horizontal and vertical directions and the displacement range is a preset value; The image pairs and their offset labels generated by the above steps constitute a synthetic dataset, which is used for training and testing the neural network model.

[0008] Furthermore, step S3 specifically includes: The feature extractor is composed of a plurality of stacked convolution blocks, each of which contains a two-dimensional convolution operation and a nonlinear activation function. Except for the last convolution block, the remaining convolution blocks are downsampled through a maximum pooling layer. The feature extractor extracts features from the reference image and the target image to form feature maps of the same size and share weights; Use feature maps of different scales for offset prediction to capture the offset information of the image; Among them, by predicting the offset step by step at different scales, the displacement of the image pair is estimated at different scales to achieve accurate prediction.

[0009] Furthermore, the offset prediction includes: At the first scale, deep feature maps are used for prediction, which contain rich semantic information; The deep feature maps of the reference image and the target image are input into the feature maximum correlation matching layer, and the correlation of different positions of the feature maps is calculated to obtain the correlation result; Inputting the correlation result into the prediction head of the first regression subnetwork to predict the displacement of the image pair in the x and y directions; At the second scale, shallow feature maps are used for prediction, which contain more edge, texture and structure information; The shallow feature map of the target image is deformed according to the prediction result of the first scale and compared with the shallow feature map of the reference image. Figure 1 Input feature maximum correlation matching layer to obtain the correlation result of the second scale; The correlation result of the second scale is input into the prediction head of the second regression sub-network to predict the displacement of the image pair in the x and y directions.

[0010] Furthermore, the final offset prediction result is obtained by fusing the prediction results of the first scale and the second scale, wherein the fusion method is to add the prediction results of the two scales.

[0011] Furthermore, in step S3, the neural network model training further includes: Construct a loss function consisting of two parts. The first part is L2 loss, which is used to make the network focus on the overall displacement information. The second part is deformation loss, which is used to make the network focus on the content differences in the overlapping areas of the image. The L2 loss is composed of the average of the L2 losses at two different scales, where the L2 loss of the first scale is based on the prediction result of a single scale, and the L2 loss of the second scale is based on the difference between the accumulated value of the prediction results of the two scales and the actual displacement; The deformation loss calculates the content difference between the deformed image and the reference image in the overlapping area by applying the same deformation matrix to the mask and the input image; The total loss is a weighted sum of the L2 loss and the deformation loss, where the weights of the L2 loss and the deformation loss are set to balance the attention paid to the overall displacement information and the content difference of the overlapping regions.

[0012] Furthermore, step S4 specifically includes: Use the trained neural network model to test on synthetic data sets to verify the accuracy and speed of the model; Deploy the model and create Web service interfaces, including the health check interface and the prediction interface. The prediction interface is used to receive input data and call the model for prediction. Collect multiple real images, resize them and correct their brightness to ensure the consistency of input data and reduce boundary artifacts; Starting from the first image, sample adjacent image pairs in sequence and call the API interface to predict the offset of the image pairs; When the predicted offset exceeds a preset threshold, the prediction of the current image group is terminated, and the image group is marked as an image group to be spliced; Repeat the above process until all image predictions are completed, and obtain multiple groups of images to be spliced; For each group of images to be stitched, the global coordinate system is calculated according to the predicted local offset, and the images are fused to obtain the stitching result.

[0013] Further, calculating the global coordinate system according to the predicted local offsets includes: multiplying each group of predicted local offsets by a scaling factor and performing calculations to obtain a global coordinate system, and gradually performing image fusion.

[0014] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for automatically and quickly stitching multiple images for a corneal laser confocal microscope.

[0015] A computer program product, when executed by a processor, implements the method for automatically and quickly stitching multiple images for a corneal laser confocal microscope.

[0016] The present invention has the following beneficial effects: The present invention proposes an innovative method for automatic rapid stitching of multiple images for corneal laser confocal microscopy, which effectively solves the problem of small imaging field of laser confocal microscopy, makes up for the hardware defects, and can handle the automatic rapid stitching of multiple images. Compared with the traditional image stitching method, the method of the present invention has significant improvements in accuracy, speed and stability. By designing a novel controlled data enhancement method, the present invention can construct a labeled synthetic data set from the original data, thereby providing high-quality data for model training. At the same time, the present invention designs a new registration neural network model, which includes three modules: a deep feature extractor, a feature maximum correlation matching layer, and a regression subnetwork prediction head, which can complete the prediction of offsets at different scales, thereby improving the prediction accuracy. The multi-scale neural network model can adapt to diverse data distributions, model complex nonlinear features, and use the different rich information contained in feature maps at different levels to perform prediction regression at multiple scales. The method of the present invention can not only improve the stitching accuracy, but also can be accelerated by GPU hardware, greatly improving the efficiency of stitching, providing doctors and researchers with a powerful tool to observe living tissues at the subcellular level, which is of great significance for studying diseases such as dry eye, corneal dystrophy, glaucoma, etc.

[0017] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a method for automatically and quickly stitching multiple corneal confocal images according to an embodiment of the present invention.

[0019] Figure 2 Schematic diagram of the controlled data enhancement process according to an embodiment of the present invention.

[0020] Figure 3 4 is a diagram of a neural network structure according to an embodiment of the present invention.

[0021] Figure 4 is a model loss diagram of an embodiment of the present invention.

[0022] Figure 5 This is splicing result example 1 of the embodiment of the present invention.

[0023] Figure 6 This is splicing result example 2 of the embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0025] A method for automatically and quickly stitching multiple images for corneal laser confocal microscopy comprises the following steps: Step S1, data acquisition: using a corneal laser confocal microscope to acquire corneal images and construct an initial data set, wherein the images contain structural information of corneal tissue.

[0026] Step S2, controlled data enhancement: screening the initial data set to remove out-of-focus or invalid images; performing brightness correction on the remaining images to make their brightness consistent; cropping a fixed-size image block in the center area of ​​the image as a reference image, and performing random displacement perturbation on the image block to generate a target image and a corresponding offset label, and construct a synthetic data set with labels.

[0027] In a preferred embodiment, step S2 specifically includes: screening the initial data set to remove images that are out of focus or do not contain corneal structure information; performing brightness correction on the screened images to make the image brightness consistent, so as to reduce boundary artifacts during splicing and highlight the main structure of the cornea; cropping a fixed-size image block in the center area of ​​the image as a reference image; performing random displacement perturbation on the cropped image block to generate a target image and a corresponding offset label, wherein the displacement perturbation remains consistent in the horizontal and vertical directions, and the displacement range is a preset value. The image pairs and their offset labels generated by the above steps constitute a synthetic data set, which is used for training and testing the neural network model.

[0028] Step S3, neural network model construction and training: build a neural network model, the model includes a feature extractor, a feature maximum correlation matching layer and a regression subnetwork prediction head; use the synthetic data set to train the model, extract image features through the feature extractor, the feature maximum correlation matching layer calculates the correlation between features, the regression subnetwork prediction head predicts the offset between image pairs based on the correlation results, and optimizes the model using a hybrid loss function until the model can accurately predict the offset.

[0029] In a preferred embodiment, step S3 specifically includes: the feature extractor is composed of a plurality of stacked convolution blocks, each convolution block includes a two-dimensional convolution operation and a nonlinear activation function, and except for the last convolution block, the remaining convolution blocks are downsampled through a maximum pooling layer; the feature extractor extracts features from the reference image and the target image to form feature maps of the same size and share weights; offset prediction is performed using feature maps of different scales to capture the offset information of the image; wherein, by predicting the offset step by step at different scales, the displacement of the image pair is estimated at different scales to achieve accurate prediction.

[0030] In a further preferred embodiment, the offset prediction includes: at a first scale, using a deep feature map for prediction, the deep feature map contains rich semantic information; inputting the deep feature maps of the reference image and the target image into the feature maximum correlation matching layer, calculating the correlation of different positions of the feature map, and obtaining the correlation result; inputting the correlation result into the first regression subnetwork prediction head to predict the displacement of the image pair in the x and y directions; at a second scale, using a shallow feature map for prediction, the shallow feature map contains more edge, texture and structure information; deforming the shallow feature map of the target image according to the prediction result of the first scale, and comparing it with the shallow feature map of the reference image; Figure 1 The feature maximum correlation matching layer is input to obtain the correlation result of the second scale; the correlation result of the second scale is input to the second regression sub-network prediction head to predict the displacement of the image pair in the x and y directions. Preferably, the final offset prediction result is the sum of the prediction results of the first scale and the second scale. The prediction results of the first scale and the second scale are fused to obtain the final offset prediction result, wherein the fusion method is to add the prediction results of the two scales.

[0031] In a preferred embodiment, in step S3, the neural network model training also includes: constructing a loss function comprising two parts, the first part is L2 loss, which is used to make the network pay attention to the overall displacement information, and the second part is deformation loss, which is used to make the network pay attention to the content difference of the overlapping area of ​​the image; the L2 loss is composed of the mean of L2 losses at two different scales, wherein the L2 loss of the first scale is based on the prediction result of a single scale, and the L2 loss of the second scale is based on the difference between the accumulated value of the prediction results of the two scales and the actual displacement; the deformation loss calculates the content difference between the deformed image and the reference image in the overlapping area by applying the same deformation matrix to the mask and the input image; wherein the total loss is the weighted sum of the L2 loss and the deformation loss, wherein the weights of the L2 loss and the deformation loss are set to balance the degree of attention to the overall displacement information and the content difference of the overlapping area. Through the design of the above loss function, the accuracy of the overall displacement and the consistency of the content of the overlapping area are optimized simultaneously during the model training process to improve the accuracy and quality of the spliced ​​image.

[0032] Step S4, image stitching: pre-process the multiple corneal images collected, adjust the image size and perform brightness correction; select adjacent image pairs in turn and input them into the trained neural network model to predict the offset between the image pairs; when the predicted offset exceeds a preset threshold, terminate the stitching of the current group of images and use the previous images as a group of images to be stitched; for each group of images to be stitched, calculate the global coordinate system according to the predicted offset, and fuse and stitch the images to generate a corneal image with a large field of view.

[0033] In a preferred embodiment, step S4 specifically includes: using the trained neural network model to test on a synthetic data set to verify the accuracy and speed of the model; deploying the model and creating a Web service interface, including a health check interface and a prediction interface, wherein the prediction interface is used to receive input data and call the model for prediction; collecting multiple real images, and resizing and luminance correction to ensure the consistency of the input data and reduce boundary artifacts; starting from the first image, sampling adjacent image pairs in sequence, and calling the API interface to predict the offset of the image pair; when the predicted offset exceeds a preset threshold, terminating the prediction of the current image group, and marking the image group as a group of images to be spliced; repeating the above process until all image predictions are completed, and obtaining multiple groups of image groups to be spliced; for each group of images to be spliced, calculating the global coordinate system according to the predicted local offset, and performing image fusion to obtain a splicing result. Preferably, the calculation of the global coordinate system according to the predicted local offset includes: multiplying each group of predicted local offsets by a scaling factor and calculating to obtain a global coordinate system, and gradually performing image fusion.

[0034] The multi-image automatic fast splicing method based on supervised learning designed by the present invention is data-driven. By training on a synthetic data set with labels, it can better learn the laws in the data, so as to be applied to unknown data for prediction. The neural network model designed by the present invention adaptively extracts features in the image through a convolutional neural network. The shallow feature map contains more basic information in the image, such as edges, textures, structures, etc., while the deep feature map contains richer high-level semantic information in the image. The multi-scale neural network model of the present invention can adapt to diverse data distributions and model complex nonlinear features. In addition, the present invention uses different rich information contained in feature maps at different levels to perform prediction regression at multiple scales to achieve higher splicing accuracy. In addition, the method of the present invention can be accelerated by GPU hardware with more computing cores, which greatly improves the efficiency of splicing.

[0035] The following further describes specific embodiments of the present invention, algorithm examples and experimental verification.

[0036] Based on the supervised learning technology in deep learning, the present invention designs a method for automatically and quickly stitching multiple images for corneal images collected by corneal laser confocal microscopy, thereby effectively increasing the imaging field of view of the corneal laser confocal microscope. The method can be implemented through the following steps to achieve automatic stitching of corneal confocal images: the first step is to collect corneal confocal images through a corneal laser confocal microscope to construct an initial data set; the second step is to perform controlled data enhancement, and construct a labeled synthetic data set for training the model from the initial data set through steps such as image brightness correction, random cropping and displacement perturbation; the third step is to build a registration neural network model for estimating the affine matrix, which is mainly composed of three modules: a deep feature extractor, a feature maximum correlation matching layer, and a regression subnetwork prediction head; the fourth step is to train the registration neural network model of the present invention with the synthetic data set, predict the affine matrix at different scales, and use a mixed loss function for supervision; the fifth step is model testing and verification, and the performance parameters of the model of the present invention are verified on the synthetic test set; the sixth step is model deployment and use, and the trained model is packaged and deployed locally through FastAPI, and then the multiple real images collected are scaled to the required size and input into the model for inference to predict the offset between adjacent images, and terminate when the predicted value exceeds the threshold. All images before the termination are regarded as a group of images to be stitched, the global coordinate system is obtained according to the predicted value calculation, and the images to be stitched are gradually fused to obtain the final multi-image stitching result.

[0037] 1. Construction of synthetic dataset First, the corneal image of the subject is taken by corneal laser confocal microscopy to obtain the initial data set. Next, the present invention proposes a novel controlled data augmentation (CDA) method to construct a synthetic data set. The specific steps are as follows: First, the initial data set is screened to filter out out-of-focus images and images that do not contain any information, and then the remaining images are brightness corrected so that the brightness of the captured images is relatively similar. This can avoid obvious boundary artifacts during splicing, and through image brightness correction, the main structure in the image can also be highlighted, such as the nerves in the anterior elastic layer of the cornea; secondly, the image is locally cropped. Here, a 256×256 image block is cropped in the center area of ​​the image as a reference image, that is, Figure 2 The image in the red box; Finally, the position of the image block is randomly perturbed. It should be noted that the offsets of the four vertices of the rectangle in the x and y directions are kept consistent, and the range of the offset is set to 64, that is:

[0038] Through this step, the present invention not only obtains the target image, that is, Figure 2The image in the green box also obtains the true label of the input image offset. and . Factors such as rotation, shearing, and scaling are ignored here, and only the movement of the microscope lens during the shooting process is considered, because the time taken to shoot each image is extremely short and the contact lens causes the local corneal area to be close to a plane rather than a curved surface. Through this data enhancement method, the present invention can construct a synthetic dataset from the initial dataset, each piece of data in the synthetic dataset contains a pair of input images to be registered and a label offset, and then the synthetic dataset is divided into a training set and a test set in proportion for model training and testing, respectively.

[0039] 2. Neural Network Model Construction and Training The neural network of the present invention is mainly composed of three modules: feature extractor, feature maximum correlation matching layer and regression subnetwork prediction head, and predicts the offset at two scales. The specific structure is as follows Figure 3 shown.

[0040] The feature extractor module is composed of 4 stacked blocks, each of which contains two Conv2d+RELU. Here Conv2d refers to the two-dimensional convolution operation. In the implementation process, the size of the convolution kernel is set to 3×3. Here RELU refers to a nonlinear activation function. Except for the last block, the remaining three blocks are downsampled through a maximum pooling layer with a stride of 2. The feature extractors used for feature extraction of the reference image (input 1) and the target image (input 2) have the same structure and share weights. It is a twin network. In other words, at each scale, the size of the feature map they obtain is exactly the same. Then the 1 / 4 size feature map before the last maximum pooling layer and the 1 / 8 size feature map output by the last block are used to predict subsequent parameters to capture the offset information of the image at different scales.

[0041] First, a 1 / 8 size feature map is used for prediction at scale 1. The deep feature map contains richer high-dimensional semantic information, and the surrounding spatial information aggregated by each feature vector is richer after multiple downsampling. The 1 / 8 size feature maps of input 1 and input 2 are input into the feature maximum correlation matching layer to establish a mapping of the correlation of different positions of the deep feature map, find the connection between the most correlated regions, obtain the correlation result at scale 1, and then input it into the regression subnetwork prediction head 1 to predict the offset parameters. Here, only the displacement in the x and y directions is predicted, while factors such as rotation, shearing, and scaling are ignored. The deformation matrix can be simplified as:

[0042] The predicted result on scale 1 is recorded as , Then, the 1 / 4-sized feature map is used to make predictions at the next scale, i.e., scale 2. This shallow feature map contains more information about edges, textures, structures, etc. on the image. First, the 1 / 4-sized feature map of input 2 is distorted according to the prediction result of scale 1 and the deformation matrix, and then it is combined with the 1 / 4-sized feature map of input 1. Figure 1 It is input into the feature maximum correlation matching layer to obtain the correlation result on scale 2, and then input into the regression subnetwork prediction head 2 for prediction. The result predicted on scale 2 is recorded as , , the final prediction result of the neural network is the sum of the two, that is, , .

[0043] The loss function consists of two parts: L2 loss and deformation loss. L2 loss hopes that the network can pay attention to the overall displacement information, while deformation loss hopes that the network can pay attention to the content differences in the overlapping areas of the image. L2 loss is composed of the mean of L2 losses at two scales:

[0044] The deformation loss is as follows, filtering the overlapping regions by a mask deformed by the same deformation matrix:

[0045] The total loss is composed of L2 loss and deformation loss weighted:

[0046] By predicting the offset step by step at different scales, the method of the present invention estimates the displacement of image pairs at different scales and achieves accurate prediction. The Adam optimizer is used for training on two Nvidia RTX 2080Ti, with the initial learning rate set to 0.0001 and an exponential decay strategy. The loss function during training is as follows: Figure 4 As shown, blue, purple, orange, pink and green represent , , , and , it can be found that the loss converges to a smaller range.

[0047] 3. Model Testing and Deployment Application After completing the training of the registration network, the Registration_Network.pth weight file was obtained, and then the model of the present invention was tested on the synthetic data set, and its accuracy and speed reached a high level.

[0048] Next, the model was deployed and applied. A Web service interface was created using FastAPI, including a health check interface, a prediction interface, etc. The prediction interface is used to receive input data transmitted by the client, call the model for prediction, and return the result. Here, the model of the present invention uses GPU to accelerate reasoning during prediction. Then, multiple real images were collected. In order to ensure the consistency of size, their sizes were adjusted before calling the interface prediction. The image size was adjusted to 256×256, and a brightness correction algorithm was used to avoid boundary artifacts. Then, starting from the first image, two adjacent images were sampled and the API interface was called to predict the image offset in the x and y directions. The predicted offset is recorded as , Here, i represents the predicted offset between the i-th and i+1-th images. When the predicted amount exceeds the threshold, it means that there is no corresponding offset relationship between the i-th and i+1-th images. The prediction group is terminated, and the images 1 to i are used as a group of images to be spliced. Then, the next group of predictions is started until all images are predicted.

[0049] After all the images are predicted, k groups of images to be stitched are found. The predicted local offset of each group is multiplied by the scaling factor and calculated to obtain the global coordinate system. The images are gradually fused to obtain the multi-image stitching results of each group of ocular surface confocal images.

[0050] Experimental Results The CCMID dataset was used for testing. A multi-image stitching result obtained by stitching 10 confocal images of the eye surface is shown in the figure. Figure 5 As shown in FIG. 1 , it can be found that the corneal anterior elastic layer nerve choroid maintains good continuity without obvious breakpoints, which proves the effectiveness of the method of the present invention. Another multi-image stitching result obtained by stitching 8 confocal images of the eye surface is shown in FIG. Figure 6 As shown, its imaging field of view is about 3 to 4 times that of the initial image.

[0051] In summary, the important features and innovative points of the present invention include: 1. A novel method for automatic rapid stitching of multiple images for corneal laser confocal microscopy is proposed. It is no longer limited to the stitching of two adjacent images, but is extended to the automatic rapid stitching of multiple images. It can make up for the hardware defects of laser confocal microscopy and effectively increase the imaging field of view. Its accuracy, speed and stability are better than existing methods.

[0052] 2. A novel controlled data augmentation method is proposed to construct a labeled synthetic dataset from the original data, which mainly includes brightness correction, cropping, displacement perturbation and other steps.

[0053] 3. A new registration neural network model is designed, which consists of three modules: deep feature extractor, feature maximum correlation matching layer and regression subnetwork prediction head. It completes the prediction of offset at different scales and improves the prediction accuracy.

[0054] In summary, the automatic fast stitching method innovatively proposed in the present invention greatly increases the imaging field of view of the corneal laser confocal microscope, which not only improves the accuracy of image analysis but also can achieve a very fast stitching speed under the acceleration of GPU, providing a new technical platform for further exploration of eye diseases.

[0055] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0056] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0057] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0058] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0059] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0060] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0061] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0062] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0063] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0064] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0065] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0066] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0067] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for automatic rapid stitching of multiple images for corneal laser confocal microscopy, characterized in that: The following steps are involved: S1. Data acquisition: corneal images are collected using a corneal laser confocal microscope to construct an initial data set, wherein the images contain structural information of corneal tissue; S2, controlled data enhancement: the initial data set is screened to remove out-of-focus or invalid images; the remaining images are brightness corrected to make their brightness consistent; a fixed-size image block is cropped in the center area of ​​the image as a reference image, and the image block is randomly displaced to generate a target image and a corresponding offset label, and a synthetic data set with labels is constructed; S3, neural network model construction and training: build a neural network model, the model includes a feature extractor, a feature maximum correlation matching layer and a regression subnetwork prediction head; use the synthetic data set to train the model, extract image features through the feature extractor, the feature maximum correlation matching layer calculates the correlation between features, the regression subnetwork prediction head predicts the offset between image pairs based on the correlation results, and optimizes the model using a hybrid loss function until the model can accurately predict the offset; S4, image stitching: pre-process the multiple corneal images collected, adjust the image size and perform brightness correction; select adjacent image pairs in turn and input them into the trained neural network model to predict the offset between the image pairs; when the predicted offset exceeds the preset threshold, terminate the stitching of the current group of images and use the previous images as a group of images to be stitched; for each group of images to be stitched, calculate the global coordinate system according to the predicted offset, and fuse and stitch the images to generate a corneal image with a large field of view.

2. The method for automatic rapid stitching of multiple images for corneal laser confocal microscopy according to claim 1, characterized in that: The specific process of step S2 includes: The initial dataset was screened to remove images that were out of focus or did not contain corneal structural information; Perform brightness correction on the screened images to make the image brightness consistent, so as to reduce boundary artifacts during stitching and highlight the main structure of the cornea; Crop a fixed-size image patch in the center area of ​​the image as a reference image; Performing random displacement perturbation on the cropped image block to generate a target image and a corresponding offset label, wherein the displacement perturbation remains consistent in the horizontal and vertical directions and the displacement range is a preset value; Through the above specific process, image pairs of reference images and target images and offset labels between the image pairs are generated to form a synthetic data set for training and testing of neural network models.

3. The method for automatic rapid stitching of multiple images for corneal laser confocal microscopy according to claim 1 or 2, characterized in that: Step S3 specifically includes: The feature extractor is composed of a plurality of stacked convolution blocks, each of which contains a two-dimensional convolution operation and a nonlinear activation function. Except for the last convolution block, the remaining convolution blocks are downsampled through a maximum pooling layer. The feature extractor extracts features from the reference image and the target image to form feature maps of the same size and share weights; Use feature maps of different scales for offset prediction to capture the offset information of the image; Among them, by predicting the offset step by step at different scales, the displacement of the image pair is estimated at different scales to achieve accurate prediction.

4. The method for automatic rapid stitching of multiple images for corneal laser confocal microscopy according to claim 3, characterized in that: The offset prediction includes: At the first scale, deep feature maps are used for prediction, which contain rich semantic information; The deep feature maps of the reference image and the target image are input into the feature maximum correlation matching layer, and the correlation of different positions of the feature maps is calculated to obtain the correlation result; Inputting the correlation result into the prediction head of the first regression subnetwork to predict the displacement of the image pair in the x and y directions; At the second scale, shallow feature maps are used for prediction, which contain more edge, texture and structure information; The shallow feature map of the target image is deformed according to the prediction result of the first scale, and is input into the feature maximum correlation matching layer together with the shallow feature map of the reference image to obtain the correlation result of the second scale; The correlation result of the second scale is input into the prediction head of the second regression sub-network to predict the displacement of the image pair in the x and y directions.

5. The method for automatic rapid stitching of multiple images for corneal laser confocal microscopy according to claim 4, characterized in that: The final offset prediction result is obtained by fusing the prediction results of the first scale and the second scale, wherein the fusion method is to add the prediction results of the two scales.

6. The method for automatic rapid stitching of multiple images for corneal laser confocal microscopy according to claim 3, characterized in that: In step S3, the neural network model training further includes: Construct a loss function consisting of two parts. The first part is L2 loss, which is used to make the network focus on the overall displacement information. The second part is deformation loss, which is used to make the network focus on the content differences in the overlapping areas of the image. The L2 loss is composed of the average of the L2 losses at two different scales, where the L2 loss of the first scale is based on the prediction result of a single scale, and the L2 loss of the second scale is based on the difference between the accumulated value of the prediction results of the two scales and the actual displacement; The deformation loss calculates the content difference between the deformed image and the reference image in the overlapping area by applying the same deformation matrix to the mask and the input image; The total loss is a weighted sum of the L2 loss and the deformation loss, where the weights of the L2 loss and the deformation loss are set to balance the attention paid to the overall displacement information and the content difference of the overlapping regions.

7. The method for automatic rapid stitching of multiple images for corneal laser confocal microscopy according to claim 1 or 2, characterized in that: The specific process of step S4 includes: Use the trained neural network model to test on synthetic data sets to verify the accuracy and speed of the model; Deploy the model and create Web service interfaces, including the health check interface and the prediction interface. The prediction interface is used to receive input data and call the model for prediction. Collect multiple real images, resize them and correct their brightness to ensure the consistency of input data and reduce boundary artifacts; Starting from the first image, sample adjacent image pairs in sequence and call the API interface to predict the offset of the image pairs; When the predicted offset exceeds a preset threshold, the prediction of the current image group is terminated, and the image group is marked as an image group to be spliced; Repeat the above specific process until all image predictions are completed, and obtain multiple groups of image groups to be spliced; For each group of images to be stitched, the global coordinate system is calculated according to the predicted local offset, and the images are fused to obtain the stitching result.

8. The method for automatic rapid stitching of multiple images for corneal laser confocal microscopy according to claim 7, characterized in that: Calculating the global coordinate system according to the predicted local offsets includes: multiplying each group of predicted local offsets by a scaling factor and performing calculations to obtain a global coordinate system, and gradually performing image fusion.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for automatically and quickly stitching multiple images for a corneal laser confocal microscope as described in any one of claims 1 to 8 is implemented.

10. A computer program product, characterized in that When the computer program is executed by a processor, the method for automatically and quickly stitching multiple images for a corneal laser confocal microscope as described in any one of claims 1 to 8 is implemented.

Images