A method and system for augmented reality navigation of lamellar keratoplasty surgery based on a joint learning strategy

Abstract

The application relates to a kind of based on the method and system of joint learning strategy's lamellar keratoplasty surgery augmented reality navigation, comprising: based on the acquisition surgery image of camera unit and conversion into single frame image, and carry out normalization processing to construct neural network model input;Based on frame-by-frame image, utilize joint learning deep neural network model to obtain surgery instrument, corneal region segmentation result and corresponding completion optical flow result;The image and corresponding completion optical flow result of current frame and forward adjacent frame are combined, and the completion information of current frame is comprehensively calculated, and the area blocked by surgery instrument is recovered by completion calculation;Utilize corneal region segmentation result to realize the augmented reality navigation result of superimposed surgery suture line and point position on the completion surgery image by ellipse fitting and suture point calculation process.The application can realize the construction of augmented reality navigation information for lamellar keratoplasty surgery, guide surgery related operation, improve surgery efficiency.

Description

Technical Field

[0001] This invention relates to the field of augmented / mixed reality, and more specifically to a method and system for augmented reality navigation in lamellar keratoplasty based on a joint learning strategy. Background Technology

[0002] With the rapid development of augmented reality (AR) technology, its application in various industries is deepening. Especially for assistive medical tasks, AR navigation technology can provide intuitive and clear visual navigation information during surgery, showing great potential and prospects for improving surgical outcomes and reducing surgical risks. Currently, a large number of new cases of blindness are caused by corneal diseases worldwide each year. Corneal transplantation is a widely used technique for treating corneal diseases, with lamellar keratoplasty being the most common. Lamellar keratoplasty has several advantages, reducing the risk of complications and shortening the recovery period. Corneal suturing is a key part of lamellar keratoplasty, requiring surgeons with extensive experience and skills. The accuracy and uniformity of the sutures are crucial during the surgery, as they directly affect the risk of postoperative complications and the need for re-suturing. In this regard, AR navigation systems show potential to enhance the suturing process.

[0003] Currently, in natural scenes, the key technical issues related to image completion and video completion mainly involve two problems. The goal of both tasks is to generate realistic missing regions to restore occluded areas, and recent state-of-the-art work can provide near-perfect image or frame completion. However, because computational efficiency is not the primary requirement, most completion methods employ complex networks to achieve better completion results. Furthermore, almost all methods require specific occlusion input to provide direct information about the completed region, necessitating additional semantic segmentation to identify and classify the occluded region during the completion process. Therefore, many methods cannot be used for real-time tasks in surgical navigation systems. Another issue is that since surgical navigation is an online task, only forward frames are available to the system. This requires completion methods to also possess high temporal consistency to generate stable completion outputs and ensure sufficiently reliable guidance. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method and system for augmented reality navigation in lamellar keratoplasty based on a joint learning strategy.

[0005] The technical solution of this invention is: a method for augmented reality navigation in lamellar keratoplasty based on a joint learning strategy, comprising:

[0006] Step S1: Based on the camera unit and corresponding image encoding and decoding of the surgical platform, the surgical scene is captured and frame-by-frame images are obtained. Based on the computational requirements, the frame-by-frame images obtained by the camera platform are downsampled and normalized to unify the input into the same form and structure.

[0007] Step S2: Based on the joint learning deep neural network, for each frame of the image, the identification and prediction of the surgical instrument mask are realized, and the corneal region mask under the condition of removing instrument occlusion and the optical flow estimation result under the condition of removing occlusion are obtained simultaneously.

[0008] Step S3: Using the completion framework, the multi-frame images including the current frame and the unobstructed optical flow estimation results are integrated. The forward neighboring frames are distorted and aligned using the optical flow propagation calculation method. The occluded areas of the current frame are restored using the unobstructed area pixels of the forward neighboring frames. The continuous occluded areas are further restored using the completion results of the previous frame. Finally, a clear image with the surgical instrument obstruction removed is obtained.

[0009] Step S4: Based on the segmentation mask result of the unobstructed corneal contour, combined with the image after removing the obstruction, the visualization information is superimposed on the image after removing the obstruction through ellipse fitting calculation and suture point calculation, so as to realize the annotation of the corneal contour and the guidance of the suture position and the suture sequence.

[0010] Compared with the prior art, the present invention has the following advantages:

[0011] 1. This invention discloses an iterative surgical navigation joint learning framework to meet the requirements of real-time suturing processes, which can provide ophthalmologists with stable, clear and accurate augmented reality navigation results in real time under severe occlusion conditions.

[0012] 2. The method disclosed in this invention is based on an iterative structure to construct various network sub-modules for segmentation, optical flow estimation and output completion, and applies a shared encoder structure to provide feature reuse characteristics. It has high configuration flexibility, can achieve a good balance between framework effect and computational pressure, and can achieve good real-time computing performance in online surgical video scenarios.

[0013] 3. The method disclosed in this invention proposes a data synthesis strategy and constructs a dataset to support the training of deep neural networks. This synthesis strategy can utilize unpaired surgical images with realistic annotations, thereby effectively reducing the network's data requirements and the difficulty of data acquisition. Attached Figure Description

[0014] Figure 1This is a flowchart illustrating an augmented reality navigation method for lamellar keratoplasty based on a joint learning strategy, as described in an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of the superimposed image of the augmented reality surgical navigation effect in an embodiment of the present invention;

[0016] Figure 3 This is a schematic diagram of the feature extraction and preprocessing submodule in the joint learning deep neural network module of this invention.

[0017] Figure 4 This is a schematic diagram of the basic result prediction and intermediate processing submodule structure in the joint learning deep neural network module of this invention embodiment;

[0018] Figure 5 This is a schematic diagram of the result completion and prediction submodule structure in the joint learning deep neural network module of this invention embodiment;

[0019] Figure 6 This is a schematic diagram of the occlusion frame completion process of the video frame completion framework module in an embodiment of the present invention;

[0020] Figure 7 This is a flowchart illustrating the implementation of an augmented reality navigation method for lamellar keratoplasty based on a joint learning strategy, as an embodiment of the present invention.

[0021] Figure 8 This is a structural block diagram of an augmented reality navigation system for lamellar keratoplasty based on a joint learning strategy, according to an embodiment of the present invention. Detailed Implementation

[0022] This invention provides an augmented reality navigation method for lamellar keratoplasty based on a joint learning strategy. It can provide ophthalmologists with stable, clear, and accurate augmented reality navigation results in real time under severe occlusion conditions. Based on an iterative structure and feature reuse design, it offers high configuration flexibility, achieving a good balance between framing performance and computational load, and delivering excellent real-time computational performance in online surgical video scenarios. Furthermore, a data synthesis strategy is used to construct a dataset supporting deep neural network training, facilitating the use of unpaired surgical images with realistic annotations, thereby effectively reducing the network's data requirements and acquisition difficulty.

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0024] The embodiments of this invention use Python as the programming language, and the devices used are an Intel(R) i7 13700K CPU (3.40GHz, 16 cores) and 64GB RAM, an Nvidia(R) RTX 4090 (24GB), running on a Windows 11 64-bit system.

[0025] Example 1

[0026] like Figure 1 As shown in the figure, an embodiment of the present invention provides an augmented reality navigation method for lamellar keratoplasty based on a joint learning strategy, comprising the following steps:

[0027] Step S1: Based on the camera unit and corresponding image encoding and decoding of the surgical platform, the surgical scene is captured and frame-by-frame images are obtained. Based on the computational requirements, the frame-by-frame images obtained by the camera platform are downsampled and normalized to unify the input into the same form and structure.

[0028] Step S2: Based on the joint learning deep neural network, for each frame of image, the surgical instrument mask is identified and predicted, and the corneal region mask and the optical flow estimation results under the condition of no instrument obstruction are predicted at the same time to provide the key information required by the navigation system.

[0029] Step S3: Using the completion framework, the images of multiple neighboring frames, including the current frame, and the results of the unobstructed optical flow estimation are integrated. The forward neighboring frames are distorted and aligned using the optical flow propagation calculation method. The occluded areas of the current frame are restored using the pixels of the unobstructed areas of the forward neighboring frames. The continuous occluded areas are further restored using the completion results saved from the previous frame. Finally, a clear image with the surgical instruments removed is obtained.

[0030] Step S4: Based on the segmentation mask results of the corneal contour under unobstructed conditions, combined with the image after removing the obstruction, the visualization information is superimposed on the unobstructed image through ellipse fitting calculation and suture point calculation, and finally the annotation and guidance of corneal contour, suture position and suture sequence are realized.

[0031] In one embodiment, step S1 above: Based on the camera unit and corresponding image encoding / decoding of the surgical platform, the surgical scene is captured and frame-by-frame images are obtained. Based on computational requirements, the frame-by-frame images obtained by the camera platform are downsampled and normalized to unify them into an input of the same form and structure. Specifically, this includes:

[0032] Step S11: Based on the camera unit of the surgical platform, the frame-by-frame images captured during the surgery are split into binocular images, and the separated left eye image is used as the system input;

[0033] Step S12: Downsample the image input to unify it to a resolution of 1280×720, and perform normalization calculations to convert the input to the range of [0,1], and convert it into the input form required for the joint learning deep neural network.

[0034] In this step, based on the video and normalized structure obtained from the surgical platform, the data structure and format required for subsequent deep neural network modules are constructed, which facilitates the use of subsequent deep neural network models and augmented reality navigation calculation methods.

[0035] Figure 2 This is a schematic diagram illustrating the overlay of images to demonstrate the augmented reality surgical navigation effect.

[0036] In one embodiment, step S2 above: Based on a joint learning deep neural network, for each frame of image, the surgical instrument mask is identified and predicted, and simultaneously the corneal region mask and the optical flow estimation results under the condition of no instrument obstruction are predicted, to provide key information required by the navigation system, specifically including:

[0037] Step S21: Based on the feature reuse framework, the features extracted from the previous frame are saved for use in the optical flow estimation calculation of the current frame. The feature completion results of the previous frame are also saved to provide prior information guidance for the feature completion calculation of the current frame. The joint learning deep neural network, based on a shared encoder structure, integrates the feature pyramid module and the hidden feature initialization module to extract multi-scale information from the image and effectively initialize the hidden features required for the iterative structure.

[0038] Step S22: Using the result propagation framework, the semantic segmentation prediction mask of the previous frame is distorted and aligned using the predicted optical flow information, providing prior information for the semantic segmentation calculation process of the current frame. Based on the joint learning strategy, the optical flow estimation module uses edge information obtained from the semantic segmentation results to guide the process and improve edge sharpness.

[0039] Step S23: Utilize an iterative structure to design multi-task sub-modules for network semantic segmentation, optical flow estimation, and feature completion. An iterative coefficient selection strategy can be employed to balance model performance and computational efficiency. By using a learning upsampling module, jointly learning a deep neural network model can achieve original-size multi-task result output.

[0040] Figure 3 This diagram illustrates the structure of the feature extraction and preprocessing submodules within the joint learning deep neural network module. The left sub-figure shows the feature processing submodule structure, where Conv represents a convolutional layer, BottleNeck represents the bottleneck module, the bottom shows the number of channels, and the top shows the scale setting. base S is the basic feature extracted. baseThis is based on the context features. The right subplot shows the preprocessing module, and SPPM is the simple feature pyramid module. Based on multi-scale features, To initialize the basic optical flow hidden features, To initialize the latent features for semantic segmentation, To initialize the semantic segmentation mask;

[0041] Figure 4 The structure of the basic result prediction and intermediate processing submodules is shown. From left to right, they are the basic optical flow estimation submodule, the basic semantic segmentation submodule, and the intermediate processing submodule. Among them, Conv Layers are multi-layer convolution, Laplacian is the Laplacian edge calculation process, ConvGRU is the convolutional gated recurrent unit, Conv RNN Cell is the basic unit of convolutional recurrent neural network, Flow Head, Occ Head, and Seg Head are the prediction heads for optical flow, occlusion, and semantic segmentation, respectively, warp is the warping operation, and Mask Calc is the mask completion calculation process. The results are presented in the following order: optical flow estimation, occlusion estimation, cross-correlation calculation, and semantic segmentation. Based on the hidden features of optical flow These are latent features for semantic segmentation. and This represents the semantic segmentation result of the previous frame at the same scale and the result after distortion and alignment. Ω s,ε Ω f,ε The masks shown in sequence are: the mask after channel separation, the semantic completion mask, and the optical flow completion mask. The components are, in order, the unoccluded mask after removing occlusion, the unoccluded contextual features, the unoccluded basic features, and the unoccluded optical flow;

[0042] Figure 5 The diagram illustrates the structure of the result completion and prediction submodules, from left to right: feature completion submodule, optical flow completion submodule, and mask completion submodule. 3D GateConv represents three-dimensional gated convolution, while 2D GateConv&MaskInpainting represents two-dimensional gated convolution and mask-based completion calculations. The results are presented in the following order: feature completion, optical flow completion, occlusion completion, cross-correlation calculation, and mask completion. and This is the same-scale semantic segmentation completion result of the previous frame and the result after distortion and alignment. and This is the result of feature completion at the same scale as the previous frame and the result after distortion and alignment. To complete the hidden features for optical flow, To complete the hidden features of the mask, This is used to initialize the mask completion result.

[0043] Step S24: Train the neural network model using a multi-variable joint loss function, which includes a multi-variable loss function for the semantic segmentation results;

[0044] L s,ε =w ce ·L ce +w oh ·L oh +w bs ·L bs +w kl ·L kl (1)

[0045] L mask =∑λ ε ·L s,ε (2)

[0046] Where L s,ε It is used for single-item mask prediction results, including upsampled basic semantic segmentation mask results. Upsampling mask completion results and initializing the semantic segmentation mask The loss function, and L mask It is used for iterative prediction results, including semantic segmentation results. Mask completion results and initialization mask completion results The loss function. L ce Let L be the cross-entropy loss function. oh For the loss function of the hard sample discovery strategy, L kl Let KL divergence loss function be used, and the prediction result of a single-frame semantic segmentation benchmark network be used as the target value. Boundary supervision loss L bs The Laplace boundary result is taken as the truth value, and supervision is provided only for the boundary region. λ is the base weight for each iteration, set to 1.2, ε is the iteration number, and w ce w oh w bs and w kl The weights for each loss function are set to 0.4, 1.0, 1.0, and 4.0, respectively.

[0047] L f,ε =w cha ·L cha +w bce ·L bce (3)

[0048] L flow =∑λ ε ·L f,ε (4)

[0049] Where L f,ε It is used for single-item output, including upsampled optical flow and occlusion completion results. and The loss function, and L flow It is used for iterative prediction results, including optical flow estimation results. Occlusion estimation results Optical flow completion results And occlusion completion results The loss function. L cha For the Chabonier loss function, L bce This is the binary cross-entropy loss function. λ is the base weight for each iteration, set to 1.2, and ε is the number of iterations. w cha w bce The weights for each loss function are all set to 1.0.

[0050]

[0051] in, This is the feature completion result of the ε-th feature completion submodule. These are the ground truth values ​​of unoccluded features extracted from the original unoccluded image.

[0052]

[0053]

[0054]

[0055] L = w base ·L base +w inp ·L inp +w F ·L F +w up ·L up (9)

[0056] Among them, L base and L inp L is the loss function output by the original and completed iterative modules. up It is the loss function of the upsampling result. Initialize semantic segmentation masks accordingly. Basic semantic segmentation results Basic optical flow and occlusion estimation results and The loss function. To correspond to the initialization mask completion results respectively Mask completion results Basic optical flow and occlusion completion results and The loss function. To correspond to the upsampled basic semantic segmentation mask results respectively Upsampling mask completion results Upsampling optical flow and occlusion completion results and The loss function. Weights w base w inp w F and w up Set them to 1.0, 1.2, 2.0 and 1.0 respectively.

[0057] Step S25: Using a synthesis strategy, based on unpaired training images and mask annotations, and employing a synthesis motion strategy and a random fusion strategy, a paired dataset containing occluded and unoccluded images with optical flow and mask ground truth values ​​is constructed through two synthesis and fusion calculations. This dataset is used for the end-to-end training process of the deep neural network, which includes image and mask fusion process and motion synthesis and fusion process.

[0058] I μ =∑w η ·B η (I bg ,I fg ,Ω B (10)

[0059] M μ =B N (M bg M fg ,Ω B (11)

[0060] Among them, B η It is a combination of three hybrid methods, including hybrid method B based on direct masking. N Gaussian Mixture Method B G Mixing method B with Laplace's pyramid L , and w η Ω is the fusion weight of each mixing method randomly sampled from the Dirichlet distribution H(1,1,1). B It is a foreground mask for surgical instruments. bg I fg I μ The images are, in order, the background image, the foreground image, and the composite image, M. bg M fg M μ For the corresponding background mask, foreground mask, and composite mask.

[0061] ω TPS (θ a,θ p ,θ t )→f TPS O TPS (12)

[0062] f μ =B N (f bg ,f fg ,Ω B (13)

[0063]

[0064] Where θ a θ p and θ t These are, in order, randomly sampled parameters for three simulated motions: affine transformation, projective transformation, and warp transformation. ω TPS The twisted mesh constructed for thin plate spline interpolation, f TPS With O TPS For the synthetic optical flow and occlusion mask corresponding to the distorted mesh, f bg f fg f μ In order, they are background optical flow, foreground optical flow, and composite optical flow. bg O fg O μ For the corresponding background occlusion mask, foreground occlusion mask, and composite occlusion mask. It is a new inconsistency region generated by the foreground object and is calculated by distorting the foreground mask of another frame.

[0065] Finally, a training dataset of 5500 sets of data was constructed using a synthetic strategy. The joint learning deep neural network model was trained for 100 epochs using a weighted adaptive moment estimation optimizer, with the learning rate initially set to 0.001 and decreasing by 5% every 4 epochs. A stochastic augmentation strategy was also used to improve the generalization performance of the joint learning deep neural network. PID-Net was used as a single-frame semantic segmentation benchmark network and trained on the dataset based on the same strategy.

[0066] In one embodiment, step S3 above involves: using a completion framework, integrating multiple neighboring images including the current frame and the unobstructed optical flow estimation results, performing distortion alignment on the forward neighboring frames using an optical flow propagation calculation method, restoring the occluded areas of the current frame using the unobstructed pixels of the forward neighboring frames, and further restoring the persistently occluded areas using the completion results saved from the previous frame, ultimately obtaining a clear image free of surgical instrument obstruction, specifically including:

[0067] Step S31: Save the completed optical flow prediction result of the current frame, and maintain the image information, grid information and occlusion information of the forward neighboring frames;

[0068] Step S32: The image information, mesh information, and occlusion information of the preceding neighboring frames are warped and aligned using the completed optical flow result of the current frame. Then, the warped mesh information is used to warp the images of the preceding neighboring frames to align with the current frame. Finally, based on the occlusion information, the occluded areas of the current frame are completed using pixel information from unoccluded neighboring frames at the same location, and the remaining occluded areas are recovered using the completion result of the previous frame. The completion result of the current frame will be saved for use in the completion calculation process of subsequent frames.

[0069] Figure 6 This diagram illustrates the occlusion frame completion process within the video frame completion framework module. (I) t For each frame of image Ω s,t For the corresponding masking layer, f t inp and For each frame, the full-size optical flow and occlusion results are completed; for the current frame, this corresponds to the upsampled optical flow and occlusion completion results. and To transmit distorted mesh information, To save the completion result of the previous frame, This is the completion result for the current frame.

[0070] In one embodiment, step S4 above: based on the segmentation mask result of the corneal contour under unobstructed conditions, combined with the image after removing the occlusion, visual information is superimposed on the unobstructed image through ellipse fitting calculation and suture point calculation, ultimately achieving the annotation and guidance of the corneal contour suture position and suture sequence, specifically including,

[0071] Step S41: Using the unobstructed corneal region mask obtained by the deep neural network, the corneal contour is extracted by Laplacian filtering, and the ellipse fitting result with the maximum intersection-union ratio with the unobstructed corneal region mask is calculated by using a point sampling strategy, and the obtained ellipse information is recorded.

[0072] Step S42: Using ellipse information, the coordinates of the ellipse contour division points are calculated according to the surgical requirements through the ellipse Fourier series. The positions of the suture points and corresponding suture lines are calculated based on the coordinates of the division points and the center coordinates of the ellipse. Augmented reality effects are then superimposed on the completion result of the current frame, and the sequence number visualization result of the suture order is further superimposed according to the surgical design.

[0073] Figure 2 The diagram and rendering show an overlay of images illustrating the augmented reality surgical navigation effect.

[0074] This invention discloses an augmented reality navigation method for lamellar keratoplasty based on a joint learning strategy. It can provide ophthalmologists with stable, clear, and accurate augmented reality navigation results in real time under severe occlusion conditions. Based on an iterative structure and feature reuse design, it offers high configuration flexibility, achieving a good balance between framework performance and computational load, and achieving good real-time computational performance in online surgical video scenarios. A data synthesis strategy is used to construct a dataset supporting deep neural network training, facilitating the use of unpaired surgical images with realistic annotations, thereby effectively reducing the data requirements and acquisition difficulty during network training. The method framework is as follows: Figure 7 As shown.

[0075] Example 2

[0076] like Figure 8 As shown, this embodiment of the invention provides an augmented reality navigation system for lamellar keratoplasty based on a joint learning strategy, comprising the following modules:

[0077] The surgical video frame acquisition and normalization module 81 is used to capture the surgical scene and obtain frame-by-frame images based on the camera unit and corresponding image encoding and decoding of the surgical platform. Based on the computational requirements, the frame-by-frame images obtained by the camera platform are downsampled and normalized to be unified into the same form and structure of input.

[0078] The joint learning deep neural network module 82 is used to identify the surgical instrument mask in the field of view for each frame of surgical video image through an iterative multi-task sub-module, and simultaneously predict the corneal region mask and the optical flow estimation results under the condition of no instrument obstruction, so as to provide the key information required by the navigation system.

[0079] The video frame completion framework module 83 is used to integrate the images of multiple neighboring frames, including the current frame, and the unobstructed optical flow estimation results using the completion framework. Through the optical flow propagation calculation method, the forward neighboring frames are distorted and aligned. The unobstructed area pixels of the forward neighboring frames are used to restore the occluded area of ​​the current frame. The completion results saved from the previous frame are used to further restore the continuously occluded area, and finally a clear image with the surgical instruments removed is obtained.

[0080] The augmented reality visualization module 84 is used to combine the segmentation mask results of the corneal contour under unobstructed conditions with the image after removing the occlusion. It achieves the overlay of visualization information on the unobstructed image through ellipse fitting calculation and suture point calculation, and finally realizes the annotation and guidance of corneal contour, suture position and suture sequence.

[0081] The above embodiments are provided merely for the purpose of describing the present invention and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.

Claims

1. A method for augmented reality navigation in lamellar keratoplasty based on a joint learning strategy, characterized in that, include: Step S1: Based on the camera unit and corresponding image encoding and decoding of the surgical platform, the surgical scene is captured and frame-by-frame images are obtained. Based on the computational requirements, the frame-by-frame images obtained by the camera unit are downsampled and normalized to unify the input into the same form and structure. Step S2: Based on the joint learning deep neural network, for each frame of image, the identification and prediction of the surgical instrument mask are realized, and the corneal region mask and the optical flow estimation results under the condition of removing instrument occlusion are obtained at the same time. Step S3: Using the completion framework, the forward multi-frame images including the current frame and the unobstructed optical flow estimation results are integrated. The forward neighboring frames are distorted and aligned using the optical flow propagation calculation method. The occluded areas of the current frame are restored using the unobstructed area pixels of the forward neighboring frames. The continuous occluded areas are further restored using the completion results of the previous frame. Finally, a clear image with surgical instruments removed is obtained. Step S4: Based on the segmentation mask result of the unobstructed corneal contour, combined with the image after removing the obstruction, the visualization information is superimposed on the image after removing the obstruction through ellipse fitting calculation and suture point calculation, so as to mark the corneal contour and provide guidance on suture position and suture sequence. Step S2 specifically includes: Step S21: Based on the feature reuse framework, the features extracted from the previous frame image are saved for use in the optical flow estimation calculation process of the current frame, and the feature completion results of the previous frame are saved to provide prior information guidance for the feature completion calculation process of the current frame. The joint learning deep neural network is based on a shared encoder structure, and integrates the feature pyramid module and the hidden feature initialization module to realize the extraction of multi-scale information of the image and the effective initialization of the hidden features required for the iterative structure. Step S22: Using the result propagation framework, the semantic segmentation prediction mask of the previous frame is distorted and aligned by the predicted optical flow information to provide prior information for the semantic segmentation calculation process of the current frame. Based on the joint learning strategy, the optical flow estimation module uses the semantic segmentation result to obtain edge information guidance to improve edge clarity. Step S23: The design of multi-task sub-modules for network semantic segmentation, optical flow estimation, and feature completion is realized by using an iterative structure. An iterative coefficient selection strategy can be used to balance model performance and computational efficiency. By using a learning upsampling module, the joint learning deep neural network model can achieve the original size multi-task result output. Step S24: Train the neural network model using a multi-joint loss function, including a multi-joint loss function for the semantic segmentation result; Step S25: Using a synthesis strategy, based on unpaired training images and mask annotations, and using a synthesis motion strategy and a random fusion strategy, through two synthesis and fusion calculations, a paired dataset containing occluded and unoccluded images with optical flow and mask real values ​​is constructed for the end-to-end training process of the deep neural network, which includes the image and mask fusion process and the motion synthesis and fusion process, respectively. Finally, a training dataset of 5500 sets of data was constructed using a synthetic strategy. The joint learning deep neural network model was trained for 100 epochs using a weighted adaptive moment estimation optimizer with an initial learning rate of 0.001 that decreased by 5% every 4 epochs. At the same time, a stochastic augmentation strategy was used to improve the generalization performance of the joint learning deep neural network. PID-Net was used as the baseline network for single-frame semantic segmentation and trained on the dataset based on the same strategy.

2. The method for augmented reality navigation in lamellar keratoplasty based on a joint learning strategy according to claim 1, characterized in that, Step S1 specifically includes: Step S11: Based on the camera unit of the surgical platform, perform binocular splitting on the frame-by-frame images captured during the surgery, and separate and acquire the left eye image as the system input; Step S12: Downsample the frame-by-frame image input to a resolution of 1280×720, and perform normalization calculation to convert the input to the range of [0,1], and convert it into the input form required by the joint learning deep neural network.

3. The method for augmented reality navigation in lamellar keratoplasty based on a joint learning strategy according to claim 1, characterized in that, In step S24, the specific multinomial loss function for the semantic segmentation result is as follows: (1) (2) in It is used for single-item mask prediction results, including upsampled basic semantic segmentation mask results. Upsampling mask completion results and initializing the semantic segmentation mask The loss function, and It is used for iterative mask prediction results, including semantic segmentation results. Mask completion results and initialization mask completion results The loss function; Let cross-entropy be the loss function. For the loss function of the hard sample discovery strategy, The loss function is KL divergence, and the semantic segmentation prediction result of the baseline network is used as the objective value, with boundary supervision loss. The Laplace boundary result is taken as the truth value and supervision is only provided for the boundary region; This is the base weight for each iteration, set to 1.

2. For the number of iterations; and , , and The weights for each loss function are set to 0.4, 1.0, 1.0, and 4.0, respectively. (3) (4) in It is used for single-item output, including upsampled optical flow and occlusion completion results. and The loss function, and It is used for iterative prediction results, including optical flow estimation results. Occlusion estimation results Optical flow completion results And occlusion completion results The loss function; For the Chabonier loss function, The binary cross-entropy loss function; This is the base weight for each iteration, set to 1.

2. For the number of iterations; and , The weights for each loss function are all set to 1.0; (5) in, It is the first Feature completion results of each feature completion submodule These are the ground truth values ​​of unoccluded features extracted from the original unoccluded image; (6) (7) (8) (9) in, and It is the loss function output by the original and completion iteration modules. It is the loss function of the upsampling result; , , Initialize semantic segmentation masks accordingly. Basic semantic segmentation results Basic optical flow and occlusion estimation results and The loss function; , , To correspond to the initialization mask completion results respectively Mask completion results Basic optical flow and occlusion completion results and The loss function; , , To correspond to the upsampled basic semantic segmentation mask results respectively Upsampling mask completion results Upsampling optical flow and occlusion completion results and Loss function; weights , , and Set them to 1.0, 1.2, 2.0, and 1.0 respectively; In step S25, the image and mask fusion process and the motion synthesis and fusion process are specifically as follows: (10) (11) in, It is a combination of three hybrid methods, including a hybrid method based on direct masking. Gaussian mixture method Hybrid method with Laplace's pyramid ,and From Dirichlet distribution The fusion weights for each mixing method in the random sampling. It is a foreground mask for surgical instruments. , , The images are arranged in the following order: background image, foreground image, and final composite image. , , For the corresponding background mask, foreground mask, and final composite mask; (12) (13) (14) in , and The parameters are, in order, random sampling parameters for three types of simulated motion: affine transformation, projection transformation, and warp transformation. Twisted meshes constructed for thin-plate spline interpolation. and The corresponding synthetic optical flow and occlusion mask for the distorted mesh, , , In order, they are background optical flow, foreground optical flow, and final composite optical flow. , , For the corresponding background occlusion mask, foreground occlusion mask, and final composite occlusion mask; It is a new inconsistency region generated by the foreground object and is calculated by distorting the foreground mask of another frame.

4. The method for augmented reality navigation in lamellar keratoplasty based on a joint learning strategy according to claim 1, characterized in that, Step S3 specifically includes: Step S31: Save the completed optical flow prediction result of the current frame, and maintain the image information, grid information and occlusion information of the forward neighboring frames; Step S32: Use the completed optical flow result of the current frame to warp and align the image information, grid information, and occlusion information of the forward neighboring frames; then use the warped grid information result to warp the forward neighboring frame image to align with the current frame; finally, based on the occlusion information, use the pixel information of the unoccluded neighboring frames at the same position to complete the occluded area of ​​the current frame, and use the completion result of the previous frame image to recover the remaining occluded area; the completion result of the current frame will be saved for the completion calculation process of subsequent frames.

5. The method for augmented reality navigation in lamellar keratoplasty based on a joint learning strategy according to claim 1, characterized in that, Step S4 specifically includes: Step S41: Using the unobstructed corneal region mask obtained by the deep neural network, the corneal contour is extracted by Laplacian filtering, and the ellipse fitting result with the maximum intersection-union ratio with the unobstructed corneal region mask is calculated using a point sampling strategy, and the obtained ellipse information is recorded. Step S42: Using the ellipse information, the coordinates of the ellipse contour division points are calculated according to the calculation requirements through the ellipse Fourier series. The positions of the suture points and corresponding suture lines are calculated according to the coordinates of the division points and the coordinates of the ellipse center. Augmented reality effects are then superimposed on the completion result of the current frame. Furthermore, the sequence number visualization result of the suture order is superimposed according to the surgical design.

6. A system for augmented reality navigation in lamellar keratoplasty based on a joint learning strategy, comprising the method of claim 1, characterized in that, Includes the following modules: The surgical video frame acquisition and normalization module is used to capture the surgical scene and obtain frame-by-frame images based on the camera unit and corresponding image encoding and decoding of the surgical platform. Based on the computational requirements, the frame-by-frame images obtained by the camera unit are downsampled and normalized to be input in the same form and structure. The joint learning deep neural network module is used to identify the surgical instrument mask in the field of view for each frame of surgical video image through iterative multi-task sub-modules, and simultaneously predict the corneal region mask and the optical flow estimation results under the condition of no instrument obstruction, so as to provide the key information required by the navigation system. The video frame completion framework module is used to integrate the images of multiple neighboring frames, including the current frame, and the unobstructed optical flow estimation results using the completion framework. Through the optical flow propagation calculation method, the forward neighboring frames are distorted and aligned. The unobstructed area pixels of the forward neighboring frames are used to restore the occluded area of ​​the current frame. The completion results saved from the previous frame are used to further restore the continuously occluded area, and finally a clear image with the surgical instrument obstruction removed is obtained. The augmented reality visualization module is used to combine the segmentation mask results of the corneal contour under unobstructed conditions with the image after removing the occlusion. Visual information is superimposed on the unobstructed image through ellipse fitting calculation and suture point calculation, and finally the annotation and guidance of the corneal contour, suture position and suture sequence are realized.

Images