An AR navigation method and system for corneal transplant suturing operation based on instance segmentation network

Through example segmentation network and AR technology, the location of corneal sutures is tracked in real time and surgical guidance is provided, which solves the problem of uneven sutures in corneal transplantation, and improves the success rate of surgery and the effect of corneal recovery.

CN114283260BActive Publication Date: 2024-11-08BEIHANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111355075.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-16
Publication Date
2024-11-08
Estimated Expiration
2041-11-16

AI Technical Summary

Technical Problem

During corneal transplant surgery, it is difficult for doctors to obtain surgical guidance in real time, resulting in uneven suture position and poor corneal recovery, increasing the risk of surgery and the probability of complications.

Method used

The AR navigation method based on the instance segmentation network is used to segment the corneal contour and sutures in real time through video images, predict the optimal suture position, and the suture position is feedback in real time during the surgery through AR technology to help doctors adjust sutures.

Benefits of technology

It improves the success rate of the surgery, reduces the risk of surgery and the occurrence of complications, and ensures the quality of corneal suture and visual recovery effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114283260B_ABST
    Figure CN114283260B_ABST
Patent Text Reader

Abstract

The present invention relates to an AR navigation method and system for corneal transplant suturing operation based on an instance segmentation network, and the method comprises: S1: obtaining a corneal surgery image, segmenting the corneal contour of the corneal surgery image using an instance segmentation network, and obtaining a segmentation mask of the corneal suture and the corneal contour; S2: obtaining the coordinate position of the corneal suture according to the segmentation mask of the corneal suture; predicting the optimal corneal suture position according to the segmentation mask of the corneal contour; S3: comparing the optimal corneal suture position with the position of the corneal suture to determine whether the position of the corneal suture needs to be re-sewn; S4: superimposing a mark in the corneal contour image of the current frame through AR to mark the corneal suture position and the optimal corneal suture position. The method provided by the present invention can track the corneal suture and the corneal contour in real time during surgery, compare the real-time corneal suture position with the optimal corneal suture position, and feedback the doctor's actual corneal suture suturing situation to improve the success rate of surgery.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of ophthalmic surgery, and in particular to an AR navigation method and system for corneal transplant suturing operations based on an instance segmentation network. Background Art

[0002] Corneal disease is the most common blinding eye disease in the world. When a person suffers from infectious keratitis and does not receive timely surgical treatment, it will cause temporary or permanent damage to corneal vision. The incidence of corneal disease in my country is second only to cataracts, with a probability of about 11 in 10,000. In recent years, my country's medical technology has developed rapidly. Some corneal diseases that cause visual impairment and cannot be treated with drugs, such as infectious corneal ulcers, can be treated with deep anterior lamellar keratoplasty (DALK). The process of deep lamellar keratoplasty is mainly to remove the epithelial cells and stroma of the human cornea until only the elastic layer is left in the cornea. After the diseased cornea is removed, the corresponding intact cornea with the same thickness is used to fill the gap. Deep lamellar keratoplasty can effectively reduce the risk of endothelial cell loss and endothelial rejection, and reduce the occurrence of surgical complications such as astigmatism.

[0003] The proposal and development of deep lamellar keratoplasty has greatly improved the success rate of corneal transplantation. At the same time, eye surgery is difficult, the visual range is small, and the surgical precision is extremely high for doctors. Therefore, lamellar keratoplasty is the most challenging type of surgery in microsurgery. Some accidents during the operation, such as incomplete removal of corneal stroma, uneven corneal suture position, and failure to divide the entire cornea equally, will affect the patient's postoperative visual recovery effect. At the same time, surgical complications such as astigmatism, glaucoma, secondary retinal detachment, and posterior membrane formation will occur when the corneal suture is incomplete. When the suture quality is seriously unqualified, it will lead to surgical failure and secondary surgery. Deep lamellar keratoplasty requires that the doctor strictly control the suture position, keep the suture interval uniform, remove the corneal thickness moderately, and suture the suture through a certain thickness of the donor cornea so that the cornea returns to its normal contour after suture. Uneven sutures will prolong the postoperative recovery time and cause scars in severe cases.

[0004] Therefore, how to provide doctors with real-time surgical guidance during surgery, enable them to make timely decisions, and reduce surgical risks has become an urgent problem to be solved. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an AR navigation method and system for corneal transplant suturing operation based on an instance segmentation network.

[0006] The technical solution of the present invention is: an AR navigation method for corneal transplant suturing operation based on an instance segmentation network, comprising:

[0007] Step S1: acquiring a video image during lamellar corneal transplantation surgery, and for each frame of corneal surgery image in the video image, segmenting the corneal contour of the corneal surgery image using an instance segmentation network to obtain a segmentation mask of the corneal suture and the corneal contour; wherein the instance segmentation network includes a prototype mask module;

[0008] Step S2: Processing the segmentation mask of the corneal suture to obtain the coordinate position of the corneal suture; using ellipse fitting to process the segmentation mask of the corneal contour to predict the optimal corneal suture position;

[0009] Step S3: comparing the optimal corneal suture position with the corneal suture position to determine whether the corneal suture position is qualified, and if unqualified, providing feedback information to inform the doctor to re-suture;

[0010] Step S4: superimposing a mark in the corneal contour image of the current frame through AR to mark the corneal suture position and the optimal corneal suture position.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] The present invention discloses an AR navigation method for corneal transplant suturing operation based on an instance segmentation network, which can track the corneal suture and corneal contour in real time during the operation, compare the real-time corneal suture position with the predicted optimal corneal suture position, and feed back the doctor the actual corneal suture suturing situation to improve the success rate of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of an AR navigation method for corneal transplant suturing operation based on an instance segmentation network in an embodiment of the present invention;

[0014] Figure 2 Schematic diagram of the minimum circumscribed rectangle of the corneal suture in an embodiment of the present invention;

[0015] Figure 3 This is a diagram showing the effect of superimposing the corneal contour image on the corneal suture position in an embodiment of the present invention;

[0016] Figure 4 This is a rendering of the optimal corneal suture position superimposed with the corneal contour image in an embodiment of the present invention;

[0017] Figure 5 This is a structural block diagram of an AR navigation system for corneal transplant suturing operation based on an instance segmentation network in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention provides an AR navigation method for corneal transplant suturing operation based on an instance segmentation network, which can track the corneal sutures and corneal contours in real time, provide feedback to the doctor on the actual suturing status of the corneal sutures, and improve the success rate of the operation.

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0020] Embodiment 1

[0021] like Figure 1 As shown, an AR navigation method for corneal transplant suturing operation based on an instance segmentation network provided by an embodiment of the present invention includes the following steps:

[0022] Step S1: acquiring a video image during lamellar corneal transplantation surgery, and segmenting the corneal contour of each frame of corneal surgery image in the video image using an instance segmentation network to obtain a segmentation mask of the corneal suture and the corneal contour; wherein the instance segmentation network includes a prototype mask module;

[0023] Step S2: Processing the segmentation mask of the corneal suture to obtain the coordinate position of the corneal suture; using ellipse fitting to process the segmentation mask of the corneal contour to predict the optimal corneal suture position;

[0024] Step S3: comparing the optimal corneal suture position with the corneal suture position to determine whether the corneal suture position is qualified, and if unqualified, providing feedback information to inform the doctor to re-suture;

[0025] Step S4: Using AR to overlay a mark in the corneal contour image of the current frame, marking the corneal suture position and the optimal corneal suture position.

[0026] In one embodiment, the above step S1: obtaining a video image during lamellar corneal transplantation surgery, segmenting the corneal contour of each frame of corneal surgery image in the video image using an instance segmentation network to obtain a segmentation mask of the corneal suture and the corneal contour; wherein the instance segmentation network includes a prototype mask module, specifically including:

[0027] Step S11: Input the corneal surgery image into the instance segmentation network, and use the prototype mask module to obtain the mask coefficient matrix C of the corneal suture and corneal contour n =(c1 c2 c3 … c k ) and the mask matrix V; where c i is a matrix of size n×k; the matrix values ​​of V are only 0 and 1, as shown in formula (1);

[0028]

[0029] Where V is a matrix of size i×j×k; k represents the number of masks;

[0030] Step S12: The mask Mask of the corneal suture and corneal contour is calculated according to the following formula (2):

[0031] Mask=σ(VC T ) (2)

[0032] Among them, σ is the sigmoid function.

[0033] The embodiment of the present invention uses an instance segmentation network to segment the corneal contour. The instance segmentation network has a lightweight characteristic and meets the real-time requirements during surgery. The instance segmentation network uses a residual network block to superimpose a feature pyramid structure. The object image classification model of the feature pyramid (FPN) determines whether the sample belongs to the corneal tissue to be segmented. In medical images, there are corneal suture features on pixels with higher zoom levels, and semantic features such as corneal contours on pixels with lower zoom levels. Compared with the previous feature extraction network, the residual network block has the main advantage of introducing a residual module using a short-circuit mechanism, which enables the neural network to retain the original information of the image during multiple downsampling processes. The residual block of the segmentation network of the example of the present invention is divided into 5 levels, and the input of the feature pyramid network comes from the upper 3 levels of the residual block. The channel sizes of these three levels are 512, 1024, and 2048 respectively.

[0034] In one embodiment, the above step S2: processing the segmentation mask of the corneal suture to obtain the coordinate position of the corneal suture; using ellipse fitting to process the segmentation mask of the corneal contour to predict the optimal corneal suture position, specifically includes:

[0035] Step S21: The contour point set of the corneal suture Mask is combined into a matrix P by row, as shown in formula (3):

[0036]

[0037] Among them, x and y are the coordinates of the contour point;

[0038] Step S22: Calculate the center point M of the corneal suture according to formula (4):

[0039]

[0040] Step S23: According to formula (5), the contour point set is normalized using M to obtain the matrix normalization B:

[0041] B=P-hM T (5)

[0042] Where h is an n×1 column vector, represented by h[i]=1, i=1,…,n;

[0043] Step S24: Use the normalized B to construct the covariance matrix D, as shown in formula (6):

[0044]

[0045] Step S25: According to formula (7), construct the matrix E so that EDE T becomes a diagonal matrix, where E is the matrix of the eigenvectors of the covariance matrix D arranged in rows, and each row is an eigenvector λ of D:

[0046] EDE T =F (7)

[0047] Among them, the matrix F is a diagonal matrix composed of the eigenvalues ​​of D, and its value is shown in formula (8):

[0048]

[0049] Among them, λ is the eigenvector;

[0050] Step S26: according to the characteristic vector in E, the direction of the corneal suture is obtained; according to the center point M of the corneal suture and its direction, the intersection points P2 and P3 of the corneal contour midline and the corneal contour can be obtained, and P2P3 divides the corneal contour into left and right parts;

[0051] Traverse the coordinates of the left and right corneal contour points to find the coordinates of the points P0 and P1 farthest from the corneal contour midline; find the minimum circumscribed rectangle x based on P0, P1, P2, and P3 R0 ,x R1 ,x R2 ,x R3 , calculation formulas (9) to (12) are as follows:

[0052]

[0053]

[0054]

[0055]

[0056] Among them, k longis the slope of the long side of the minimum circumscribed rectangle, k short is the slope of its short side, P0 and P1 are the points on both sides of the contour farthest from the midline, and P2 and P3 are the intersection points of the corneal contour midline and the contour;

[0057] like Figure 2 , which is a schematic diagram of the minimum circumscribed rectangle of a corneal suture obtained by solving P0, P1, P2, and P3;

[0058] Step S27: Construct the joint loss function L total Solve the final mask M and the joint loss function L total It consists of the candidate box loss function, the confidence loss function, the pixel-level mask loss function and the semantic segmentation loss function:

[0059] The confidence loss function is shown in formula (13):

[0060]

[0061] in, Indicates whether the i-th predicted box matches the j-th true box regarding category p, and its value is 0 or 1;

[0062] The candidate box loss function is shown in formula (14):

[0063]

[0064] in, Indicates the candidate box position, Indicates the real frame position;

[0065] The pixel-level mask loss function is shown in formula (15):

[0066]

[0067] Among them, x represents the label of the sample, and its value is 0 or 1. represents the predicted probability of the network;

[0068] The semantic segmentation loss function is shown in formula (16):

[0069]

[0070] Constructing the joint loss function L total As shown in formula (17):

[0071] L total =αL bbox +βL class +γL mask +δL seg (17)

[0072] Among them, α, β, γ, and δ are weight coefficients respectively;

[0073] Step S28: Perform ellipse fitting using ellipse Fourier coefficients according to the mask of the corneal contour to predict the optimal corneal suture position.

[0074] Through this step, the real-time corneal suture position during the operation is calculated, and the optimal corneal suture position is predicted. By comparing the real-time corneal suture position with the optimal corneal suture position, it is determined whether the corneal suture position needs to be adjusted to make it closer to the optimal corneal suture position.

[0075] In one embodiment, the above step S3: comparing the best corneal suture with the position of the corneal suture to determine whether the position of the corneal suture is qualified, and if it is unqualified, providing feedback information to inform the doctor to re-suture, specifically includes:

[0076] The corneal suture and the optimal corneal suture coordinates are calculated according to formulas (18) to (19):

[0077]

[0078] α=|θ G -θ F | (19)

[0079] Where d is the position deviation of the corneal suture, α is the angle deviation of the corneal suture; G represents the optimal corneal suture, F represents the corneal suture; θ G is the optimal corneal suture angle, θ F is the angle of the corneal suture;

[0080] Determine whether the positional deviation and angle deviation of the corneal suture relative to the optimal corneal suture exceed a preset threshold. If so, the corneal suture at this position is unqualified, and feedback information is given to inform the doctor to re-sew; otherwise, no re-sew is required.

[0081] In the embodiment of the present invention, when the deviation d between the position of the corneal suture and the optimal corneal suture position exceeds 30% of the corneal suture length, or the α value exceeds 15 degrees, it indicates that the cornea at this position is sutured to a large extent and needs to be re-sutured.

[0082] In order to better visualize and display the corneal suture and the optimal corneal suture, the embodiment of the present invention uses AR technology to superimpose the position marks of the two in the corneal contour image.

[0083] In one embodiment, the above step S4: superimposing a mark in the corneal contour image of the current frame by AR to mark the corneal suture position and the optimal corneal suture position specifically includes:

[0084] Ellipse fitting is performed on the foreground point set obtained from the segmentation result of the corneal suture image of the current frame, the corneal contour is marked with the fitted ellipse, and the fitted ellipse is equally divided and uniformly sampled to mark multiple uniformly distributed optimal corneal suture positions.

[0085] Figure 3 The effect of superimposing the corneal contour image on the corneal suture position is shown;

[0086] Figure 4 The effect of superimposing the corneal contour image on the optimal corneal suture position is shown.

[0087] The present invention discloses an AR navigation method for corneal transplant suturing operation based on an instance segmentation network, which can track the corneal suture and corneal contour in real time during the operation, compare the real-time corneal suture position with the optimal corneal suture position, and provide feedback to the doctor on the actual corneal suture suturing situation, thereby improving the success rate of the operation.

[0088] Embodiment 2

[0089] like Figure 5 As shown, an embodiment of the present invention provides an AR navigation system for corneal transplant suturing operation based on an instance segmentation network, comprising the following modules:

[0090] An instance segmentation module 51 is used to obtain video images during lamellar corneal transplantation surgery, and for each frame of corneal surgery image in the video image, segment the corneal contour of the corneal surgery image using an instance segmentation network to obtain a segmentation mask of the corneal suture and the corneal contour, wherein the instance segmentation network includes a prototype mask module;

[0091] The module 52 for obtaining the position of corneal sutures is used to process the segmentation mask of the corneal sutures to obtain the coordinate position of the corneal sutures; the segmentation mask of the corneal contour is processed using ellipse fitting to predict the optimal corneal suture position;

[0092] The corneal suture position determination module 53 is used to compare the optimal corneal suture position with the corneal suture position to determine whether the corneal suture position is qualified. If it is unqualified, feedback information is given to inform the doctor to re-sew;

[0093] The AR marking module 54 is used to superimpose a mark in the corneal contour image of the current frame through AR to mark the corneal suture position and the optimal corneal suture position.

[0094] The above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should all be included within the scope of the present invention.

Claims

1. An AR navigation method for corneal transplant suturing operation based on instance segmentation network, characterized in that: include: Step S1: acquiring a video image during lamellar corneal transplantation surgery, and for each frame of corneal surgery image in the video image, segmenting the corneal contour of the corneal surgery image using an instance segmentation network to obtain a segmentation mask of the corneal suture and the corneal contour; wherein the instance segmentation network includes a prototype mask module; Step S2: Processing the segmentation mask of the corneal suture to obtain the coordinate position of the corneal suture; using ellipse fitting to process the segmentation mask of the corneal contour to predict the optimal corneal suture position; Step S3: comparing the optimal corneal suture position with the corneal suture position to determine whether the corneal suture position is qualified, and if unqualified, providing feedback information to inform the doctor to re-suture; Step S4: superimposing a mark in the corneal contour image of the current frame through AR to mark the corneal suture position and the optimal corneal suture position.

2. The AR navigation method for corneal transplant suturing operation based on instance segmentation network according to claim 1, characterized in that: The step S1: obtaining a video image during lamellar corneal transplantation surgery, and for each frame of corneal surgery image in the video image, segmenting the corneal contour of the corneal surgery image using an instance segmentation network to obtain a segmentation mask of the corneal suture and the corneal contour; wherein the instance segmentation network includes a prototype mask module, specifically including: Step S11: Input the corneal surgery image into the instance segmentation network, and use the prototype mask module to obtain the mask coefficient matrix of the corneal suture and corneal contour and mask matrix V; where, is a matrix of size n×k; the matrix values ​​of V are only 0 and 1, as shown in formula (1); (1) Where V is a matrix of size i×j×k; k represents the number of masks; Step S12: The mask Mask of the corneal suture and corneal contour is calculated according to the following formula (2): (2) in, It is the sigmoid function.

3. The AR navigation method for corneal transplant suturing operation based on instance segmentation network according to claim 1, characterized in that: The step S2: processing the segmentation mask of the corneal suture to obtain the coordinate position of the corneal suture; using ellipse fitting to process the segmentation mask of the corneal contour to predict the optimal corneal suture position, specifically includes: Step S21: Combining the contour point set of the corneal suture Mask into a matrix P by row, as shown in formula (3): (3) Among them, x and y are the coordinates of the contour point; Step S22: Calculate the center point M of the corneal suture according to formula (4): (4) Step S23: According to formula (5), the contour point set is normalized using M to obtain a normalized matrix B: (5) Where h is an n×1 column vector, expressed as ; Step S24: Use the normalized matrix B to construct a covariance matrix D, as shown in formula (6): , Step S25: According to formula (7), construct the matrix E so that EDE T becomes a diagonal matrix, where E is the matrix of the covariance matrix D eigenvectors arranged in rows, and each row is an eigenvector of D : (7) Among them, the matrix F is a diagonal matrix composed of the eigenvalues ​​of D, and its value is shown in formula (8): (8) in, is the feature vector; Step S26: according to the characteristic vector in E, the direction of the corneal suture is obtained; according to the center point M of the corneal suture and its direction, the intersection points P2 and P3 of the corneal contour midline and the corneal contour can be obtained, and P2P3 divides the corneal contour into left and right parts; Traverse the coordinates of the left and right corneal contour points to find the coordinates of the point farthest from the corneal contour midline P0, P1; 1, Find the minimum enclosing rectangle of P2 and P3 , the calculation formulas (9)~(12) are as follows: (9) (10) (11) (12) Among them, k long is the slope of the long side of the minimum circumscribed rectangle, k short is the slope of its short side, P0 and P1 are the points on both sides of the contour farthest from the midline, and P2 and P3 are the intersection points of the corneal contour midline and the contour; Step S27: Construct joint loss function Solving for the final mask, the joint loss function It consists of the candidate box loss function, the confidence loss function, the pixel-level mask loss function and the semantic segmentation loss function: The confidence loss function is shown in formula (13): (13) in, Indicates whether the i-th predicted box matches the j-th true box regarding category p, and its value is 0 or 1; The candidate box loss function is shown in formula (14): (14) in, , Indicates the candidate box position, Indicates the real frame position; The pixel-level mask loss function is shown in formula (15): (15) Among them, x represents the label of the sample, and its value is 0 or 1. represents the predicted probability of the network; The semantic segmentation loss function is shown in formula (16): (16) Construct the joint loss function As shown in formula (17): (17) in, are weight coefficients respectively; Step S28: performing ellipse fitting using ellipse Fourier coefficients according to the Mask of the corneal contour to predict the optimal corneal suture position.

4. The AR navigation method for corneal transplant suturing operation based on instance segmentation network according to claim 1, characterized in that: The step S3: comparing the best corneal suture with the position of the corneal suture to determine whether the position of the corneal suture is qualified, and if it is unqualified, providing feedback information to inform the doctor to re-suture, specifically includes: The corneal suture and the optimal corneal suture coordinates are calculated according to formulas (18) to (19): (18) (19) Wherein, d is the position deviation of the corneal suture, α is the angle deviation of the corneal suture; G represents the optimal corneal suture, and F represents the corneal suture; is the optimal corneal suture angle, is the angle of the corneal suture; Determine whether the positional deviation and angle deviation of the corneal suture relative to the optimal corneal suture exceed a preset threshold. If so, the corneal suture at this position is unqualified, and feedback information is given to inform the doctor to re-sew; otherwise, no re-sew is required.

5. The AR navigation method for corneal transplant suturing operation based on instance segmentation network according to claim 1, characterized in that: The step S4: superimposing a mark in the corneal contour image of the current frame by AR to mark the corneal suture position and the optimal corneal suture position, specifically includes: Ellipse fitting is performed on the foreground point set obtained from the segmentation result of the corneal suture image of the current frame, the corneal contour is marked with the fitted ellipse, and the fitted ellipse is equally divided and uniformly sampled to mark multiple uniformly distributed optimal corneal suture positions.

6. An AR navigation system for corneal transplant suturing based on instance segmentation network, characterized in that: Includes the following modules: An instance segmentation module is used to obtain video images during lamellar corneal transplantation surgery, and for each frame of corneal surgery image in the video image, segment the corneal contour of the corneal surgery image using an instance segmentation network to obtain a segmentation mask of the corneal suture and the corneal contour, wherein the instance segmentation network includes a prototype mask module; A module for obtaining the position of corneal sutures is used to process the segmentation mask of the corneal sutures to obtain the coordinate position of the corneal sutures; the segmentation mask of the corneal contour is processed using ellipse fitting to predict the optimal corneal suture position; A corneal suture position judgment module is used to compare the optimal corneal suture position with the corneal suture position to judge whether the corneal suture position is qualified, and if unqualified, to give feedback information to inform the doctor to re-sew; The AR marking module is used to superimpose a mark in the corneal contour image of the current frame through AR to mark the corneal suture position and the optimal corneal suture position.