Method and system for acquiring rotation angle of eyeball

Abstract

The invention discloses a method and a system for acquiring the rotation angle of an eyeball, which can acquire the rotation angle of the eyeball in dynamic movement of the eyeball, and belongs to the technical field of computer-aided medical treatment. The method for acquiring the rotation angle of the eyeball comprises the following steps: 1, preparing data; step 2, positioning corneal limbus and eye center in the eye picture; step 3, calculating eyeball rotation angles in different eye pictures; 4, the corneal meridian and the surgical incision position in the eye picture are obtained, S41, the corneal meridian and surgical incision rotation angle is obtained through the eyeball rotation angle, and therefore the real-time motion state of the corneal meridian and the surgical incision in the operation is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer-aided medical treatment, and in particular relates to a method and a system for obtaining an eyeball rotation angle. Background Art

[0003] Studies have shown that the location of the surgical incision will have a certain impact on postoperative corneal astigmatism. When the limbal tunnel incision is made in the direction of the corneal meridian, postoperative corneal astigmatism can be effectively reduced. In addition, the corneal meridian is also an important reference for the position of intraocular lens implantation. Accurately aligning the intraocular lens along the corneal meridian is a prerequisite for achieving the best visual results. In preoperative work for cataracts, the steepest meridian of the cornea is determined by corneal topography or keratometer. However, there is displacement, rotation and deformation between the intraoperative eye image and the preoperative eye image, resulting in changes in the planned corneal meridian and incision position. Doctors need to rely on clinical experience for dynamic adjustments. In order to improve the success rate and visual effect of the operation, some doctors usually use markers to mark the corneal meridian.

[0004] The technical defects of using markers are: the traditional markers used to mark the corneal meridian may be erased or become blurred during surgery due to factors such as eye secretions and flushing fluids, making it difficult for doctors to accurately identify the meridian position during surgery. Eye tissue is relatively soft and will be affected by certain external forces during surgery. The patient's eye movement and the slight contact of the surgical instrument with the surface of the eye may cause the marker to shift. Once the marker shifts, it will seriously interfere with the precise positioning of the surgery and affect the surgical effect.

[0005] Traditional image processing methods are prone to tracking failures in complex cataract surgery scenarios, and the results vary greatly for different patients. Deep learning-based methods have excellent representation and learning capabilities and can handle complex image tasks, thereby achieving more accurate and efficient results.

[0006] The deep model itself is complex and redundant, with a large number of parameters and taking up a lot of memory, and it is also very time-consuming. Therefore, it is considered to compress and optimize the relevant models, reduce the number of model parameters, improve the model's computational efficiency, and enhance the model's practicality. Summary of the invention

[0007] The purpose of the present invention is to provide a method and system for obtaining eyeball rotation angle, which can obtain the eyeball rotation angle during the dynamic movement of the eyeball.

[0008] The present invention is achieved by adopting the following technical solutions:

[0009] A method for obtaining eye rotation angle, step 1, data preparation;

[0010] S11, extracting eye images from eye videos;

[0011] S12, generating input data for the limbus segmentation model using thresholding and normalization operations.

[0012] Step 2: locate the corneal limbus and eye center in the eye image;

[0013] S21, inputting the input data generated by S12 into the trained limbus segmentation model, and using the limbus segmentation model to generate a limbus segmentation map;

[0014] S22, according to the corneal limbus segmentation map, contour detection and ellipse fitting are used to calculate the positions of the corneal limbus and the eye center.

[0015] Step 3, calculating the eye rotation angle in different eye images;

[0016] S31, inputting the corneal limbus segmentation map generated in step 2 into the trained feature point matching model, and calculating the coordinates of the matching points through the feature point matching model;

[0017] S32, obtaining the angle of the matching point through the eye center position and the matching point coordinates;

[0018] S33, comparing the angles of the same matching point in two different eye images, and calculating the angle difference of the same matching point in the compared two eye images;

[0019] S34, the obtained angle difference is the rotation angle of the eyeballs in the two compared eye images.

[0020] Further preferably, S11, after extracting the eye image from the eye video, adjust the pixel size of the eye image and the pixel size of the limbus segmentation label, and the adjustment method is as follows: the eye image is adjusted to a size of 640*480 pixels, S12, the pixel value of the eye image is divided by 255 for normalization to generate the input data of the limbus segmentation model, and the training data and test data of the limbus segmentation model are generated using thresholding and normalization operations, which are divided into training data and test data according to the number of eye videos. A total of 20 patients' eye videos are obtained, of which 10 groups are used as training sets, 3 groups are used as validation sets, and 7 groups are used as test sets. In the training process of the limbus segmentation model, step one also includes S13, finding the limbus edge in the eye image by manual marking.

[0021] Further preferably, S21 is subdivided into the following steps: S2101, the limbal segmentation model is constructed using the U-net network, the encoding end uses ResNet18 to replace the feature extraction network of the U-net network, and the limbal segmentation model replaces the long connection operation deep learning in the U-net network through a cross-attention fusion module; S2102, the decoding end subdivides and segments the limbal segmentation map through the edge attention module.

[0022] Further preferably, the input of the cross attention fusion model is the low-order feature map at the encoding end and the high-order feature map at the decoding end. First, the high-order feature map is adjusted using bilinear interpolation upsampling and 1×1 convolution and kept consistent with the size of the low-order feature map. The channel dimension of the feature map is average pooled and maximum pooled. After merging, the spatial attention weight is generated by the Sigmoid function; the high-order attention weight is multiplied by the low-order feature map, the low-order attention weight is multiplied by the high-order feature map, and the two are concatenated; convolution and tangent hyperbolic functions are used to generate offsets of the deformable convolution layer, and the features also need to be input into conventional convolution for structural feature extraction; these features are fused to induce the shape and Perception of deformation; The edge attention module is based on the adaptive boundary Gaussian heat map. The edge attention module uses the morphological corrosion operation to corrode the corneal limbus segmentation map generated by the corneal limbus segmentation model. After each corrosion, the target boundary is reduced by one circle, and the Gaussian result is calculated and assigned to the pixel of this corrosion, and the operation is repeated until the corrosion is completed; when generating the attention weight, a convolution layer with an output dimension of 1 is used to compress the multi-dimensional features, and then the attention weight is normalized by the Sigmoid function, the attention weight is upsampled and the loss value is calculated with the adaptive boundary Gaussian heat map obtained above, and then it is multiplied with the original input feature map to enhance the network's ability to pay attention to the edge.

[0023] Further preferably, in S22, a corneal limbus segmentation image is obtained by predicting the trained corneal limbus segmentation model, the coordinate values ​​of the pixel points at the corneal limbus are obtained by contour detection, and then the best fitting ellipse one of these coordinate sets is obtained by using an ellipse fitting algorithm based on the least squares method, and the ellipse center of the fitted ellipse one is used as the eye center, and the fitted ellipse one is used as the corneal limbus; the fitted ellipse two is generated by the ellipse center of the fitted ellipse one, and the fitted ellipse two and the fitted ellipse one form concentric ellipses, wherein the fitted ellipse one is the inner ellipse of the concentric ellipse, and the fitted ellipse two is the outer ellipse of the concentric ellipse, the major axis and the minor axis of the outer ellipse are 1.2 times the major axis and the minor axis of the inner ellipse, respectively, and the area of ​​the concentric ellipse is the corneal limbus area.

[0024] Further preferably, step 2 further includes S23, after calculating the positions of the corneal limbus and the eye center, drawing auxiliary lines of the positions of the corneal limbus and the eye center on the corresponding eye image.

[0025] Further preferably, training data for the feature point matching model is generated through affine transformation, and the method of training the feature point matching model is: the first input picture is an eye picture, and the second input picture is a picture generated by affine transformation of the eye picture. The point-to-point correspondence between the two pictures can be obtained through the affine matrix, which is used for calculating the loss function.

[0026] Further preferably, a long-term matching feature point matching model is constructed through the LoFTR framework, the LoFTR model is compressed and optimized using the knowledge distillation algorithm, the pre-trained LoFTR model is used as the teacher network model, the training data is passed through LoFTR to generate soft label data, the training data is passed through T-LoFTR (feature point matching model) to generate predicted data, and the predicted data and the soft label are used to calculate the distillation loss, and the student network model loss is calculated with the real hard label. The final loss is generated for back propagation to update the network parameters so that it fits the teacher network model LoFTR model, and ORB feature matching is used as the short-term matching method, and T-LoFTR feature matching is used as the long-term matching method. The two eye pictures for short-term matching are two eye pictures of adjacent frames, and the two eye pictures for long-term matching are the eye picture of the current frame and the standard eye picture.

[0027] Further preferably, in step four, the corneal meridian and surgical incision position in the eye image are obtained.

[0028] S41, obtaining the rotation angle of the corneal meridian and the surgical incision from the eyeball rotation angle, thereby knowing the real-time movement status of the corneal meridian and the surgical incision during the operation.

[0029] S42, after obtaining the corneal meridian and the surgical incision position, draw the corneal limbus meridian and the surgical incision position on the corneal limbus segmentation map. 0 ,y 0 ) coordinates to draw the eye center auxiliary line, with coordinates (x 0 -10,y 0 ) is the starting point, (x 0 +10,y 0 ) as the end point, draw a horizontal line; with coordinates (x 0 ,y 0 -10) as the starting point, (x 0 ,y 0 +10) as the end point, draw a vertical line; the cross auxiliary line is the eye center auxiliary line; with O(x 0 ,y 0 ), R 1 , R 2 , Ellipse structure parameters, draw the corneal margin auxiliary line; set the angle of the corneal meridian of the front eye image to θ 1, according to the calculated rotation angle difference δ m It can be seen that the angle of the corneal meridian of the current image is θ 1 +δ m , and then according to the center point O(x 0 ,y 0 ) can draw the corneal meridian; set the angle of the predetermined surgical incision in the front eye image to θ 2 , according to the calculated rotation angle difference δ m It can be seen that the angle of the current surgical incision is θ 2 +δ m , the point of this angle in the corneal limbus auxiliary line is the surgical incision position, and then a triangular surgical incision auxiliary line is drawn.

[0030] A system for obtaining an eyeball rotation angle is implemented by the above method for obtaining an eyeball rotation angle.

[0031] The invention comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein before the processor executes the computer program, the system model framework is first converted into an ONNX format and stored in the memory, so that the model can be deployed and inferred on different platforms;

[0032] The computer program includes two modules: a static phase and a dynamic phase. The static phase processes a single image, while the dynamic phase supports real-time processing of videos.

[0033] When executing the program, the system displays three submodules in the static positioning module in the static stage: limbus segmentation, feature matching area and matching result visualization; the limbus binary image is displayed on the system interface through the limbus segmentation model, the limbus position and candidate area selection scheme are displayed on the system interface through the feature point matching model, and the matching result displays the matching point difference result on the system interface, that is, the eyeball rotation angle;

[0034] The dynamic stage system displays four sub-modules of the dynamic positioning module: limbus, eye center, corneal meridian and surgical incision; the limbus real-time positioning module realizes rapid processing of video images, and realizes real-time positioning and display of the limbus position. The eye center positioning module can quickly process the displacement state of the eyeball in the video image during the operation. The real-time positioning of the corneal meridian and surgical incision confirms the rotation state of the eyeball from the side.

[0035] The beneficial effects of the present invention are as follows: (1) by drawing auxiliary lines to mark the rotation angle on the eyeball without leaving any mark, the eyeball movement state can be observed in real time. (2) the corneal limbus segmentation model used is a lightweight model, which reduces the number of model parameters, improves the model's computational efficiency, and enhances the model's practicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 Represents the overall technical solution of the present invention.

[0039] Figure 2 A structural diagram of the corneal limbus segmentation model of the present invention is shown.

[0040] Figure 3 Represents the cross-attention fusion module of the present invention.

[0041] Figure 4 Represents the edge attention fusion module of the present invention.

[0042] Figure 5 A flow chart showing the limbus segmentation and eye center positioning of the present invention.

[0043] Figure 6 A schematic diagram showing the eye rotation angle measurement of the present invention.

[0044] Figure 7 The flowchart of the eyeball rotation angle algorithm of the present invention is shown. DETAILED DESCRIPTION

[0045] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the scheme of the present invention will be further described below. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0046] In the description, it should be noted that the terms "first" and "second" are only used for descriptive purposes and should not be understood as indicating or implying relative importance. It should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be internal communication between two elements. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0047] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all of the embodiments.

[0048] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0049] Embodiment 1, a method for obtaining eyeball rotation angle, such as Figure 1 shown.

[0050] Step 1: Data preparation;

[0051] S11, extracting an eye image from the eye video. After extracting the eye image from the eye video, adjusting the pixel size of the eye image and the pixel size of the corneal limbus segmentation label.

[0052] Due to the special framing of medical eye videos, it is necessary to adjust the pixel size of the eye image and the corneal limbus segmentation label. The corneal limbus segmentation label can be understood as a segmentation label for the corneal part in the eye image. In this embodiment, the adjustment method is as follows: the acquired eye images are all collected from 1920×1080, 25 frames of eye video, and the eye images are adjusted to 640×480 pixels.

[0053] The eye image is adjusted to a size of 640*480 pixels. In S12, the pixel value of the eye image is divided by 255 for normalization to generate training data and test data for the corneal limbus segmentation model.

[0054] In this embodiment, the eye video data comes from (desensitized) videos of 20 cataract patients undergoing routine cataract surgery in an ophthalmology hospital, 10 groups of which are used as training sets (videos 11-20), 3 groups as validation sets (videos 8-10), and 7 groups as test sets (videos 1-7).

[0055] S13, find the limbus edge in the eye image by manual marking.

[0056] It should be noted that only the limbus segmentation model training stage requires manual marking of the limbus edge. After the limbus segmentation model is trained, it is not necessary to manually mark the limbus edge in the eye image when using it.

[0057] Step 2: locate the corneal limbus and eye center in the eye image;

[0058] S21, inputting the training data generated by S12 into the limbus segmentation model, and using the limbus segmentation model to generate a limbus segmentation map.

[0059] S2101, the corneal limbus segmentation model is constructed using the U-net network. The encoding end uses ResNet18 to replace the feature extraction network of the U-net network. The corneal limbus segmentation model replaces the long connection operation deep learning in the U-net network through the cross-attention fusion module.

[0060] The input of the cross-attention fusion model is the low-order feature map on the encoding end and the high-order feature map on the decoding end. First, the high-order feature map is adjusted using bilinear interpolation upsampling and 1×1 convolution and kept consistent with the size of the low-order feature map. The channel dimensions of the low-order feature map on the encoding end and the high-order feature map on the decoding end are average pooled and maximum pooled respectively. After merging, the spatial attention weights are generated by the Sigmoid function; the high-order attention weights are multiplied by the low-order feature map, the low-order attention weights are multiplied by the high-order feature map, and the two are concatenated; convolution and tangent hyperbolic functions are used to generate offsets for the deformable convolution layer, and the features also need to be input into conventional convolution for structural feature extraction; these features are fused to induce perception of the shape and deformation of the network.

[0061] Specifically, the structure of the corneal limbus segmentation model is as follows: Figure 2 As shown, the RCH-Net encoder structure uses ResNet18 as the feature extraction network and inputs the training data (i.e. Figure 2 The size of the middle eye is X∈R 512×512×3 , after feature extraction, the encoding end feature maps of different dimensions are generated, expressed as Figure 2 The rectangular block on the left side of the middle layer cross-fuses the feature maps of the encoding and decoding ends of adjacent layers through the cross-attention fusion module, that is, Figure 2 The CAF module in , enters the edge attention module to generate the previous level output feature map, that is Figure 2 In addition, during the training phase, a multi-scale loss function is used to supervise the corneal limbus segmentation maps at different levels so that the network can achieve better performance.

[0062] The cross attention fusion module is as follows Figure 3 As shown, the input is the low-level feature map (low-level feature) F at the encoding end low ∈R H×W×C2 And decoder high-level feature map (advanced features) Where H, W are the feature map sizes, C1, C2 are the feature map channel dimensions. The high-order feature map is upsampled using bilinear interpolation and adjusted with 1×1 convolution and kept consistent with the low-order feature map size, as shown in the following formula: high2 =Conv 1×1 (Up(F high )) Among them, Conv 1×1 is a 1×1 convolution, and Up is a bilinear interpolation upsampling.

[0063] Perform average pooling and maximum pooling on the feature map channel dimension, merge and generate spatial attention weights through the Sigmoid function. Multiply the high-order attention weights with the low-order feature map, multiply the low-order attention weights with the high-order feature map, and concatenate the two, as shown in the following formula:

[0064] S low =Sigmoid(MP(F low )+AP(F low ))

[0065] S high =Sigmoid(MP(F high2 )+AP(F high2 ))

[0066] F l ow1 =F l ow ×S hi g h

[0067] F hi g h 1 =F hi g h2 ×S l ow

[0068] Among them, MP is the maximum pooling, AP is the average pooling, S low , S high for low-order and high-order attention, F low1 , F high1 are the low-order and high-order feature maps of the output.

[0069] Convolution and tangent hyperbolic functions are used to generate offsets of the deformable convolution layer. The features also need to be input into regular convolution for structural feature extraction; these features are fused to induce the perception of the shape and deformation of the network, as shown in the following formula:

[0070] F′=Conv 3×3 (Cat(F low1 ,F high1 ))+DConv 3×3 (Cat(F low1 ,F high1 ))

[0071] Among them, Conv 3×3 For 3×3 convolution, DConv 3×3is a 3×3 deformable convolution, Cat is a concatenation operation, and F′ is the final output feature map.

[0072] S2102, the decoding end subdivides and segments the corneal limbus segmentation map through the edge attention module.

[0073] The edge attention module is based on the adaptive boundary Gaussian heat map. The edge attention module uses the morphological erosion operation to erode the corneal limbus segmentation map generated by the corneal limbus segmentation model. After each erosion, the target boundary is reduced by one circle, and the Gaussian result is calculated and assigned to the eroded pixel, and the operation is repeated until the erosion ends.

[0074] Specifically, the edge attention structure is as follows Figure 4 As shown, input: corneal limbus segmentation map, output: boundary Gaussian heat map H corresponding to the corneal limbus segmentation map (reference Figure 4 Prediction heatmap in ).

[0075] Calculate the target area S, calculate the width and height w and h of the target circumscribed rectangle, and calculate the area:

[0076] According to h, w, S, S r Counting loop iterations

[0077] Calculate the variance of the Gaussian function

[0078] Set the kernel function Kernel and copy the corneal limbus segmentation map record as N;

[0079] Start loop, the number of loops is k, i = 1, 2, ... k;

[0080] Use the kernel function Kernel to erode the corneal limbus segmentation map to obtain the eroded image er;

[0081] by As the threshold, the binary erosion image N is obtained to obtain the image th 0 ;

[0082] by is the threshold, and the binary erosion image er is obtained to obtain the image th 1 ;

[0083] If i is 0, the image th in N is 0 The pixel in the image is 255. 1 Update the pixel at position 0 in the middle pixel

[0084] If i is not 0, the image th in N is b The pixel in the image is 255. 1Update the pixel at position 0 in the middle pixel

[0085] Copy 1 for b ;

[0086] The loop ends when the number of loops is reached.

[0087] Update the pixel in N that satisfies the position of pixel 255 in image th_1 to: pixel = 0;

[0088] According to the above steps, an adaptive boundary Gaussian heat map for model training can be generated.

[0089] When generating attention weights, a convolutional layer with an output dimension of 1 is used to compress the multidimensional features, and then the attention weights are normalized by the Sigmoid function. The attention weights are upsampled and the loss value is calculated with the adaptive boundary Gaussian heat map obtained above, and then multiplied with the original input feature map to enhance the network's ability to pay attention to the edges.

[0090] S22, according to the corneal limbus segmentation map, contour detection and ellipse fitting are used to calculate the positions of the corneal limbus and the eye center.

[0091] The corneal limbus segmentation image is obtained by predicting the trained corneal limbus segmentation model (i.e., semantic segmentation is performed using the corneal limbus segmentation model). The coordinate values ​​of the corneal limbus pixel points are obtained by contour detection. Then, the ellipse fitting algorithm based on the least squares method is used to obtain the best fitting ellipse 1 of these coordinate sets. The important structural parameters of the fitting ellipse 1 include the ellipse center O(x 0 ,y 0 ), the major axis of the ellipse R 1 and short axis R 2 , the rotation angle of the ellipse, using the ellipse center of the fitted ellipse 1 as the eye center, and the fitted ellipse 1 as the corneal limbus,

[0092] Generate the fitted ellipse 2 by fitting the ellipse center of the ellipse 1, r 1 and r 2 As the major axis and minor axis, fitted ellipse 2 and fitted ellipse 1 form concentric ellipses, wherein fitted ellipse 1 is the inner ellipse of the concentric ellipse, and fitted ellipse 2 is the outer ellipse of the concentric ellipse, the major axis and minor axis of the outer ellipse are 1.2 times the major axis and minor axis of the inner ellipse respectively, and the area of ​​the concentric ellipse is the corneal limbus area.

[0093] S23, after calculating the positions of the corneal limbus and the eye center, drawing auxiliary lines of the positions of the corneal limbus and the eye center on the corresponding eye image.

[0094] Step 3, calculating the eye rotation angle in different eye images;

[0095] S31, inputting the corneal limbus segmentation map generated in step 2 into the trained feature point matching model, and calculating the matching point coordinates through the feature point matching model.

[0096] The feature point matching model training is as follows:

[0097] The training data of the feature point matching model is generated by affine transformation. The method of training the feature point matching model is as follows: the first input image is the eye image, the second input image is the image generated by the eye image after affine transformation, and the coordinates P obtained by affine transformation of a point P(x, y) in the eye image ′ (x ′ ,y ′ ) can be expressed as:

[0098]

[0099] Among them, M represents the affine matrix, R 00 , R 01 , R 10 , R 11 , T x , T y is the affine matrix parameter. It can be seen that the conversion between the coordinates of two points in two-dimensional space can be realized through the parameter matrix M. Specifically, R in the matrix M 00 , R 11 Controls the image scaling, R 00 , R 01 , R 10 , R 11 Controls the rotation angle of the image, and finally T x , T y The overall position of the image on the plane is controlled, and the image translation operation is realized. The point-to-point correspondence between the two images can be obtained through the affine matrix, which is used to calculate the loss function.

[0100] A long-term matching feature point matching model is constructed through the LoFTR framework, and the knowledge distillation algorithm is used to compress and optimize the LoFTR model. The pre-trained LoFTR model is used as the teacher network model. The training data is processed by LoFTR to generate soft label data. The training data is processed by T-LoFTR to generate predicted data. The predicted data and the soft label are used to calculate the distillation loss, and the student network model loss is calculated with the real hard label. The final loss is generated for backpropagation to update the network parameters so that it fits the teacher network model LoFTR model.

[0101] For ease of understanding, the following additional explanations are made: the soft label data is generated by the pre-trained LoFTR model (teacher network model). Its function is to calculate the distillation loss with the predicted data generated by T-LoFTR in the knowledge distillation algorithm, so as to help the T-LoFTR model better learn and fit the output features of the LoFTR model; the real hard label is obtained by manual marking in the eye picture. During the training process of the feature point matching model T-LoFTR, the input training data is processed based on its own structure and parameters, and the predicted data is output. The predicted data represents the understanding and judgment results of the T-LoFTR model on the input data in the current training stage, so that T-LoFTR can inherit the advantages of LoFTR to a certain extent. At the same time, through the constraints of the distillation loss, T-LoFTR is prompted to continuously optimize its own parameters, improve model performance, and respond quickly to input.

[0102] Oriented FAST and Rotated BRIEF (ORB) feature matching is used as a short-term matching method. The two images for short-term matching are two images of adjacent frames. The final output result is the coordinate values ​​of n groups of matching points in the two images. The short-term matching method can monitor the short-term dynamic changes of the eyeball during surgery and can quickly capture these changes. It outputs the coordinate values ​​of n groups of matching points in adjacent frame images, combines the center coordinates obtained when locating the eye center, calculates the angle between the matching point and the center coordinate, and then obtains the angle of eyeball rotation between adjacent frames; the short-term matching method can complement the long-term matching and jointly serve the needs of precise surgical operations.

[0103] T-LoFTR feature matching is used as the long-term matching method. The two images for long-term matching are the current frame image and the preoperative standard image. The final output result is the coordinate values ​​of n groups of matching points in the two images.

[0104] S32, obtaining the angle of the matching point through the eye center position and the matching point coordinates;

[0105] Among them, long-term matching is performed once every 1 second, that is, every 25 frames, and short-term matching is performed in the remaining 24 frames within 1 second.

[0106] S33, comparing the angles of the same matching point in two different eye images, and calculating the angle difference of the same matching point in the two compared eye images.

[0107] S34, the obtained angle difference is the rotation angle of the eyeballs in the two compared eye images;

[0108] The specific operations are as follows:

[0109] The coordinate system is established with the center of the fitted ellipse 1 as the origin of the coordinate system, the horizontal rightward direction as the positive direction of the horizontal axis, and the vertical downward direction as the positive direction of the vertical axis.

[0110] According to the coordinates of the matching points and connecting the coordinates of the origin, calculate the rotation angle θ with respect to the positive direction of the horizontal axis.

[0111] Obtain a set of matching points (i.e., matching pairs) of intraoperative eye images and standard eye images (i.e., preoperative eye images), and calculate the corresponding rotation angles θ 1 and θ 2 , and calculate the angle difference δ = θ 1 -θ 2 In this embodiment, the clockwise direction is defined as the positive direction. When δ>0, it means that the intraoperative eye image is rotated clockwise by δ° relative to the standard eye image. When δ<0, it means that the intraoperative eye image is rotated counterclockwise by δ° relative to the standard eye image. When δ=0, the eyeball does not rotate.

[0112] Find the median of all group matching points δ as the rotation angle difference δ m . Eye rotation angle measurement Figure 6 shown.

[0113] Step 4: Obtain the corneal meridian and surgical incision position in the eye image.

[0114] S41, obtaining the rotation angle of the corneal meridian and the surgical incision from the eyeball rotation angle, thereby knowing the real-time movement status of the corneal meridian and the surgical incision during the operation.

[0115] S42, after obtaining the corneal meridian and the surgical incision position, draw the corneal limbus meridian and the surgical incision position on the corneal limbus segmentation map.

[0116] According to the eye center O(x 0 ,y 0 ) coordinates to draw the eye center auxiliary line, with coordinates (x 0 -10,y 0 ) is the starting point, (x 0 +10,y 0 ) as the end point, draw a horizontal line; with coordinates (x 0 ,y 0 -10) as the starting point, (x 0 ,y 0 +10) as the end point, draw a vertical line; the cross auxiliary line is the eye center auxiliary line; with O(x 0 ,y 0 ), R 1 , R 2 , Ellipse structure parameters, draw the corneal limbus auxiliary line; set the angle of the corneal meridian of the standard eye image to θ1 , according to the calculated rotation angle difference δ m It can be seen that the angle of the corneal meridian of the current image is θ 1 +δ m , and then according to the center point O(x 0 ,y 0 ) can draw the corneal meridian; set the angle of the surgical incision predetermined in the standard eye picture to θ 2 , according to the calculated rotation angle difference δ m It can be seen that the angle of the current surgical incision is θ 2 +δ m , the point of this angle in the corneal limbus auxiliary line is the surgical incision position, and then a triangular surgical incision auxiliary line is drawn.

[0117] Embodiment 2: The specific method for obtaining the corneal limbus position and eye center positioning in Embodiment 1 is:

[0118] The model training is started based on the processed training data obtained in step 1 and the corneal limbus segmentation model designed in step 2. After the training is completed, the model weights are saved. When the corneal limbus segmentation model is tested, the model weights are loaded into the corneal limbus segmentation model and the test data is used for prediction. At this time, the predicted result P∈R 640×480×2 , where 640×480 is the size of the prediction result, 2 represents the output category, the first category is the background (not the limbus area), and the second category is the limbus. Finally, the category of the pixel is determined by the output probability of the background and limbus at the same position to generate a limbus segmentation map.

[0119] The limbal segmentation map is obtained by predicting the trained limbal segmentation model. At this time, the limbal segmentation map is a binary image. The findContours function in OpenCV is used to find the contour points in the image. The output is the coordinates of n contour points. Then, the fitEllipse function in OpenCV is used to fit the ellipse based on these coordinates. The output parameter is the center of the ellipse O(x 0 ,y 0 ), the major axis of the ellipse R 1 and short axis R 2 , the rotation angle of the ellipse The center of the ellipse is used as the eye center, the fitted ellipse is used as the corneal limbus, and auxiliary lines are drawn as visual cues. The process of corneal limbus segmentation and eye center positioning is as follows: Figure 5 shown.

[0120] Embodiment 3: The specific method for generating the limbal region in Embodiment 1 is as follows:

[0121] Using the above O(x 0 ,y 0 ), R 1 , R 2 , The fitting ellipse generated by the ellipse coefficient is the inner ellipse of the concentric ellipse, r 1 and r 2 As the major and minor axes, where r 1 =1.2R 1 , r 2 =1.2R 2 , generate the outer ellipse, the specific steps are as follows:

[0122] Create a completely black mask the same size as the eye image;

[0123] The mask uses O(x 0 ,y 0 ), r 1 , r 2 Draw an ellipse and set the values ​​of all pixels inside the ellipse to 255;

[0124] The mask uses O(x 0 ,y 0 ), R 1 , R 2 Draw an ellipse and set the values ​​of all pixels inside the ellipse to 0;

[0125] At this time, the white pixels in the mask are concentric ellipses, the black pixels in the corresponding mask in the eye image are set to 0, and the remaining area is the region of interest for feature point matching (i.e., the limbus area).

[0126] Embodiment 4, T-LoFTR uses the knowledge distillation algorithm to compress and optimize the LoFTR model. The specific operation is as follows: the lightweight LoFTR matching algorithm is generated based on LoFTR through channel pruning, knowledge distillation, lightweight module design and other schemes. Tables 1 and 2 are the configuration parameters between the original LoFTR and T-LoFTR, respectively. The lightweight design includes three parts. In the feature extraction network, the network is replaced from FPN+ResNet_8_2 to FPN+ResNet_16_4. The numbers represent the scaling ratio of coarse-grained features and fine-grained features compared to the original input dimension. Initial_dim is the initial channel dimension ready to enter the first feature extraction module, and block_dims is the channel dimension size corresponding to the module after each downsampling. In the coarse-grained stage, d_model corresponds to the last dimension of block_dims, which is the input dimension of the position entering the Transformer. nhead is the number of heads used in self-attention or cross-attention, and layer_names is the number of self-attention or cross-attention used.

[0127] Table 1 LoFTR related parameters

[0128]

[0129] Table 2 T-LoFTR related parameters

[0130]

[0131] The above is only a specific implementation of the present invention, which enables those skilled in the art to understand or implement the present invention. Although detailed descriptions are given with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments, and they should all be covered by the protection scope of the claims.

Claims

1. A method for obtaining an eyeball rotation angle, characterized in that: Step 1: Data preparation; S11, extracting eye images from eye videos; S12, generating input data for the limbus segmentation model using thresholding and normalization operations; Step 2: locate the corneal limbus and eye center in the eye image; S21, inputting the input data generated by S12 into the trained limbus segmentation model, and using the limbus segmentation model to generate a limbus segmentation map; S22, according to the corneal limbus segmentation map, the positions of the corneal limbus and the eye center are calculated by using contour detection and ellipse fitting; Step 3, calculating the eye rotation angle in different eye images; S31, inputting the corneal limbus segmentation map generated in step 2 into the trained feature point matching model, and calculating the coordinates of the matching points through the feature point matching model; S32, obtaining the angle of the matching point through the eye center position and the matching point coordinates; S33, comparing the angles of the same matching point in two different eye images, and calculating the angle difference of the same matching point in the compared two eye images; S34, the obtained angle difference is the rotation angle of the eyeballs in the two compared eye images.

2. A method for obtaining eyeball rotation angle according to claim 1, characterized in that: S11, after extracting the eye image from the eye video, adjust the pixel size of the eye image and the pixel size of the limbus segmentation label, the adjustment method is as follows: the eye image is adjusted to 640*480 pixel size, S12, the pixel value of the eye image is divided by 255 for normalization to generate the input data of the limbus segmentation model, and the training data and test data of the limbus segmentation model are generated using thresholding and normalization operations. A total of 20 patients' eye videos are obtained, of which 10 groups are used as training sets, 3 groups are used as validation sets, and 7 groups are used as test sets. In the training process of the limbus segmentation model, step one also includes S13, finding the limbus edge in the eye image by manual marking.

3. A method for obtaining eyeball rotation angle according to claim 1, characterized in that: S21 is subdivided into the following steps: S2101, the limbal segmentation model is constructed using the U-net network, the encoding end uses ResNet18 to replace the feature extraction network of the U-net network, and the limbal segmentation model replaces the long connection operation deep learning in the U-net network through the cross-attention fusion module; S2102, the decoding end uses the edge attention module to subdivide and segment the limbal segmentation map.

4. A method for obtaining eyeball rotation angle according to claim 3, characterized in that: The input of the cross attention fusion model is the low-order feature map at the encoding end and the high-order feature map at the decoding end. First, the high-order feature map is adjusted using bilinear interpolation upsampling and 1×1 convolution and kept consistent with the size of the low-order feature map. The channel dimensions of the low-order feature map at the encoding end and the high-order feature map at the decoding end are average pooled and maximum pooled respectively. After merging, the spatial attention weight is generated by the Sigmoid function; the high-order attention weight is multiplied by the low-order feature map, the low-order attention weight is multiplied by the high-order feature map, and the two are concatenated; convolution and tangent hyperbolic functions are used to generate offsets for the deformable convolution layer, and regular convolution is also input for structural feature extraction; these features are fused to induce the network Perception of shape and deformation; the edge attention module is based on the adaptive boundary Gaussian heat map. The edge attention module uses the morphological corrosion operation to corrode the corneal limbus segmentation map generated by the corneal limbus segmentation model. After each corrosion, the target boundary is reduced by one circle, and the Gaussian result is calculated and assigned to the pixel of this corrosion, and the operation is repeated until the corrosion is completed; when generating the attention weight, a convolution layer with an output dimension of 1 is used to compress the multi-dimensional features, and then the attention weight is normalized by the Sigmoid function, the attention weight is upsampled and the loss value is calculated with the adaptive boundary Gaussian heat map obtained above, and then it is multiplied with the original input feature map to enhance the network's ability to pay attention to the edge.

5. A method for obtaining eyeball rotation angle according to claim 1, characterized in that: In S22, a limbal segmentation image is obtained by predicting the trained limbal segmentation model, the coordinate values ​​of the pixel points at the limbus are obtained by contour detection, and then the best fitting ellipse one of these coordinate sets is obtained by using an ellipse fitting algorithm based on the least squares method, the ellipse center of the fitting ellipse one is used as the eye center, and the fitting ellipse one is used as the limbus; a fitting ellipse two is generated by the ellipse center of the fitting ellipse one, and the fitting ellipse two and the fitting ellipse one form concentric ellipses, wherein the fitting ellipse one is the inner ellipse of the concentric ellipse, and the fitting ellipse two is the outer ellipse of the concentric ellipse, the major axis and the minor axis of the outer ellipse are 1.2 times the major axis and the minor axis of the inner ellipse, respectively, and the area of ​​the concentric ellipse is the limbus area.

6. A method for obtaining eyeball rotation angle according to claim 5, characterized in that: Step 2 also includes S23, after calculating the positions of the corneal limbus and the eye center, drawing auxiliary lines of the positions of the corneal limbus and the eye center on the corresponding eye image.

7. A method for obtaining eyeball rotation angle according to claim 1, characterized in that: The training data of the feature point matching model is generated through affine transformation. The method of training the feature point matching model is as follows: the first input image is an eye image, and the second input image is an image generated by affine transformation of the eye image. The point-to-point correspondence between the two images can be obtained through the affine matrix, which is used to calculate the loss function.

8. A method for obtaining eyeball rotation angle according to claim 7, characterized in that: A long-term matching feature point matching model is constructed through the LoFTR framework, and the knowledge distillation algorithm is used to compress and optimize the LoFTR model. The pre-trained LoFTR model is used as the teacher network model. The training data is passed through LoFTR to generate soft label data, and the training data is passed through T-LoFTR to generate predicted data. The predicted data and the soft label are used to calculate the distillation loss, and the student network model loss is calculated with the real hard label. The final loss is generated for backpropagation to update the network parameters to make it fit the teacher network model LoFTR model. ORB feature matching is used as the short-term matching method, and T-LoFTR feature matching is used as the long-term matching method. The two eye pictures for short-term matching are the two eye pictures of adjacent frames, and the two eye pictures for long-term matching are the eye picture of the current frame and the standard eye picture.

9. A method for obtaining eyeball rotation angle according to claim 1, characterized in that: The method also includes step 4, obtaining the corneal meridian and the surgical incision position in the eye image, S41, obtaining the corneal meridian and the surgical incision rotation angle from the eyeball rotation angle, so as to know the real-time movement state of the corneal meridian and the surgical incision during the operation; S42, after obtaining the corneal meridian and the surgical incision position, drawing the corneal meridian and the surgical incision position on the corneal limbus segmentation map; drawing the eye center auxiliary line according to the coordinates of the eye center O(x0, y0), taking the coordinates (x0-10, y0) as the starting point and (x0+10, y0) as the end point, drawing a horizontal line; taking the coordinates (x0, y0-10) as the starting point and (x0, y0+10) as the end point, drawing a vertical line; the cross auxiliary line is the eye center auxiliary line; taking O(x0, y0), R1, R2, Ellipse structure parameters, drawing the auxiliary line of the corneal limbus; Set the angle of the corneal meridian of the front eye image to θ1, and calculate the rotation angle difference δ m It can be seen that the angle of the corneal meridian of the current image is θ1+δ m , and then the corneal meridian can be drawn according to the center point O(x0, y0); the angle of the predetermined surgical incision in the front eye image is set to θ2, and the calculated rotation angle difference δ m It can be seen that the current surgical incision angle is θ2+δ m , the point of this angle in the corneal limbus auxiliary line is the surgical incision position, and then a triangular surgical incision auxiliary line is drawn.

10. A system for obtaining eyeball rotation angle, characterized in that: Used to implement a method for obtaining an eyeball rotation angle as described in any one of claims 1 to 9; The invention comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein before the processor executes the computer program, the system model framework is first converted into an ONNX format and stored in the memory, so that the model can be deployed and inferred on different platforms; The computer program includes two modules: a static phase and a dynamic phase. The static phase processes a single image, while the dynamic phase supports real-time processing of videos. When executing the program, the system displays three submodules in the static positioning module in the static stage: limbus segmentation, feature matching area and matching result visualization; the limbus binary image is displayed on the system interface through the limbus segmentation model, the limbus position and candidate area selection scheme are displayed on the system interface through the feature point matching model, and the matching result displays the matching point difference result on the system interface, that is, the eyeball rotation angle; The dynamic stage system displays four sub-modules of the dynamic positioning module: limbus, eye center, corneal meridian and surgical incision; the limbus real-time positioning module realizes rapid processing of video images, and realizes real-time positioning and display of the limbus position. The eye center positioning module can quickly process the displacement state of the eyeball in the video image during the operation. The real-time positioning of the corneal meridian and surgical incision confirms the rotation state of the eyeball from the side.

Images