An end-to-end ophthalmic surgical positioning navigation method and device

Abstract

The present application relates to the field of medical image processing, and particularly relates to an end-to-end ophthalmic surgery positioning navigation method and device. First, a coding and decoding network is used to extract eyeball features, and a multi-class segmentation layer constructed according to the eyeball features is used for semantic segmentation to determine the center position of the eyeball, so as to construct a polar coordinate sampler, so that the real-time collected intraoperative image is directly projected under the polar coordinate, and the eyeball feature map, the semantic segmentation result map and the preoperative image are converted to the polar coordinate system. Under the polar coordinate system, the current image and the target image are registered to complete eyeball rotation tracking. In the registration process, a twin network is combined with a correlation filter, the tracking target position, number and receptive field size are defined by defining a Gaussian template, the calculation efficiency is improved, the weighted sum of the inter-frame registration result and the long-time registration result is taken as the final rotation amount, the cumulative error is eliminated, and the stability and registration accuracy of the eyeball rotation continuous registration are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, in particular to an end-to-end ophthalmic surgery positioning navigation method and device. BACKGROUND

[0002] In ophthalmic surgery, there is usually a need for eye positioning and rotation registration. For example, in cataract surgery, the position and size of the capsulorhexis and the implantation angle of the intraocular lens all have a crucial influence on the postoperative visual recovery of the patient. In traditional surgery, the doctor uses a slit lamp marking method to make an invasive color marking on the eye to determine the position, size and implantation angle of the intraocular lens. This method is low in safety, poor in accuracy and easy to cause discomfort to the patient. In recent years, with the development of computer navigation assisted ophthalmic surgery, ophthalmologists have the ability to accurately position the eye, effectively improving the safety and efficiency of surgery, and the patient's recovery effect is better.

[0003] For example, CN111616800A discloses an ophthalmic surgery navigation system integrated with a surgical microscope, which includes a surgical microscope, a video recording system, an eye positioning navigation module and an eyepiece projection module. The ophthalmic surgery navigation system is used to complete iris boundary segmentation and sclera multi-position capillary rotation tracking, and the cumulative error is eliminated by registration with a reference image. Since the rotation tracking method adopted selects several boxes at the iris boundary and uses multiple single target trackers for tracking, the ability to cope with large-scale rotation is weak.

[0004] For example, CN112043383B discloses an ophthalmic surgery navigation system and electronic equipment, which enriches the intraoperative information feedback by registering the OCT three-dimensional image and the two-dimensional surgical microscope image, and presents the surgical picture by using a 3D display in the process, so as to improve the safety and accuracy of the surgery. However, the registration method of the OCT three-dimensional image and the two-dimensional surgical image and the lesion segmentation method are introduced more vaguely. In addition, the three-dimensional registration of high-resolution images has a large amount of calculation, and it is difficult to realize real-time navigation during surgery.

[0005] The ophthalmic surgery navigation systems disclosed in the prior art all have certain defects, so it is of positive significance to study an ophthalmic surgery navigation method with higher registration accuracy. SUMMARY

[0006] The purpose of the present application is to provide an end-to-end ophthalmic surgery positioning navigation method and device to improve the working efficiency and registration accuracy of ophthalmic surgery navigation.

[0007] To achieve the above purpose, the present application adopts the following technical solutions:

[0008] An end-to-end ophthalmic surgery navigation positioning method, comprising the following steps:

[0009] Step 1, obtaining preoperative images of the patient's eye;

[0010] Step 2, constructing a coding-decoding network, the coding-decoding network is a neural network based on U-Net structure, wherein the encoding network inputs the patient's eyeball image, and outputs the extracted image deep and shallow feature maps; the image deep feature represents global information, and the shallow feature represents local information; the decoding network inputs the image deep and shallow feature maps, and outputs the eyeball feature map fused with the deep and shallow features; wherein the encoder adopts TransU-Net or the encoder of EfficientNet;

[0011] Step 3, constructing a multi-class segmentation layer for semantic segmentation of the iris, sclera and cornea regions of the eyeball; the multi-class segmentation layer is composed of multiple convolution layers with a convolution kernel size of 1, and the input of the multi-class segmentation layer is the eyeball feature map, and the output of the multi-class segmentation layer is the semantic segmentation result, i.e. the segmentation prediction map of the iris, sclera and cornea;

[0012] Step 4, calculating the center of the iris region according to the semantic segmentation result, and taking the center as the eyeball center position; constructing a polar coordinate sampler with the eyeball center position as the pole;

[0013] Step 5, using the polar coordinate sampler constructed in step 4 to collect the intraoperative images in real time and convert them to the polar coordinate system, and inputting the preoperative images, the semantic segmentation result and the eyeball feature map into the polar coordinate sampler to convert them into polar coordinate images;

[0014] Step 6, in the polar coordinate system, registering the current image and the target image to obtain the rotation information of the eyeball, thereby completing the rotation navigation of the eyeball;

[0015] Step 6.1, constructing and training a twin network to extract the scleral capillary features of the search image and the target image in the polar coordinate system; the search image is the current frame image in the real-time image, and the target image includes the preoperative image and the previous frame image, wherein the previous frame image is the target image for inter-frame registration, and the preoperative image is the target image for long-time registration;

[0016] Step 6.2, defining a multi-point Gaussian template, the Gaussian template is an image in which the image gray value is distributed in a Gaussian function, the peak value position of the gray value is the target position, and the position of the Gaussian distribution in the Gaussian template is the position to be registered in the target image; training a correlation filter online using the Gaussian template and the target image, calculating the cross-correlation of the correlation filter and the search image, and obtaining a correlation response map;

[0017] Step 6.3. Building a confidence network with the correlation response map as input to obtain a target position area prediction map; multiplying the target position area prediction map with the correlation response map, and the product is taken as the registration result output; the peak position in the registration result is the target position, the correlation response value of the target position is the confidence of the registration result, and the displacement of the peak position compared with the angle coordinate axis of the Gaussian template in the polar coordinate system is the rotation amount of the point;

[0018] Step 6.4. Calculating the average of the displacement amounts of the multiple target positions in the search image in the Gaussian template, and taking the average as the rotation amount of the search image compared with the target image, i.e. the inter-frame registration result and the long-time registration result; calculating the weighted sum of the inter-frame registration result and the long-time registration result as the final rotation amount of the eyeball of the current frame image; calculating the iris area of the current frame image according to the final rotation amount, and completing the eyeball center positioning and eyeball rotation tracking by registering the current image with the target image.

[0019] Further, the above-mentioned end-to-end ophthalmic surgery positioning and navigation method further comprises:

[0020] Step 7. Data augmentation is performed on the registered images, and the process is as follows:

[0021] Superimposing capillary feature information from random positions of other eye surgery videos at the registration position, the capillary features superimposed at the registration position by each pair of registered images are the same, and the capillary features superimposed by different pairs of registered images are different.

[0022] Further, the Gaussian template defined in step 6.2 has multiple Gaussian templates, and a dense registration point set is established according to the multiple Gaussian templates to perform non-rigid registration of the target image and the search image.

[0023] Further, the twin network comprises a first neural network and a second neural network, the first neural network is connected by convolution layers and deformation or composition in sequence; the second neural network has the same structure as the first neural network and shares the weights; the convolution layers of the first neural network and the second neural network are connected with a polar coordinate sampler, receive preoperative image or real-time intraoperative image data in the polar coordinate system, and extract capillary feature maps as output; the deformation network receives the semantic segmentation result and the capillary feature map in the polar coordinate system, and performs deformation compensation on the capillary feature map in the polar coordinate system according to the limbus edge in the semantic segmentation result converted to the polar coordinate system, to obtain an iris boundary capillary feature map eliminating deformation.

[0024] Further, step 6.4 further comprises calculation of the translation amount in the process of eyeball center positioning and rotation tracking, and the calculation method of the translation amount is as follows:

[0025] Step 6.4.1, a segmentation branch of the twin network is constructed, the segmentation branch is composed of a convolutional layer and a classification layer; the convolutional layer receives the iris boundary capillary feature map extracted by the twin network and the eyeball feature map transformed from the rectangular coordinate system, and performs fusion and then performs iris region segmentation through the classification layer, and the centroid of the iris region in the rectangular coordinate system is the final eyeball center;

[0026] Step 6.4.2, the displacement amount of the eyeball center obtained in step 6.4.1 compared with the initial eyeball center of the surgery is calculated, and the displacement amount is the translation amount of the eyeball.

[0027] Further, the weighting and calculation method of the inter-frame registration and long-time registration results in step 6.4 is a Kalman filtering algorithm; the detailed calculation process is as follows:

[0028] The covariance of the inter-frame registration result and the covariance of the long-time registration are summed; the weight m is calculated according to the covariance summation result:

[0029] m=P / (P+Q)

[0030] Wherein P represents the covariance of the inter-frame registration, and Q represents the covariance of the long-time registration.

[0031] Further, the step 3 further comprises: setting a full connection layer after the coding and decoding network, the input of the full connection layer is the eyeball feature map output by the decoder, which is used to judge the current image state to exclude abnormal state images.

[0032] Further, the encoder in step 2 is provided with two branches of CNN and Token Mixing; wherein the CNN branch is used to obtain local information, and the branch processes the surgical image through n convolutional layers, n is a natural number greater than 1; the branch based on Token Mixing is used to obtain global information, the image is divided into S non-overlapping image blocks as the input of the Token Mixing branch, all image blocks are linearly mapped to the hidden layer, and the position embedding is added to the mapped features to save the relative position of each image block; the MLP network is used to operate on each channel dimension of the features to output the results containing feature interaction; the local information obtained by the CNN branch and the global information obtained by the Token Mixing branch are fused using an attention gate;

[0033] The decoder part adopts the same structure as the Attention U-Net, and uses the attention gate to strengthen the features of the encoding part.

[0034] An ophthalmic surgical microscope device, comprising an objective lens system, an eyepiece system, a beam splitter system, a video recording assembly, a surgical navigation module, and a projection and display module.

[0035] The objective lens system focuses the light beam to the patient's eye to achieve clear imaging;

[0036] The eyepiece lens system projects the image plane into the doctor's eye, so that the microscope image and the projection pattern generated by the surgical navigation module can be directly observed by the doctor;

[0037] The beam splitter system divides the light beam into two or more paths, one of which leads to the eyepiece lens system, and the other leads to the video recording assembly;

[0038] The video recording assembly transmits the image or video into the surgical navigation module;

[0039] The surgical navigation module is connected to the projection and display module for end-to-end ophthalmic surgery positioning navigation method to obtain the navigation information of the eyeball segmentation; The projection and display module projects the received navigation information into the microscope field of view through the beam splitter system and the eyepiece lens system, or displays the real-time video collected by the video recording assembly on the display end through the external display to assist the doctor in surgery.

[0040] Further, the ophthalmic surgical microscope device is a binocular stereoscopic vision ophthalmic surgical microscope, which has left and right eye video recording assemblies, and its projection and display module is a three-dimensional projection and display module;

[0041] The objective lens system divides into two beams of light and projects them into the left and right eyepieces, respectively. The light path of the left eyepiece passes through the beam splitter to divide a beam into the left eye video recording assembly, and the light path of the right eyepiece passes through the beam splitter to divide a beam into the right eye video recording assembly;

[0042] The surgical navigation module receives the images obtained by the left and right eye video recording assemblies, and calibrates the internal and external parameters of the left and right eyepieces. According to the calibration results, a binocular stereoscopic vision system is established to obtain the depth information of the eyeball in the left and right eyepiece images. Then, the end-to-end ophthalmic surgery positioning navigation method is executed respectively to obtain the navigation information of the left eyeball segmentation and the navigation information of the right eyeball segmentation, and the average position is taken in the three-dimensional space;

[0043] The three-dimensional projection and display module projects the navigation information after taking the average position into the microscope field of view through the beam splitter system and the eyepiece lens system, or displays the real-time video collected by the video recording assembly on the display end through the external display to assist the doctor in surgery.

[0044] The application provides an end-to-end ophthalmic surgery positioning navigation method, which extracts eyeball features by using a coding and decoding network, performs semantic segmentation on a multi-class segmentation layer constructed according to the eyeball features to determine the center position of the eyeball, and constructs a polar coordinate sampler with the center position of the eyeball. The coordinate sampler can directly project the real-time collected intraoperative image to the polar coordinate system on one hand, and can convert the eyeball feature map, the semantic segmentation result map and the preoperative image to the polar coordinate system on the other hand. In the polar coordinate system, the current image and the target image are registered to complete eyeball center positioning and iris boundary segmentation. In the registration process, a twin network is combined with a correlation filter to define the target position, number and receptive field size by defining a Gaussian template, thereby avoiding the dependence on prior parameters such as size, scale and number caused by anchor setting, improving the calculation efficiency, and being more reasonable and accurate for blood vessel tracking which has no clear boundary. The weighted sum of the results of the inter-frame registration and the long-term registration is taken as the final rotation amount, thereby eliminating the cumulative error, improving the stability of the eyeball rotation continuous registration, and weakening the interference of the error registration result on the rotation registration. A dense registration point set is established by using multiple defined Gaussian templates to realize the non-rigid registration of the target image and the search image, and the inter-frame registration and the long-term registration are realized at the same time. The increase of the number of registration targets can be realized by increasing the Gaussian templates, without multiple feature extractions. Compared with the use of multiple single target trackers, the method has higher efficiency. In addition, the entire registration process is completed in the polar coordinate system, the rotation of the target is changed into the translation of the target by using the polar coordinate system, and after the rotation amount is changed into the translation amount, the calculation process is simpler and the calculation efficiency is higher.

[0045] The end-to-end ophthalmic surgery positioning navigation method is used as a navigation module in an ophthalmic surgery micro-imaging system, which is beneficial to assisting the doctor to complete the surgery task efficiently, accurately and safely, and lays a good foundation for the postoperative recovery of the patient. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a workflow diagram of the end-to-end ophthalmic surgery positioning navigation method of the embodiment applied to ophthalmic surgery and equipment;

[0047] Figure 2 is a flowchart of the end-to-end ophthalmic surgery positioning navigation method of the application;

[0048] Figure 3 is a structure diagram of the twin network part in the application;

[0049] Figure 4 is a method diagram of the network training set registration label data enhancement of the application;

[0050] Figure 5 is a Gaussian template setting and registration response and target region prediction map in the application;

[0051] Figure 6 is a schematic diagram of processing binocular stereovision microscope images according to the present application;

[0052] Figure 7 is an ophthalmic surgical microscope device according to the present application;

[0053] Figure 8 is a binocular stereovision surgical microscope according to the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings.

[0055] An end-to-end ophthalmic surgical positioning navigation method provided by the present embodiment, as shown in the figure, includes the following steps: Figure 2

[0056] Step 1: Obtain the preoperative image of the patient's eye, which needs to be clear, without trailing, containing the iris and part of the sclera, and the pupil position is in the middle.

[0057] Step 2: Construct a coding and decoding network, which is a neural network based on the U-Net structure, including an encoder 201 and a decoder 202, the encoder 201 is connected with the decoder 202; wherein the encoding network inputs the patient's eyeball image, and outputs the extracted image deep and shallow feature maps; the deep feature of the image represents the global information, and the shallow feature represents the local information. The input of the decoding network is the deep and shallow feature maps of the image, and the output is the eyeball feature map fused with the deep and shallow features. In the coding and decoding network of the present embodiment, the encoder 201 can adopt TransU-Net or the encoder of EfficientNet.

[0058] ​In addition, due to the large amount of parameters of the Transformer network, the calculation cost is large, and the CNN convolution receptive field is limited. In practical applications, the CNN and Token Mixing combination method can be used to obtain local and global information of the image. That is, two branches of CNN and Token Mixing are set in the encoder, where the CNN branch is used to obtain local information, and the branch processes the surgical image through n convolution layers, where n is a natural number greater than 1, such as 2, 3, 4, and 5. The branch based on Token Mixing is used to obtain global information. Assuming that the original image is H*W, the branch image is divided into S non-overlapping image patches as input, and the size of each image patch is P*P, then S = H*W / P^2. All image patches are linearly mapped to the hidden layer, and similar to the Transformer, the mapped features are added with position embedding to save the relative position of each image patch. In order to obtain global information, an MLP is used to operate on each channel dimension of the feature, and the output result contains the interaction result of the feature. Finally, an attention gate is used to fuse local and global information. The decoder part adopts the same structure as the Attention U-Net, and uses the attention gate to strengthen the features of the encoding part, making the algorithm more robust.

[0059] Using the combination of CNN and Token Mixing, the local and global information is fully utilized, and both low-level spatial information and high-level semantic information can be obtained at the same time. At the same time, the Token Mixing method is based on MLP, and the parameter amount is much smaller than that of the Transformer, which is suitable for the deployment of real-time systems.

[0060] Step 3, a multi-class segmentation layer 204 is constructed for semantic segmentation of the iris, sclera and cornea regions of the eyeball. The multi-class segmentation layer 204 is composed of multiple convolution layers with a convolution kernel size of 1, and the input is the eyeball feature map, and the output is the semantic segmentation result, i.e. the segmentation prediction map of the iris, sclera and cornea.

[0061] Step 4, the center of the iris region is calculated according to the semantic segmentation result, and is taken as the eyeball center position X center ,Y center ; the eyeball center position is taken as the pole to construct a polar coordinate sampler.

[0062] Step 5, the polar coordinate sampler constructed in step 4 is used to collect the intraoperative image in real time, and convert it to the polar coordinate system; the preoperative image, the feature map and the semantic segmentation result are input into the polar coordinate sampler to be converted into a polar coordinate image. In the polar coordinate sampler, the conversion method between the rectangular coordinate and the polar coordinate is:

[0063] A polar coordinate image pixel grid matrix r and θ, R ij , A ij are the elements of r and θ, respectively, to establish the coordinate mapping relationship of the polar coordinate system and the rectangular coordinate system with the eyeball center (X center , Y center ) as the origin:

[0064] r = R h×w , R ij = j

[0065] θ = A h×w , A ij = i * 2 / h - 1

[0066]

[0067] where w and h are the width and height of the polar coordinate image, the matrix size of r and θ is h x w, S eye is the area of the iris and pupil region, b is a constant, f is the scaling ratio of the polar coordinate image, which changes according to the change of the iris radius, and t is the initial angle of the polar coordinate transformation, wherein the multiplication of r and θ is dot multiplication, and the multiplication of other matrices is cross multiplication, x and y are the mapping positions of the polar coordinate image in the rectangular coordinate system, and the image in the rectangular coordinate system is sampled 209 using the coordinate mapping, and then the image can be projected into the polar coordinate system 210.

[0068] Step 6, in the polar coordinate system, the current image and the target image are registered to obtain the navigation information of the eyeball rotation, so as to complete the eyeball center positioning and rotation tracking. In this embodiment, the eyeball registration in the polar coordinate system is realized by registering the capillary features of multiple points on the target image and the search image. Continuous inter-frame image registration can ensure good continuity of eyeball rotation registration, but there is cumulative error in long-term registration. Registration of the current frame image and the reference image with the reference image as the target image (i.e. long-time registration) can avoid error accumulation, but it is difficult to maintain stability and continuity for complex movement, occlusion and deformation of the eyeball during the operation. The end-to-end method of the present application simultaneously registers two types of targets, so that the network has good accuracy, stability and continuity. The detailed process is as follows:

[0069] Step 6.1, construct and train a twin network, the input of the twin network is the preoperative image and real-time intraoperative image data in the polar coordinate system, which is used to extract the scleral capillary features of the search image and the target image in the polar coordinate system. The search image is the current frame image in the real-time image, and the target image includes the preoperative image and the previous frame image, wherein the previous frame image is the target image of inter-frame registration, and the preoperative image is the target image of long-time registration.

[0070] In the registration process, the deformation of the blood vessel features is one of the main difficulties. Therefore, the deformation network is arranged in the twin network of the embodiment to solve the deformation problem of the blood vessel features, so that the extracted features are more robust. The structure of the twin network of the embodiment includes a first neural network and a second neural network, the first neural network is composed of a convolution layer 213 and a deformation network 214 connected in sequence; the second neural network has the same structure as the first neural network and shares the weights; the convolution layers of the first neural network and the second neural network are connected with the polar coordinate sampler 209, and the deformation network 214 is connected with the convolution layer 209.

[0071] Since the eyeball is not a regular shape during the operation, the deformation is often caused by the extrusion and dragging of the intervention instrument, which is quite different from the reference image, thereby reducing the success rate of long-term registration. The arrangement of the deformation network can restore the corneal limbus in the classification prediction map obtained in step 3 to the polar coordinate system, and always restore the eyeball edge to a circular shape according to the deformation and stretching of the edge, thereby eliminating the registration difficulty caused by the eyeball deformation and improving the stability of the long-term registration. The embodiment takes the projection v of the corneal limbus segmentation result in the polar coordinate system as an example to illustrate the process of deformation elimination.

[0072] As shown in Figure 3 , assuming that the gray value of the target position in v is 1 and the gray value of the background position is 0, in the polar coordinate sampler, the coordinate mapping relationship between the polar coordinate system and the rectangular coordinate system is established:

[0073]

[0074] In the formula, x and y are points in v, the size matrix d is h x w, and the difference between the sum of the corneal limbus segmentation v result in the y direction and the axis position in the image is the deformation variable of each row in the polar coordinate image. x' and y' are the mapping positions of the deformation elimination feature map in the original polar coordinate feature map. By using the coordinate mapping to sample and project 301 the feature map in the polar coordinate system, the deformation of the image can be eliminated, and the extracted features are more robust.

[0075] Step 6.2, define a multi-point Gaussian template 402, train a correlation filter online using the Gaussian template and the target image, calculate the cross-correlation of the correlation filter and the search image, and obtain a correlation response map. Specifically:

[0076] The Gaussian template of the embodiment is an image with Gaussian function distribution of image gray values, the peak position of the gray values is the target position, and the position of the Gaussian distribution in the Gaussian template is the position to which the target image is expected to be registered. There are multiple Gaussian templates, and a dense registration point set is established according to the multiple Gaussian templates to perform non-rigid registration of the target image and the search image. The update strategy of the inter-frame registration Gaussian template can be to update frame by frame according to the previous frame position prediction result to continuously track the same target, or to update only with the sclera region transformation, to be uniformly distributed in the sclera region at the sclera boundary, and to perform registration only on the inter-frame deformation. The correlation filter is updated with the update of the target image and the Gaussian template.

[0077] The embodiment adopts correlation filtering to calculate the cross-correlation of the correlation filter and the search image, and the working mechanism is that the correlation filter discriminates and regresses the surgical image 215 to generate a Gaussian distribution response reaching a peak at the target position. That is, the correlation filter is learned through the Gaussian label G 216 and the target image T 211, and the multi-vessel feature target tracking of the frame-by-frame image is realized. The correlation filter w can be calculated by calculating the correlation response i *T i Linear ridge regression with the Gaussian label G is performed to learn online:

[0078]

[0079] wherein λ is a regularization parameter to prevent overfitting.

[0080] The correlation filter w performs bit-by-bit correlation calculation on the search image S 210 through a circulant matrix. To improve the calculation efficiency, the target image T 211 and the search image S 210 are converted to the frequency domain for calculation through discrete Fourier transform F 218, thereby improving the calculation efficiency. The multiplication in the frequency domain corresponds to the convolution in the time domain.

[0081]

[0082] wherein F*(S) is the complex conjugate of F(S), and · is the Hadamard product.

[0083] response=F -1 (F * (w)·F(S))

[0084] The correlation filter w is multiplied by the search image 210 to obtain a response map in the frequency domain. The response map is converted back to the frequency domain through inverse discrete Fourier transform 219 to obtain a time domain response map response 220. The numerical value of the response map represents the correlation size of the position and the target.

[0085] Step 6.3. The confidence network 221 is constructed with the correlation response map as input to predict the target location area in the current frame image, and a target location area prediction map 405 is obtained. The confidence network 221 is composed of k layers of convolution-normalization-activation layers, and a layer of convolution kernel with a value of 1. k is an integer greater than or equal to 2, and is usually 2, 3, or 4. The target location area prediction map 405 is multiplied by the response map 404, and the peak position in the registration result 222 is the target position in the current frame. The correlation response value of the peak position is the confidence of the registration result. The angle axis offset of the response map peak position 404 relative to the Gaussian template 402 is the rotation amount of the point position. If the tracking is abnormal, the classification result 405 is empty, i.e., no tracking position is output. The classification branch is used to classify the response map 404, eliminate non-target area interference, and improve the robustness of registration.

[0086] Step 6.4. The Gaussian template 402 of multiple point positions is defined, and the average value of the displacement amount of multiple target positions in the search image in the Gaussian template is calculated as the rotation amount of the search image relative to the target image, thereby obtaining the inter-frame registration result C and the long-time registration result L. As shown in FIG. 6, the Gaussian template 402 is defined as a circle with a radius of 2 pixels, and the center of the circle is the target position in the target image. The search image is searched for the target position in the Gaussian template, and the displacement amount of the target position in the search image in the Gaussian template is calculated. The average value of the displacement amounts of multiple target positions in the search image in the Gaussian template is calculated as the rotation amount of the search image relative to the target image, thereby obtaining the inter-frame registration result C and the long-time registration result L. Figure 4 The long-time registration with the preoperative image can uniformly distribute the center position of the Gaussian function, or can set the Gaussian template at a specified position of interest of the doctor. In the inter-frame registration, the Gaussian template with a center position uniformly distributed in the sclera region can be selected for each frame, i.e., fixed position registration; or the Gaussian template can be set with the target position registered in the previous frame as the center, i.e., continuous tracking of the same feature.

[0087] To eliminate accumulated errors, improve the stability of continuous rotation registration of the eyeball, and weaken the interference of false registration results on rotation registration, the embodiment adopts a weighted sum of the inter-frame registration result and the long-time registration result as the final rotation amount 226:

[0088] Z = C + m × (C - L)

[0089] m is the weight of the weighted sum of the inter-frame registration result and the long-time registration result, and Z is the final rotation amount.

[0090] The weighted sum calculation of the inter-frame registration result and the long-time registration result in the embodiment adopts the Kalman filtering algorithm. Kalman filtering is an optimization algorithm for optimal state estimation by fusing multiple observation information, and can suppress the interference of noise in the system result. E is the real rotation amount of the eyeball, and the results C and L of the inter-frame registration and the long-time registration can be regarded as systems containing noise γ and ε, respectively.

[0091] C = E + γ

[0092] L = E + ε

[0093] In the Kalman filter method, the weight m is the Kalman gain.

[0094] m = P / (P+Q)

[0095] P and Q are the covariances of inter-frame registration and long-time registration respectively. The weight of the weighted sum of the results of inter-frame registration and long-time registration dynamically changes in the surgical engineering according to the registration results.

[0096] According to the final rotation amount, the iris region of the current frame image is calculated, and the registration of the current image and the target image is completed to position the eyeball center and segment the iris boundary.

[0097] In the process of positioning the eyeball center and segmenting the iris boundary, the translation amount is also calculated, and the calculation method of the translation amount is as follows:

[0098] The segmentation branch of the twin network is constructed, the segmentation branch is composed of a convolution layer and a classification layer; the convolution layer receives the iris boundary capillary feature map extracted by the twin network and the eyeball feature map converted from the rectangular coordinate system, and performs fusion and iris region segmentation through the classification layer, and the centroid of the iris region in the rectangular coordinate system is the final eyeball center; the displacement of the eyeball center compared to the initial eyeball center of the surgery is the translation amount of the eyeball.

[0099] In step 3 of the embodiment, a fully connected layer is further set after the encoding and decoding network, the input of the fully connected layer is the eyeball feature map output by the decoder, and the current image state is judged according to the feature map to exclude abnormal state images. In actual use, the registered images are subjected to data enhancement, and the method of data enhancement is as follows: the capillary feature information from a random position of other eye surgery videos is superimposed at the registration position, and the capillary features superimposed at the registration position of each pair of registered images are the same, and the capillary features superimposed by different pairs of registered images are different.

[0100] In the network construction and training process of the embodiment, the labels corresponding to the data are labeled by using a computer-aided method. In the process of semantic segmentation, a small amount of data is labeled by using artificial labeling, the iris boundary region and the sclera region are pixel-level labeled, and the image abnormal state is binary labeled, the image abnormality is 0 and the normality is 1. The network is trained with the existing labels to predict a large amount of data. The correct labels are selected by artificial screening, and the coverage rate of correct prediction is improved by adjusting the network parameters. For samples that are difficult to predict correctly, the computer-labeled results are modified by artificial modification or labeled by artificial labeling.

[0101] For the tracking problem, a large number of surgical videos are labeled with tracking labels. For example, Figure 5, the artificial selects the sclera blood vessel feature point 502 located at the iris boundary 506 on the initial frame image 501 as the tracking point, adopts a single target tracker such as KCF based on correlation filtering or SiamFc++ based on a twin network to track the blood vessel feature point, the artificial selects the tracking result, retains the correct result as the blood vessel feature registration label, and modifies the error result. Each video selects scattered u blood vessel feature points 502, 503, 504, 505 for labeling, and u is an integer greater than 2. When training the end-to-end network, the data is enhanced, the image is subjected to affine transformation, Gaussian blur processing, and brightness non-uniform change processing, which enhances the adaptability of the network to the deformation of the eyeball in the operation, the image blur problem, and the uneven light of the picture. The network needs two images 507, 508 with the same blood vessel feature label 509, 510 as a group of labels to train the registration branch. For each group of labels, an additional randomly selected surgical image 511 with completed sclera segmentation is extracted, the sclera region containing the blood vessel information is subjected to affine transformation 512 of scaling and rotation, and then superimposed 513, 514 with the training image according to the eyeball center position, so that the same blood vessel feature is superimposed at the labeled position of the same group of training images under the constraint condition of affine transformation, to obtain a new label 515, 516, thereby enhancing the diversity of the labeled blood vessel feature and improving the generalization ability of the network.

[0102] The embodiment also provides an application of the above-mentioned end-to-end ophthalmic surgery positioning navigation method in an ophthalmic surgery navigation system. Figure 1 is a surgical navigation method and device of the embodiment in a surgical workflow diagram. As shown in Figure 1 , a surgical physician 115 observes a patient's eye 113 through an ophthalmic surgery microscope 112, uses various intervention instruments 114 to perform surgical operations, and an eye surgery image 103 is collected to a computer by an image collection system 101 and input to a processing module 108. The processing module 108 uses the above-mentioned end-to-end ophthalmic surgery positioning navigation method to sequentially perform segmentation of eyeball key information, i.e., semantic segmentation 104, polar coordinate conversion 105, and eyeball rotation registration, wherein the eyeball rotation tracking includes inter-frame registration 106 and long-time registration 107 between frames, to obtain navigation information 109 output and present in a projection system 116. The reference image 111 is a preoperative image collected by a doctor before the operation, which is used for long-time registration 107. If the image comes from a binocular stereo vision microscope imaging system, the output navigation information 109 will also be subjected to point cloud processing 110 of depth calculation.

[0103] Figure 6 is a schematic diagram of the embodiment processing binocular stereo vision microscope images, as Figure 6As shown, if the image information 603 captured by the image acquisition system 601 comes from the binocular stereoscopic vision ophthalmic surgery system 602, the network 604 processes the left and right eyepiece images in dual channels, and outputs left and right eyepiece image navigation information 605, which will then be processed by point cloud processing 606 to calculate depth information. A specific example of depth calculation is to use the set of iris boundary points P of the left and right eyepiece images... iris 607, and the intersection point 608 of the rotation axis and the iris boundary are converted to three-dimensional space according to the calibration parameters of the binocular microscopic stereoscopic vision system. The centroid 610 of the iris boundary point set in three-dimensional space serves as the eyeball center. The average position 611 of the rotation axis of the left and right eyepiece images and the focal point of the iris boundary serves as the identifier of eyeball rotation in three-dimensional space. The three-dimensional navigation information can be displayed in the 3D projection system 612, or the navigation information can be displayed in the eyepiece field of view through the beam splitter 613.

[0104] like Figure 7 As shown, the ophthalmic surgical microscope consists of an objective lens 701, a beam splitter 703, an eyepiece 704, and a CMOS component 702. Surgical images are projected onto the eyepiece 704 through the objective lens 701. The surgeon observes the surgical images through the eyepiece. The beam splitter passes the light beam into the CMOS component, which transmits the images to the computer. The surgical navigation module processes the surgical images and calculates navigation information. Figure 8 The image shown is a binocular stereoscopic surgical microscope. The objective system splits two beams of light, which are projected onto the left and right eyepieces, forming two pathways. The images from the left and right eyepieces can provide the surgeon with a stereoscopic visual effect. The light paths of the left and right eyepieces are split into beams by beam splitters 804 and 803, respectively, and then sent to the left and right eyepiece CMOS components 802 and 801. The computer then acquires the images from the left and right eyepieces. By calibrating the internal and external parameters of the left and right eyepieces, a binocular stereoscopic vision system can be established to obtain the depth information of the eye body in the image. The surgical navigation method of this invention can process the images from the left and right eyepieces to obtain navigation information with depth information.

[0105] The embodiments described above are merely some, not all, embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. The present invention aims to achieve eyeball localization and rotation navigation in ophthalmic surgery using an end-to-end method.

Claims

1. An ophthalmic surgical microscope device, comprising an objective lens system, an eyepiece system, a beam splitter system, a video recording component, a surgical navigation module, and a projection and display module; The objective lens system focuses the light beam into the patient's eye to achieve clear imaging; The eyepiece system projects the image plane into the doctor's eyes, allowing the doctor to directly observe the microscope image and the projection pattern generated by the surgical navigation module. The beam splitter system divides the light beam into two paths, one leading to the eyepiece system and the other to the camera / video recording assembly; The camera / video recording component transmits images or videos to the surgical navigation module; Its features are: The surgical navigation module is connected to the projection and display module and is used to execute an end-to-end ophthalmic surgical positioning and navigation method to obtain navigation information for eyeball segmentation. The projection and display module projects the received navigation information into the microscope's field of view through the beam splitter system and eyepiece system, or displays the real-time video collected by the camera and video recording components on the display end through an external display to assist doctors in performing surgery; The end-to-end ophthalmic surgical positioning and navigation method includes the following steps: Step 1: Obtain preoperative images of the patient's eyes; Step 2: Construct an encoder-decoder network, which is a neural network based on the U-Net structure. The encoder network takes the patient's eye image as input and outputs the extracted deep and shallow feature maps of the image. The deep features represent global information, and the shallow features represent local information. The decoder network takes the deep and shallow feature maps of the image as input and outputs an eye feature map that fuses the deep and shallow features. The encoder uses either TransU-Net or EfficientNet. Step 3: Construct a multi-class segmentation layer for semantic segmentation of the iris, sclera, and cornea regions of the eyeball; the multi-class segmentation layer consists of multiple convolutional layers with a kernel size of 1, the input of which is the eyeball feature map, and the output is the semantic segmentation result, namely the segmentation prediction map of the iris, sclera, and cornea; Step 4: Calculate the center of the iris region based on the semantic segmentation results and use it as the center of the eyeball; construct a polar coordinate sampler with the center of the eyeball as the pole; Step 5: Use the polar coordinate sampler constructed in Step 4 to acquire intraoperative images in real time and convert them to the polar coordinate system. Input the preoperative images, semantic segmentation results and eye feature maps into the polar coordinate sampler to convert them into polar coordinate images. Step 6: In the polar coordinate system, register the current image and the target image to obtain the rotation information of the eyeball, thereby completing the rotation navigation of the eyeball; Step 6.1: Construct and train a Siamese network to extract scleral capillary features from the search image and the target image in polar coordinates; the search image is the current frame image in the real-time image, and the target image includes the preoperative image and the previous frame image, wherein the previous frame image is the target image for inter-frame registration, and the preoperative image is the target image for long-term registration. Step 6.2: Define a multi-point Gaussian template. The Gaussian template is an image in which the gray values ​​are distributed according to a Gaussian function. The position of the gray value peak is the target position, and the position of the Gaussian distribution in the Gaussian template is the position of the target image to be registered. Use the Gaussian template and the target image to train a correlation filter online, calculate the cross-correlation between the correlation filter and the search image, and obtain the correlation response map. Step 6.3: Construct a confidence network using the correlation response map as input to obtain the target location region prediction map; multiply the target location region prediction map with the correlation response map, and output the product as the registration result; the peak position in the registration result is the target position, the correlation response value of this position is the confidence of the registration result, and the displacement of the peak position relative to the Gaussian template on the angular coordinate axis in the polar coordinate system is the rotation of this point; Step 6.4: Calculate the average displacement of multiple target positions in the search image within the Gaussian template, and use this as the rotation of the search image relative to the target image, i.e., the inter-frame registration result and the long-term registration result; calculate the weighted sum of the inter-frame registration result and the long-term registration result as the final rotation of the eyeball in the current frame image. The weights of the weighted sum of the inter-frame registration and the long-term registration are dynamically adjusted according to the registration results during the operation; calculate the iris region of the current frame image based on the final rotation, register the current image with the target image, and complete the eyeball center localization and eyeball rotation tracking.

2. The ophthalmic surgical microscope device according to claim 1, characterized in that, The end-to-end ophthalmic surgical positioning and navigation method also includes: Step 7: Perform data augmentation on the registered images. The process is as follows: Capillary feature information from random locations in other ocular surgery videos is superimposed at the registration location. The capillary features superimposed at the registration location are the same for each pair of registered images, but the capillary features superimposed on different pairs of registered images are different.

3. The ophthalmic surgical microscope device according to claim 1, characterized in that: In step 6.2, multiple Gaussian templates are defined. A dense registration point set is established based on the multiple Gaussian templates to perform non-rigid registration between the target image and the search image.

4. The ophthalmic surgical microscope device according to claim 1, characterized in that: The twin network includes a first neural network and a second neural network. The first neural network consists of convolutional layers and a deformable network connected in sequence. The second neural network has the same structure as the first neural network and shares weights. The convolutional layers of both the first and second neural networks are connected to polar coordinate samplers to receive preoperative images or real-time intraoperative image data in polar coordinates and extract them as capillary feature maps for output. The deformation network receives the semantic segmentation results and capillary feature maps in polar coordinates. Based on the corneal limbal edge in the semantic segmentation results in polar coordinates, it performs deformation compensation on the capillary feature maps in polar coordinates to obtain the iris boundary capillary feature maps without deformation.

5. The ophthalmic surgical microscope device according to claim 1, characterized in that: Step 6.4, during the eye center localization and rotation tracking process, also includes the calculation of translation amount. The calculation method for translation amount is as follows: Step 6.4.1: Construct the segmentation branch of the Siamese network. The segmentation branch consists of a convolutional layer and a classification layer. The convolutional layer receives the iris boundary capillary feature map extracted by the Siamese network and the eye feature map transformed from the Cartesian coordinate system. After fusing them, the iris region is segmented by the classification layer. The centroid of the iris region in the Cartesian coordinate system is the final eye center. Step 6.4.2: Calculate the displacement of the eyeball center obtained in step 6.4.1 relative to the initial eyeball center during surgery. This displacement is the translation of the eyeball.

6. The ophthalmic surgical microscope device according to claim 1, characterized in that: The weighted sum calculation method for the inter-frame registration and long-term registration results in step 6.4 is the Kalman filter algorithm; its detailed calculation process is as follows: sum the covariance of the inter-frame registration result and the covariance of the long-term registration result; calculate the weight based on the covariance summation result. m : m = P / ( P + Q ) in P This represents the covariance of inter-frame registration. Q This represents the covariance of long-term registration.

7. The ophthalmic surgical microscope device according to claim 1, characterized in that: Before step 3, a fully connected layer is set after the encoding and decoding network. The fully connected layer receives the eye feature map output by the decoder and determines the current image state, excluding abnormal state images.

8. The ophthalmic surgical microscope device according to claim 1, characterized in that: The encoder-decoder network in the ophthalmic surgery localization and navigation method comprises two branches: a CNN and a Token Mixing branch. The CNN branch acquires local information by processing the surgical image through n convolutional layers, where n is a natural number greater than 1. The Token Mixing branch acquires global information by segmenting the image into S non-overlapping image patches as input. All image patches are linearly mapped to hidden layers, and position embeddings are added to the mapped features to preserve the relative position of each image patch. An MLP network is used to operate on each channel dimension of the features, outputting a result containing feature interactions. Attention gates are used to fuse the local information acquired by the CNN branch and the global information acquired by the Token Mixing branch. The decoder part of the encoder-decoder network adopts the same structure as AttentionU-Net, and uses attention gates to enhance the features of the encoding part.

9. The ophthalmic surgical microscope device according to claim 1, characterized in that: The ophthalmic surgical microscope device is a binocular stereo vision ophthalmic surgical microscope, which has two sets of camera and video recording components for the left and right eyes, and its projection and display module is a three-dimensional projection and display module. The objective lens system splits into two beams of light, which are projected onto the left and right eyepieces respectively. The light path of the left eyepiece is split into a beam by a beam splitter and enters the left eyepiece video recording component. The light path of the right eyepiece is split into a beam by a beam splitter and enters the right eyepiece video recording component. The surgical navigation module receives images acquired by the left and right eyepiece video recording components, calibrates the internal and external parameters of the left and right eyepieces, establishes a binocular stereo vision system based on the calibration results, and acquires the depth information of the eye body in the left and right eyepiece images; then, it executes the end-to-end ophthalmic surgical positioning and navigation method as described in claim 1 respectively to obtain navigation information of the left eyeball segmentation and navigation information of the right eyeball segmentation, and takes the average position after being transferred to three-dimensional space; The 3D projection and display module projects the averaged position navigation information into the microscope's field of view through the beam splitter system and eyepiece system, or displays the real-time video collected by the camera and video recording components on the display end through an external display to assist doctors in performing surgery.

Images