A cataract surgery evaluation method and system based on deep learning and a medium

Abstract

The application belongs to the technical field of deep learning, and provides a cataract surgery evaluation method and system based on deep learning and a medium, comprising the following steps: S1, classifying the surgery stage in which a video frame is located by using the surgery instrument features in the video frame in combination with the eye background features; S2, training a general network according to the initial feature labels of each link of the surgery to extract evaluation features in the video frames corresponding to each link for surgery evaluation; and S3, obtaining the quantitative information of each link of the surgery according to the preset labels of the extracted evaluation features, and inputting the quantitative information into the trained preset classification evaluation network to perform classification evaluation on each link of the surgery. The application has the advantage that the quantitative relationship between the descriptive evaluation indexes in the ICO-OSCAR standard and the learnable surgery features of the deep learning network is established, so that the full-process participation of expert doctors in surgery training is replaced by artificial intelligence technology, and the objectivity, reliability and response speed of the training effect are improved.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method, system, and medium for evaluating cataract surgery based on deep learning. Background Technology

[0002] Cataracts are the leading cause of blindness, affecting approximately half of the blind population in my country. Surgery is the primary way to help patients regain their sight. Currently, the surgery rate in my country remains relatively low. Increasing the cataract surgery rate and ensuring surgical outcomes are among the most pressing issues in current blindness prevention and treatment efforts. Cataract surgery has entered the refractive era. Patients' individualized needs and pursuit of postoperative visual outcomes place higher demands on incision location, capsulorhexis size, and the centralization of the intraocular lens after implantation. Timely feedback on incision, capsulorhexis, and intraocular lens implantation during surgery is particularly important for improving the surgical skills of novice surgeons in cataract surgery training.

[0003] Standardized cataract surgery training aims to shorten the learning curve, standardize surgical procedures, and reduce surgical complications. Traditional cataract surgery training feedback typically uses the International Council of Ophthalmologists' Surgical Competency Assessment Standard (ICO-OSCAR) to track and evaluate doctors. Expert doctors score each step of the cataract surgery according to the standard. This process is time-consuming and subject to significant subjective differences, making it difficult to meet clinical requirements through training in a short period of time. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, and medium for evaluating cataract surgery based on deep learning, in order to solve the above-mentioned problems.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A deep learning-based method for evaluating cataract surgery, comprising the following steps:

[0007] S1. Using the features of surgical instruments in the video frame combined with the features of the eye background, the surgical stage of the video frame is classified through a preset classification network; the surgical stage includes the incision stage, the capsulorhexis stage, and the intraocular lens implantation stage.

[0008] S2. Train a general network based on the initial feature labels of each stage of the surgery to extract evaluation features for surgical evaluation from the corresponding video frames of each stage.

[0009] S3. The extracted evaluation features are used to obtain quantitative information of each stage of the surgery according to the preset labels. The quantitative information is then input into the trained preset classification evaluation network to classify and evaluate each stage of the surgery.

[0010] Furthermore, the steps for classifying the surgical stage of the video frame include:

[0011] S11. Perform layered sampling processing on the video frames;

[0012] S12. Using the trained preset target detection model, obtain the surgical instruments and eye background regions in the video frame, and perform batch cropping of the surgical instruments and eye background regions.

[0013] S13. The cropped surgical instruments and eye background area, as well as the corresponding video frames, are fed into a trained classifier, which outputs the classification result of the surgical stage of the video frame.

[0014] Furthermore, the steps for training the general-purpose network include:

[0015] T1. Randomly crop an image block from the video frame, and input it along with the images of the preceding and following frames into the spatial feature encoder to calculate the corresponding spatial features.

[0016] T2. Calculate the localization parameters of the image block that best matches the image block in the spatial features of the corresponding images in the previous and next frames using a differentiable tracker, and perform bilinear sampling using a bilinear sampler to obtain the spatial features of the best matching image block.

[0017] T3. The spatial feature encoder and differentiable tracker are trained end-to-end through steps T1 and T2 to obtain the trained general network.

[0018] Furthermore, the evaluation features used for surgical evaluation include surgical instrument position information, optical flow field information, limbal morphology and position features, and intraocular lens position features.

[0019] Furthermore, the steps for classifying and evaluating surgical incision procedures include:

[0020] A1. Fit the limbus to determine the limbus center based on the described limbus position characteristics;

[0021] A2. Using the center of the limbus as a reference point, the relative motion trajectory of the surgical instruments is obtained based on the position information of the surgical instruments;

[0022] A3. Input the relative motion trajectory of the surgical instruments and the morphological characteristics of the corneal limbus into the preset classification and evaluation network to evaluate the surgical incision process according to the ICO-OSCAR standard and obtain the operation score of the surgical incision process.

[0023] Furthermore, the steps for classifying and evaluating the capsulorhexis procedure during surgery are as follows:

[0024] The optical flow field information is input into a preset classification and evaluation network to evaluate the capsulorhexis procedure according to the ICO-OSCAR standard, and an operational score for the capsulorhexis procedure is obtained.

[0025] Furthermore, the steps for classifying and evaluating the intraocular lens implantation process during surgery are as follows:

[0026] B1. Fit the corneal limbal position features and intraocular lens position features extracted by the general network to obtain the center point position information of each.

[0027] B2. Input the center point location information of both into the preset classification evaluation network to evaluate the intraocular lens implantation process during surgery according to the ICO-OSCAR standard, and obtain the operation score of the intraocular lens implantation process during surgery.

[0028] A second aspect of the present invention provides a deep learning-based cataract surgery evaluation system, comprising at least one processor and at least one memory, wherein the memory stores a computer program that, when executed by the processor, enables the processor to perform the deep learning-based cataract surgery evaluation method.

[0029] A third aspect of the present invention provides a computer-readable storage medium that, when instructions in the storage medium are executed by a processor within a device, enables the device to perform the deep learning-based cataract surgery evaluation method.

[0030] Compared with the prior art, the present invention has at least the following beneficial effects:

[0031] (1) Based on the surgical instruments involved in different surgical procedures as the main learning features, combined with eye background information, this invention achieves accurate classification of video frames through a trained accurate classifier, providing algorithmic support for data processing before surgical evaluation.

[0032] (2) This invention develops a general-purpose multi-feature extraction network to achieve the extraction of quantifiable surgical features from different surgical procedures using the same network. Information such as the position of surgical instruments, optical flow and ocular deformation features obtained through this general-purpose network, combined with classification labels, enables skill evaluation of surgical incision, capsulorhexis and intraocular lens implantation procedures.

[0033] (3) Establish a quantitative relationship between the descriptive evaluation indicators in the ICO-OSCAR standard and the surgical path, corneal limbal morphology, and optical flow information of surgical instruments that can be learned by deep learning networks, so as to realize the replacement of expert doctors with artificial intelligence technology to participate in the entire surgical training process, thereby improving the objectivity, reliability and response rate of the training effect. Attached Figure Description

[0034] Figure 1 This is a flowchart of a deep learning-based cataract surgery evaluation method in an embodiment of the present invention;

[0035] Figure 2 This is a flowchart illustrating the classification of surgical stages of video frames in an embodiment of the present invention;

[0036] Figure 3 This is a flowchart of training a general-purpose network in an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of general network training in an embodiment of the present invention;

[0038] Figure 5 This is a flowchart illustrating the classification and evaluation of surgical incision procedures in this embodiment of the invention.

[0039] Figure 6 This is a flowchart illustrating the classification and evaluation of the intraocular lens implantation process during surgery, as described in this embodiment of the invention. Detailed Implementation

[0040] It should be noted that the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0041] The following are specific embodiments of the present invention, and the technical solutions of the present invention will be further described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0042] like Figure 1 As shown, this invention provides a deep learning-based method for evaluating cataract surgery, comprising the following steps:

[0043] S1. Using the features of surgical instruments in the video frame combined with the features of the eye background, the surgical stage of the video frame is classified through a preset classification network; the surgical stage includes the incision stage, the capsulorhexis stage, and the intraocular lens implantation stage.

[0044] Among them, such as Figure 2 As shown, the steps for classifying the surgical stage of the video frame include:

[0045] S11. Perform layered sampling processing on the video frames.

[0046] Due to the varying skill levels of surgeons and the different requirements at each stage of surgery, the video recordings are often quite long, with varying durations at each stage and complex and variable surgical scenarios. This invention employs hierarchical sampling to process video frames, which can effectively improve the training efficiency of the network model.

[0047] S12. Using the trained preset target detection model, obtain the surgical instruments and eye background regions in the video frame, and then perform batch cropping of the surgical instruments and eye background regions.

[0048] This invention trains a Yolov3-based target detection model based on surgical instruments and eye background labels to obtain key surgical instruments and eye background regions in video frames. These regions are then batch-cropped and used as input to a classifier, thereby improving the fine-grained layout information of video frames.

[0049] S13. The cropped surgical instruments and eye background area, as well as the corresponding video frames, are fed into a trained classifier, which outputs the classification result of the surgical stage of the video frame.

[0050] By feeding surgical instruments, the eye background region, and the corresponding video frames into a ResNet network, global and local features are extracted. Then, the features output from the fully connected layers of each classifier are concatenated to form a fine-grained feature representation of the entire frame, improving the accuracy of video frame classification.

[0051] This invention uses surgical instruments involved in different surgical procedures as the main learning features, combined with eye background information, to achieve accurate classification of video frames, providing algorithmic support for subsequent data processing.

[0052] S2. Train a general network based on the initial feature labels of each stage of the surgery to extract evaluation features for surgical evaluation from the corresponding video frames of each stage.

[0053] This invention aims to improve the versatility of feature extraction networks, reduce the workload of labeling, and increase feature extraction efficiency. It learns visual correspondences from unlabeled videos, training the network to learn a feature space by tracking patches extracted from images across several consecutive frames. The network then calculates the visual similarity of target patches across consecutive frames in a time series. When extracting features from incision, capsulorhexis, and intraocular lens implantation stages, given labels for these stages, the network can directly extract the features required for evaluating each stage of the surgery.

[0054] Among them, such as Figure 3 and Figure 4 As shown, the steps for training the general-purpose network in this invention include:

[0055] T1. A random image block is cropped from the video frame and input together with the images of the preceding and following frames into the spatial feature encoder to calculate the corresponding spatial features.

[0056] T2. The localization parameters of the image block that best matches the image block in the spatial features of the preceding and following frames are calculated by a differentiable tracker, and bilinear sampling is performed by a bilinear sampler to obtain the spatial features of the best matching image block.

[0057] T3. The spatial feature encoder and differentiable tracker are trained end-to-end through steps T1 and T2 to obtain the trained general network.

[0058] This invention utilizes randomly cropped patches (P) from video frames. t ) and previous and next frame images (I t-1 The spatial features x are respectively input into the spatial feature encoder Φ based on the ResNet network and the spatial features x are calculated respectively. P and x I Furthermore, the channel dimension of the spatial features is normalized to facilitate subsequent similarity calculations.

[0059] The differentiable tracker T first focuses on x I With x P The similarity between coordinates is measured, and then the feature x is calculated. I In and x P The localization parameter θ corresponding to the best matching image patch is obtained by using a bilinear sampler to sample the image features x. I Bilinear sampling is performed with θ to generate new patch features x. P’ Then, the similarity between the new patch features and the images in the preceding and following frames is calculated, thus forming a loop to train the network.

[0060] After training, this general-purpose network can extract features from surgical procedures such as incision, capsulorhexis, and intraocular lens implantation by providing different feature labels. It can also obtain information such as the position of surgical instruments, optical flow field, limbal morphology, and intraocular lens position, which can be used for subsequent classification and evaluation of each stage.

[0061] S3. The extracted evaluation features are used to obtain quantitative information of each stage of the surgery according to the preset labels. The quantitative information is then input into the trained preset classification evaluation network to classify and evaluate each stage of the surgery.

[0062] Based on information such as the position of surgical instruments, optical flow field, limbal morphology, and intraocular lens position obtained from the general network, this information is converted into quantitative information that can be evaluated, such as the trajectory of surgical instruments and the speed and direction of movement of surgical instruments (i.e., optical flow field information). Combined with information such as the position and morphological changes of the limbus, and with the classification labels of expert doctors according to the ICO-OSCAR standard, classification models are trained to achieve the classification and evaluation of incision, capsulorhexis, and intraocular lens implantation.

[0063] Among them, such as Figure 5 As shown, the steps for classifying and evaluating surgical incision procedures include:

[0064] A1. Fit the limbus to determine the limbus center based on the described limbus position characteristics;

[0065] A2. Using the center of the limbus as a reference point, the relative motion trajectory of the surgical instruments is obtained based on the position information of the surgical instruments;

[0066] A3. Input the relative motion trajectory of the surgical instruments and the morphological characteristics of the corneal limbus into the preset classification and evaluation network to evaluate the surgical incision process according to the ICO-OSCAR standard and obtain the operation score of the surgical incision process.

[0067] The steps for classifying and evaluating the capsulorhexis procedure during surgery are as follows:

[0068] The optical flow field information is input into a preset classification and evaluation network to evaluate the capsulorhexis procedure according to the ICO-OSCAR standard, and an operational score for the capsulorhexis procedure is obtained.

[0069] like Figure 6 As shown, the steps for classifying and evaluating the intraocular lens implantation process during surgery are as follows:

[0070] B1. Fit the corneal limbal position features and intraocular lens position features extracted by the general network to obtain the center point position information of each.

[0071] B2. Input the center point location information of both into the preset classification evaluation network to evaluate the intraocular lens implantation process during surgery according to the ICO-OSCAR standard, and obtain the operation score of the intraocular lens implantation process during surgery.

[0072] This invention establishes a quantitative relationship between descriptive evaluation indicators in the ICO-OSCAR standard and learnable surgical instrument paths, limbal morphology, and optical flow information of surgical instruments by deep learning networks. This enables artificial intelligence technology to replace expert doctors in the entire surgical training process, thereby improving the objectivity, reliability, and response speed of the training effect.

[0073] In another embodiment of the present invention, a deep learning-based cataract surgery evaluation system is also provided, including at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor is able to perform the above-described deep learning-based cataract surgery evaluation method.

[0074] In another embodiment of the present invention, a computer-readable storage medium is also provided, which, when the instructions in the storage medium are executed by a processor within a device, enables the device to perform the above-described deep learning-based cataract surgery evaluation method.

[0075] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A deep learning-based method for evaluating cataract surgery, characterized in that, Including the following steps: S1. Using the features of surgical instruments in the video frame combined with the features of the eye background, the surgical stage of the video frame is classified through a preset classification network; the surgical stage includes the incision stage, the capsulorhexis stage, and the intraocular lens implantation stage. S2. A general network is trained based on the initial feature labels of each stage of the surgery to extract evaluation features for surgical evaluation from the corresponding video frames of each stage; the evaluation features for surgical evaluation include surgical instrument position information, optical flow field information, corneal limbus morphology and position features, and intraocular lens position features; S3. Obtain quantitative information of each stage of the surgery based on the extracted evaluation features according to the preset labels, and input the quantitative information into the trained preset classification evaluation network to classify and evaluate each stage of the surgery. The steps for classifying and evaluating the intraocular lens implantation process during surgery are as follows: B1. Fit the corneal limbal position features extracted by the general network with the intraocular lens position features to obtain the center point position information of each; B2. Input the center point location information of both into the preset classification evaluation network to evaluate the intraocular lens implantation process during surgery according to the ICO-OSCAR standard, and obtain the operation score of the intraocular lens implantation process during surgery.

2. The method for evaluating cataract surgery based on deep learning according to claim 1, characterized in that, The steps for classifying the surgical stage of the video frame include: S11. Perform layered sampling processing on the video frames; S12. Using the trained preset target detection model, obtain the surgical instruments and eye background regions in the video frame, and perform batch cropping of the surgical instruments and eye background regions. S13. The cropped surgical instruments and eye background area, as well as the corresponding video frames, are fed into a trained classifier, which outputs the classification result of the surgical stage of the video frame.

3. The method for evaluating cataract surgery based on deep learning according to claim 1, characterized in that, The steps for training the general-purpose network include: T1. Randomly crop an image block from the video frame, and input it along with the images of the preceding and following frames into the spatial feature encoder to calculate the corresponding spatial features. T2. Calculate the localization parameters of the image block that best matches the image block in the spatial features of the corresponding images in the previous and next frames using a differentiable tracker, and perform bilinear sampling using a bilinear sampler to obtain the spatial features of the best matching image block. T3. The spatial feature encoder and differentiable tracker are trained end-to-end through steps T1 and T2 to obtain the trained general network.

4. The cataract surgery evaluation method based on deep learning according to claim 1, characterized in that, The steps for classifying and evaluating surgical incisions include: A1. Fit the limbus to determine the limbus center based on the described limbus position characteristics; A2. Using the center of the limbus as a reference point, the relative motion trajectory of the surgical instruments is obtained based on the position information of the surgical instruments; A3. Input the relative motion trajectory of the surgical instruments and the morphological characteristics of the corneal limbus into the preset classification and evaluation network to evaluate the surgical incision process according to the ICO-OSCAR standard and obtain the operation score of the surgical incision process.

5. The method for evaluating cataract surgery based on deep learning according to claim 1, characterized in that, The steps for classifying and evaluating the capsulorhexis procedure during surgery are as follows: input the optical flow field information into the preset classification and evaluation network, evaluate the capsulorhexis procedure according to the ICO-OSCAR standard, and obtain the operation score of the capsulorhexis procedure during surgery.

6. A deep learning-based cataract surgery evaluation system, characterized in that, It includes at least one processor and at least one memory, wherein the memory stores a computer program that, when executed by the processor, enables the processor to perform the deep learning-based cataract surgery evaluation method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor within the device, the device is able to perform the deep learning-based cataract surgery evaluation method according to any one of claims 1-5.

Images