A method for identifying cataract surgery procedures

Abstract

This invention discloses a method for identifying cataract surgery procedures, including using a target detection model to detect the lens and surgical instruments in video frames; employing a three-stream convolutional network to extract fine-grained features from the video frames; and using a multi-stage temporal convolutional network to extract temporal information from the video for pre-identification of cataract surgery stages. This invention provides a fine-grained identification method for cataract surgery procedures, specifically by extracting fine-grained features from the video to reduce the probability of misclassifying similar stages in cataract videos, and using a multi-stage temporal convolutional network for accurate pre-identification of cataract surgery stages, providing an effective way to reduce complications caused by improper doctor operation.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a method for recognizing the process of cataract surgery. Background Technology

[0002] Cataracts are an eye disease caused by the denaturation of lens proteins, leading to clouding and obstructing light from entering the eye, eventually causing vision loss or even blindness. With my country's aging population, environmental changes, and genetic factors, the number of cataract patients is gradually increasing, resulting in a growing demand for cataract surgery. Furthermore, postoperative complications such as infection, vitreous hemorrhage, and corneal endothelial damage are common after cataract surgery. Automated surgical procedure recognition can significantly improve patient safety during surgery. Doctors can anticipate complications based on the surgical plan and receive early warnings during the procedure, reducing the risk of errors.

[0003] A key step in surgical procedure modeling is the recording, collection, and analysis of data during the procedure. In cataract surgery, surgeons observe the patient's eyes using an optical microscope and record video using a camera attached to the microscope. The surgical video obtained from the camera can be used alone or in combination with other data. More importantly, surgical video data is a near-perfect representation of the surgery, containing a wealth of information about the surgical process. Therefore, surgical video has become a primary data source for surgical procedure identification.

[0004] Prior to this, several traditional surgical procedure recognition methods had been proposed. Early research first utilized manually designed descriptors to extract features such as intensity, gradient, shape, color, and texture, then used linear statistical models to capture the temporal structure of surgical videos. However, manually designed descriptors are very time-consuming when processing video frames, thus failing to meet the demands of fast and accurate automated surgical video recognition. Recently, due to the increase in data volume and the availability of computing power, deep learning techniques have also demonstrated their advantages in low-level computer vision tasks, leading to the introduction of deep learning methods in surgical procedure recognition. Current deep learning methods seamlessly integrate deep residual networks and recurrent neural networks into a unified framework through end-to-end recurrent convolutional models, jointly capturing the spatiotemporal features of videos. Although joint training in time and space is achieved, memory limitations prevent the learning of temporal and spatial features over longer time dimensions.

[0005] Although there are strict requirements for the tools used at each stage of cataract surgery, in reality, different tools may be used at the same stage of cataract surgery or the same tools may be used at different stages, which reduces the efficiency of determining the steps of cataract surgery.

[0006] Therefore, how to provide a novel deep learning-based cataract surgery procedure recognition scheme to reduce the probability of misclassifying similar stages in cataract videos and improve the efficiency and accuracy of determining cataract surgery steps is one of the current challenges that the academic community needs to address. Summary of the Invention

[0007] The main objective of this invention is to provide a method for recognizing cataract surgery procedures, thereby overcoming the shortcomings of existing cataract surgery procedure prediction algorithms that cannot identify similar stages or capture information over a longer period of time for prediction.

[0008] To achieve the aforementioned objectives, the technical solution adopted by this invention includes: a method for identifying cataract surgery procedures, comprising:

[0009] S1, using an object detection model to detect the lens and surgical instruments in the video frame;

[0010] S2 employs a three-stream convolutional network, taking the lens, surgical instruments, and original images as input, to extract fine-grained features from video frames that contribute to the prediction of surgical stages.

[0011] S3. A multi-stage temporal convolutional network is used to extract temporal information from the video and combined with the fine-grained features extracted by the three-stream convolutional network to perform pre-identification of the cataract surgery stage.

[0012] In a preferred embodiment, S1 includes:

[0013] S11, the lens and surgical instruments in the video frame are labeled using an image annotation tool and sent to the target detection model for training as a training set. The target detection model is a target detection model for the lens and surgical instruments.

[0014] S12, detect the target area of ​​the lens and surgical instruments, and crop out the target area as input for the next step of feature extraction.

[0015] In a preferred embodiment, the target detection model is the YOLOv3 target detection model.

[0016] In a preferred embodiment, S2 includes: S21, feeding the local area of ​​the surgical instrument, the local area of ​​the lens, and the entire image into the residual network to extract the corresponding first local feature, second local feature, and global feature respectively;

[0017] S22, the first local feature, the second local feature, and the global feature are concatenated to form a fine-grained feature representation of the entire video frame;

[0018] S23 feeds the tandem features into the fully connected layer and outputs a preliminary prediction of the cataract surgery stage.

[0019] In a preferred embodiment, the multi-stage temporal convolutional network includes a first-stage temporal convolutional network and a second-stage temporal convolutional network. The first-stage temporal convolutional network consists of multiple double-expanded temporal convolutional layers, and the second-stage temporal convolutional network consists of multiple single-expanded temporal convolutional layers.

[0020] In a preferred embodiment, S3 specifically includes:

[0021] S31, the fine-grained features of each frame extracted by the three-stream convolutional network are convolved by the first-stage temporal convolutional network and the first-stage prediction result is output to the second-stage temporal convolutional network.

[0022] S32, the prediction result of the first stage is convolved by the second stage temporal convolutional network and then the prediction result of the second stage is output;

[0023] S33, the prediction results of the second stage are concatenated with the fine-grained features extracted by the three-stream convolutional network and fed into a 1×1 convolutional layer for dimensionality reduction;

[0024] S34, the second-stage prediction result after dimensionality reduction is fed into the classifier for the final surgical stage prediction.

[0025] In a preferred embodiment, the first-stage temporal convolutional network and the second-stage temporal convolutional network are trained using cross-entropy loss with class weights, the class weights being obtained by using median frequency balancing.

[0026] In a preferred embodiment, each of the dual-expanded temporal convolutional layers includes two first-expanded temporal convolutional layers and a second-expanded temporal convolutional layer with different dilation factors. The dilation factor of the first-expanded temporal convolutional layer increases exponentially with the number of layers, while the dilation factor of the second-expanded temporal convolutional layer decreases exponentially with the number of layers.

[0027] In a preferred embodiment, the operation of each layer is represented as follows:

[0028]

[0029]

[0030]

[0031] D l =D l-1 +W 3，l *Q l +b 3，l (4)

[0032] in, and These are the outputs of the first and second dilated temporal convolutional layers, respectively, W. 1，l W 2，l b represents the weights of the first and second dilated temporal convolutional layers, respectively. 1，l b 2，l Q represents the bias of the first and second dilated temporal convolutional layers, respectively. l express and The connection, D l It is l th The output of the double extended temporal convolutional layer, where W 3，l b is the weight of the 1×1 convolutional layer in formula (3). 3，l It is the bias of the 1×1 convolutional layer in formula (3).

[0033] In a preferred embodiment, each single-expansion temporal convolutional layer in the second-stage temporal convolutional network undergoes normalization and ReLU activation operations, and the expansion rate of the second-stage temporal convolutional layers increases exponentially.

[0034] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0035] 1. This invention provides a fine-grained identification method for cataract surgery procedures. Specifically, it extracts fine-grained features from the video to reduce the probability of misclassification of similar stages in cataract videos, and uses a multi-stage temporal convolutional network for accurate pre-identification of cataract surgery stages, providing an effective way to reduce complications caused by improper operation by doctors.

[0036] 2. This invention detects important local areas in the surgical procedure using target detection technology, and then feeds them into a three-stream convolutional network to obtain global information of the video frame as well as local fine-grained information contained in the surgical instruments and pupils, thereby improving the ability to recognize similar frames.

[0037] 3. This invention uses a multi-level temporal convolutional network for extracting temporal features. Through its dual convolutional layers, temporal features can be extracted from different temporal lengths. Moreover, the temporal convolutional network can remember longer temporal relationships than traditional recurrent neural networks, thus solving the problem that previous algorithms could not capture the temporal information of long sequences. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating a method according to one embodiment of the present invention;

[0040] Figure 2 This is a flowchart illustrating the detection of the lens and surgical instruments according to one embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram of a three-stream spatial feature extraction network in one embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of a surgical procedure recognition framework according to one embodiment of the present invention;

[0043] Figure 5 This is a dual extended temporal convolutional layer in one embodiment of the present invention. Detailed Implementation

[0044] The invention will be more fully understood through the following detailed description, which should be read in conjunction with the accompanying drawings. Detailed embodiments of the invention are disclosed herein; however, it should be understood that the disclosed embodiments are merely exemplary of the invention, which may be embodied in various forms. Therefore, the specific functional details disclosed herein should not be construed as limiting, but rather as the basis for the claims and as intended to teach those skilled in the art to employ the representative basis of the invention in different ways in any suitable detailed embodiment.

[0045] The present invention discloses a method for recognizing cataract surgery procedures. In view of the problems that existing cataract surgery procedure prediction algorithms cannot identify similar stages and cannot capture long-term information for prediction, the present invention proposes to extract fine-grained features from the video and use a multi-stage temporal convolutional network for accurate pre-identification of cataract surgery stages. It can be used for the identification of cataract surgery procedures and for assisting surgical systems.

[0046] like Figure 1 As shown in the embodiment of the present invention, a method for identifying the cataract surgery process specifically includes the following steps:

[0047] S1 uses an object detection model to detect the lens and surgical instruments in the video frame.

[0048] Specifically, S1 includes the following steps:

[0049] S11, the lens and surgical instruments in the video frame are labeled using an image annotation tool and sent to the target detection model for training. The target detection model is a target detection model for the lens and surgical instruments.

[0050] Specifically, in this embodiment, considering that the same surgical instruments may appear at different stages of cataract surgery videos, object detection technology is proposed to detect the lens and surgical instruments, thereby improving the model's focus on these key local areas. Since YOLOv3 is a fast and accurate object detection model, it is used as the object detection tool in this invention. Specifically, the lens and surgical instruments in some video frames are labeled using image annotation software (AlbeImg) and fed into YOLOv3 as a training set to train an object detection model for the lens and surgical instruments.

[0051] S12, cropping out the detected target region as input for the next step of feature extraction.

[0052] Specifically, it detects the target areas of the lens and surgical instruments, such as Figure 2 As shown, the detected target region is cropped out and used as input for the next step of feature extraction.

[0053] S2 employs a three-stream convolutional network, taking the aforementioned lens, surgical instruments, and original images as input, to extract fine-grained features from video frames that contribute to the prediction of surgical stages.

[0054] Specifically, to address the difficulty in distinguishing familiar frames in cataract video frames, this invention proposes a three-stream convolutional network to extract discriminative fine-grained features, such as... Figure 3 As shown.

[0055] In this embodiment, step S2 specifically includes the following steps:

[0056] S21, the local area of ​​the surgical instrument, the local area of ​​the lens, and the whole image are respectively fed into the residual network to extract the corresponding first local feature, second local feature, and global feature.

[0057] Specifically, given the outstanding performance of the ResNet50 deep learning network in biomedical image processing tasks, this invention uses it as the backbone of the model. Important local regions such as surgical instruments and the lens, as well as the entire image, are fed into the ResNet deep learning network to extract corresponding first local features, second local features, and global features, respectively.

[0058] S22, the first local feature, the second local feature and the global feature are concatenated together as a fine-grained feature representation of the entire video frame.

[0059] S23 feeds the tandem features into the fully connected layer and outputs a preliminary prediction of the cataract surgery stage.

[0060] Through steps S21 to S23 described above, the three-stream convolutional network proposed in this invention can simultaneously utilize global information from video frames as well as local fine-grained information contained in surgical instruments and lenses. Furthermore, when the same surgical instruments or lenses appear at different surgical stages, the proposed three-stream convolutional network will selectively perform classification based on surgical instruments or lenses, thereby improving the proposed method's ability to identify similar frames.

[0061] S3. A multi-stage temporal convolutional network is used to extract temporal information from the video and combined with the fine-grained features extracted by the three-stream convolutional network to perform pre-identification of the cataract surgery stage.

[0062] Specifically, the multi-stage temporal convolutional network proposed in this embodiment of the invention includes two stages: a first-stage temporal convolutional network and a second-stage temporal convolutional network. For ease of description, the first-stage temporal convolutional network is denoted as S1, which specifically consists of N double-expanded temporal convolutional layers. The second-stage temporal convolutional network is denoted as S2, which specifically consists of N single-expanded temporal convolutional layers, as follows: Figure 4 As shown.

[0063] In this embodiment, step S3 specifically includes the following steps:

[0064] S31, the fine-grained features of each frame extracted by the three-stream convolutional network are convolved by the first-stage temporal convolutional network and the first-stage prediction result is output to the second-stage temporal convolutional network.

[0065] Specifically, the input to S1 is the fine-grained feature X = (x1, x2, ..., x...) extracted from each frame by the three-stream convolutional network proposed in this invention. t ), t∈[1,T], where T is the total number of frames. After convolution by the first-stage temporal convolutional network, the first-stage prediction result is output. In this embodiment, the first-stage prediction result is represented as Y1=Γ S1 (x1, x2, ..., x t ), t∈[1,T], where Γ S1 This represents a double-expanded temporal convolutional layer. Then, Y1 is used as the input to S2.

[0066] S32, the prediction results of the first stage are convolved by the second stage temporal convolutional network to output the prediction results of the second stage.

[0067] Specifically, in this embodiment, Y1 is used as the input of S2 to obtain the output of the second-stage prediction result, which is represented as Y2.

[0068] S33, the prediction results of the second stage are concatenated with the fine-grained features extracted by the three-stream convolutional network and fed into a 1×1 convolutional layer for dimensionality reduction.

[0069] Specifically, the second-stage prediction result Y2 is concatenated with the fine-grained features X extracted from each frame by the three-stream convolutional network, and then fed into a 1×1 convolutional layer for dimensionality reduction.

[0070] S34, the second-stage prediction results after dimensionality reduction are fed into the classifier for the final surgical stage prediction.

[0071] That is, the second-stage prediction result Y2 is fed into the classifier for the final surgical stage prediction. Since the number of frames for each stage in a cataract surgery video differs, the identification of surgical stages becomes an imbalanced pattern. Therefore, this invention uses cross-entropy loss with class weights (WCE Loss) to train the proposed S1 and S2, where the class weights are obtained by balancing median frequencies.

[0072] Additionally, in each double-expanded temporal convolutional layer of the first-stage temporal convolutional network, such as Figure 5 As shown, it contains two dilation-time convolutional layers, a first dilation-time layer and a second dilation-time convolutional layer, with different dilation factors. The dilation factor of the first dilation-time convolutional layer increases exponentially with the number of layers: DR n =2 n-1 , of which DR n Indicates the nth th The dilation rate of the layer. The second dilated temporal convolutional layer is exactly the opposite of the first dilated temporal convolutional layer. That is, as the number of layers increases, the dilation rate decreases exponentially: DR n =2 N-n .

[0073] The operations at each level can be represented as:

[0074]

[0075]

[0076]

[0077] D l =D l-1 +W 3，l *Q l +b 3，l (4);

[0078] in and These are the outputs of the first and second dilated temporal convolutional layers, respectively, W. 1，l W 2，l b represents the weights of the first and second dilated temporal convolutional layers, respectively. 1，l b 2，l Q represents the bias of the first and second dilated temporal convolutional layers, respectively. l express and The connection, D l It is l th The output of the double extended temporal convolutional layer, where W 3，l b is the weight of the 1×1 convolutional layer in formula (3). 3，l It is the bias of the 1×1 convolutional layer in formula (3).

[0079] In the second-stage temporal convolutional network, each single-expansion temporal convolutional layer undergoes layer normalization and ReLU activation. Similarly, the expansion rate of the second-stage temporal convolutional layers increases exponentially, i.e., n... th The expansion rate of the layer is: DR n =2 n-1 .

[0080] This invention has the following advantages: 1. This invention provides a fine-grained recognition method for cataract surgery procedures. Specifically, it extracts fine-grained features from the video to reduce the probability of misclassifying similar stages in cataract videos, and uses a multi-stage temporal convolutional network for accurate pre-identification of cataract surgery stages, providing an effective way to reduce complications caused by improper operation by doctors; 2. This invention detects important local areas in the surgical procedure using object detection technology, and then feeds them into a three-stream convolutional network to obtain global information of video frames as well as local fine-grained information contained in surgical instruments and pupils, thereby improving the ability to recognize similar frames; 3. This invention uses a multi-level temporal convolutional network for temporal feature extraction. Through its double convolutional layers, temporal features can be extracted from different temporal lengths, and the temporal convolutional network can remember longer temporal relationships than traditional recurrent neural networks, thus solving the problem that previous algorithms could not capture the temporal information of long sequences.

[0081] All aspects, embodiments, features, and examples of this invention are to be regarded as illustrative in all respects and are not intended to limit the invention, the scope of which is defined only by the claims. Other embodiments, modifications, and uses will become apparent to those skilled in the art without departing from the spirit and scope of the invention as claimed.

[0082] The use of headings and sections in this invention is not intended to limit the invention; each section can be applied to any aspect, embodiment or feature of the invention.

Claims

1. A method for identifying the cataract surgery procedure, characterized in that, The method includes: S1, using a target detection model to detect the lens and surgical instruments in the video frame, S1 includes: S11, the lens and surgical instruments in the video frame are labeled using an image annotation tool and sent to the target detection model for training as a training set. The target detection model is a target detection model for the lens and surgical instruments. S12, detect the target area of ​​the lens and surgical instruments, and crop out the target area as input for the next step of feature extraction; S2 employs a three-stream convolutional network, taking the lens, surgical instruments, and original image as input, to extract fine-grained features from video frames that contribute to the prediction of surgical stages. S2 includes: S21, the local area of ​​the surgical instrument, the local area of ​​the lens, and the whole image are respectively fed into the residual network to extract the corresponding first local feature, second local feature, and global feature; S22, the first local feature, the second local feature, and the global feature are concatenated to form a fine-grained feature representation of the entire video frame; S23, feeds the tandem features into the fully connected layer and outputs a preliminary prediction of the cataract surgery stage; S3, employing a multi-stage temporal convolutional network to extract temporal information from the video and combining it with the fine-grained features extracted by a three-stream convolutional network, performs pre-identification of the cataract surgery stage. The multi-stage temporal convolutional network includes a first-stage temporal convolutional network and a second-stage temporal convolutional network. The first-stage temporal convolutional network consists of multiple double-expanded temporal convolutional layers, and the second-stage temporal convolutional network consists of multiple single-expanded temporal convolutional layers. S3 includes: S31, the fine-grained features of each frame extracted by the three-stream convolutional network are convolved by the first-stage temporal convolutional network and the first-stage prediction result is output to the second-stage temporal convolutional network. S32, the prediction result of the first stage is convolved by the second stage temporal convolutional network and then the prediction result of the second stage is output; S33, the prediction results of the second stage are concatenated with the fine-grained features extracted by the three-stream convolutional network and fed into a 1×1 convolutional layer for dimensionality reduction; S34, the second-stage prediction result after dimensionality reduction is fed into the classifier for the final surgical stage prediction.

2. The cataract surgery procedure identification method according to claim 1, characterized in that: The target detection model is the YOLOv3 target detection model.

3. The cataract surgery procedure identification method according to claim 1, characterized in that: The first-stage temporal convolutional network and the second-stage temporal convolutional network are trained using cross-entropy loss with class weights, the class weights being obtained by using median frequency balancing.

4. The cataract surgery procedure identification method according to claim 1, characterized in that: Each of the dual-expanded temporal convolutional layers includes two first-expanded temporal convolutional layers and a second-expanded temporal convolutional layer with different dilation factors. The dilation factor of the first-expanded temporal convolutional layer increases exponentially with the number of layers, while the dilation factor of the second-expanded temporal convolutional layer decreases exponentially with the number of layers.

5. The cataract surgery procedure identification method according to claim 4, characterized in that: The operations at each level are represented as follows: D l =D l-1 +W 3，l *Q l +b 3，l ; (4) in, and These are the outputs of the first and second dilated temporal convolutional layers, respectively, W. 1，l W 2，l b represents the weights of the first and second dilated temporal convolutional layers, respectively. 1,l b 2,l Q represents the bias of the first and second dilated temporal convolutional layers, respectively. l express and The connection, D l It is l th The output of the double extended temporal convolutional layer, where W 3,l b is the weight of the 1×1 convolutional layer in formula (3). 3，l It is the bias of the 1×1 convolutional layer in formula (3).

6. The cataract surgery procedure identification method according to claim 1, characterized in that: In the second-stage temporal convolutional network, each single-expansion temporal convolutional layer undergoes normalization and ReLU activation operations, and the expansion rate of the second-stage temporal convolutional layers increases exponentially.

Images