A video segmentation method and system for laparoscopic liver surgery

Abstract

The application discloses a video segmentation method and system for laparoscopic liver surgery, and belongs to the technical field of video segmentation. The method comprises the following steps: pre-processing a laparoscopic liver surgery video to obtain training data; constructing a video liver segmentation network, including a backbone network for segmenting a liver from an input video frame and an auxiliary network for performing time consistency constraint and occlusion estimation on the input video frame; training the video liver segmentation network through a knowledge distillation method, and extracting the backbone network as a video liver segmentation model after the training is completed; and segmenting real-time video data input through the video liver segmentation model to obtain a liver segmentation result. The training process based on the spatiotemporal consistency constraint can fully utilize unannotated frames in the video, and ensures high precision. The model is compressed based on the knowledge distillation method, which saves the computing power while ensuring the real-time prediction, and provides real-time assistance for doctors during the operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video segmentation, in particular to a video segmentation method and system for laparoscopic liver surgery. BACKGROUND

[0002] Laparoscopic surgery is a very useful minimally invasive surgery (MIS) with significant clinical advantages, including reducing postoperative complications, reducing blood loss, shortening recovery time, reducing scars and tissue trauma. In the past few decades, interventional endoscopy has been widely used in various organs in the body cavity, including laparoscopic gastrectomy, cholecystectomy, and pancreatic resection.

[0003] However, in actual laparoscopic surgery, due to the limited field of view of the surgeon, he can only see partial information of the organs in the abdominal cavity during the operation, so this has a certain impact on the diagnosis and treatment of the disease.

[0004] Currently, computer-aided diagnosis systems can effectively help doctors find lesions and reduce misdiagnosis rates by taking appropriate surgical procedures. Currently, although video segmentation algorithms have been widely used in polyp detection, due to the different visual features of polyps and livers, and the complexity of liver blood supply and the variety of lesions, the algorithm for video polyp segmentation cannot be directly and well migrated to video liver segmentation.

[0005] For example, the Chinese invention patent with publication number CN114972293A discloses a video polyp segmentation method and device based on a semi-supervised spatio-temporal attention network. The technical solution provided by the invention constructs a U-shaped network architecture through a multi-layer encoder and decoder, and obtains a relatively accurate polyp segmentation result through a time local context attention module and an inter-frame spatio-temporal attention module. However, due to the different visual features of the liver and polyps, and the complexity of liver lesions, deformation occurs during the operation, and the network architecture proposed by the invention cannot be well migrated to the intraoperative liver segmentation task, and the training method used by the technology cannot utilize all the data in the data set.

[0006] For example, the liver CT image segmentation method, device, equipment and readable storage medium disclosed in Chinese invention patent CN115272366A. The technical scheme provided by the invention uses the liver part CT image obtained during the operation, based on the MIMO-FAN model, and performs multi-scale analysis on three different layer slice images through spatial pyramid pooling, which can explain the image-level context features of the whole scene from different scales, and then segments the liver part in the CT image according to the image-level context features. The technical means of the invention can achieve good accuracy and speed for segmenting a single ct image, but if it is used for intraoperative real-time liver segmentation, it cannot achieve the purpose of dynamic positioning and real-time assistance of the surgeon's operating area along with the operation process.

[0007] As can be seen from the above, the defects of the prior art scheme applied directly to video liver segmentation are mainly:

[0008] 1. Video liver segmentation task is more complex than ordinary segmentation task. Limited by the difficulty of obtaining real surgery video data and the complexity of video annotation, the existing segmentation method is difficult to realize the video liver segmentation task.

[0009] 2. High computing power is required. If the picture segmentation method is used to segment each frame of the video, it is difficult to provide timely and effective help to the surgeon in the operation scene.

[0010] 3. Poor real-time performance, cannot perform real-time video segmentation on the surgeon's operating area along with the operation process.

[0011] 4. In the actual laparoscopic liver surgery environment, the surgeon's surgical equipment is likely to cause occlusion of the liver, and the existing liver semantic segmentation method is difficult to cope with liver segmentation with occlusion. SUMMARY

[0012] In view of the problem that the existing technical scheme cannot be well migrated to video liver segmentation, the purpose of the present application is to provide a video segmentation method and system for laparoscopic liver surgery, so as to at least partially solve the above problems.

[0013] To achieve the above purpose, the technical scheme of the present application is:

[0014] In a first aspect, the present application provides a video segmentation method for laparoscopic liver surgery, comprising the following steps:

[0015] S1, obtaining a laparoscopic liver surgery video and preprocessing it to obtain training data comprising a plurality of video frames;

[0016] S2, a video liver segmentation network is constructed, which includes a backbone network for segmenting a liver from an input video frame and an auxiliary network for temporal consistency constraint and occlusion estimation of the input video frame;

[0017] S3, the video liver segmentation network is trained by a method of knowledge distillation, and the backbone network is extracted as a video liver segmentation model after the training is completed;

[0018] S4, the input real-time video data is segmented by the video liver segmentation model to obtain a liver segmentation result.

[0019] Preferably, in step S1, the step of preprocessing the obtained laparoscopic liver surgery video specifically includes:

[0020] Part of the video frames in the obtained laparoscopic liver surgery video are labeled;

[0021] A plurality of labeled video frames are randomly sampled, and each labeled video frame and its two preceding frames and two subsequent frames are constructed into a five-tuple;

[0022] Each video frame in the five-tuple is uniformly scaled to the same size, and after normalization, the training data is formed.

[0023] Preferably, the label for labeling the video frame includes a liver surface, an upper edge and a lower edge of the liver within a visible range in the image, and a falciform ligament of the liver and a lesion of the liver.

[0024] Preferably, in step S2, the backbone network is a Unet network or a Resnet network, and the auxiliary network is a MaskFlownet network.

[0025] Preferably, in step S3, the step of training the video liver segmentation network by the method of knowledge distillation specifically includes:

[0026] A teacher network T is pre-trained using a cross-entropy loss and a temporal consistency loss, and the backbone network is used as a student network S;

[0027] During the training process, the parameters of the teacher network T are frozen, and the student network S and the teacher network T are aligned using double-frame dependency.

[0028] Preferably, the teacher network T is obtained based on the combination of Resnet101 and Unet segmentation architecture.

[0029] Preferably, the auxiliary network is trained in a self-supervised manner, that is, the t-th video image is transformed to the t+k-th video image by optical flow transformation, and then trained using photometric consistency and motion smoothness.

[0030] Preferably, in step S2, the video liver segmentation network further comprises a spatial transformer, so as to make the video liver segmentation network more sensitive to the liver region in the video frame.

[0031] In a second aspect, the present application also provides a video segmentation system for laparoscopic liver surgery, comprising:

[0032] A video recording module is configured to acquire a real-time surgery video of a liver part under a surgeon's perspective in a laparoscopic surgery.

[0033] A preprocessing module is configured to pre-process the acquired real-time surgery video.

[0034] A video liver segmentation module is configured to call a pre-trained video liver segmentation model to perform semantic segmentation on the pre-processed real-time surgery video, and obtain a video liver segmentation result.

[0035] Preferably, the system further comprises a dynamic visualization module configured to visualize the video liver segmentation result, and the visualized content includes the upper and lower edges of the liver, the falciform ligament, the liver surface and the liver lesion area within the visual range under the surgeon's perspective.

[0036] By adopting the above technical solutions, the present application has the following advantages:

[0037] 1. To solve the problem of difficulty in annotating video liver segmentation dataset, the present application provides a scheme based on joint learning of optical flow and semantic segmentation, which un-supervises the learning of optical flow in the video, and uses the optical flow to impose constraints on the feature maps of the semantic segmentation of the front and back frames, so that the optical flow and the semantic segmentation mutually gain, and such constraints can help to utilize all the data in the dataset including the un-annotated video frames to learn more robust segmentation features to improve the accuracy.

[0038] 2. To solve the problems of high computing power requirement and poor real-time performance of existing methods, the present application provides a network model compressed based on a knowledge distillation method, which uses a compact network to make predictions in the application stage, improves the prediction efficiency while ensuring the accuracy, does not increase additional calculation in the test stage, and can be well utilized in the intraoperative real-time liver segmentation task to provide timely and effective help to the surgeon in the surgery scene. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The flowchart of the video segmentation method for laparoscopic liver surgery in the first embodiment of the present application;

[0040] Figure 2 The structural schematic diagram of the video segmentation system for laparoscopic liver surgery in the first embodiment of the present application;

[0041] Figure 3 Figure 3 is a schematic diagram of the structure of an electronic device in Embodiment Three of the present application. DETAILED DESCRIPTION

[0042] The specific embodiments of the present application will be further described below with reference to the drawings. It should be noted that the description of these embodiments is intended to help understand the present application, but does not constitute a limitation on the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0043] It should be noted that in the description of the present application, the terms "upper", "lower", "left", "right", "front", "back" and the like indicate the orientation or positional relationship of the structure of the present application shown in the drawings, and are only for the convenience of describing the present application simply, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0044] For "first" and "second" in the present technical solution, it is only a distinction of the same or similar structure, or the corresponding structure with similar functions, and is not an arrangement of the importance of these structures, nor a size comparison, or other meanings.

[0045] In addition, unless otherwise explicitly specified and limited, the terms "mounting", "connection" should be understood broadly, for example, the connection can be fixed connection, or detachable connection, or integral connection; can be mechanical connection, or electrical connection; can be directly connected, or indirectly connected through intermediate medium, can be the internal communication of two structures. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the overall idea of the present application and the specific circumstances of the present solution.

[0046] Embodiment One

[0047] A video segmentation method for laparoscopic liver surgery, as shown in Figure 1 The method comprises the following steps:

[0048] S1, obtaining a laparoscopic liver surgery video and pre-processing it to obtain training data comprising a plurality of video frames.

[0049] The step of pre-processing the obtained laparoscopic liver surgery video specifically comprises:

[0050] The reason for labeling part of the video frames in the obtained laparoscopic liver surgery video is that labeling all video frames is time-consuming and laborious. The labels include the liver surface, the upper and lower edges of the liver in the visible range of the image, and the falciform ligament and lesions of the liver.

[0051] After labeling is completed, a number of labeled video frames are randomly sampled, and each labeled video frame and its two predecessor frames and two subsequent frames are constructed into a five-tuple, i.e., (frame_1, frame_2, labelled frame, frame_3, frame_4).

[0052] Each video frame in the five-tuple is uniformly scaled to the same size, for example, 512x512x3, and then normalized to form training data. That is, the training data usually includes a number of five-tuples, and each five-tuple can be used to train the network. In this embodiment, the laparoscopic liver surgery video is actually obtained from different patients. The training data formed according to the above steps is divided into a training set and a validation set according to a ratio of 9:1. The training set is used to train the network model, and the validation set is used to verify the trained network model.

[0053] S2, a video liver segmentation network is constructed, which includes a backbone network for segmenting the liver from an input video frame and an auxiliary network for temporal consistency constraint and occlusion estimation of the input video frame.

[0054] The backbone network is selected as a Unet network or a Resnet network, and the auxiliary network is selected as a MaskFlownet network (i.e., a neural network-based prediction method).

[0055] For a continuously played frame sequence, the predicted classification of a certain pixel point of a predecessor frame should be consistent with the predicted classification of the corresponding pixel point in a subsequent frame. Therefore, temporal consistency constraint is used to limit the predicted classification of the video frame that has not been labeled. Specifically, the loss function is as follows:

[0056]

[0057] wherein I t , I t+k represent the tth frame and the t+kth frame images input to the video liver segmentation network, is the predicted result of the tth frame image at position i, is the predicted result of the t+kth frame image at position i when the t+kth frame image is transformed to the tth frame image by optical flow. In this way, the segmentation maps of the two input images will be aligned by this optical flow motion guidance.

[0058] In this way, based on the spatiotemporal consistency constraint, the unlabeled frames in the video can also be utilized in the training process, solving the problem of the data set of the video liver segmentation, while ensuring high precision of the intraoperative segmentation of the liver region, and good generalization for different patients, providing strong support for the laparoscopic liver surgery, thereby solving the problems of great difficulty in obtaining the data set and time-consuming and laborious data set labeling in the video liver semantic segmentation task.

[0059] In addition, since the surgical instruments of the doctor may block the liver part in the actual laparoscopic liver surgery, an occlusion estimation is also needed to determine whether a certain region of interest is occluded, so as to eliminate the error caused by the optical flow transformation. Among them, O occ is introduced for this purpose, which represents the occlusion probability and is also calculated by the auxiliary network; if there is no occlusion, (1-O occ ) is close to 1, the temporal consistency loss is calculated as real, otherwise, this item is close to 0, and the temporal consistency loss of this pixel position can be ignored.

[0060] S3, training the video liver segmentation network by the method of knowledge distillation, and extracting the backbone network as the video liver segmentation model after the training is completed.

[0061] Among them, the step of training the video liver segmentation network by the method of knowledge distillation specifically includes:

[0062] The teacher network T is pre-trained by using the cross-entropy loss and the temporal consistency loss, and the above-mentioned backbone network is used as the student network S;

[0063] In the training process, the parameters of the teacher network T are frozen, and the student network S and the teacher network T are aligned by using the double-frame dependence.

[0064] The principle of knowledge distillation is: for two tensors M t , M t+k , that is, the segmentation results of the t-th frame and the t+k-th frame, as the input of this part, after processing, the similarity graph of the two tensors is obtained For the pixel r ij in R, there are That is and cosine similarity, where and are the i-th row and the j-th row of the features of M t , M t+k . If the position i in the t-th frame is transformed by the optical flow to the position j in the t+k-th frame, then their similarity r ij will be high, so the double-frame dependence can reflect the motion correlation between the two video frames. Specifically, the loss function of this part is as follows:

[0065]

[0066] wherein the superscript S represents the parameters of the student network S, and the superscript T represents the parameters of the teacher network T, and the teacher network T is obtained based on the combination of Resnet101 and Unet segmentation architecture. The auxiliary network is trained in a self-supervised manner, that is, the t-th frame of video image is transformed into the t+k-th frame of video image through optical flow transformation, and then trained by using photometric consistency and motion smoothness.

[0067] In this step, the teacher network T is used to compensate for the difficulty that the intra-abdominal image color is similar, the feature is not obvious, and the student network S cannot capture the feature well, so as to ensure that the student network S (i.e. the backbone network) has good accuracy and robustness.

[0068] After the network is trained, the auxiliary network and the teacher network T (i.e. the auxiliary network and the teacher network T are only used as auxiliary tools in the training process to calculate the temporal consistency loss and the compressed model) can be discarded, and only the student network S (i.e. the backbone network) is retained as the video liver segmentation model, so as to effectively improve the prediction rate and achieve the function of real-time assistance in the surgical scene.

[0069] S4, segmenting the input real-time video data by using the video liver segmentation model to obtain a liver segmentation result.

[0070] Embodiment two

[0071] In this embodiment, for the above step S2, the video liver segmentation network further includes a spatial transformer, so as to make the video liver segmentation network more sensitive to the liver region in the video frame.

[0072] That is, a spatial attention mechanism is introduced to identify the liver part under laparoscopy. Since the video captured in the laparoscopic surgery scene will be distorted when projected onto the screen, and the visual features inside the patient's abdominal cavity are similar, a spatial attention mechanism needs to be introduced, that is, a spatial transformer is used to perform corresponding spatial transformation on the spatial domain information in the image, so as to extract the key information and make the network more sensitive to the liver features. Specifically, for an image of NxCxHxW dimensions, weights are assigned to different positions (HxW) of the same channel C, so that the network pays more attention to different regions.

[0073] Embodiment three

[0074] A video segmentation system for laparoscopic liver surgery, as shown in Figure 2 , comprising:

[0075] The video recording module is used to acquire real-time surgical videos of the liver from the surgeon's perspective during laparoscopic surgery.

[0076] The preprocessing module is used to preprocess the acquired real-time surgical video;

[0077] The video liver segmentation module is used to call a pre-trained video liver segmentation model to perform semantic segmentation on pre-processed real-time surgical videos and obtain video liver segmentation results.

[0078] Preferably, the system also includes a dynamic visualization module for visualizing the liver segmentation results in the video. The visualized content includes the upper and lower edges of the liver, the falciform ligament, the liver surface, and the liver lesion area within the visible range from the doctor's perspective.

[0079] Example 4

[0080] An electronic device, such as Figure 3 As shown, it includes a memory storing executable program code and a processor coupled to the memory; wherein the processor calls the executable program code stored in the memory to execute the method steps disclosed in the above embodiments.

[0081] Example 5

[0082] A computer storage medium storing a computer program, wherein the computer program is executed by a processor to perform the method steps disclosed in the above embodiments.

[0083] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable devices provide operational steps for implementing the function specified in the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0086] The embodiments of the present application described above are merely intended to illustrate the present application, but the present application is not limited to the described embodiments. Various changes, modifications, replacements and variations of the embodiments can be made by those skilled in the art without departing from the principles and spirits of the present application, and still fall within the protection scope of the present application.

Claims

1. A video segmentation method for laparoscopic liver surgery, characterized in that: The method comprises the following steps: S1, acquiring a laparoscopic liver surgery video and pre-processing the same to obtain training data comprising a plurality of video frames; S2, constructing a video liver segmentation network comprising a backbone network for segmenting a liver from an input video frame and an auxiliary network for temporal consistency constraint and occlusion estimation of the input video frame; S3, training the video liver segmentation network by a method of knowledge distillation and extracting the backbone network as a video liver segmentation model after the training is completed; S4, segmenting input real-time video data by the video liver segmentation model to obtain a liver segmentation result; In step S1, the pre-processing of the acquired laparoscopic liver surgery video comprises: annotating part of the video frames in the acquired laparoscopic liver surgery video; randomly sampling a plurality of annotated video frames and constructing each annotated video frame and its two preceding frames and two subsequent frames into a five-tuple; uniformly scaling each video frame in the five-tuple to the same size and then performing normalization to form the training data; The auxiliary network is trained in a self-supervised manner, that is, the t-th frame of video image is transformed to the t+1-th frame of video image through optical flow transformation, and then training is performed by using photometric consistency and motion smoothness, and a loss function is as follows: L = (I t+1 - I t) 2 + (I t+1 - I t) 2 ； wherein, respectively represent the first frame and the second frame image input to the video liver segmentation network, is a prediction result of the second frame image at position , is a prediction result of the first frame image when the position is transformed to the second frame image by optical flow, represents an occlusion probability at position .

2. The method of claim 1, wherein: the labels for annotating the video frames include a liver surface, an upper edge and a lower edge of the liver within a visible range in the image, a falciform ligament of the liver and a lesion of the liver.

3. The method of claim 1, wherein: In step S2, the backbone network is a Unet network or a Resnet network, and the auxiliary network is a MaskFlownet network.

4. The method of claim 1, wherein: In step S3, the training of the video liver segmentation network by the method of knowledge distillation comprises: pre-training a teacher network T by a cross-entropy loss and a temporal consistency loss, and the backbone network is a student network S; in the training process, the parameters of the teacher network T are frozen, and the student network S is aligned with the teacher network T by double-frame dependence.

5. The method of claim 4, wherein: The teacher network T is obtained based on the combination of Resnet101 and Unet segmentation architecture.

6. The method of claim 1, wherein: In step S2, the video liver segmentation network further comprises a spatial transformer for making the video liver segmentation network more sensitive to the liver region in the video frame.

7. A video segmentation system for laparoscopic liver surgery, the system being configured to perform the method of any one of claims 1-6, characterized in that: The system comprises: a video recording module for acquiring a real-time surgery video of a liver part under a surgeon's perspective in a laparoscopic surgery; a pre-processing module for pre-processing the acquired real-time surgery video; a video liver segmentation module for calling a pre-trained video liver segmentation model to perform semantic segmentation on the pre-processed real-time surgery video to obtain a video liver segmentation result.

8. The system of claim 7, wherein: The system further comprises a dynamic visualization module for visualizing the video liver segmentation result, and the visualized content includes an upper edge and a lower edge of the liver, a falciform ligament, a liver surface and a liver lesion region within a visible range under the surgeon's perspective.

Images