Liver vein recognition method in liver operation based on computer vision technology

Abstract

The embodiment of the invention discloses a liver intraoperative hepatic vein recognition method based on a computer vision technology. The method comprises the following steps: acquiring a to-be-recognized liver intraoperative video stream; sequentially executing the following operations on each video frame in the video stream: performing target detection on the current video frame by using an improved YOLO model to frame a hepatic vein region in the current video frame; performing instance segmentation on the features of the hepatic vein region by using a UNet model to obtain a pixel-level segmentation boundary of the hepatic vein structure; wherein the improved YOLO model and the UNet model share a convolutional layer used for feature extraction. According to the embodiment of the invention, the instantaneity and accuracy of intraoperative hepatic vein recognition are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to biomedical engineering technology and computer vision technology, and in particular to a method for identifying hepatic veins during liver surgery based on computer vision technology. Background Art

[0002] With the development of minimally invasive surgical techniques, the application of laparoscopy in liver surgery has become one of the standard treatment methods. Compared with traditional open surgery, minimally invasive surgery has significant advantages such as less trauma, faster recovery, and shorter hospital stay. However, minimally invasive surgery also has many technical difficulties, especially in liver surgery. How to accurately identify the hepatic vein structure has become a key challenge in the operation.

[0003] In traditional laparotomy, surgeons can determine the distribution and morphology of blood vessels by directly observing the surface of the liver and its vascular structure. However, in minimally invasive surgery, doctors can only rely on the field of view provided by laparoscopic equipment. This limitation makes it extremely difficult to identify the hepatic vein and other important vascular structures. Once liver vascular damage occurs, especially hepatic vein damage, it will not only cause severe bleeding, but also increase the complexity of the operation, and may even lead to conversion to laparotomy, increasing the patient's postoperative recovery time and risk.

[0004] Currently, intraoperative identification of hepatic veins mainly relies on the following techniques:

[0005] 1) Intraoperative ultrasound technology: Intraoperative ultrasound can provide real-time images of the liver, but its image quality and operation accuracy are greatly affected by the operator's experience. In addition, not all hospitals have highly skilled ultrasound operators, making it difficult to popularize.

[0006] 2) Three-dimensional reconstruction and virtual reality technology: These technologies use preoperative imaging data (such as CT and MRI) to perform three-dimensional modeling to assist doctors in preoperative planning and intraoperative navigation. However, due to the flexible characteristics of the liver and its dynamic changes during surgery, it is difficult to accurately match the intraoperative three-dimensional reconstructed image with the actual liver morphology, which affects the recognition accuracy.

[0007] 3) Molecular fluorescence imaging technology: uses near-infrared imaging technology and fluorescent contrast agents to mark blood vessels and tissues, but its penetration depth is only a few centimeters, making it difficult to effectively identify deep structures, and may cause side effects during use.

[0008] 4) Augmented reality technology (AR): Augmented reality technology can superimpose preoperative three-dimensional images with real-time images during surgery to enhance the doctor's visual perception ability. However, since it relies on preoperative images, it cannot cope with changes in liver morphology during surgery and it is difficult to accurately align with the flexible structure of the liver.

[0009] At present, although various imaging technologies have been widely used in liver surgery, there are still obvious shortcomings in their real-time performance, operational complexity, accuracy, etc., and most technologies are highly dependent on the operator's experience and are not universally applicable. Summary of the invention

[0010] The embodiment of the present invention provides a method for identifying hepatic veins during liver surgery based on computer vision technology, which improves the real-time performance and accuracy of hepatic vein identification during surgery.

[0011] In a first aspect, an embodiment of the present invention provides a method for identifying hepatic veins during liver surgery based on computer vision technology, comprising:

[0012] Acquire a liver intraoperative video stream to be identified;

[0013] The following operations are performed on each video frame in the video stream in sequence:

[0014] S1-1, using an improved YOLO model to perform target detection on the current video frame to frame the hepatic vein region in the current video frame;

[0015] S1-2, using the UNet model to perform instance segmentation on the features of the hepatic vein region to obtain pixel-level segmentation boundaries of the hepatic vein structure;

[0016] The improved YOLO model and the UNet model share a convolutional layer for feature extraction.

[0017] In a second aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0018] one or more processors;

[0019] a memory for storing one or more programs,

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for identifying hepatic veins during liver surgery based on computer vision technology as described in any embodiment.

[0021] Embodiments of the invention The embodiments of the invention apply computer vision technology and image processing algorithms, combined with deep learning models, to propose a method for identifying hepatic veins during liver surgery based on computer vision technology. The method can obtain the vascular information of the liver in real time during complex liver surgery, assist surgeons in accurately locating the hepatic veins in a dynamic surgical environment, and solve the accuracy and real-time problems of hepatic vein identification during surgery.

[0022] Specifically, this embodiment uses target detection technology to detect targets in video frames, frames the hepatic vein region, and then performs instance segmentation on the image features of the hepatic vein region, which can improve segmentation efficiency and eliminate interference from other regions, thereby improving the real-time and accuracy of hepatic vein structure recognition. At the same time, the target detection model and the instance segmentation model in this embodiment share the front convolution layer, and realize joint training and recognition of multiple tasks in the same model framework, ensuring that the two tasks have the same basic features, ensuring the consistency of subsequent feature processing, and improving training efficiency and recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 Schematic diagram of the basic structure of a YOLO model provided by an embodiment of the present invention;

[0025] Figure 2 is a schematic diagram of a shared convolutional layer sliding window provided by an embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of a shared convolutional layer structure and parameters provided by an embodiment of the present invention;

[0027] Figure 4 is a flow chart of a method for identifying hepatic veins during liver surgery based on computer vision technology provided by an embodiment of the present invention;

[0028] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0030] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0031] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0032] As described in the background art, although the prior art has applied a variety of methods to identify and locate the hepatic vein in liver surgery, these technologies still have certain defects, especially in terms of real-time performance, accuracy, operation dependence and universality. Specifically, the shortcomings of the prior art are mainly reflected in the following aspects:

[0033] 1) Lack of real-time performance

[0034] Intraoperative ultrasound technology: Although intraoperative ultrasound can provide real-time images, its display effect is affected by many factors such as operator skills, equipment performance, and real-time interference. Especially in complex liver surgery, ultrasound images are difficult to provide sufficiently clear vascular locations, and they need to rely on experienced doctors for operation and interpretation, lacking a real-time feedback mechanism.

[0035] 3D reconstruction and virtual reality (VR) technology: These technologies rely on preoperative imaging data for 3D modeling and reconstruction. Although they have certain advantages in some fixed structure surgeries, the liver is a flexible organ, and its shape and position will change significantly during the operation with the surgical operation and the patient's position. Traditional 3D reconstruction technology is difficult to adapt to the dynamic changes of the liver, and therefore cannot provide doctors with real-time and accurate vascular positioning.

[0036] Augmented reality technology: Although augmented reality can combine preoperative images with real-time intraoperative images for display, it still relies on preoperative images and three-dimensional reconstruction. Therefore, when the liver deforms and changes during surgery, it is difficult to ensure accurate matching with the actual intraoperative situation, which affects the accuracy of the surgery.

[0037] 2) Operational complexity and technology dependence

[0038] High technical threshold of intraoperative ultrasound technology: The use of intraoperative ultrasound technology is highly dependent on the experience and skills of the operator, requiring not only the operating doctor to have superb skills, but also the entire surgical team to have the ability to cooperate and understand. Due to the high skill requirements, the popularity of intraoperative ultrasound operation is poor, especially in small and medium-sized hospitals or in the case of non-professional ultrasound doctors, the identification of hepatic veins during surgery is often greatly limited.

[0039] Difficulty of molecular fluorescence imaging technology: This technology requires the injection and monitoring of specific fluorescent contrast agents during surgery, and is usually used in conjunction with intraoperative ultrasound, which is difficult to operate. At the same time, the penetration depth of near-infrared imaging technology is only 1-2cm, so it is difficult to effectively identify deep blood vessels or hepatic vein structures, and the penetration ability is greatly limited.

[0040] 3) Poor adaptability and insufficient accuracy

[0041] Poor adaptability of 3D reconstruction and virtual reality technologies: Although these technologies can be prepared before surgery, they rely on the registration of preoperative images with intraoperative images. Since the liver is a flexible organ, changes in its morphology and position during surgery can lead to deviations between preoperative images and actual intraoperative images, which in turn affects the accuracy of intraoperative navigation and positioning images.

[0042] Technology relies on preoperative image data: Both 3D reconstruction and augmented reality technologies rely on preoperative CT or MRI data for reconstruction, which means that when the liver morphology changes during surgery, the image cannot be adjusted in real time and cannot accurately reflect the dynamic changes of the liver.

[0043] 4) Technical limitations

[0044] Limitations in identifying deep blood vessels: Most of the existing intraoperative imaging technologies are unable to effectively identify deep vascular structures, especially tiny hepatic veins and vascular branches. The penetration depth of intraoperative ultrasound and fluorescence imaging is limited, making it difficult to obtain complete vascular information, especially when dealing with tiny blood vessels or blood vessels hidden inside the liver, the accuracy is significantly reduced.

[0045] The present invention proposes a method for identifying hepatic veins during liver surgery based on computer vision technology, aiming to solve the deficiencies of existing technologies in terms of real-time, accuracy, and adaptability of hepatic vein identification during minimally invasive liver surgery. This method combines target detection and instance segmentation technology, and uses a deep learning model to achieve accurate identification and real-time positioning of hepatic veins, thereby providing doctors with efficient and reliable surgical navigation images. In order to illustrate this method, a method for constructing a model for identifying hepatic veins during liver surgery based on computer vision technology is first introduced. Specifically, the method includes data acquisition, data preprocessing, data annotation, algorithm model establishment, model training and optimization, system integration, and hardware adaptation.

[0046] 1. Abstract medical problems into engineering problems

[0047] The invention of this embodiment is to abstract the problem of hepatic vein recognition in liver surgery into a target detection and instance segmentation problem, and solve it by applying computer vision and deep learning technology. The recognition of vascular structures in liver resection, especially the dynamic recognition of hepatic veins, involves real-time processing of intraoperative images, and in this process, accurately calibrating the position and morphology of the hepatic veins.

[0048] In traditional surgery, the identification of hepatic veins is highly dependent on the surgeon's experience and preoperative imaging data. However, as a flexible organ, the liver's position and morphology change greatly during surgery. Preoperative images cannot perfectly match the actual situation of the liver during surgery, making it difficult to effectively apply traditional technologies. This embodiment abstracts the hepatic vein recognition problem into a target detection and instance segmentation problem in computer vision, and develops a real-time recognition and segmentation method and system for hepatic vein structure based on the improved YOLO and UNet deep learning frameworks. Through this method, this embodiment can provide real-time and precise hepatic vein positioning in minimally invasive liver surgery, and provide accurate and safe intraoperative images.

[0049] 2. Data Collection and Preprocessing

[0050] The data here refers to training data. In the data collection stage, this embodiment combines laparoscopic equipment and intraoperative video streams to capture intraoperative vascular images in real time through high-resolution camera technology. Different from the traditional static image acquisition method, this embodiment uses dynamic video stream acquisition to ensure that the real-time changes of the liver and blood vessels during the operation can be captured and used for subsequent analysis.

[0051] Specific methods:

[0052] 1) Data acquisition equipment: High-definition laparoscopic equipment is used in conjunction with intraoperative ultrasound imaging technology to achieve real-time capture of vascular structures during liver surgery. Through high-speed dynamic video streaming, the position changes of the hepatic veins can be accurately tracked to avoid vascular positioning errors caused by liver deformation during surgery.

[0053] 2) Data preprocessing: The collected intraoperative videos are processed by denoising, image enhancement, and light balancing to ensure high-quality data input. Image denoising technology is used to remove interference signals (such as blood, water vapor, etc.) generated during surgery, and standardized processing is performed to meet the input requirements of subsequent deep learning models. At the same time, the collected intraoperative dynamic images are screened and processed, the available data is retained, the abnormal data is eliminated, and whether to use data enhancement processing is decided based on the specific situation;

[0054] 3. Data labeling and segmentation

[0055] This embodiment uses a semi-automatic annotation tool in the data annotation process, combined with the verification of medical experts, to ensure the accuracy and consistency of vascular annotation. At the same time, the annotation process takes into account the dynamic variability of liver surgery, and annotates each video segment (such as before, during, and after liver parenchyma separation) in stages to provide multi-dimensional vascular annotation information.

[0056] Specific methods:

[0057] 1) Annotation content: Under the guidance of medical experts, the hepatic vein and its branch vessels are annotated in detail. The annotation content includes: main blood vessels, small blood vessels, microvessels, etc., and their spatial relationships and anatomical levels are further annotated.

[0058] 2) Labeling tool: Use semi-automatic labeling tools and image segmentation technology to calibrate the blood vessel area in each frame of the image to ensure the efficiency and accuracy of the labeling.

[0059] 3) Data segmentation: The labeled data set is divided into training set, validation set and test set to ensure the generalization ability of model training. At the same time, data enhancement (such as rotation, flipping, scaling, etc.) is used to improve the model's adaptability to different image changes.

[0060] 4. Construction and training of recognition model

[0061] 4.1 Selection and Design of Object Detection and Instance Segmentation Models

[0062] This embodiment combines the two technologies of target detection and instance segmentation, and proposes a multi-task learning model for the precise positioning and pixel-level segmentation of hepatic veins, including an improved YOLO model for target detection and a UNet model for instance segmentation. Through this innovative combination, not only can the accurate position of the hepatic vein be located, but also high-precision pixel-level segmentation can be performed on it. This design can effectively cope with the complex vascular morphology and real-time dynamic changes in liver surgery, solving the bottlenecks of existing methods in terms of accuracy and real-time performance.

[0063] 4.2 Improved YOLO target detection model

[0064] In this embodiment, the improved YOLO (based on the YOLOv5 model) is used to detect the location of the hepatic vein and accurately define the spatial location of each blood vessel. Compared with the traditional sliding window or region proposal method, the improved YOLO uses global feature learning and can output the detection task through a single forward pass, greatly improving the detection speed.

[0065] Specifically, the improvements of the YOLO model in this embodiment are:

[0066] 1) Optimized network structure: The improved YOLO model is optimized based on the traditional YOLO model, and introduces the CSPDarknet network structure and SiLU activation function, which effectively improves the performance of the model in complex image scenes. Specifically, the basic structure of the YOLO model is as follows: Figure 1 As shown, this embodiment introduces CSPDarknet and SiLU activation functions in YOLOv5, so that they work together in the feature extraction and activation process, forming an important part of the model. These two functions help YOLOv5 achieve efficient and accurate target detection in complex image scenes by optimizing the capabilities of feature extraction and nonlinear expression, and are particularly suitable for scenes with high real-time requirements such as liver surgery. The connection and role of these two functions in YOLOv5 will be specifically described below.

[0067] CSPDarknet is the backbone network of YOLOv5, which replaces the CSP part (including CSP1_X and CSP2_X) in the original network with CSPDarknet to extract features from the input image. It improves the feature extraction efficiency of YOLOv5 in multi-level and complex scenes by optimizing the calculation in the feature extraction module.

[0068] CSPDarknet is the backbone of YOLOv5 and is located after the input layer of the model. It is responsible for extracting preliminary feature information from the input image. These features will serve as the basis for the subsequent input of the detection head (YOLO detection part). Specifically, CSPDarknet processes the low-level and mid-level features of the input image and extracts the features of targets such as the hepatic vein and other blood vessels through multi-layer convolution.

[0069] Furthermore, CSPDarknet adopts the Cross-Stage Partial (CSP) structure, which divides the features into two parts, one of which is more calculated and the other is less calculated, and finally the two parts are merged. In this way, CSPDarknet can reduce the amount of calculation and improve the efficiency of feature extraction. Through residual connection, CSPDarknet ensures full utilization of feature information, so that the network can efficiently extract and retain key image information at different levels.

[0070] After the SiLU activation function is introduced into each convolutional layer of the YOLOv5 model, it plays a role in improving the nonlinear mapping ability of the network. The SiLU activation function is connected after each convolutional layer of CSPDarknet. Whenever CSPDarknet extracts the features of the image, the SiLU activation function will perform nonlinear transformation on these features to help the network capture more complex patterns and details. This processing method improves the detection ability of YOLOv5 in complex backgrounds and complex target structures (such as liver blood vessels).

[0071] The SiLU activation function provides a smooth activation effect by multiplying the Sigmoid function with the linear part. Compared with the traditional ReLU activation function, SiLU avoids the dead neuron problem that may occur in ReLU, and can better transmit information and improve the network's ability to learn details, especially on small and difficult-to-identify targets such as liver blood vessels.

[0072] With the cooperation of CSPDarknet and SiLU, the input image is subjected to preliminary feature extraction by CSPDarknet. CSPDarknet extracts low-level and mid-level features in the image, such as the rough morphology of liver tissue and vascular regions, through its optimized convolutional network architecture. These extracted features are then processed nonlinearly by the SiLU activation function, thereby enhancing the model's ability to learn image details, especially for targets with complex structures and fuzzy boundaries (such as hepatic veins). Among them, CSPDarknet provides more efficient feature extraction capabilities, especially in multi-level and multi-scale feature learning. It ensures that the network can effectively capture the preliminary features of small blood vessels such as hepatic veins. After feature extraction, the SiLU activation function increases nonlinear capabilities, allowing the network to learn complex patterns in the image more accurately. For vascular overlap or occlusion that may occur during liver surgery, the SiLU activation function can help the model better distinguish different levels of vascular information.

[0073] The combination of CSPDarknet and SiLU activation function enables YOLOv5 to achieve efficient and accurate target detection in complex scenes and dynamic surgical environments. The following are several key factors that improve the performance of the model:

[0074] Better feature extraction and fusion: CSPDarknet reduces the amount of calculation and improves efficiency in processing complex scenarios through its unique segmented calculation design. During surgery, the morphology and vascular structure of the liver constantly change, and CSPDarknet can effectively adapt to these changes and extract more detailed features.

[0075] Enhanced nonlinear modeling capabilities: The introduction of the SiLU activation function enables YOLOv5 to better handle complex nonlinear features, especially when the boundaries of blood vessels are fuzzy or overlap. SiLU helps the model better identify small and complex targets such as hepatic veins and tiny vascular structures.

[0076] Adapt to complex backgrounds: In surgical images, there is often background noise or interference from other organs, which may affect the detection of blood vessels. CSPDarknet enhances the adaptability of the network to complex backgrounds through a multi-level feature extraction mechanism, ensuring that the recognition of the target is not interfered by the background.

[0077] Real-time and accuracy: Combining the efficient feature extraction of CSPDarknet and the ability of SiLU to improve nonlinear modeling, YOLOv5 can perform accurate hepatic vein detection in a real-time environment.

[0078] 2) Adaptive anchor frame: The improved YOLO model adopts an adaptive anchor frame method, which can automatically adjust the scale of the detection frame to ensure detection accuracy according to different surgical scenarios and changes in hepatic vein structure.

[0079] In the improved YOLOv5 model, the realization of automatically adjusting the scale of the detection box relies on the combination of the feature extraction capability of the CSPDarknet network structure and the multi-scale training mechanism. Through k-means clustering to initialize the anchor box, multi-scale training and adaptive adjustment of the anchor box, YOLOv5 can automatically adjust the scale of the anchor box according to different target sizes and image features. In each training, the system optimizes the matching degree of the anchor box according to the IoU value, and ensures the detection accuracy by automatically selecting and adjusting the anchor box in the inference stage. This technological innovation enables YOLOv5 to achieve efficient and accurate target detection in complex scenarios such as minimally invasive liver surgery.

[0080] Based on the above model structure, the training method of the improved YOLO model includes:

[0081] 1) Input processing: First, each frame image in the surgical video stream is converted into an input format of uniform size through data preprocessing (such as grayscale, normalization, etc.) to meet the requirements of the improved YOLO model.

[0082] 2) Training set and test set: The labeled hepatic vein image data is used to split the data set, and the annotated hepatic vein area in each frame is used as the target for training. The output of the target detection model includes the hepatic vein bounding box and the vascular category label.

[0083] 4.3 UNet Instance Segmentation Model

[0084] Based on target detection, UNet is applied to instance segmentation to achieve pixel-level accurate segmentation of hepatic vein structure. UNet is a typical fully convolutional network (FCN), which can extract high-level features of the image and restore the spatial resolution of the image by encoding and decoding the input image. UNet is particularly suitable for processing fine-grained segmentation problems in medical images and can effectively distinguish the boundary between the hepatic vein and other liver tissues.

[0085] Specifically, UNet uses skip connections to enable low-level spatial information to be directly transmitted to the high-level feature extraction process, which is particularly important for vascular structures with detailed features such as the hepatic vein. In addition, UNet uses an encoder-decoder structure, in which the encoder extracts features through convolution and pooling, and the decoder restores the feature map to the original image resolution through deconvolution to achieve pixel-level segmentation.

[0086] The UNet model is very suitable for accurate segmentation of blood vessels in liver surgery, especially when dealing with small and intricate vascular structures, showing its unique advantages. The encoding-decoding structure of UNet enables the network to gradually compress the information of the input image, extract multi-level features, and then restore the resolution of the image through deconvolution (decoding stage), ensuring the details and accuracy of the segmentation results. Especially when dealing with slender structures with unclear edges such as liver blood vessels, UNet can combine low-level detail information with high-level semantic information through skip connections, which not only helps to restore image details, but also enhances the recognition ability of blood vessel boundaries. Even when the blood vessels are blocked by other tissues or the background is complex, the contours of the blood vessels can still be accurately identified and segmented. Therefore, the UNet model can effectively solve the complexity of vascular structures and capture fine-grained features, provide higher accuracy for vascular positioning in liver surgery, and reduce the risk of unnecessary damage during surgery.

[0087] Based on the above model structure, the training methods of the UNet network include:

[0088] 1) Input processing: The image in each frame of the video is input into the UNet network for pixel-level segmentation. The system outputs the fine boundary of each hepatic vein structure to provide accurate visual information for surgical navigation.

[0089] 2) Model optimization: Pre-train the network and adjust the parameters through fine-tuning to ensure the accuracy of the segmentation results on fine structures.

[0090] 4.4 Multi-task Learning and Joint Training

[0091] In order to complete the target detection and instance segmentation tasks of hepatic veins at the same time, this embodiment adopts a multi-task learning (MTL) framework for joint training. Multi-task learning can better utilize the correlation between tasks and improve the generalization ability and accuracy of the model by sharing some parameters in the same network and solving multiple tasks at the same time. This sharing mechanism enables the model to learn global positioning information and local detail information at the same time, making full use of the complementarity between tasks, and at the same time enables the model to learn two types of tasks on the same network structure at the same time, improving efficiency and accuracy.

[0092] 1) Shared convolutional layers enable joint training of object detection and instance segmentation

[0093] In the framework of multi-task learning, object detection and instance segmentation tasks extract the basic features of the image by sharing convolutional layers. Specifically, the object detection task is mainly responsible for locating the target (such as the location information of blood vessels), while the instance segmentation task further focuses on the precise boundary segmentation of these targets.

[0094] Shared convolutional layers: In the early layers of the network, the role of the convolutional layer is to extract low-level features (such as edges, textures, shapes, etc.) from the input image. These features are required for both object detection and instance segmentation, so they share these convolutional layers. By sharing convolutional layers, the network can simultaneously learn the spatial location of blood vessels when performing object detection, and capture the detailed structure of blood vessels when performing instance segmentation. This sharing feature enables the network to efficiently learn common features in multiple tasks while reducing the amount of computation.

[0095] Task separation: Although the convolutional layers are shared in the front layer, the subsequent tasks of object detection and instance segmentation are handled separately through different branches. The object detection part locates the position of the blood vessels (generates bounding boxes) through a specific detection head, while the instance segmentation part performs pixel-level segmentation of the blood vessel area through another branch.

[0096] Specifically, this embodiment provides two optional implementations of convolutional layer sharing and task separation:

[0097] In a first optional implementation, the improved YOLO model and the UNet model share a convolutional layer sliding window (i.e., sliding window) for extracting global features of an image. Figure 2 As shown in the figure, when a certain video frame is input into the hepatic vein recognition model, first, the pixel information of the current video frame is extracted using the same convolutional layer sliding window, and the pixel information is simultaneously input into the convolutional layer of the improved YOLO model and the convolutional layer of the UNet model for feature extraction; then, the sliding window is moved to extract the pixel information of the next window, and the new pixel information is simultaneously input into the convolutional layer of the improved YOLO model and the convolutional layer of the UNet model for feature extraction; this cycle is repeated until all the convolutional layer sliding window positions are moved and calculated, and the basic features of the improved YOLO model and the UNet model are obtained respectively; the basic features are input into the subsequent structure of the improved YOLO model for target detection to frame the hepatic vein region in the current video frame; the basic features are input into the subsequent structure of the UNet model encoder to obtain the final encoding features; the encoding features of the hepatic vein region are intercepted and input into the decoder of the UNet model to complete instance segmentation.

[0098] A second optional implementation manner is that the improved YOLO model and the UNet model share a convolutional layer structure and parameters for extracting image features, wherein the shared convolutional layer may be part or all of the feature extraction network of the improved YOLO model, or may be part or all of the encoder in the UNet model. Figure 3 Taking the shared convolutional layer as part of the backbone network of the improved YOLO model and the entire encoder in the UNet model as an example, the overall model structure of the shared convolutional layer is exemplified. In this structure, part of the convolutional layer of the backbone network of the YOLO model is reused as the encoder of the UNet model, and a decoder structure symmetrical to the encoder is constructed. Accordingly, when a video frame is input into the hepatic vein recognition model, first, the shared convolutional layer is used to perform a convolution operation on the current video frame to obtain an image feature, which is referred to as a shared feature (i.e., Figure 3 Then, the shared features are input into the subsequent structure of the improved YOLO model for target detection to frame the hepatic vein region in the current video frame; finally, a portion of the hepatic vein region is intercepted from the shared features and input into the subsequent structure of the UNet model to complete instance segmentation.

[0099] Of course, the multi-scale features output by each layer of the encoder in the above two methods can also intercept the part corresponding to the hepatic vein area, and jump to the corresponding layers of the decoder to complete the boundary segmentation of the hepatic vein result.

[0100] 2. Learning global and local information

[0101] In joint training, the network needs to learn both global positioning information and local detail information at the same time, which are crucial for object detection and instance segmentation tasks respectively.

[0102] Learning global positioning information: In the object detection task, the network needs to understand the overall structure of the image and identify the approximate location and overall shape of the blood vessels. This type of information is usually extracted by the convolution layer, which captures larger regional features through global convolution kernels or pooling operations. Since the receptive field of the convolution layer is large, the network can recognize global information in the image, such as the overall outline of the liver and the approximate location of the blood vessels.

[0103] Learning of local detail information: In the instance segmentation task, the network needs to accurately identify the details of the blood vessels, such as the branching structure and precise contours of the blood vessels. These details are usually extracted through smaller convolution kernels and usually rely on the local receptive field of the network. Smaller convolution kernels can better capture local features in the image, such as the branching points of blood vessels and fine texture information. Therefore, in the instance segmentation task, the network will focus on the details of the local area to help the model distinguish the tiny structures of the blood vessels.

[0104] In this embodiment, through shared convolutional layers and multi-task learning, the model can learn global information and local information at the same time, and these two types of information will be effectively integrated in subsequent network layers:

[0105] Shared convolution layer of the front layer: The shared convolution layer can effectively extract the global features of the image and some basic local detail features, providing basic information for target detection and instance segmentation.

[0106] Task-specific convolutional layers in subsequent branches: In the subsequent task branches, the object detection branch is responsible for extracting global features to locate the target (the approximate location of the blood vessel), while the instance segmentation branch pays more attention to details and further extracts local information such as the branches and edges of the blood vessels through smaller convolution kernels. In this way, global information and local information are effectively distinguished and fused, ensuring the simultaneous optimization of the two types of tasks.

[0107] Among them, global positioning information is crucial for target detection tasks, which can help the network quickly determine the approximate location of blood vessels in the image and reduce the complexity of target detection. Local detail information is the key to instance segmentation tasks, which can help the network accurately identify the contours and branches of blood vessels, especially during surgery, which is very important for the protection and precise cutting of blood vessels.

[0108] Multi-task learning allows the network to optimize global positioning and local detail learning within the same model framework, avoiding repeated calculations between tasks and improving overall detection and segmentation efficiency. Through joint training, the model can simultaneously learn the location and shape of the target, while accurately segmenting the specific details of the target, which is crucial for the accurate recognition of slender structures such as blood vessels.

[0109] In addition, in order to achieve joint training of target detection and instance segmentation, this embodiment designs a composite loss function, which combines the cross entropy loss (for classification tasks) and the Dice loss (for segmentation tasks), while optimizing the performance of the two tasks to ensure the precise positioning of the target box and the accuracy of blood vessel segmentation.

[0110] During the joint training process, the training data includes images with annotated vessel bounding boxes and vessel masks. The loss function of the object detection part is used to optimize the bounding box position, and the loss function of the instance segmentation part is used to optimize the segmentation accuracy. The training strategy uses the Adam optimizer and the learning rate decay strategy, and cross-validation is used to avoid model overfitting. After each round of training, the system evaluates the IoU (Intersection over Union) and Dice coefficient to quantify the performance of the model in object detection and instance segmentation.

[0111] 4.5 Data Augmentation and Transfer Learning

[0112] In order to solve the problem of insufficient data and improve the generalization ability of the model, this embodiment also introduces two technologies: data enhancement and transfer learning.

[0113] The data enhancement techniques include:

[0114] Image enhancement technology: During the model training process, diversified training data is generated by rotating, cropping, scaling, flipping, and other operations on video frames. By increasing the diversity of training data, the model's adaptability to different intraoperative scenarios can be effectively improved.

[0115] Lighting and noise processing: During the image enhancement process, brightness adjustment, contrast adjustment, and Gaussian noise addition are performed, especially for uneven lighting and image noise that may occur during surgery, to simulate image changes in different surgical environments and enhance the robustness of the model.

[0116] Transfer learning technology includes: This embodiment uses transfer learning technology to migrate the model weights trained on a large medical imaging dataset to this project, avoiding training from scratch, reducing training time, and significantly improving the initial accuracy of hepatic vein recognition.

[0117] 4.6 Model Optimization and Reasoning Acceleration

[0118] After completing the model training, this embodiment optimizes the model through pruning, quantization, and distillation techniques to ensure efficient reasoning on the edge computing platform. Through these optimization measures, the model size can be greatly reduced, the computational complexity can be reduced, and real-time reasoning can be ensured on low-power devices.

[0119] Among them, model pruning reduces the model's computational workload and improves inference speed by pruning unimportant neurons and connections. Quantization technology converts floating-precision values ​​into low-bit fixed-point values ​​to reduce storage and computing resource requirements. Model distillation uses a large model to guide the training of a small model, allowing the small model to reduce computational workload while retaining accuracy.

[0120] Based on the hepatic vein recognition model during liver surgery constructed in any of the above embodiments, Figure 4 1 is a flowchart of a method for identifying hepatic veins during liver surgery based on computer vision technology provided by an embodiment of the present invention. The method is applicable to the case of dynamically identifying hepatic vein structures in an intraoperative video stream and is executed by an electronic device. Figure 4 As shown, the method specifically includes:

[0121] S110: Acquire a liver intraoperative video stream to be identified.

[0122] S120, performing the following operations on each video frame in the video stream in sequence:

[0123] S1-1. Use the improved YOLO model to perform target detection on the current video frame to frame the hepatic vein area in the current video frame.

[0124] S1-2. Use the UNet model to perform instance segmentation on the features of the hepatic vein region to obtain the pixel-level segmentation boundary of the hepatic vein structure.

[0125] Optionally, the improved YOLO model improves each CSP network in the backbone network into a CSPDarknet network, and each CSPDarknet network is used to extract low-level features and intermediate features of the input image, including the rough morphology of liver tissue and blood vessel areas.

[0126] SiLU activation functions are added after the convolutional layers of each CSPDarknet network. Each SiLU activation function is used to perform nonlinear transformation on the features extracted by each CSPDarknet module to learn the image patterns of blurred blood vessels or overlapping and occluded multi-layer blood vessels during liver surgery.

[0127] Optionally, each CSPDarknet network adopts a CSP structure, which divides the features into two parts, one part undergoes more calculations and the other part keeps less calculations, and finally merges the two parts to reduce the amount of calculations and adapt to the characteristics of frequent changes in liver morphology and vascular structure during surgery; through residual connections, each CSPDarknet network can retain key image information at different levels.

[0128] Optionally, the UNet model uses skip connections to transfer low-level spatial information to high-level feature extraction to enhance the learning ability of detailed vascular structures.

[0129] Optionally, the improved YOLO model and the UNet model share a convolutional layer sliding window for extracting basic features of an image; accordingly, the operations of S1-1 and S1-2 may include the following specific steps:

[0130] S2-1, using the same convolutional layer sliding window to extract pixel information of the current video frame;

[0131] S2-2, inputting the pixel information into the convolution layer of the improved YOLO model and the convolution layer of the UNet model simultaneously to extract basic features;

[0132] S2-3, moving the convolutional layer sliding window, and repeating the operations of S2-1 and S2-2 until all sliding window positions are calculated, and obtaining the basic features of the improved YOLO model and the UNet model respectively;

[0133] S2-4, inputting the basic features into the subsequent structure of the improved YOLO model for target detection to frame the hepatic vein region in the current video frame; at the same time, inputting the basic features into the subsequent structure of the UNet model encoder to obtain the final encoding features;

[0134] S2-5. The encoded features of the hepatic vein region are intercepted and input into the decoder of the UNet model to complete instance segmentation.

[0135] Optionally, the improved YOLO model and UNet model may also share convolutional layer structures and parameters for extracting image features; accordingly, the operations of S1-1 and S1-2 may include the following specific steps: performing a convolution operation on the current video frame using a shared convolutional layer to obtain shared features of the image; inputting the shared features into a subsequent structure of the improved YOLO model for target detection to frame the hepatic vein region in the current video frame; and inputting the shared features of the hepatic vein region into a subsequent structure of the UNet model for instance segmentation.

[0136] Furthermore, after obtaining the hepatic vein structure of each video frame during the operation through the above hepatic vein recognition model, a dynamic tuning mechanism can be used to automatically optimize the hepatic vein recognition model based on real-time feedback during the operation. Through the error feedback mechanism, the accuracy of the system is continuously improved, so that the hepatic vein recognition can be automatically adjusted as the operation progresses. Specifically, the following steps may be included:

[0137] Real-time error feedback: During the operation, the recognition effect of the hepatic vein is monitored in real time, and feedback is provided based on the actual situation of the operation (such as vascular position and liver morphological changes). If the recognition is wrong, the feedback information will be automatically incorporated into the training data to update the model parameters.

[0138] Dynamic tuning: Based on real-time feedback data, the model weights are adjusted to continuously improve the recognition accuracy of the hepatic venous structure and ensure effective operation in different surgical environments.

[0139] In summary, the embodiment of the present invention applies computer vision technology and image processing algorithms, combined with a deep learning model, to propose a method for identifying hepatic veins during liver surgery based on computer vision technology. This method can obtain the vascular information of the liver in real time during complex liver surgery, assist surgeons in accurately locating the hepatic veins in a dynamic surgical environment, and solve the accuracy and real-time problems of hepatic vein identification during surgery.

[0140] Specifically, this embodiment uses target detection technology to detect targets in video frames, frame the hepatic vein area, and then perform instance segmentation on the image features of the hepatic vein area, which can improve the segmentation efficiency, eliminate interference from other areas, and improve the real-time and accuracy of hepatic vein structure recognition. Among them, the target detection network adopts an improved YOLO model, and introduces the CSPDarknet network and SiLU activation function into the basic YOLO model. Among them, CSPDarknet provides more efficient feature extraction capabilities, especially in multi-level and multi-scale feature learning, ensuring that the network can effectively capture the preliminary features of small blood vessels such as hepatic veins; the SiLU activation function increases the nonlinear ability after feature extraction, so that the network can learn complex patterns in the image more accurately. For vascular overlap or occlusion that may occur in liver surgery, the SiLU activation function can help the model better distinguish different levels of vascular information.

[0141] At the same time, the target detection model and the instance segmentation model in this embodiment share the front convolutional layer to achieve joint training and recognition of multiple tasks in the same model framework. The sharing methods include sharing sliding windows, or sharing structures and parameters. The shared sliding window ensures that the two tasks have the same basic feature window, ensures the consistency of subsequent feature processing, and improves the training efficiency and recognition efficiency; the shared structure and parameter method further reduces the amount of calculation of shared features. Both sharing methods extract features from the entire image at the feature extraction stage, taking into account global information, and focus on the target instance at the instance segmentation stage to eliminate other interference, thereby improving the accuracy and real-time performance of hepatic vein recognition.

[0142] In addition, the method is highly adaptable and supports offline real-time operation. Through edge computing deployment, the system used to implement the method can achieve real-time reasoning in a network-free environment, ensuring stable transmission and instant feedback of data during surgery. This low-power, high-efficiency design enhances the adaptability and flexibility of the recognition system, making it suitable for various complex surgical environments.

[0143] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the device includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device can be one or more. Figure 5 A processor 60 is taken as an example; the processor 60, the memory 61, the input device 62 and the output device 63 in the device can be connected by a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0144] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the method for identifying hepatic veins during liver surgery based on computer vision technology in the embodiment of the present invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 61, that is, realizing the above-mentioned method for identifying hepatic veins during liver surgery based on computer vision technology.

[0145] The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 61 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include a memory remotely arranged relative to the processor 60, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The input device 62 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 63 may include a display device such as a display screen.

[0147] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for identifying hepatic veins during liver surgery based on computer vision technology according to any embodiment is implemented.

[0148] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it.

[0149] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0150] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0151] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying hepatic veins during liver surgery based on computer vision technology, characterized in that: include: Acquire a liver intraoperative video stream to be identified; The following operations are performed on each video frame in the video stream in sequence: S1-1, using an improved YOLO model to perform target detection on the current video frame to frame the hepatic vein region in the current video frame; S1-2, using the UNet model to perform instance segmentation on the features of the hepatic vein region to obtain pixel-level segmentation boundaries of the hepatic vein structure; The improved YOLO model and the UNet model share a convolutional layer for feature extraction.

2. The method according to claim 1, characterized in that The improved YOLO model improves each CSP network in the backbone network into a CSPDarknet network, and each CSPDarknet network is used to extract low-level features and intermediate features of the input image, including the rough morphology of liver tissue and blood vessel areas; SiLU activation functions are added after the convolutional layers of each CSPDarknet network. Each SiLU activation function is used to perform nonlinear transformation on the features extracted by each CSPDarknet module to learn the image patterns of blurred blood vessels or overlapping and occluded multi-layer blood vessels during liver surgery.

3. The method according to claim 2, characterized in that Each CSPDarknet network adopts the CSP structure, which divides the features into two parts, one part undergoes more calculations, and the other part keeps less calculations. Finally, the two parts are merged to reduce the amount of calculations and adapt to the characteristics of frequent changes in liver morphology and vascular structure during surgery; Through residual connections, each CSPDarknet network can retain key image information at different levels.

4. The method according to claim 3, characterized in that The UNet model uses skip connections to transfer low-level spatial information to high-level feature extraction to enhance the learning ability of detailed vascular structures.

5. The method according to claim 4, characterized in that The improved YOLO model and the UNet model share a convolutional layer sliding window for extracting basic features of an image; Accordingly, the operations of S1-1 and S1-2 include: S2-1, using the same convolutional layer sliding window to extract pixel information of feature points in the current video frame; S2-2, inputting the pixel information into the convolution layer of the improved YOLO model and the convolution layer of the UNet model simultaneously for feature extraction; S2-3, moving the convolutional layer sliding window, and repeating the operations of S2-1 and S2-2 until all sliding window positions are calculated, and obtaining the basic features of the improved YOLO model and the UNet model respectively; S2-4, inputting the basic features into the subsequent structure of the improved YOLO model for target detection to frame the hepatic vein region in the current video frame; at the same time, inputting the basic features into the subsequent structure of the UNet model encoder to obtain the final encoding features; S2-5. The encoded features of the hepatic vein region are intercepted and input into the decoder of the UNet model to complete instance segmentation.

6. The method according to claim 4, characterized in that The improved YOLO model and the UNet model share the convolutional layer structure and parameters for extracting image features; Accordingly, the operations of S1-1 and S1-2 include: Use the shared convolution layer to perform convolution operation on the current video frame to obtain the shared features of the image; Inputting the shared features into a subsequent structure of the improved YOLO model for target detection to frame the hepatic vein region in the current video frame; The shared features of the hepatic vein region are input into the subsequent structure of the UNet model for instance segmentation.

7. The method according to claim 1, characterized in that Before sequentially performing the following operations on each video frame in the video stream, the method further includes: The intraoperative video streams of the liver used to construct the dataset were acquired by laparoscopic equipment; The video stream is annotated in stages, wherein the stages include before, during and after liver parenchyma separation, and the annotated contents include the spatial relationship and anatomical layers of the main blood vessels, small blood vessels and microvessels; Using the labeled data set, jointly training the improved YOLO model and the UNet model; During training, the labeled blood vessel bounding boxes are used to calculate the cross entropy loss function, which is used to optimize the bounding box position output by the improved YOLO model; The Dice loss function is calculated using the labeled image mask image, and the Dice loss function is used to optimize the segmentation accuracy of the UNet model.

8. The method according to claim 7, characterized in that After performing instance segmentation on the hepatic vein region using the UNet model to obtain the pixel-level segmentation boundary of the hepatic vein structure, the method further includes: Provide feedback based on actual surgical conditions; If the hepatic veins are identified incorrectly, feedback information is automatically incorporated into the data set, and the model parameters are dynamically updated using the new data set.

9. The method according to claim 8, characterized in that In the training stage, for the improved YOLO model: the anchor frame is initialized by K-means clustering, and the matching degree of the anchor frame is optimized according to the IoU value of the multi-scale target extracted by the CSPDarknet network during each training; Accordingly, in the inference stage, the improved YOLO model automatically selects an anchor box that matches the target according to the training results.

10. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method for identifying hepatic veins during liver surgery based on computer vision technology as described in any one of claims 1-9.

Images