A blood vessel distortion prediction method in a minimally invasive surgery video

Abstract

A method for predicting blood vessel course in minimally invasive surgical videos, relating to the field of image recognition technology, includes real-time acquisition of a minimally invasive surgical video stream, the video stream containing source frames showing exposed blood vessels and target frames showing blood vessels obscured by tissue; based on a temporal reverse annotation method, using the source frames and their blood vessel annotation information, generating blood vessel course labels for the target frames through temporal backtracking to construct a training dataset; constructing a deep learning neural network model, training the model using the training dataset, enabling the model to learn the ability to predict the course of obscured blood vessels from a single frame of obscured image; inputting the real-time minimally invasive surgical footage into the trained deep learning neural network model, outputting the predicted course information of the obscured blood vessels in the current image; this method addresses the problems of complex operation or inaccurate prediction results in traditional methods.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically to a method for predicting the course of blood vessels in minimally invasive surgical videos. Background Technology

[0002] During minimally invasive surgery, blood vessels are often obscured by fat, connective tissue, or other anatomical structures, preventing surgeons from directly observing their course. This blind spot increases the risk of accidental injury to blood vessels during surgery, potentially leading to serious complications such as massive bleeding, and affecting the safety and efficiency of the procedure. Therefore, accurately predicting the course of obscured blood vessels during surgery has become one of the key issues that urgently need to be addressed in the field of minimally invasive surgery.

[0003] Existing technologies explore methods for vascular detection and prediction from multiple directions, including: fluorescence imaging via contrast agent injection, tissue component identification using hyperspectral imaging, acquisition of deep vascular information using photoacoustic imaging, and augmented reality registration of preoperative images with intraoperative images. These methods have improved vascular recognition capabilities to some extent, but still have the following limitations:

[0004] a) Some techniques rely on contrast agent injection or specialized imaging equipment, which increases the complexity of preoperative preparation and intraoperative procedures and affects the smoothness of the surgical process;

[0005] b) Some methods require additional incisions or contact testing, increasing the risk of trauma to patients;

[0006] c) Existing methods generally do not make full use of the visual information available during the operation and fail to fully explore the implicit features such as temporal texture and tissue pulsation contained in the endoscopic video, making it difficult to achieve efficient and low-cost real-time vascular prediction.

[0007] Therefore, we propose a method that can predict the course of obscured blood vessels using intraoperative video information. Summary of the Invention

[0008] The purpose of this invention is to provide a method for predicting blood vessel course in minimally invasive surgical videos, which solves the problems of complex operation or inaccurate prediction results of traditional methods.

[0009] This invention is achieved through the following technical solution:

[0010] A method for predicting blood vessel course in minimally invasive surgical videos, specifically including:

[0011] Real-time acquisition of minimally invasive surgical video streams, the video streams including source frames of exposed blood vessels and target frames of blood vessels being obscured by tissue;

[0012] Based on the temporal reverse annotation method, the blood vessel course labels of the target frame are generated by temporal backtracking using the source frame and its blood vessel annotation information, and a training dataset is constructed.

[0013] Construct a deep learning neural network model and train the model using a training dataset to enable the model to learn the ability to predict the course of occluded blood vessels from a single frame of occluded image.

[0014] The real-time minimally invasive surgical footage is input into a trained deep learning neural network model, which outputs the predicted course of the obscured blood vessels in the current image.

[0015] Furthermore, the time-series reverse annotation method specifically includes:

[0016] Perform pixel-level segmentation on the blood vessels in the source frame to obtain a reference mask;

[0017] The unit displacement vector between adjacent frames in a video sequence is extracted using optical flow, and the cumulative dense displacement field from the exposed source frame to the target frame is calculated by temporal integration.

[0018] Based on the accumulated dense displacement field, the reference mask is gradually distorted in reverse time and projected back onto the target frame to generate the blood vessel course label for that frame.

[0019] Furthermore, the formula for calculating the reference mask is as follows:

[0020]

[0021] In the formula, In order to be exposed at the moment The acquired raw endoscope image frames, It is a pixel-level semantic segmentation operator. This is the generated blood vessel reference mask.

[0022] Furthermore, the formula for calculating the cumulative dense displacement field is as follows:

[0023]

[0024] in, For the target frame, It is a unit displacement vector.

[0025] Furthermore, based on the accumulated dense displacement field, the reference mask is gradually distorted against the time axis and projected backwards onto the target frame to generate the blood vessel course label for that frame. The calculation formula is as follows:

[0026]

[0027] In the formula, This is an image distortion operation.

[0028] Furthermore, in the training dataset, each sample consists of a target frame image and its corresponding blood vessel course label. The temporal reverse annotation method also includes an automatic verification mechanism, specifically:

[0029] Calculate the relative positional relationship between the backtracked blood vessel course labels in each sample and the known anatomical landmarks in the target frame;

[0030] If there is an overlapping area between the two, it is determined that a logical conflict has occurred, and the sample will be automatically removed.

[0031] If there is no overlap between the two, the verification is considered successful, and the sample and its blood vessel course label are retained and included in the training dataset.

[0032] Furthermore, the deep learning neural network model includes:

[0033] The backbone network is used to extract multi-scale features from the input image, and the backbone network adopts a convolutional neural network structure that includes cross-stage local networks and attention mechanisms.

[0034] The neck network is used to construct a feature pyramid, fusing deep semantic information with shallow detail texture information;

[0035] The prediction head includes a classification branch, a regression branch, and a segmentation branch, which are used for pixel-level vessel classification, vessel bounding box regression, and pixel-level mask generation for vessel course, respectively.

[0036] Furthermore, the attention mechanism is an efficient channel attention module that captures cross-channel interaction information through one-dimensional convolution, enhancing the sensitivity to extremely low contrast areas and the subtle periodic pulsations and micro-color differences in light and shadow on the tissue surface.

[0037] Furthermore, the training process using the training dataset employs a composite loss function, including location regression loss, classification loss, and segmentation loss, which are used to supervise the geometric deviation of the blood vessel bounding box, the classification accuracy of blood vessels and non-blood vessels, and the pixel-level segmentation accuracy of the blood vessel contour, respectively.

[0038] Furthermore, the formula for calculating the composite loss function is as follows:

[0039]

[0040] In the formula, For position regression loss, For classifying losses, To divide the loss, , and These are the weights for different loss functions.

[0041] The technical solution of the present invention has at least the following advantages and beneficial effects:

[0042] This invention discloses a method for predicting blood vessel course in minimally invasive surgical videos. It uses only the existing endoscopic video stream during the minimally invasive surgery as the data source, without the need for contrast agents, special imaging equipment such as hyperspectral or photoacoustic imaging, or additional incisions or contact detection. Furthermore, the training phase utilizes the temporal context of the video itself to generate labels, and the inference phase only requires a single frame to complete the prediction, significantly simplifying the surgical procedure and avoiding the operational complexity and patient trauma risks caused by external intervention.

[0043] In addition, by using the temporal reverse annotation method, the clear shape of the blood vessel at the moment of exposure is projected back to the moment of occlusion, thus constructing a deep correlation model of "the surface texture of the occluded tissue - the deep vascular pulsation - the anatomical topological rules". Furthermore, the neural network can learn the implicit features that are difficult for the human eye to capture, such as the weak periodic pulsation of the fat surface and the slight color difference of light and shadow. Therefore, it can accurately predict the course of blood vessels occluded by fat and connective tissue without relying on external information, thus making full use of intraoperative visual information.

[0044] Furthermore, by introducing an automatic verification mechanism, logical conflicts are detected between the backtracked blood vessel course labels and known anatomical landmarks in the target frame, and erroneous samples are automatically removed, ensuring the purity and accuracy of the training dataset and providing a high-quality data foundation for model training. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the process for predicting blood vessel course in a minimally invasive surgical video according to the present invention;

[0046] Figure 2 This is a schematic diagram of the deep learning neural network model structure of the present invention;

[0047] Figure 3 This is a schematic diagram of the structure of a blood vessel course prediction system in a minimally invasive surgical video according to the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0049] Example 1

[0050] like Figure 1The method for predicting blood vessel course in a minimally invasive surgical video, as shown, specifically includes:

[0051] Real-time acquisition of minimally invasive surgical video streams, the video streams including source frames of exposed blood vessels and target frames of blood vessels being obscured by tissue;

[0052] The video stream for minimally invasive surgery is formatted with a resolution of 1920×1080 and a frame rate of 60fps. This resolution ensures that subtle latent features such as the weak periodic pulsations of the fat surface and minor color differences in light and shadow can be effectively sampled, avoiding information loss due to insufficient resolution. After downsampling, the high-resolution image can still retain the local details required by the superficial network (such as the slight bulges at the edges of blood vessels) and provide sufficient semantic information for the deep network to support the feature pyramid fusion of the neck network. In addition, the temporal reverse annotation method (RTL) relies on optical flow methods (such as RAFT) to calculate the unit displacement vector between adjacent frames. The high frame rate of 60fps results in smaller tissue deformation amplitude between adjacent frames, lower optical flow estimation error, and more accurate calculation of the cumulative displacement field. Furthermore, under the typical motion speed of surgical operations (such as tissue detachment and instrument movement), 60fps can ensure that the displacement between each frame does not exceed the pixel-level trackable range, avoiding motion blur or tracking loss caused by excessively low frame rates.

[0053] In addition, the 60fps input-output matching enables the entire inference pipeline (preprocessing → backbone network → neck network → prediction head) to run with a single-frame latency of <16.7ms. Combined with TensorRT quantization optimization, it meets the physiological requirements for real-time overlay display during surgery.

[0054] Based on the temporal reverse annotation method, the blood vessel course labels of the target frame are generated by temporal backtracking using the source frame and its blood vessel annotation information, and a training dataset is constructed.

[0055] Specifically, the time-series reverse annotation method includes:

[0056] Pixel-level segmentation of blood vessels in the source frame is performed to obtain a reference mask. The calculation formula is as follows:

[0057]

[0058] In the formula, In order to be exposed at the moment The acquired raw endoscope image frames, Pixel-level semantic segmentation operators, such as manually labeled + pre-trained segmentation models. This step generates a vascular reference mask; it is used to establish the "gold standard" anatomical truth at the moment when the vascular morphology is clearest.

[0059] The Recurrent All-Pairs Field Transforms (RAFT) method is used to extract unit displacement vectors between adjacent frames in a video sequence, and the cumulative dense displacement field from the exposed source frame to the target frame is calculated through temporal integration. The calculation formula is:

[0060]

[0061] in, For the target frame, This is a unit displacement vector; this step is used to quantify the continuous deformation process of the tissue from the moment of exposure to the moment of occlusion.

[0062] Based on the accumulated dense displacement field, the reference mask is progressively distorted against the time axis and projected backwards onto the target frame to generate the blood vessel course label for that frame. The calculation formula is as follows:

[0063]

[0064] In the formula, The image distortion operation is mathematically essentially a coordinate remapping process. This step is used to "upside down" the blood vessel morphology at the time of exposure and paste it back into the image at the time of occlusion.

[0065] This method first obtains a reference mask through pixel-level semantic segmentation on the source frame where the anatomical structure of the blood vessel is most clearly defined. Then, the RAFT optical flow algorithm is used to extract unit displacement vectors between adjacent frames, and these vectors are accumulated through temporal integration to form a dense displacement field from the exposure moment to the occlusion moment. Finally, image distortion and coordinate remapping are performed on the reference mask based on this displacement field to achieve accurate back projection. This process transforms the forward process of "tissue dissection exposing blood vessels" in clinical practice into a mathematically reversible transformation, allowing anatomical knowledge from future moments to be transferred to past moments, providing previously unavailable supervisory signals for model training.

[0066] Furthermore, the temporal reverse annotation method utilizes the temporal context of the video during the training phase—specifically, the vascular morphology of exposed frames—as privileged information to generate labels. This allows the model to predict the course of deep blood vessels during the inference phase based solely on a single occluded frame. This "privileged during training, unprivileged during inference" design compresses the implicit temporal dimension information in the video into the network weights, achieving a mapping learning from surface visual features to deep anatomical structures. More importantly, this method constructs a three-in-one correlation model of "occluded tissue surface texture—deep blood vessel pulsation—anatomical topological patterns": by forcing the network to predict blood vessel labels inferred from the real displacement field on occluded images, the model is forced to learn to recognize clues that are difficult for the human eye to detect, such as the weak periodic pulsation of the fat surface, the subtle color difference caused by the movement of light and shadow with blood vessels, and the texture features of tissue gaps. It then establishes a causal relationship between these implicit features and the geometric course of blood vessels. This learning mechanism avoids the shallow pattern recognition of traditional image segmentation that relies solely on color and shape, enabling the model to truly "understand" the behavior patterns of blood vessels when they are occluded.

[0067] Furthermore, the temporal reverse annotation method achieves a qualitative leap in data augmentation—it doesn't simply increase the number of samples within the same data distribution, but rather injects anatomical information from future moments into each occluded image, enabling the model to learn deeper knowledge beyond the visual information of the current frame. Simultaneously, this method establishes a channel for transforming clinical experience into model knowledge: the implicit experience of senior doctors in judging blood vessel locations through touch and intuition is implicitly encoded into the network weights through a large number of labels deduced from real anatomical morphology, providing younger doctors with "X-ray"-like assistance. Moreover, the physical realism of the displacement field ensures that the generated labels have anatomical accuracy and temporal consistency, forming a close synergy with subsequent attention mechanisms, composite loss functions, automatic verification, and other modules, jointly supporting the system's accurate predictive capabilities.

[0068] Furthermore, in the training dataset, each sample consists of a target frame image and its corresponding blood vessel course label. The temporal reverse annotation method also includes an automatic verification mechanism, specifically:

[0069] Calculate the relative positional relationship between the backtracked blood vessel routing labels in each sample and known anatomical landmarks in the target frame, such as the gallbladder outline and liver edge;

[0070] If there is an overlapping area between the two, it is determined that a logical conflict has occurred. For example, if the blood vessel label is inside the liver parenchyma, the sample will be automatically removed.

[0071] If there is no overlap between the two, the verification is considered successful. The sample and its blood vessel course label are then retained and included in the training dataset to ensure the key quality of the training dataset in terms of purity and anatomical rationality.

[0072] The calculation formula involved in this step is as follows:

[0073]

[0074] For the blood vessel label dataset, A mask for known anatomical landmarks in the target frame. This is the area calculation function. It is an empty set;

[0075] In minimally invasive surgery, blood vessels follow specific anatomical patterns. For example, major blood vessels such as the hepatic artery and portal vein are distributed in specific areas such as the hepatoduodenal ligament and do not pass through the liver parenchyma. Although the temporal reverse labeling method achieves accurate reverse projection based on physical motion (optical flow), factors such as cumulative displacement field error and tissue deformation estimation bias may still lead to unreasonable anatomical phenomena such as "pattern crossing" in the generated labels. Therefore, an automatic verification mechanism can use these anatomical constraints as a "litmus test" to quickly identify and filter out such erroneous samples, preventing the model from learning false associations that violate physiological laws.

[0076] Furthermore, unlike traditional data cleaning methods that rely on inefficient manual review, this mechanism operates fully automatically, requiring no additional manpower. By eliminating logically conflicting samples, the training dataset retains only "gold standard" labels that conform to anatomical patterns, fundamentally ensuring the correctness of the model's learning direction. Experiments show that after introducing this mechanism, the spatial consistency between the model's predicted blood vessel course and the actual anatomical structure is significantly improved in subsequent inferences, thereby enhancing the accuracy of the model's predictions.

[0077] Secondly, the temporal reverse annotation method relies on the optical flow displacement field, which is a modeling at the physical motion level; while the automatic verification mechanism introduces anatomical landmarks, which is a constraint at the prior knowledge level. The combination of the two forms a dual guarantee: the former ensures that the labels are aligned with the tissue deformation in the image, and the latter ensures that the labels conform to anatomical rationality. This multimodal verification approach enhances the robustness of the entire data production process. Even if optical flow estimation drifts in certain scenarios, anatomical constraints can promptly intercept erroneous samples.

[0078] Subsequently, although the rejected samples are not used for training, their "logical conflicts" themselves contain important pedagogical significance. These samples often correspond to rare cases such as failure of displacement field estimation, abnormal tissue deformation, or anatomical variations. Although they cannot be used as positive examples, the model avoids being misled during training by "not seeing" these errors. At the same time, this mechanism also lays the foundation for the "anomaly detection" function that may be introduced in the future.

[0079] Ultimately, this mechanism directly serves the temporal reverse labeling method, providing quality assurance for its generated labels; it also complements the subsequent composite loss function—only high-quality label input can truly optimize the model's predictive ability. Furthermore, the existence of the automatic verification mechanism makes the entire data construction process interpretable: each retained sample can be traced back to which anatomical constraints it passed, enhancing the credibility of the technical solution in clinical application.

[0080] Construct a deep learning neural network model and train the model using a training dataset to enable the model to learn the ability to predict the course of occluded blood vessels from a single frame of occluded image.

[0081] like Figure 2 As shown, this model is an improvement on the YOLOv11 architecture, specifically designed to enhance the ability to capture subtle textures. Its basic working principle is as follows: the input image enters through the input terminal and undergoes a series of image preprocessing steps, including scaling, image equalization, normalization, and texture enhancement, before entering the backbone network. The backbone network consists of multiple layers of convolutional neural networks. Through deliberate design, the image size is halved and the dimension doubled with each convolution calculation. Multiple convolution calculations transform the texture and structural information of the image into dimensionality; this process is called feature extraction. The high-dimensional images (or feature maps) after feature extraction at different layers enter the neck network and are fused through upsampling and downsampling. The aim is to couple the shallow feature maps containing local details with the deep feature maps containing semantic information, further enhancing the expressive power of the final model. Finally, the coupled feature maps at different scales are output through the detection head at the output terminal, providing predicted bounding boxes, categories, and masks.

[0082] The input preprocessing involves: inputting a single-frame RGB image, scaling it to 640×640. CLAHE (Contrast-Limited Adaptive Histogram Equalization) is applied to enhance the contrast between highlights and shadows on the fat surface, highlighting subtle texture differences caused by subcutaneous blood vessel bulges. The backbone network uses the C3k2 module as the main body, extracting features through cross-stage local networks. An ECA-Net (Efficient Channel Attention) module is embedded after each residual block of the C3k2 module. The ECA module does not reduce dimensionality; it captures cross-channel interaction information through one-dimensional convolution, forcing the network to focus on features other than "color" (such as changes in gloss and subtle changes in surface curvature).

[0083] In addition, the attention mechanism is an efficient channel attention module that captures cross-channel interaction information through one-dimensional convolution, enhancing the sensitivity to extremely low contrast areas and the subtle periodic pulsations and light and shadow color differences on the tissue surface.

[0084] The neck network constructs a Bidirectional Feature Pyramid Network (BiFPN) to bidirectionally fuse deep semantic information (the general direction of blood vessels) with superficial detailed information (the texture of the fat surface). This allows the model to predict both major blood vessels (coarse-grained) and small branch vessels (fine-grained).

[0085] The prediction head includes a classification branch, a regression branch, and a segmentation branch. The classification branch uses a series of 3×3 convolutions to output an N-dimensional feature vector for pixel-level blood vessel / background classification prediction. The regression branch predicts the bounding boxes of blood vessels by performing independent convolution operations, ultimately outputting a 4-dimensional feature vector for predicting the precise bounding box location of blood vessels. The segmentation branch outputs a full-resolution binary mask for fine-grained display of blood vessel morphology.

[0086] It is important to note that Figure 2 The entire model is divided into three layers: the backbone network, the neck network, and the prediction head. The meanings of each module in the figure are as follows: Convolution-Normalization-Activation Unit (CBS); Multi-scale Residual Feature Extraction Module (C3k2); Efficient Channel Attention Module (ECA); Spatial Pyramid Pooling Module (SPPF); Dual-Path Attention Enhancement Module (C2PSA); Upsampling Unit (Up-Sample); Feature Fusion Node (Concat).

[0087] Furthermore, the training process using the training dataset employs a composite loss function, including location regression loss, classification loss, and segmentation loss, which are used to supervise the geometric deviation of the blood vessel bounding box, the classification accuracy of blood vessels and non-blood vessels, and the pixel-level segmentation accuracy of the blood vessel contour, respectively.

[0088] The formula for calculating the composite loss function is as follows:

[0089]

[0090] In the formula, For position regression loss, For classifying losses, To divide the loss, , and These are the weights for different loss functions. Since the numerical ranges of different losses are different (e.g., Dice loss is between 0 and 1, and cross-entropy may be larger), the weights need to be adjusted to make their contributions comparable.

[0091] In particular, to prevent the model from rote memorization and to simulate the real surgical environment, semi-transparent Perlin noise can be superimposed on the image to simulate electrosurgical smoke, and random highlight areas can be added to simulate shadowless lamp reflections; this forces the model to make inferences by comprehensively utilizing anatomical structures and textures instead of relying on a single visual feature.

[0092] The real-time minimally invasive surgical footage is input into a trained deep learning neural network model, which outputs the predicted course of the obscured blood vessels in the current image.

[0093] Example 2

[0094] To ensure zero latency during the surgical procedure, this embodiment employs the following technical means to optimize inference efficiency:

[0095] TensorRT is used to quantize the trained model using FP16 or INT8, and the model is deployed on an embedded computing platform (such as NVIDIA Jetson or a dedicated medical workstation) in the operating room to ensure that the single-frame processing latency is controlled within 15ms, meeting the real-time screen synchronization requirements of 60fps.

[0096] Furthermore, the video stream acquisition, model inference stream, and rendering display stream are decoupled and asynchronously processed to achieve multi-threaded synchronous processing, avoiding inference fluctuations that could cause stuttering in the endoscopic view and ensuring smooth visual feedback for doctors.

[0097] In addition, in deep learning segmentation models, the probability that each pixel belongs to the blood vessel category, i.e., the confidence score, is calculated using the Softmax function. It reflects the model's judgment on the signal-to-noise ratio of the input signal: the stronger and clearer the signal, the higher the confidence score; the weaker and blurrier the signal, the lower the confidence score.

[0098] By setting an adjustable confidence threshold (Default 0.5); The system only renders values ​​with a confidence level greater than 0.5. In areas with a confidence level of less than [a certain value], the specific rendering method involves outlining the blood vessel contours with solid lines or filling the blood vessel area with a semi-transparent solid-color overlay. The area is displayed using dashed lines or light-colored point clouds to indicate to the doctor that there may be anatomical variations or excessive occlusion in this area, requiring careful operation.

[0099] Furthermore, the original output of the segmentation branch is a pixel-level probability map, representing the confidence level of each pixel belonging to the blood vessel category. In practical applications, this probability map can be used for two different visualization purposes:

[0100] 1) A binary mask is generated through thresholding to accurately depict the geometry of blood vessels;

[0101] 2) Directly used as the basis for depth mapping, the probability value is linearly mapped to the color space, and different color depths represent the coverage depth of blood vessels relative to the tissue surface;

[0102] Therefore, by mapping the mask probability value output by the segmentation branch to "coverage depth", the darker the color (such as dark red) in the coverage depth, the closer the blood vessel is to the tissue surface and the higher the risk; the lighter the color (such as translucent pink) represents the deeper the blood vessel is buried, providing doctors with an intuitive reference for the anatomical layers.

[0103] As needed, an intraoperative risk real-time early warning mechanism can be added. When a large-scale tissue displacement or bleeding is detected that causes severe obstruction, the system will automatically switch to "anatomical prior mode" to provide reference patterns based on the topological rules learned by RTL, rather than blindly tracking visual signals.

[0104] In addition, automatic correction capabilities can be added to address common interference factors during surgery:

[0105] By leveraging the attention-enhancing mechanism in the backbone network, the system can maintain stable output of vascular alignment by capturing the weak periodic pulsations on the tissue surface, even when the contrast is reduced due to the smoke generated by the electrosurgical unit.

[0106] Alternatively, when the camera is obscured by blood or temporarily moved away and then re-enters the surgical area, global feature matching (GFM) can be used to quickly retrieve the anatomical location and restore the vascular prediction overlay within 0.2 seconds.

[0107] In addition, to address the issue of predicted bounding box "drifting" or "flickering" caused by tissue deformation, smoke obscuring, or camera movement during surgery, a target tracking mechanism based on deep features is integrated, specifically:

[0108] The coordinates of the vessel centerline output by the detection head are processed by Kalman filtering. By establishing a constant velocity or constant acceleration motion model, the position of the vessel in the next frame is predicted, so that logical tracking can still be maintained for 5-10 frames even when visual features are temporarily missing (such as when the electrosurgical smoke completely obscures the image), ensuring the smooth display of the AR enhancement layer.

[0109] In the real-time inference phase, a linear motion model is established for each tracked blood vessel segment in order to smooth the coordinates of the vessel centerline and cope with instantaneous occlusion.

[0110] The linear motion model includes the definition of the state vector:

[0111] Let the first The state vector of the center point of the blood vessel target at frame time :

[0112]

[0113] in, The pixel coordinates of the blood vessel center. The area of ​​the bounding box. Aspect ratio, These are the corresponding instantaneous rates of change (velocities). This is the transpose symbol.

[0114] Prediction phase: Using the state estimate from the previous time step. Predict the prior state value at the current moment. And error covariance matrix :

[0115]

[0116]

[0117] in, Here is the state transition matrix. For the system process noise covariance, For the first The error covariance matrix at time t.

[0118] Update phase:

[0119] When the YOLOv11 prediction head obtains new observations in the current frame When calculating the Kalman gain (i.e., the location of the blood vessel detected in the current frame),... And correct the state vector:

[0120]

[0121]

[0122]

[0123] in, For the observation matrix, To measure the noise covariance, To predict residuals, It is the identity matrix. For the first Error covariance matrix at time step. Occlusion handling logic: When the detection head cannot output due to smoke, bleeding, or tissue covering the blood vessel. When an observation becomes inaccurate, the system will stop the update phase and only execute the prediction phase. This is achieved by analyzing the instantaneous rate of change at historical moments. The velocity value estimated in the previous frame is used to maintain the dynamic position of the blood vessel's course until the detection head reacquires valid visual features, thereby achieving a smooth tracking effect.

[0124] Then, using the Appearance Embedding extracted from the YOLOv11 backbone network, the system assigns a unique tracking ID to each major blood vessel segment within the field of view. When multiple blood vessels (such as arteries and veins) intersect or are close to each other, the system performs data association by calculating the cosine similarity of the feature vectors to prevent the predicted labels from jumping between different blood vessels.

[0125] To address the non-rigid deformation of tissues during surgery, a key point tracking operator is introduced. This operator extracts anatomical key points such as vascular branches and sharp bends, and combines them with the dense displacement field calculated in previous steps. This allows the mask to deform synchronously with tissue stretching, ensuring that the predicted vascular course always closely follows the anatomical location.

[0126] The moment a blood vessel is re-exposed after being completely obscured by the instrument, the system uses the Re-ID module to compare the current features with the cached "prior feature pool". If a match is found, the system immediately restores the historical morphology parameters of that ID, avoiding the perception delay caused by re-initialization and achieving "resuming tracking from the breakpoint".

[0127] Ultimately, by having the tracking and detection modules work in parallel, the system ensures that the update frequency of blood vessel alignment is synchronized with the surgical procedure at 60fps video streaming, with no visible lag. Furthermore, doctors can lock onto specific "focused blood vessels" via voice or foot switch, and the system will enhance tracking of that specific ID with a highlighted border and automatically increase alarm sensitivity when instruments approach the tracked target.

[0128] Example 3

[0129] like Figure 3 The system shown is a blood vessel course prediction system in a minimally invasive surgical video, specifically including:

[0130] The video stream acquisition module is used to acquire minimally invasive surgical video streams in real time.

[0131] The data annotation module, based on the temporal reverse annotation method, uses the source frame and its blood vessel annotation information to generate blood vessel course labels for the target frame through temporal backtracking, and constructs a training dataset;

[0132] The model building module is used to build deep learning neural network models. The model is trained using a training dataset, enabling it to learn the ability to predict the course of occluded blood vessels from a single frame of occluded image.

[0133] The prediction output module is used to input real-time minimally invasive surgical images into a trained deep learning neural network model and output the predicted course information of obscured blood vessels in the current image.

[0134] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for predicting blood vessel course in minimally invasive surgical videos, characterized in that, Specifically, it includes: Real-time acquisition of minimally invasive surgical video streams, the video streams including source frames of exposed blood vessels and target frames of blood vessels being obscured by tissue; Based on the temporal reverse annotation method, the blood vessel course labels of the target frame are generated by temporal backtracking using the source frame and its blood vessel annotation information, and a training dataset is constructed. The time-series reverse annotation method specifically includes: Perform pixel-level segmentation on the blood vessels in the source frame to obtain a reference mask; The unit displacement vector between adjacent frames in a video sequence is extracted using optical flow, and the cumulative dense displacement field from the exposed source frame to the target frame is calculated by temporal integration. Based on the accumulated dense displacement field, the reference mask is gradually distorted in reverse time and projected back onto the target frame to generate the blood vessel course label for that frame. Construct a deep learning neural network model and train the model using a training dataset to enable the model to learn the ability to predict the course of occluded blood vessels from a single frame of occluded image. The real-time minimally invasive surgical footage is input into a trained deep learning neural network model, which outputs the predicted course of the obscured blood vessels in the current image.

2. The method for predicting blood vessel course in minimally invasive surgical videos according to claim 1, characterized in that: The formula for calculating the reference mask is: In the formula, In order to be exposed at the moment The acquired raw endoscope image frames, It is a pixel-level semantic segmentation operator. This is the generated blood vessel reference mask.

3. The method for predicting blood vessel course in minimally invasive surgical videos according to claim 2, characterized in that: The formula for calculating the cumulative dense displacement field is: in, For the target frame, It is a unit displacement vector.

4. The method for predicting blood vessel course in minimally invasive surgical videos according to claim 3, characterized in that: The process involves progressively twisting the reference mask against the time axis based on the accumulated dense displacement field, then projecting it backwards onto the target frame to generate a blood vessel course label for that frame. The calculation formula is as follows: In the formula, This is an image distortion operation.

5. The method for predicting blood vessel course in minimally invasive surgical videos according to claim 1, characterized in that: In the training dataset, each sample consists of a target frame image and its corresponding blood vessel course label. The temporal reverse annotation method also includes an automatic verification mechanism, specifically: Calculate the relative positional relationship between the backtracked blood vessel course labels in each sample and the known anatomical landmarks in the target frame; If there is an overlapping area between the two, it is determined that a logical conflict has occurred, and the sample will be automatically removed. If there is no overlap between the two, the verification is considered successful, and the sample and its blood vessel course label are retained and included in the training dataset.

6. The method for predicting blood vessel course in minimally invasive surgical videos according to claim 1, characterized in that: The deep learning neural network model includes: The backbone network is used to extract multi-scale features from the input image, and the backbone network adopts a convolutional neural network structure that includes cross-stage local networks and attention mechanisms. The neck network is used to construct a feature pyramid, fusing deep semantic information with shallow detail texture information; The prediction head includes a classification branch, a regression branch, and a segmentation branch, which are used for pixel-level vessel classification, vessel bounding box regression, and pixel-level mask generation for vessel course, respectively.

7. The method for predicting blood vessel course in minimally invasive surgical videos according to claim 6, characterized in that: The attention mechanism is a high-efficiency channel attention module that captures cross-channel interaction information through one-dimensional convolution, enhancing the sensitivity to weak periodic pulsations and subtle color differences in light and shadow in extremely low contrast areas and tissue surfaces.

8. The method for predicting blood vessel course in minimally invasive surgical videos according to claim 1, characterized in that: The training process using the training dataset employs a composite loss function, including location regression loss, classification loss, and segmentation loss, which are used to supervise the geometric deviation of the blood vessel bounding box, the classification accuracy of blood vessels and non-blood vessels, and the pixel-level segmentation accuracy of the blood vessel contour, respectively.

9. The method for predicting blood vessel course in minimally invasive surgical videos according to claim 8, characterized in that: The formula for calculating the composite loss function is as follows: In the formula, For position regression loss, For classifying losses, To divide the loss, , and These are the weights for different loss functions.

Images