Vascular anastomosis skill automatic evaluation method based on computer vision technology

Abstract

The invention provides a vascular anastomosis skill automatic evaluation method based on a computer vision technology, and relates to the technical field of video processing, and the method comprises the steps: training a visual feature extraction model through the unmarked vascular anastomosis operation video data, and obtaining a pre-trained visual feature extraction model; performing feature extraction on the vascular anastomosis video data to obtain vascular anastomosis skill visual features; performing up-sampling on the vascular anastomosis skill visual features by using a high-resolution pyramid to obtain a high-spatial-resolution feature map; performing operation action category recognition, target segmentation and tip position recognition on the high-spatial-resolution feature map by using multiple downstream task models; calculating action time consumption by using operation action categories, obtaining a movement track by using the tip position of the instrument, identifying a suture binary mask by using target segmentation, calculating a multi-dimensional index, and obtaining an automatic assessment result of the vascular anastomosis skill. The problem that the small target recognition precision is difficult to improve in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of video processing, in particular to a blood vessel anastomosis skill automatic evaluation method based on computer vision technology. BACKGROUND

[0002] Blood vessel anastomosis is a core surgical technique in the fields of cardiovascular surgery, microsurgery, etc., which requires the operator to have high operation accuracy and stability. The traditional training method mainly relies on expert manual evaluation, which has strong subjectivity, non-uniform evaluation standard, difficulty in quantification, low training efficiency, etc. In recent years, offline training systems such as the low-cost plastic basin anastomosis simulation system developed by Japan evaluate skills by quantifying operation time, needle pitch and bite coefficient of variation (CV), but still require manual measurement and scoring, and cannot realize real-time feedback. In addition, some studies use Mask R-CNN model to detect and segment blood vessels and sutures in blood vessel anastomosis image <Quantitative Outcome-Oriented Assessment of Microsurgical Anastomosis>, but it can only perform offline, single-dimensional "result evaluation" on postoperative static photos, and some patents are related to blood vessel anastomosis devices or auxiliary devices, but less attention is paid to visual-based automatic training evaluation. Although there are video object segmentation methods and video action recognition methods based on self-supervised learning, and technologies for analyzing surgical instrument motion trajectories, there is still no systematic integration of these technologies, and no specific optimization for blood vessel anastomosis training scenarios, especially the lack of region locking strategies centered on the recipient blood vessel to improve small target recognition accuracy.

[0003] The current evaluation techniques for vascular anastomosis skill training have many limitations, which are difficult to meet the requirements of modern surgical education for objectivity, real-time and comprehensiveness, mainly in the following aspects: strong subjectivity of evaluation, lack of quantitative standard: traditional evaluation highly depends on on-site observation and subjective scoring of experienced doctors (such as OPRS scoring system), different evaluators have different standards, the results are easily affected by personal experience and preference, lack of objective and unified quantitative indicators, difficult to make horizontal comparison and longitudinal tracking. The evaluation is lagging behind, and it is impossible to provide real-time feedback: although the mainstream offline training system (such as the system described in PMC6326066) introduces quantitative indicators such as operation time, needle distance and bite edge coefficient of variation, it still needs to be measured and calculated manually after the training is completed, which cannot provide immediate and dynamic feedback during the operation of the student, greatly reducing the learning efficiency. The evaluation dimension is single and not comprehensive: some automation attempts, such as using Mask R-CNN model to analyze static images, can only conduct "result-oriented" single-dimensional evaluation on the final suture shape. They cannot capture and analyze key information in the operation process, such as the movement trajectory of the instrument and the smoothness of the operation, which are crucial for the overall improvement of skills. The integration of technology is low, and there is a lack of targeted optimization: although there are advanced technologies such as video action recognition, self-supervised learning and instrument trajectory tracking in the field of computer vision, these technologies have not been systematically integrated and optimized specifically for the "vascular anastomosis" surgical scene with high precision requirements. In particular, there is a lack of dynamic region locking mechanism centered on the recipient blood vessel, which leads to insufficient recognition accuracy of small key targets such as needle tip and suture in complex background and fast operation, making it difficult to support high-precision geometric parameter (such as needle entry angle) calculation. Strong data dependence, high cost: many deep learning-based solutions require a large amount of accurately labeled video or image data for supervised training, and obtaining high-quality surgical operation labeled data is extremely costly and time-consuming, which seriously hinders the development, iteration and application of the model in different institutions. SUMMARY

[0004] In view of the above deficiencies in the prior art, the blood vessel anastomosis skill automatic evaluation method based on computer vision technology provided by the present application solves the problem of difficult improvement of small target recognition accuracy in the prior art through the "two-stage adaptive target region locking" strategy and the multi-task cooperative mechanism.

[0005] In order to achieve the above invention purpose, the technical scheme adopted by the present application is: a blood vessel anastomosis skill automatic evaluation method based on computer vision technology, S1: using unlabeled vascular anastomosis surgery video data, pre-training a visual feature extraction model using a self-supervised learning method to obtain a pre-trained visual feature extraction model; S2: performing feature extraction on the anastomosis operation video data by using a pre-trained visual feature extraction model to obtain anastomosis skill visual features; S3: performing up-sampling on the anastomosis skill visual features by using a high-resolution pyramid to obtain a high spatial resolution feature map; S4: performing action recognition, target segmentation and tip position recognition on the high spatial resolution feature map by using a plurality of downstream task models to obtain a current frame action class probability, a binary mask and a tip position probability heat map, and taking the maximum probability class in the current frame action class probability as an operation action class; wherein the plurality of downstream task models are obtained by joint training based on the current frame action class probability, the binary mask and the tip position probability heat map; S5: based on the operation action class, calculating an action time consumption; based on the binary mask and the tip position probability heat map, analyzing the instrument tip position to calculate a motion trajectory and a suture binary mask, calculating a multi-dimensional index, and obtaining an anastomosis skill automatic evaluation result, to complete the anastomosis skill automatic evaluation.

[0006] The present application provides an anastomosis skill automatic evaluation method based on computer vision technology, which realizes end-to-end, multi-dimensional and real-time automatic evaluation of operation action flow, suture geometric parameters and instrument motion trajectory through self-supervised pre-training, two-stage adaptive target region locking, multi-task collaborative neural network architecture and customized loss function system based on clinical prior knowledge. (1) Key point heat map regression is used instead of boundary box detection to realize sub-pixel level positioning and naturally support multi-instrument tip identification. (2) According to the action recognition result, a spatial attention map centered on the blood vessel edge is dynamically generated to modulate the HR-FPN feature, realize semantic alignment between tasks, and effectively suppress false positives. (3) Through the needle distance variance and edge deviation term, the model output is guided to conform to the clinical standard of suture point distribution. Using surgical rules such as "the needle tip must be close to the blood vessel when the needle is inserted", a differentiable supervision signal is constructed to improve the anatomical rationality of the detection result. (4) Taking the recipient blood vessel as the center, a fixed size ROI is dynamically locked for all downstream task inputs, which significantly improves the signal-to-noise ratio of small targets, and further reduces the ROI according to the action recognition result to realize fine calculation of high-precision geometric parameters such as needle insertion / needle extraction angle.

[0007] Further, the S4 comprises: performing action recognition on the anastomosis skill visual features by using an action recognition branch of the downstream task model to obtain a current frame action class probability; performing target segmentation on the high spatial resolution feature map by using a target segmentation branch of the downstream task model to obtain a binary mask; performing cross-task attention fusion on the current frame action class probability to obtain a dynamically enhanced attention map; Based on the dynamic enhanced attention graph, an instrument head detection sub-model of a downstream task model is used to identify the tip position through key point heat map regression to obtain a probability heat map of the tip position.

[0008] Further, the expression of the loss function of the downstream task model is: ; ; ; ; ; ; wherein, denotes the total loss, denotes the action recognition loss, denotes the target segmentation loss, denotes the instrument head detection loss, denotes the geometric constraint loss, denotes the cross-task consistency loss, denotes the weight of the action recognition loss, denotes the weight of the target segmentation loss, denotes the weight of the instrument head detection loss, denotes the weight of the geometric constraint loss, denotes the weight of the cross-task consistency loss, denotes, denotes the real action class, denotes the action class output by the downstream task model, denotes the loss of the prediction and the real area overlap, denotes the real label mask, denotes the mask predicted by the downstream task model, denotes the classification error of each pixel point, denotes the number of samples in the batch, i denotes the i-th sample in the batch, denotes the real heat map of the i-th sample, denotes the predicted heat map of the i-th sample, denotes the square of the L2 norm, denotes the variance of the calculation set, denotes the predicted coordinates of the j-th seam point, denotes the predicted coordinates of the j+1-th seam point, denotes the Euclidean distance, denotes the mean value of the set, denotes the Euclidean distance, represents a blood vessel anastomosis edge, represents an ideal margin value, represents an indicator function, represents the first action category prediction result of the frame, represents a ReLU function, represents the predicted coordinates of the needle tip, represents the point on the blood vessel edge closest to the needle tip, represents a tolerance threshold.

[0009] Further, the action recognition branch using the downstream task model performs action recognition on the vascular anastomosis skill visual features to obtain the current frame action category probability, which includes: The pre-trained visual feature extraction model is used to analyze the vascular anastomosis surgery video data to obtain the vascular anastomosis skill visual features; The action recognition branch using the downstream task model performs action recognition on the vascular anastomosis skill visual features to locate the center coordinates of the recipient blood vessel to obtain the recipient blood vessel positioning result; Based on the recipient blood vessel positioning result, the region is expanded and locked to obtain the locked region; The locked region is dynamically tracked and the operation action recognition is performed to obtain the current frame action category probability.

[0010] The recognized operation action category is used to adaptively lock the ROI region to improve the accuracy of the downstream small target recognition subtask.

[0011] Further, the S5 includes: Based on the operation action category, the action time consumption is calculated; Based on the binary mask and the probability heat map of the tip position, the instrument tip position is analyzed to calculate the motion trajectory and the suture binary mask, and the operation action category is calculated to calculate the instrument tip motion trajectory, the trajectory parameters, and the suture geometric parameters, respectively, to obtain the action score, the motion score, and the geometric score; Based on the action score, the motion score, and the geometric score, the comprehensive quality evaluation index is calculated by weighting to obtain the vascular anastomosis skill automatic evaluation result, and the vascular anastomosis skill automatic evaluation is completed.

[0012] Further, the trajectory parameters include path length and average speed, wherein the expression of the path length is: ; The expression of the average speed is: ; The expression of the vascular anastomosis skill automatic evaluation result is: ; wherein, denotes the path length, and denotes the two-dimensional coordinates of the instrument tip at time t+1, and denotes the two-dimensional coordinates of the instrument tip at time t, t denotes the index of the data point in chronological order, denotes the average velocity, denotes the total number of trajectory data points, denotes the video frame rate, denotes the result of the automatic evaluation of the vascular anastomosis skill, denotes the motion score, denotes the motion score, denotes the geometry score, denotes the weight of the motion score, denotes the weight of the motion score, denotes the weight of the geometry score. BRIEF DESCRIPTION OF DRAWINGS

[0013] The present specification will further illustrate by way of example embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, in these embodiments, like numerals refer to like structures wherein: Figure 1 is an exemplary flowchart of a method for automatic evaluation of a vascular anastomosis skill based on computer vision technology according to some embodiments of the present specification; Figure 2 is a schematic diagram of a framework of a method for automatic evaluation of a vascular anastomosis skill based on computer vision technology according to some embodiments of the present specification. DETAILED DESCRIPTION

[0014] The specific embodiments of the present application are described below to facilitate the understanding of the present application for those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application as defined in the appended claims, and all the applications utilizing the concept of the present application are within the scope of protection.

[0015] EMBODIMENT Figure 1 is an exemplary flowchart of a method for automatic evaluation of a vascular anastomosis skill based on computer vision technology according to some embodiments of the present specification. As shown in Figure 1 and Figure 2 , the flow includes the following steps. In some embodiments, the flow can be executed by a processor.

[0016] S1: Pre-training a visual feature extraction model using unsupervised learning method with unannotated vascular anastomosis surgery video data to obtain a pre-trained visual feature extraction model.

[0017] The vascular anastomosis surgery video data is video data related to the vascular anastomosis surgery task.

[0018] The visual feature extraction model is a general visual feature extraction model such as Vision Transformer Base.

[0019] In some embodiments, the processor can crop the vascular anastomosis surgery video data to obtain an ROI cropped image with a size of 384x384x3, input the visual feature extraction model, initialize it using BYOL self-supervised pre-training weights, and obtain a 14x14 vascular anastomosis skill visual feature with a channel number of 768.

[0020] S2: Feature extraction of the vascular anastomosis surgery video data using the pre-trained visual feature extraction model to obtain vascular anastomosis skill visual features.

[0021] The vascular anastomosis skill visual features are feature maps (ViT) related to the vascular anastomosis skill, with a channel number of 768 and a size of 14x14.

[0022] S3: Upsampling of the vascular anastomosis skill visual features using a high-resolution pyramid to obtain a high spatial resolution feature map.

[0023] The high spatial resolution feature map is a feature map with a resolution of 96x96x256.

[0024] In some embodiments, since the vascular anastomosis skill visual features have low resolution, which is not conducive to small target positioning, the last 4 stage outputs (different resolutions) of the ViT are upsampled to 1 / 4 of the original image resolution using a high-resolution feature pyramid (HR-FPN), and a high spatial resolution feature map (96x96x256) is output through horizontal connection + 3x3 convolution fusion, which facilitates subsequent segmentation and detection.

[0025] S4: Action recognition, target segmentation, and tip position recognition of the high spatial resolution feature map using multiple downstream task models to obtain current frame action class probability, binary mask, and tip position probability heat map, and the maximum probability class in the current frame action class probability is taken as the operation action class; wherein the multiple downstream task models are obtained through joint training based on the current frame action class probability, binary mask, and tip position probability heat map.

[0026] The downstream task model includes an action recognition branch (Temporal Convolutional Network, TCN model), a target segmentation branch (lightweight U-Net decoder), and an instrument head detection model (key point heatmap regression).

[0027] The current frame action category probability includes needle entry probability and needle exit probability.

[0028] The binary mask is a mask reflecting the suture edge.

[0029] The probability heatmap of the tip position is a probability heatmap of the peak coordinate position, used to reflect the probability of the instrument tip.

[0030] In some embodiments, the processor can utilize the action recognition branch of the downstream task model to perform action recognition on the vascular anastomosis skill visual features, to obtain the current frame action category probability; utilize the target segmentation branch of the downstream task model to perform target segmentation on the high spatial resolution feature map, to obtain the binary mask; perform cross-task attention fusion on the current frame action category probability, to obtain a dynamically enhanced attention map; based on the dynamically enhanced attention map, utilize the instrument head detection sub-model of the downstream task model to perform tip position recognition through key point heatmap regression, to obtain the probability heatmap of the tip position.

[0031] In some embodiments, the processor can utilize the action recognition branch of the downstream task model to analyze the vascular anastomosis skill visual features, to identify key operation steps such as “needle entry”, “needle exit”, “suture transfer”, and “knot tying”, to obtain the current frame action category probability; utilize the target segmentation branch (4-layer up-sampling lightweight U-Net decoder) of the downstream task model to perform target segmentation on the high spatial resolution feature map, to accurately segment the “vascular anastomosis port” region and identify the “suture edge”, for subsequent calculation of geometric parameters such as needle distance and edge distance, to obtain a 384×384 binary mask; utilize the instrument head detection sub-model of the downstream task model to perform action recognition on the current frame action category probability and the binary mask through key point heatmap regression, to detect and locate the tip position of a key surgical instrument (such as “needle holder” and “forceps”), to obtain the probability heatmap of the tip position (96×96×1); wherein the key point heatmap regression includes 3 layers of Conv (256→128→64→1).

[0032] A large number of unlabeled surgical videos are utilized to pre-train the ViT backbone network through BYOL self-supervised contrastive learning, greatly reducing the dependence on expensive pixel-level / frame-level manual annotation data, and improving the model generalization ability and development efficiency; the obtained probability heatmap of the tip position has sub-pixel accuracy, is robust to occlusion, and naturally supports multiple instrument tips (strong expansibility).

[0033] In some embodiments, the processor can jointly train a plurality of downstream task models to obtain a trained downstream task model.

[0034] In some embodiments, the processor can select a spatial attention map matching the current operation semantics from a preset attention template library according to the action category output by the action recognition model. For example, when the recognition result is 'needle entry', the system automatically selects a Gaussian attenuation weight map centered on the blood vessel edge; when the recognition result is 'knot tying', a weight map centered on the suture intersection area is selected. The attention map is multiplied element by element with the HR-FPN feature to dynamically enhance the key area and then sent to the segmentation / detection head.

[0035] Using the action recognition result to dynamically modulate the attention of the segmentation / detection branch can align the semantics between tasks and reduce false positives (e.g., mistaking forceps for a needle tip).

[0036] In some embodiments, the downstream task model can be obtained by jointly training a plurality of labeled training samples. For example, a plurality of labeled training samples can be input into an initial downstream task model, a loss function can be constructed based on the labels and the results of the initial downstream task model, and the parameters of the initial downstream task model can be iteratively updated based on the loss function by gradient descent or other methods. When a preset condition is met, the model training is completed, and a trained downstream task model is obtained. The preset condition can be that the loss function converges, the number of iterations reaches a threshold, etc.

[0037] In some embodiments, the training samples can include historical blood vessel anastomosis surgery video data. The labels can be corresponding operation action categories. The labels can be manually annotated.

[0038] In some embodiments, the loss function of the downstream task model has the following expression: ; ; ; ; ; ; wherein, Ltotal represents the total loss, Laction represents the action recognition loss, Lseg represents the target segmentation loss, Lseg represents the target segmentation loss, Lseg represents the target segmentation loss, Lseg represents the target segmentation loss, Laction represents the weight of the action recognition loss, Lseg represents the weight of the target segmentation loss, The weights represented The weights represented The weights represented express, express, express, The loss represents the degree of overlap between the prediction and the actual region. This represents the actual label mask. This represents the mask used for predictions by the downstream task model. This represents the classification error measured for each pixel. This represents the number of samples in the batch, where i represents the i-th sample in the batch. This represents the true heatmap of the i-th sample. This represents the predicted heatmap for the i-th sample. Denotes the square of the L2 norm. This indicates the calculation of the variance of the set. This represents the predicted coordinates of the j-th seam point. This represents the predicted coordinates of the (j+1)th seam point. Represents Euclidean distance. Represents the average of the set. Represents Euclidean distance. Indicates the edge of the vascular anastomosis. This represents the ideal margin value. Indicates an indicator function, Indicates the first Frame action category prediction results Represents the ReLU function. This represents the predicted coordinates of the needle tip. This indicates the point on the edge of the blood vessel closest to the needle tip. This indicates the tolerance threshold.

[0039] Dice Loss focuses on global overlap, and BCE Loss focuses on local pixel classification. The combination of the two can optimize both macro and micro structures. When the current scene (such as the suture line) accounts for a very small proportion, BCE Loss may be dominated by the background, while Dice Loss can force the model to focus on the foreground area. BCE Loss provides stable gradients, and Dice Loss provides stronger semantic constraints. The joint use can accelerate convergence and avoid falling into local optimum. Accurate segmentation of the "suture line edge" is crucial. Dice Loss is sensitive to boundaries, and BCE Loss can refine the probability transition at boundaries. In clinical practice, the ideal needle interval (StitchInterval, SI) is usually 0.8-1.2 mm, and the difference between adjacent needle intervals is required to be small (i.e. high consistency). Large fluctuations in needle interval will lead to uneven tension, blood leakage or stenosis. Therefore, the "low coefficient of variation" of the needle interval itself is an important indicator of skill proficiency. The model can be encouraged to predict that the suture points are located on an ideal "kiss edge" distance, avoiding being too close or too far. In blood vessel anastomosis, the "edge distance" (Edge Distance, ED) refers to the distance from the needle point to the blood vessel wall, and the ideal value is usually about 1 mm. Too large or too small will affect the quality of suture. The role of this loss term is to "pull" the model's prediction to the clinical standard value, making its output more consistent with the actual operation specifications.

[0040] S3: Analyzing the current frame action class probability, the binary mask, and the probability heat map of the tip position to obtain an operation action class.

[0041] The operation action class is a blood vessel anastomosis operation action class. For example, the operation action class can include "needle entry", "needle exit", "suture transfer", and "knot tying", etc.

[0042] In some embodiments, the action recognition branch of the downstream task model is used to perform action recognition on the blood vessel anastomosis skill visual features to obtain the current frame action class probability, including: using a pre-trained visual feature extraction model to analyze the blood vessel anastomosis surgery video data to obtain blood vessel anastomosis skill visual features; using the action recognition branch of the downstream task model to perform action recognition on the blood vessel anastomosis skill visual features to locate the center coordinates of the recipient blood vessel to obtain a recipient blood vessel positioning result; based on the recipient blood vessel positioning result, performing region expansion and locking to obtain a locked region; performing dynamic tracking and operation action recognition on the locked region to obtain the current frame action class probability.

[0043] The locking region is a square or rectangular region of a fixed size (for example, the side length is 1 / 4 or 1 / 3 of the original picture width) that is extended outward from the located blood vessel center (cx, cy) as the reference. This region is the "target region" (Region of Interest, ROI) for subsequent analysis. Wherein, ROI = [cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2] wherein w and h are the preset ROI width and height.

[0044] In some embodiments, if the blood vessel can move slightly during operation, a simple tracking algorithm (such as based on optical flow or Kalman filtering) can be introduced to predict the position of the current frame according to the blood vessel position of the previous frame, and dynamically adjust the ROI center to ensure that the blood vessel is always located near the ROI center.

[0045] In some embodiments, the processor can crop the above-defined ROI region for each frame in the video, and then input this ROI image (rather than the full image) into the trained operation action recognition model. This greatly reduces background interference and makes small targets occupy a larger relative proportion in the input image, thereby improving the accuracy of recognition. According to the recognized operation action type, the ROI region is further adjusted adaptively, such as when "needle out" and "needle in" actions are recognized, the ROI region needs to be further reduced for more fine-grained judgment of the needle in angle and needle out angle.

[0046] For small key targets such as needle tips and sutures, a "two-stage adaptive target region locking" strategy is proposed: the first stage dynamically locks a fixed ROI centering on the recipient blood vessel to improve the signal-to-noise ratio; the second stage further focuses on the local area according to the action recognition result (such as "needle in") to support high-precision calculation of millimeter-level geometric parameters such as needle in / out angle. At the same time, key point heat map regression is used instead of traditional bounding box detection, and HR-FPN high resolution feature pyramid is introduced to significantly improve the positioning accuracy of small targets.

[0047] S5: Based on computer vision technology, the operation action category is calculated to obtain the blood vessel anastomosis skill automatic evaluation result, and the blood vessel anastomosis skill automatic evaluation is completed.

[0048] The blood vessel anastomosis skill automatic evaluation result is the result of automatic scoring and feedback on the operation quality of the blood vessel anastomosis skill.

[0049] In some embodiments, the processor can calculate the action duration based on the operation action category, analyze the instrument tip position to calculate the motion trajectory and the suture binary mask based on the probability heat map of the binary mask and the tip position, perform instrument tip motion trajectory calculation, trajectory parameter calculation and suture geometry parameter calculation on the operation action category to obtain an action score, a motion score and a geometry score respectively, and perform comprehensive quality evaluation index calculation by weighting based on the action score, the motion score and the geometry score to obtain the automatic evaluation result of the vascular anastomosis skill and complete the automatic evaluation of the vascular anastomosis skill.

[0050] The action score is a score based on instrument tip motion trajectory calculation.

[0051] The motion score is a score based on trajectory parameter calculation.

[0052] The geometry score is a score based on suture geometry parameter calculation. The suture geometry parameters can include needle distance, edge distance and needle entry angle, etc.

[0053] In some embodiments, the processor can calculate the kinematics indexes such as the motion trajectory, speed and path length of the instrument (such as needle holder and forceps) based on the instrument tip position sequence output by the target detection model. Combined with the action recognition result and the suture geometry parameters (needle distance and edge distance), a comprehensive evaluation index is constructed to automatically score and feedback the operation quality of the student.

[0054] In some embodiments, the processor can obtain the pixel coordinates (x_t, y_t) of the “needle holder tip” and the “forceps tip” in each frame through the target detection model. The coordinate points of the consecutive frames are connected to form an original motion trajectory sequence. To eliminate detection noise, the original trajectory is smoothed, and algorithms such as moving average filtering, Savitzky-Golay filtering or Kalman filtering can be used.

[0055] In some embodiments, the trajectory parameters include path length and average speed, wherein the expression of the path length is: ; wherein, denotes the path length, and denotes the two-dimensional coordinates of the instrument tip at t+1 time, and denotes the two-dimensional coordinates of the instrument tip at t time, and t denotes the index of the data points in time sequence.

[0056] In some embodiments, the expression of the average speed is: ; wherein, denotes the average speed, denotes the total number of trajectory data points, denotes the video frame rate.

[0057] In some embodiments, the processor can obtain the accurate contour of the "suture edge" through the target segmentation model. The straight-line distance between the two adjacent needle entry points (or needle exit points) is calculated as the stitch interval (SI). The distance from each needle entry point (or needle exit point) to the nearest "vascular anastomosis edge" is calculated as the edge distance (ED).

[0058] In some embodiments, the expression of the automatic evaluation result of the vascular anastomosis skill is: ; wherein, denotes the automatic evaluation result of the vascular anastomosis skill, denotes the action score, denotes the motion score, denotes the geometric score, denotes the weight of the action score, denotes the weight of the motion score, denotes the weight of the geometric score.

[0059] An evaluation process completely without human intervention can be constructed, abandoning the traditional way of relying on expert subjective scoring. Through quantifiable action recognition results, geometric parameters and kinematic indicators, an objective, repeatable and horizontally comparable scoring standard is provided.

[0060] The three dimensions of action recognition, suture geometric quality and motion are unified in an end-to-end framework, and through a cross-task attention fusion mechanism and a customized loss function (such as a geometric constraint loss and a cross-task consistency loss), semantic alignment between tasks is achieved, ensuring that the evaluation result not only conforms to the operation logic, but also meets the clinical anatomical specifications, providing comprehensive, accurate and real-time training feedback for students.

[0061] In some embodiments of the present specification, a computer vision technology-based automatic evaluation method for vascular anastomosis skills is provided. Through self-supervised pre-training, two-stage adaptive target region locking, multi-task collaborative neural network architecture, and customized loss function system integrating clinical prior knowledge, end-to-end, multi-dimensional, and real-time automatic evaluation of operation action process, suture geometric parameters, and instrument motion trajectory is achieved. (1) Key point heat map regression is used instead of bounding box detection to achieve sub-pixel level positioning and naturally support multi-instrument tip identification. (2) According to the action recognition result, a spatial attention map centered on the blood vessel edge is dynamically generated to modulate the HR-FPN feature, achieve semantic alignment between tasks, and effectively suppress false positives. (3) Through needle distance variance and edge deviation terms, the model is guided to output suture point distribution that conforms to clinical specifications. Surgical rules such as "the needle tip must be close to the blood vessel when the needle is inserted" are used to construct a differentiable supervision signal to improve the anatomical rationality of the detection result. (4) Taking the recipient blood vessel as the center, a fixed-size ROI is dynamically locked for all downstream task inputs, significantly improving the signal-to-noise ratio of small targets, and further reducing the ROI according to the action recognition result to achieve fine calculation of high-precision geometric parameters such as needle insertion and needle extraction angles.

Claims

1. A method for automatic evaluation of a blood vessel anastomosis skill based on computer vision technology, characterized in that, The method comprises the following steps: S1: using unlabeled anastomosis operation video data, a self-supervised learning method is used to pre-train a visual feature extraction model to obtain a pre-trained visual feature extraction model; S2: using the pre-trained visual feature extraction model to extract features from the anastomosis operation video data to obtain anastomosis skill visual features; S3: using a high-resolution pyramid to upsample the anastomosis skill visual features to obtain a high spatial resolution feature map; S4: using multiple downstream task models, action recognition, target segmentation and tip position recognition are performed on the high spatial resolution feature map to obtain current frame action class probability, binary mask and tip position probability heat map, and the maximum probability class in the current frame action class probability is taken as the operation action class; wherein the multiple downstream task models are obtained through joint training based on the current frame action class probability, the binary mask and the tip position probability heat map; S5: based on the operation action class, the action time consumption is calculated; based on the binary mask and the tip position probability heat map, the instrument tip position is analyzed to calculate the motion trajectory and the suture binary mask, and the multi-dimensional index is calculated to obtain the anastomosis skill automatic evaluation result, and the anastomosis skill automatic evaluation is completed. 2.The computer vision technology based blood vessel anastomosis skill automatic evaluation method according to claim 1, characterized in that, The S4 comprises: using the action recognition branch of the downstream task model to perform action recognition on the anastomosis skill visual features to obtain the current frame action class probability; using the target segmentation branch of the downstream task model to perform target segmentation on the high spatial resolution feature map to obtain the binary mask; performing cross-task attention fusion on the current frame action class probability to obtain a dynamically enhanced attention map; based on the dynamically enhanced attention map, using the instrument head detection sub-model of the downstream task model, performing tip position recognition through key point heat map regression to obtain the tip position probability heat map. 3.The computer vision technology based blood vessel anastomosis skill automatic evaluation method according to claim 1, characterized in that, The expression of the loss function of the downstream task model is: ； ； ； ； ； ； in, Indicates the total loss. This indicates the loss in action recognition. Indicates the target segmentation loss. This indicates the loss detected by the instrument head. Represents geometric constraint loss. This indicates cross-task consistency loss. The weights represent the action recognition loss. The weights represent the target segmentation loss. This indicates the weight of the detection loss by the instrument head. The weights representing the geometric constraint loss, The weights representing the cross-task consistency loss express, Indicates the actual action category, This indicates the action category output by the downstream task model. The loss represents the degree of overlap between the prediction and the actual region. This represents the actual label mask. This represents the mask used for predictions by the downstream task model. This represents the classification error for each pixel. This represents the number of samples in the batch, where i represents the i-th sample in the batch. This represents the true heatmap of the i-th sample. This represents the predicted heatmap for the i-th sample. Denotes the square of the L2 norm. This indicates the calculation of the variance of the set. This represents the predicted coordinates of the j-th seam point. This represents the predicted coordinates of the (j+1)th seam point. Represents Euclidean distance. Represents the average of the set. Represents Euclidean distance. Indicates the edge of the vascular anastomosis. This represents the ideal margin value. Indicates an indicator function, Indicates the first Frame action category prediction results Represents the ReLU function. This represents the predicted coordinates of the needle tip. This indicates the point on the edge of the blood vessel closest to the needle tip. This indicates the tolerance threshold. 4.The computer vision technology based blood vessel anastomosis skill automatic evaluation method according to claim 2, characterized in that, The S4 comprises: using the action recognition branch of the downstream task model to perform action recognition on the anastomosis skill visual features to obtain the current frame action class probability; using the target segmentation branch of the downstream task model to perform target segmentation on the high spatial resolution feature map to obtain the binary mask; performing cross-task attention fusion on the current frame action class probability to obtain a dynamically enhanced attention map; based on the dynamically enhanced attention map, using the instrument head detection sub-model of the downstream task model, performing tip position recognition through key point heat map regression to obtain the tip position probability heat map. 5.The computer vision technology based blood vessel anastomosis skill automatic evaluation method according to claim 1, characterized in that, The expression of the loss function of the downstream task model is: The S4 comprises: using the action recognition branch of the downstream task model to perform action recognition on the anastomosis skill visual features to obtain the current frame action class probability; using the target segmentation branch of the downstream task model to perform target segmentation on the high spatial resolution feature map to obtain the binary mask; performing cross-task attention fusion on the current frame action class probability to obtain a dynamically enhanced attention map; based on the dynamically enhanced attention map, using the instrument head detection sub-model of the downstream task model, performing tip position recognition through key point heat map regression to obtain the tip position probability heat map. The expression of the loss function of the downstream task model is: The S5 comprises: based on the operation action class, the action time consumption is calculated; based on the binary mask and the tip position probability heat map, the instrument tip position is analyzed to calculate the motion trajectory and the suture binary mask, and the operation action class is calculated to obtain the action score, the motion score and the geometric score; based on the action score, the motion score and the geometric score, the comprehensive quality evaluation index is calculated through weighting to obtain the anastomosis skill automatic evaluation result, and the anastomosis skill automatic evaluation is completed.

6. The method of claim 5, wherein the method further comprises: The trajectory parameters include a path length and an average speed, wherein an expression of the path length is: ； An expression of the average speed is: ； An expression of the automatic evaluation result of the vascular anastomosis skill is: ； wherein, denotes the path length, and denotes the two-dimensional coordinates of the instrument tip at time t+1, and denotes the two-dimensional coordinates of the instrument tip at time t, t denotes the index of the data points in chronological order, denotes the average speed, denotes the total number of trajectory data points, denotes the video frame rate, denotes the result of the automatic assessment of the vascular anastomosis skill, denotes the motion score, denotes the motion score, denotes the geometry score, denotes the weight of the motion score, denotes the weight of the motion score, denotes the weight of the geometry score.

Images