A deep learning-based surgical quality assessment method and system

Abstract

This invention discloses a deep learning-based surgical quality assessment method and system, relating to the field of medical information. The method includes: acquiring preoperative three-dimensional images and generating a surgical operation space baseline map using a first deep neural network; continuously acquiring real-time instrument pose data and high-definition video frame sequences of the surgical area tissue during surgery; using a spatiotemporal attention mechanism for dynamic registration to construct an instrument-tissue interaction spatiotemporal feature set; calculating instrument operation standardization scores and tissue processing accuracy scores using a second deep neural network; acquiring postoperative physiological recovery indicators; inputting the two scores and indicators into a causal inference model to quantify their respective marginal contributions and generate a comprehensive quality assessment index; and outputting a graded assessment conclusion based on the comparison results of this index with multi-level thresholds. This invention achieves an objective, interpretable, and clinically predictive quantitative assessment of surgical quality by integrating data from the entire chain of preoperative planning, intraoperative operation, and postoperative recovery.

Description

Technical Field

[0001] This application relates to the field of medical information technology, and in particular to a surgical quality assessment method and system based on deep learning, which is specifically applicable to the quantitative analysis and quality evaluation of surgical procedures. Background Technology

[0002] Currently, surgical quality assessment mainly relies on two methods: one is retrospective scoring of surgical videos by experts after surgery, and the other is statistical evaluation based on discrete indicators such as operation time, blood loss, and complication rate. However, retrospective scoring by experts suffers from drawbacks such as strong subjectivity, time-consuming nature, and difficulty in large-scale application; while statistical evaluation of discrete indicators ignores the specific details of the surgical procedure and cannot pinpoint which step led to poor prognosis. More importantly, existing assessment methods mostly sever the intrinsic connection between preoperative planning, intraoperative operation, and postoperative recovery. Preoperative images are only used for navigation, intraoperative data is only used for monitoring, and postoperative indicators are only used for summarization, lacking a unified quantitative mapping framework among the three.

[0003] The root cause of these problems lies in the fact that surgical quality is essentially a causal chain from planning benchmarks to operational execution and physiological response. Current technologies lack multimodal data fusion models capable of simultaneously processing three-dimensional image space, two-dimensional video space, and instrument kinematic space, and also lack inference mechanisms that can causally correlate operational behavior scores with clinical outcomes. Traditional machine learning methods often treat intraoperative features and postoperative indicators as independent inputs and outputs, ignoring the structured dependencies between them. This leads to difficult-to-interpret assessment results and fails to provide surgeons with forward-looking guidance for operational correction.

[0004] Therefore, there is an urgent need for a surgical quality assessment scheme that can span the entire process from preoperative to postoperative and can automatically learn the complex mapping relationships between multimodal data. Summary of the Invention

[0005] This application provides a surgical quality assessment method and system based on deep learning, which at least addresses the problems existing in the prior art.

[0006] A first aspect of this application provides a deep learning-based surgical quality assessment method, comprising the following steps: S1: Obtain preoperative three-dimensional medical image data of the target object, and based on the preoperative three-dimensional medical image data, use the first deep neural network to generate a surgical operation space reference map of the anatomical structure. S2: During the surgical procedure, real-time positional data of surgical instruments and high-definition video frame sequences of the surgical area tissue are collected. S3: Based on the surgical operation space reference map, a spatiotemporal attention mechanism is used to dynamically register real-time pose data with high-definition video frame sequences in order to construct a spatiotemporal feature set of surgical instruments-surgical area tissue interaction. S4: Based on the spatiotemporal feature set of surgical instrument-surgical area tissue interaction, the second deep neural network is used to calculate the instrument operation standardization score and tissue processing accuracy score of the current surgical stage respectively; S5: After the surgery, obtain at least one physiological recovery indicator of the target subject in the short term after the surgery, and input the instrument operation standardization score, tissue processing accuracy score and physiological recovery indicator into the preset causal inference model; S6: Using a causal inference model, the marginal contribution of the instrument operation standardization score and the tissue processing accuracy score to physiological recovery indicators is quantified, and a comprehensive quality assessment index is generated. S7: Based on the comparison between the comprehensive quality assessment index and the preset multi-level quality thresholds, output the graded assessment conclusion corresponding to the surgical operation behavior.

[0007] This application's embodiments incorporate personalized baseline maps generated from preoperative images, multimodal data of intraoperative instruments and tissues, and postoperative physiological recovery indicators into a single deep neural network and causal inference cascade framework, achieving full-chain quantitative evaluation from surgical planning to operation execution and clinical outcome. Specifically, the spatiotemporal attention mechanism ensures precise alignment of instrument movement and tissue deformation under the constraints of the baseline map, avoiding feature misalignment caused by inconsistencies in coordinate systems. The introduction of marginal contribution ensures that the final comprehensive evaluation index not only reflects the quality of the operation itself but also predicts the actual impact of the operation on patient recovery, overcoming the shortcomings of traditional assessments that are disconnected from clinical outcomes.

[0008] In some embodiments of this application, the dynamic registration in step S3 specifically includes: Based on high-definition video frame sequences, the first motion trajectory of the instrument tip in the tissue coordinate system is extracted; Based on real-time pose data, the second motion trajectory of the instrument end effector in the coordinate system of the positioning device is obtained; By employing a differentiable spatial transformation network, affine transformation parameters are learned from the positioning device coordinate system to the organization coordinate system, thereby minimizing the dynamic time warping distance between the second motion trajectory and the first motion trajectory.

[0009] This application's embodiments achieve joint alignment of the device motion trajectory and tissue image trajectory in both spatial and temporal dimensions by introducing a differentiable spatial transformation network and dynamic temporal warping distance. Compared to traditional fixed registration matrices, this scheme can adaptively correct non-rigid offsets caused by patient positional movements, respiratory motions, etc., significantly improving the accuracy of subsequent feature extraction, while providing a sub-pixel-level spatial benchmark for device operation standardization scoring.

[0010] In some embodiments of this application, the second deep neural network is a multi-task learning network. The multi-task learning network shares a feature extraction backbone and branches into a first regression head and a second regression head. The first regression head is used to output the instrument operation standardization score, and the second regression head is used to output the tissue processing accuracy score. The loss function of the multi-task learning network is the sum of the mean square error of the instrument operation standardization label, the mean square error of the tissue processing accuracy label, and the maximum mean difference penalty term of the feature vectors of the two regression heads.

[0011] This application embodiment enables joint representation learning of instrument operation features and tissue processing features at the bottom layer by sharing the feature extraction backbone, avoiding feature redundancy caused by training two networks separately; at the same time, the maximum mean difference penalty term forces the feature distributions of the two regression heads to be close to each other, preventing the network from being biased towards learning private noise of a certain task, thereby improving the generalization ability and consistency of the two scores.

[0012] In some embodiments of this application, the causal inference model is a non-parametric model based on structural causal equations; the process of calculating the marginal contribution using the non-parametric model includes: fixing the instrument operation standardization score, changing the tissue treatment accuracy score, and observing the first change in the physiological recovery index; fixing the tissue treatment accuracy score, changing the instrument operation standardization score, and observing the second change in the physiological recovery index; normalizing the first change and the second change, and using them as the marginal contribution of the instrument operation standardization score and the tissue treatment accuracy score, respectively.

[0013] This application's embodiments calculate marginal contribution using a non-parametric structural causal model, enabling the accurate quantification of the independent impact of different operational dimensions on patient recovery without presupposing linear relationships. This approach effectively distinguishes the respective action paths of "device manipulation" and "tissue treatment," avoiding collinearity interference in traditional regression models and ensuring clear interpretability of the comprehensive quality assessment index.

[0014] In some embodiments of this application, step S3 further includes a stage quality assessment step: S31: The spatiotemporal feature set of surgical instrument-surgical area tissue interaction is divided according to surgical action units, and each surgical action unit corresponds to a complete cutting, suturing or hemostasis operation; S32: For each surgical action unit, calculate its sub-action score using a pre-trained motion quality assessment network; S33: Input the scores of each sub-action into a bidirectional long short-term memory network in chronological order, and output the overall quality evolution trend curve of the current surgical stage.

[0015] This application embodiment, by segmenting the continuous surgical process into meaningful action units and using a bidirectional long short-term memory network to capture the sequential dependencies between sub-action scores, can identify early signs of quality decline (such as a continuous decline in the score of a certain suturing action), thereby providing a more sensitive dynamic indicator for real-time intraoperative early warning than the stage average score.

[0016] In some embodiments of this application, the motion quality assessment network includes a first convolutional module, a second convolutional module, and a comparison module; the first convolutional module is used to extract the velocity and acceleration features of the instrument movement in the current surgical motion unit; the second convolutional module is used to extract the strain rate features of tissue deformation in the same motion unit; the comparison module is used to calculate the first cosine similarity of the velocity and acceleration features relative to the standard motion template, and the second cosine similarity of the strain rate features relative to the standard tissue response template, and to use the weighted sum of the first cosine similarity and the second cosine similarity as the sub-motion score.

[0017] This application embodiment uses convolutional encoding to encode the kinematic characteristics of the device and the biomechanical characteristics of the tissue, and compares them with the cosine similarity of the standard template. This gives the sub-action score a clear physical meaning—the score directly reflects the degree of deviation between the actual action and the expert demonstration action in terms of velocity profile and tissue response, thus enhancing the guiding value of the evaluation results for clinical teaching.

[0018] In some embodiments of this application, the process of generating the surgical operation space reference map in step S1 further includes: extracting multi-scale spatial features of preoperative three-dimensional medical image data using a three-dimensional convolutional layer in a first deep neural network; resampling the multi-scale spatial features using a spatial transformation layer to correct individual pose differences of the target object; and using an upsampling layer and a skip connection structure to output a probability map with the same resolution as the original image, marked with safety boundaries and key avoidance areas, as the surgical operation space reference map.

[0019] This application's embodiments, by embedding a spatial transformation layer and skip connections in the baseline map generation network, eliminate the interference of individual anatomical differences on subsequent registration while preserving high-resolution safety boundary information. The output probability map can be directly used as a common reference frame for intraoperative navigation and assessment, making the detection of instrument trajectory deviations from the safety boundary more accurate.

[0020] In some embodiments of this application, S7 further includes a visual feedback step: S71: In response to the grading assessment conclusion being lower than the first preset threshold, the spatiotemporal segment corresponding to the low score in the spatiotemporal feature set of the surgical instrument-surgical area tissue interaction is superimposed on the surgical display interface. S72: In the spatiotemporal segment, heatmaps are used to identify the trajectory segments where the instrument pose deviates from the safety boundary in the surgical operation space baseline map, as well as video frames where tissue processing exceeds the critical avoidance area. S73: Generate text prompts that advise surgeons to adjust instrument entry angles or tissue traction in subsequent surgical procedures.

[0021] This application's embodiments present low-scoring spatiotemporal segments, deviation heatmaps, and correction suggestions in a linked manner, providing surgeons with an intuitive feedback loop of "where the error occurred - where the error lies - how to correct it." Compared to simply providing a total score, this visualization method significantly reduces cognitive load and helps to improve procedures in real time during surgery.

[0022] In some embodiments of this application, an incremental model update step is also included: After each surgery is completed, the instrument operation standardization score, tissue processing accuracy score, comprehensive quality assessment index, and corresponding real-time pose data and high-definition video frame sequence generated in this surgery are used as an incremental training sample. Store incremental training samples into the recurrent experience replay buffer; When the number of samples in the loop experience replay buffer reaches the batch threshold, a batch of samples is randomly sampled from the buffer to jointly fine-tune the second deep neural network and the causal inference model.

[0023] This application's embodiments utilize a cyclical experience replay buffer and a joint fine-tuning mechanism to enable the evaluation model to continuously adapt to new surgical techniques, instruments, and the operating habits of different surgeons as surgical data accumulates, thus preventing model degradation. Simultaneously, random sampling and batch updates prevent overfitting of recent samples, maintaining model stability across different cases.

[0024] In some embodiments of this application, the first deep neural network in step S1 and the second deep neural network in step S4 are pre-trained using a contrastive learning approach: constructing positive sample pairs, which include instrument pose subsequences and corresponding tissue video sub-segments under the same surgical operation; constructing negative sample pairs, which include instrument pose subsequences and tissue video sub-segments under different surgical operations or with temporal misalignment; obtaining pre-trained feature extractor weights by minimizing the feature distance of the positive sample pairs and maximizing the feature distance of the negative sample pairs, and initializing at least a portion of the layers of the first deep neural network and the second deep neural network with these weights respectively.

[0025] This application's embodiments utilize contrastive learning pre-training, enabling the network to learn cross-modal alignment representations between instruments and tissues on a large amount of unlabeled surgical data. This initialization strategy significantly reduces the need for expert-annotated scoring data during subsequent supervised training, while simultaneously improving the model's generalization ability across small sample surgical types.

[0026] In some embodiments of this application, the formula for calculating the comprehensive quality assessment index E is as follows: E = α•(w1•S_instr + w2•S_tissue) + (1-α)•(w3•ΔR) Wherein, S_instr is the instrument operation standardization score, S_tissue is the tissue processing accuracy score, ΔR is the change in physiological recovery index relative to historical baseline data, α is the balance weight coefficient, which is negatively correlated with the complexity level of the current surgery, and w1, w2, and w3 are the dynamic weight coefficients learned through the causal inference model.

[0027] This application's embodiments introduce a balanced weighting coefficient α that is negatively correlated with the complexity of the surgery, enabling the comprehensive assessment index to adaptively adjust the relative importance between intraoperative operation scores and postoperative predicted outcomes: for complex surgeries, α is reduced to rely more on postoperative recovery predictions, avoiding false positive warnings caused by overly stringent intraoperative scoring; for simple surgeries, α is increased to emphasize operational standardization, thus achieving personalized assessment strategies.

[0028] A first aspect of this application provides a deep learning-based surgical quality assessment system, comprising: The preoperative planning unit is used to acquire preoperative three-dimensional medical image data of the target object, and based on the preoperative three-dimensional medical image data, it uses a first deep neural network to generate a surgical operation space reference map of the anatomical structure. The intraoperative acquisition unit is used to continuously acquire real-time pose data of surgical instruments and high-definition video frame sequences of surgical area tissues during the operation. The spatiotemporal registration unit is used to dynamically register real-time pose data with high-definition video frame sequences based on the surgical operation space reference map using a spatiotemporal attention mechanism, so as to construct a spatiotemporal feature set of surgical instruments-surgical area tissue interaction. The stage evaluation unit is used to calculate the instrument operation standardization score and tissue processing accuracy score of the current surgical stage based on the spatiotemporal feature set of surgical instrument-surgical area tissue interaction and the second deep neural network. The causal analysis unit is used to obtain at least one physiological recovery indicator of the target subject in the short term after surgery, and input the instrument operation standardization score, tissue processing accuracy score and physiological recovery indicator into the preset causal inference model; through the causal inference model, the marginal contribution of the instrument operation standardization score and tissue processing accuracy score to the physiological recovery indicator is quantified, and a comprehensive quality assessment index is generated. The conclusion output unit is used to output a graded evaluation conclusion corresponding to the surgical operation behavior based on the comparison result between the comprehensive quality assessment index and the preset multi-level quality threshold. And control processors connected to each of the above units, used to coordinate the working timing and data transmission of each unit.

[0029] This application embodiment achieves a fully automated hardware system from preoperative planning to intraoperative assessment and postoperative causal analysis by setting up functional units that correspond one-to-one with the method steps and configuring unified and coordinated timing and control processors for each unit. The units use standardized data interfaces to transmit intermediate results such as surgical operation space baseline maps and interactive spatiotemporal feature sets, ensuring the system's deployability and scalability in different operating room equipment environments. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating a deep learning-based surgical quality assessment method provided in one embodiment of this application. Detailed Implementation

[0031] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0032] To make the purpose, technical solution, and advantages of this application clearer, the following will be described in conjunction with the appendix. Figure 1 The following is an explanation using specific examples.

[0033] Current surgical quality assessment methods mainly fall into two categories. One involves subjective scoring by experts reviewing surgical videos post-operatively. This method relies heavily on individual expert experience, resulting in low consistency among different experts and a lack of real-time feedback during the procedure. The other method uses statistical evaluation based on discrete indicators such as surgical duration, blood loss, and complication rates. While these indicators are objective, they overlook specific details of the surgical procedure, such as whether the instrument insertion angle is optimal or whether tissue traction is excessive. More critically, existing methods often treat preoperative planning, intraoperative manipulation, and postoperative recovery as separate stages: preoperative images are used only for navigation planning, intraoperative data is used only for real-time display, and postoperative indicators are used only for review and summary. There is a lack of a quantitative bridge connecting these three stages. Therefore, developing a method that can span the entire surgical process and correlate surgical actions with clinical outcomes is a pressing technical challenge.

[0034] Analyzing the underlying causes of the above problems reveals that surgical quality is essentially a causal chain from anatomical benchmarks to operational execution and physiological responses. Preoperative 3D images contain individual anatomical information about the patient, such as the location of important blood vessels and tumor boundaries. This information can be transformed into a safety benchmark in the operational space. The movement trajectory of instruments and the interaction between instruments and tissues (such as cutting depth and the range of heat diffusion during hemostasis) directly determine whether the operation deviates from this benchmark. Postoperative recovery indicators (such as inflammatory factor levels and organ function recovery time) are the final physiological manifestation of the operational results. However, current technologies lack a multimodal data fusion model that can simultaneously process 3D image space, 2D video space, and instrument kinematic space. Traditional methods typically treat image data, pose data, and video data as independent information sources, extracting features separately and then simply stitching them together. This approach ignores the spatiotemporal consistency constraints of instrument movement and tissue deformation and lacks a statistical framework that can causally infer operational scores from recovery indicators. Therefore, new solutions need to improve both the data fusion method and the evaluation logic simultaneously.

[0035] In exploring solutions, the inventors noticed that the spatiotemporal attention mechanism in deep neural networks can automatically learn the alignment relationships between data from different modalities. For example, in the field of video understanding, attention mechanisms are used to synchronize speech signals with lip-sync videos. Inspired by this, the inventors attempted to apply this mechanism to surgical scenarios: the preoperative 3D image-generated operational space baseline map can be considered a static reference coordinate system, while the real-time instrument pose data and surgical area tissue video frames acquired during surgery are dynamic observation data. Through the spatiotemporal attention mechanism, the network can automatically focus on instrument pose moments that deviate significantly from the baseline map and the corresponding tissue deformation frames, thereby achieving dynamic registration. Furthermore, in economics and epidemiology, structural causal models are widely used to quantify the marginal contribution of different factors to a particular outcome. Drawing on this idea, the inventors considered instrument operation standardization scores and tissue treatment accuracy scores as two intervention variables, and postoperative physiological recovery indicators as outcome variables, estimating the independent contribution of each operational dimension to the recovery effect through non-parametric structural causal equations. In this way, the final comprehensive quality assessment index is no longer a simple weighted sum, but a quantitative indicator with clinical predictive significance. Therefore, this application proposes a complete technical solution that combines preoperative baseline generation, intraoperative multimodal registration and evaluation, and postoperative causal inference.

[0036] Please refer to Figure 1 , Figure 1 A deep learning-based surgical quality assessment method provided in this application includes the following steps: S1: Obtain preoperative three-dimensional medical image data of the target object, and based on the preoperative three-dimensional medical image data, use the first deep neural network to generate a surgical operation space reference map of the anatomical structure. S2: During the surgical procedure, real-time positional data of surgical instruments and high-definition video frame sequences of the surgical area tissue are collected. S3: Based on the surgical operation space reference map, a spatiotemporal attention mechanism is used to dynamically register real-time pose data with high-definition video frame sequences in order to construct a spatiotemporal feature set of surgical instruments-surgical area tissue interaction. S4: Based on the spatiotemporal feature set of surgical instrument-surgical area tissue interaction, the second deep neural network is used to calculate the instrument operation standardization score and tissue processing accuracy score of the current surgical stage respectively; S5: After the surgery, obtain at least one physiological recovery indicator of the target subject in the short term after the surgery, and input the instrument operation standardization score, tissue processing accuracy score and physiological recovery indicator into the preset causal inference model; S6: Using a causal inference model, the marginal contribution of the instrument operation standardization score and the tissue processing accuracy score to physiological recovery indicators is quantified, and a comprehensive quality assessment index is generated. S7: Based on the comparison between the comprehensive quality assessment index and the preset multi-level quality thresholds, output the graded assessment conclusion corresponding to the surgical operation behavior.

[0037] In this application, the preoperative three-dimensional medical imaging data in step S1 may include computed tomography (CT) images, magnetic resonance imaging (MRI) images, or three-dimensional ultrasound images, which serve to provide input for subsequent anatomical structure recognition. The first deep neural network is a pre-trained convolutional neural network used to segment anatomical structures from the three-dimensional images, such as tumor boundaries in the liver or important blood vessels in the Calot's triangle. The surgical operation space baseline map can be understood as a probability map corresponding to the surgical area. The value of each pixel or voxel in the map indicates the degree to which the location belongs to a safe operating area, a danger avoidance area, or a transitional area; for example, areas near important blood vessels are marked as high-probability avoidance areas.

[0038] The real-time pose data in step S2 is usually provided by electromagnetic or optical positioning sensors installed on the surgical instruments, including the spatial coordinates and rotation angle of the instrument end; the high-definition video frame sequence is acquired by the surgical field camera to record the real-time interaction between the instruments and tissues.

[0039] The spatiotemporal attention mechanism in step S3 is a network module that can simultaneously calculate attention weights in both the spatial dimension (different locations in the image) and the temporal dimension (between different frames), allowing the network to dynamically focus on frames and pixel regions that deviate significantly from the reference image. Dynamic registration refers to aligning the instrument trajectory in the positioning device coordinate system with the image trajectory in the tissue coordinate system, similar to overlaying a transparent trajectory image onto another reference image and continuously adjusting their relative positions to make them coincide. The spatiotemporal feature set of surgical instrument-surgical area tissue interaction is a structured data set containing multidimensional information such as timestamps, instrument positions, instrument postures, tissue deformation vectors, and attention weights.

[0040] The second deep neural network in step S4 is a multi-task learning structure. Its input is the feature set obtained in the previous step, and its output is two scalar values: the instrument operation standardization score reflects the degree of conformity between the instrument movement trajectory and the standard path, such as whether the cutting edge of the electrosurgical knife is smooth and whether it repeatedly cuts in the same position; the tissue processing accuracy score reflects whether the force, heat or shearing effect applied by the instrument to the tissue is within the safe range, such as whether excessive burning during hemostasis leads to carbonization of the surrounding tissue.

[0041] The physiological recovery indicators in step S5 can be the concentration of inflammatory factors, the characteristics of drainage fluid, pain scores, or organ function test values ​​within 24 hours post-surgery. These indicators can reflect the degree of impact of surgical trauma on the patient's body. The causal inference model can be implemented using structural causal equations or Bayesian networks. Its role is to estimate, while controlling for other variables, the change in physiological recovery indicators caused by a one-unit change in the score of each operation. This change is the marginal contribution.

[0042] The comprehensive quality assessment index in step S6 is a value obtained by fusing the two operation scores and marginal contribution. The value can be set between 0 and 100, and the higher the value, the better the surgical quality.

[0043] The multi-level quality threshold in step S7 can be preset to three levels, for example, below 60 points is to be improved, 60 to 85 points is good, and above 85 points is excellent; the graded evaluation conclusion can be output to the display device in the form of text, color coding or graphics.

[0044] In this embodiment, the surgical operation space baseline map generated in step S1 provides a personalized anatomical reference system for all subsequent assessments. Without this baseline map, the dynamic registration in step S3 would lack an alignment target, and the operational standardization score in step S4 would be unable to determine whether the instrument has deviated from the safety boundary. The introduction of the baseline map makes the assessment no longer generic, but customized for each patient's specific anatomical structure, thereby improving the clinical relevance of the score.

[0045] In step S2, both instrument pose data and video frame sequences are acquired simultaneously in time, providing the necessary input for the spatiotemporal attention mechanism in step S3. Using pose data alone only reveals where the instrument is moving, but not the tissue deformation that occurs during the interaction between the instrument and the tissue; using video data alone shows tissue changes, but lacks quantifiable parameters of instrument movement. The two types of data complement each other: pose data provides precise physical quantities (such as velocity and acceleration), while video data provides intuitive morphological information (such as whether the tissue is blackened or bleeding). The spatiotemporal attention mechanism jointly registers both data with a baseline image, allowing subsequent feature extraction to focus on truly important spatiotemporal locations. For example, when the instrument approaches a blood vessel, the attention weight automatically increases, enabling more refined analysis of the data in that time period.

[0046] Step S4 utilizes a second deep neural network to extract two scores from the registered feature set. These two scores are not isolated indicators. The instrument operation standardization score focuses on whether the instrument's behavior conforms to standards, while the tissue processing accuracy score focuses on whether the instrument's behavior produces the expected results on the tissue. For example, in an electrosurgical resection operation, the instrument's movement speed conforms to standards (high standardization score), but improper power settings lead to tissue edge carbonization (low accuracy score). The two scores characterize the same operation from different dimensions, mutually verifying each other and preventing a single indicator from being masked by a high score for a low score.

[0047] Steps S5 and S6 introduce postoperative physiological recovery indicators and a causal inference model, which is a key difference between this approach and existing methods. Traditional methods end after calculating the intraoperative score, but does a high intraoperative score necessarily mean a good patient recovery? Not necessarily, as individual differences such as physiological reserves and underlying diseases can affect the recovery process. The causal inference model answers the question by calculating marginal contribution: after excluding other interfering factors, how much does a 10-point increase in the instrument operation standardization score, on average, reduce the level of postoperative inflammatory factors? This quantitative relationship gives the comprehensive quality assessment index clinical predictive power. Step S7 outputs a grading conclusion based on the comprehensive index. This conclusion is not only a post-operative evaluation of past procedures but also implicitly estimates the patient's future recovery trend, thus providing physicians with more valuable decision-making references.

[0048] Steps S1 to S7 in this application form a complete closed loop, from static baseline generation to dynamic data acquisition and registration, to multi-dimensional scoring, and finally to causal prediction and conclusion output. The output of each step serves as the input for the next step. The baseline map provides the target for registration, the registration result provides aligned features for scoring, the scoring result and postoperative indicators jointly drive causal inference, and the inference result ultimately generates the evaluation conclusion. This interlocking design avoids isolated feature extraction, enabling the entire method to process heterogeneous data from the preoperative, intraoperative, and postoperative stages within a unified technical framework, achieving a quantitative correlation between operational quality and clinical outcomes.

[0049] In some embodiments of this application, the dynamic registration in step S3 specifically includes: extracting the first motion trajectory of the instrument end in the tissue coordinate system based on the high-definition video frame sequence; converting the instrument end in the positioning device coordinate system based on real-time pose data to obtain the second motion trajectory of the instrument end in the positioning device coordinate system; and using a differentiable spatial transformation network to learn the affine transformation parameters from the positioning device coordinate system to the tissue coordinate system, so as to minimize the dynamic time warping distance between the second motion trajectory and the first motion trajectory.

[0050] In this embodiment, the first motion trajectory refers to the sequence of continuous position points of the instrument tip in the tissue coordinate system extracted from a high-definition video frame sequence using image processing algorithms (e.g., target detection and tracking). The tissue coordinate system is a coordinate system established with reference to the surgical area tissue, for example, with a bony landmark or soft tissue marker in the surgical field as the origin. It is understood that, due to possible tissue deformation or displacement, the tissue coordinate system is not absolutely fixed, but dynamically updated with the movement of the tissue in the video frame. The second motion trajectory is the sequence of position points of the instrument tip in the positioning device coordinate system obtained based on real-time pose data conversion. The positioning device coordinate system is determined by the spatial measurement range of the electromagnetic positioning instrument or optical positioning instrument, such as the spatial coordinate origin of the electromagnetic field generator. A differentiable spatial transformation network is a neural network structure whose special feature is that the transformation parameters in the network can be updated by gradient through backpropagation algorithm, which means that the network can learn the mapping relationship from the positioning device coordinate system to the tissue coordinate system in an end-to-end manner. Affine transformation parameters are a set of values ​​describing the linear mapping between two coordinate systems, typically including rotation, translation, scaling, and shearing. For example, if a point in the positioning device coordinate system has coordinates (1,0,0), after an affine transformation, it is mapped to a point in the tissue coordinate system. The transformation parameters determine the specific method of this mapping. Dynamic time warping distance is a metric for comparing the similarity of two time series. It allows for non-linear alignment of the two series along the time axis, such as stretching or compressing faster segments in the second motion trajectory to match the temporal position of the corresponding operation in the first motion trajectory. By minimizing this distance, the spatial transformation network can learn the optimal affine transformation parameters, achieving the best match between the two trajectories in both space and time.

[0051] In this embodiment, the first motion trajectory is derived from video images, reflecting how the instrument's end effector moves within the tissue's own reference frame. Since the video images directly record the relative position of the instrument and the tissue, the first motion trajectory can be considered a true or near-true instrument movement path. However, video images are susceptible to occlusion, lighting variations, and image noise; trajectories extracted solely from video may be discontinuous or lack sufficient accuracy at certain times. The second motion trajectory is derived from a positioning sensor, providing high-precision, high-sampling-rate spatial position data for the instrument. However, there is a lack of direct correspondence between the positioning device's coordinate system and the tissue's coordinate system; for example, the coordinate values ​​measured by the positioning device cannot directly tell the physician which anatomical point on the tissue corresponds to that position. Both trajectories have their advantages and limitations; neither can provide both accurate and tissue-aligned instrument movement information when used alone.

[0052] To address this issue, step S3 introduces a differentiable spatial transformation network to learn the affine transformation parameters between the two coordinate systems. This network takes the second motion trajectory as input, performs an affine transformation, and outputs a transformed trajectory. This transformed trajectory is then compared to the first motion trajectory. Since the first motion trajectory is considered the reference standard, the network continuously adjusts the affine transformation parameters to make the transformed trajectory as close as possible to the first motion trajectory. Dynamic time warping distance is used as the comparison metric because even if two trajectories describe the same physical process, they may exhibit local scaling on the time axis—for example, a surgeon's operating speed may vary, but the sequence and spatial form of the operation are similar. Dynamic time warping distance can tolerate such temporal misalignment, thus calculating a more meaningful similarity.

[0053] First, the first motion trajectory provides a spatially aligned anchor point for the second motion trajectory, enabling data from the positioning device's coordinate system to be transformed into an anatomically meaningful tissue coordinate system. Second, a differentiable spatial transformation network and a dynamic temporal warping distance together constitute a trainable registration module. This module can not only learn static coordinate transformation parameters but also adaptively align the temporal differences between the two trajectories. This spatiotemporal joint alignment method reduces error accumulation compared to the traditional method of first synchronizing time and then spatially registering. Finally, the registered instrument trajectory retains the high accuracy of the positioning sensor while acquiring spatial semantics aligned with the tissue image, thus providing more reliable input features for subsequent scoring calculations. For example, when it is necessary to determine whether the instrument has crossed the blood vessel avoidance zone, the registered trajectory can be directly compared pixel-level with the safety boundary in the surgical operation space reference map without additional coordinate transformation steps.

[0054] In some embodiments of this application, the second deep neural network is a multi-task learning network. The multi-task learning network shares a feature extraction backbone and branches into a first regression head and a second regression head. The first regression head is used to output the instrument operation standardization score, and the second regression head is used to output the tissue processing accuracy score. The loss function of the multi-task learning network is the sum of the mean square error of the instrument operation standardization label, the mean square error of the tissue processing accuracy label, and the maximum mean difference penalty term of the feature vectors of the two regression heads.

[0055] In this embodiment, the second deep neural network adopts a multi-task learning network structure. A multi-task learning network is a neural network architecture that learns multiple related tasks simultaneously. In this scheme, these two tasks are predicting the instrument operation standardization score and predicting the tissue processing accuracy score. The shared feature extraction backbone means that the first few layers of the network (e.g., multiple convolutional layers or fully connected layers) are shared for both tasks. These layers extract common feature representations from the spatiotemporal feature set of the input surgical instrument-surgical area tissue interaction, such as the smoothness of the instrument trajectory and the uniformity of tissue deformation. After the shared backbone, the network splits into two independent branches, each called a regression head. The regression head typically consists of several fully connected layers, with its output layer having only one neuron, used to output a continuous numerical value as a score. The first regression head outputs the instrument operation standardization score, and the second regression head outputs the tissue processing accuracy score. The loss function is used to measure the difference between the network's predicted value and the true label. During training, the network parameters are updated by minimizing the loss function. The loss function in this scheme includes three terms. The first term is the mean squared error of the instrument operation standardization label, which is the average of the squared differences between the standardization score predicted by the network and the standardization label annotated by experts. The second term is the mean squared error of the tissue processing accuracy label, with a similar meaning. The third term is the maximum mean difference penalty term. The maximum mean difference is a statistic that measures the difference between two distributions. Its basic idea is to map the samples in the two distributions to a high-dimensional regenerating kernel Hilbert space, and then calculate the distance between the mean vectors of the two distributions in this space. In this scheme, this penalty term calculates the maximum mean difference between the feature vectors of the two regression heads after sharing the backbone and before entering the final output layer (i.e., the output of the penultimate layer of each head). The purpose of the penalty term is to encourage the feature distributions of the two heads to be close to each other, avoiding the network learning contradictory or redundant feature representations in order to fit the two tasks separately.

[0056] In this embodiment, the introduction of a shared feature extraction backbone transforms the instrument operation standardization score and tissue processing accuracy score from two independently learned models. Since both tasks rely on the same set of input data (i.e., the spatiotemporal feature set of instrument-tissue interaction), their required underlying features overlap significantly. For example, changes in instrument movement speed affect both operation standardization (excessive speed may lead to uneven cutting edges) and tissue processing accuracy (excessive speed may lead to tissue tearing). The shared backbone forces the network to learn representations of these underlying features simultaneously, thereby reducing the overall number of parameters and mitigating the risk of overfitting. Without a shared backbone, if two independent networks were trained separately, each network would independently learn to extract motion and tissue deformation features. This not only increases computational overhead but may also lead to inconsistent understandings of the same physical phenomena by the two networks.

[0057] The branching structure of the two regression heads allows the network to perform specialized learning for its respective tasks after sharing underlying features. For example, the scoring of instrument handling standardization might focus more on kinematic features such as the trajectory curvature and acceleration changes of the instrument's end effector, while the scoring of tissue processing accuracy might focus more on the spatiotemporal patterns of tissue strain and the extent of thermal damage. The branching structure allows each head to adjust weights in its own fully connected layers to adapt to the nonlinear mapping requirements of its task. This design, with a shared backbone and branching heads, avoids the information silos problem of completely independent training and the interference between the two tasks when all parameters are shared.

[0058] The maximum mean difference penalty term in the loss function is a key mechanism connecting the two regression heads. If only two mean squared error terms are used for training, the network may exhibit a phenomenon where the feature vectors extracted by the two heads have significantly different distributions; that is, the feature space learned by the instrument standardization head and the feature space learned by the tissue precision head lie on different manifolds. This leads to a potential problem: when the input data includes new surgical scenarios, the two heads may exhibit inconsistent generalization behaviors. The maximum mean difference penalty term forces the feature distributions of the two heads to converge, making the two scores comparable at the feature level. Understandably, this constraint does not require the two scores themselves to be numerically similar (because scores can have different dimensions), but rather that the intermediate features used to generate the scores have similar distributional patterns. In this way, when the network encounters instrument operation patterns not present in the training data, the two heads can still provide relatively consistent judgment logic based on similar feature representations, thereby improving the model's stability and robustness across surgical types and physician operating habits. In addition, the penalty term also acts as a regularization, suppressing overfitting of a single head to the training data, since overfitting usually causes the feature distribution to deviate from the normal distribution range of the other head.

[0059] In some embodiments of this application, the causal inference model is a non-parametric model based on structural causal equations; the process of calculating the marginal contribution using the non-parametric model includes: fixing the instrument operation standardization score, changing the tissue treatment accuracy score, and observing the first change in the physiological recovery index; fixing the tissue treatment accuracy score, changing the instrument operation standardization score, and observing the second change in the physiological recovery index; normalizing the first change and the second change, and using them as the marginal contribution of the instrument operation standardization score and the tissue treatment accuracy score, respectively.

[0060] In this embodiment, structural causal equations are a mathematical expression used to describe causal relationships between variables. Unlike traditional correlation and regression equations, they can distinguish the causal direction between variables. Non-parametric models do not pre-assume that variables follow a specific functional form (such as linear or exponential relationships). Their advantage lies in their ability to fit mapping relationships of arbitrary shapes. In this scheme, using a non-parametric causal inference model means that the causal relationship between instrument operation standardization scores, tissue treatment accuracy scores, and physiological recovery indicators is not pre-set to be linear or monotonic, but rather learned through data-driven learning. "Fixed" refers to maintaining the value of a variable at a constant level when calculating marginal contribution, for example, fixing the instrument operation standardization score to the median or a typical value of its actual observed values. "Changed" refers to adjusting the value of another variable while keeping the fixed variable unchanged, for example, changing the tissue treatment accuracy score from one percentile to another of its actual values. The change in a physiological recovery indicator refers to the numerical change in the physiological recovery indicator caused by a change in the operational score. The first change corresponds to the magnitude of change in the physiological recovery indicator when the tissue processing accuracy score is changed, and the second change corresponds to the magnitude of change in the physiological recovery indicator when the instrument operation standardization score is changed. Normalization transforms changes of different dimensions or ranges to the same scale (e.g., between 0 and 1), allowing direct comparison of two marginal contributions. The marginal contribution can be understood as the independent change in the physiological recovery indicator caused by each unit change in a particular operational score, assuming other conditions remain constant; the larger the value, the more significant the impact of that operational dimension on the patient's recovery. For example, suppose that after a certain surgery, the instrument operation standardization score is fixed at 80 points, and then the tissue handling accuracy score is changed from 70 points to 90 points. It is observed that physiological recovery indicators (such as postoperative inflammatory factor concentration) decrease from 10 units to 6 units, and the first change is a decrease of 4 units. Similarly, by fixing the tissue handling accuracy score and changing the instrument operation standardization score, a second change is obtained. After normalization, if the first change is larger, it indicates that the tissue handling accuracy score has a greater impact on recovery than the instrument operation standardization score.

[0061] In conventional assessment methods, instrument handling standardization scores and tissue processing precision scores are often directly weighted and summed to obtain a composite score. This approach implicitly assumes that the two scores have independent effects on postoperative recovery and that their contribution ratios remain constant. However, in actual surgery, there are complex interactions between the two. For example, when instrument handling is highly standardized, minor deviations in tissue processing may not lead to significant recovery problems; conversely, if instrument handling has deviated significantly from standard procedures, even high precision in tissue processing cannot compensate for this. This interaction makes simple linear weighting inaccurate in reflecting the true contribution of each dimension. This proposed approach uses a non-parametric structural causal equation model to estimate the independent effects of each score by fixing one score and changing the other, essentially controlling for confounding variables in an experimental design and observing the net effect of the target variable. This method mathematically separates the causal effects of each score on the recovery indicators, avoiding estimation bias caused by correlations between dependent variables.

[0062] Furthermore, normalization allows the two marginal contributions to be compared on the same scale. Assuming the normalized value of the first variation is 0.6 and the normalized value of the second variation is 0.3, this means that, all other things being equal, the impact of tissue handling precision score on patient recovery is approximately twice that of instrument handling standardization score. This information has significant clinical guidance value for surgeons: if a surgeon has significant deficiencies in tissue handling, prioritizing improvements in tissue handling precision will yield greater benefits to patient recovery than improving instrument handling standardization. Moreover, non-parametric models do not require the pre-assuming a linear relationship between scores and recovery indicators, thus capturing threshold or saturation effects. For example, the impact of instrument handling standardization score on recovery may only be significant when it falls below a certain threshold; beyond this threshold, the marginal benefit of further improving the standardization score gradually diminishes. This non-linear relationship is easily overlooked in parametric models, but non-parametric models can automatically learn this pattern from the data.

[0063] In this embodiment, the fixed-change operation is combined with a non-parametric structural causal equation. Unlike traditional randomized controlled trials, this method allows for post-operative causal inference based entirely on collected surgical data, without the need to deliberately alter the operation in actual surgery, thus not increasing the patient's surgical risk. Furthermore, since the marginal contribution is calculated from data from the same surgery, it reflects the unique contextual characteristics of that surgery, such as the complexity of the patient's anatomy and the type of surgical instruments. This gives the final comprehensive quality assessment index an individualized attribute, rather than simply an absolute score detached from the specific context. In this way, the assessment system can provide feedback to the surgeon: under the specific conditions of the surgery, whether the standardization of instrument trajectory or the precision of tissue interaction has a greater impact on patient recovery, thereby providing more targeted suggestions for behavioral modifications in subsequent surgeries.

[0064] In some embodiments of this application, step S3 further includes a stage quality assessment step: S31: The spatiotemporal feature set of surgical instrument-surgical area tissue interaction is divided according to surgical action units, and each surgical action unit corresponds to a complete cutting, suturing or hemostasis operation; S32: For each surgical action unit, calculate its sub-action score using a pre-trained motion quality assessment network; S33: Input the scores of each sub-action into a bidirectional long short-term memory network in chronological order, and output the overall quality evolution trend curve of the current surgical stage.

[0065] In this embodiment, the surgical action unit in step S31 refers to the smallest meaningful segment into which a continuous surgical process is divided according to the operation type and start and end times. For example, a cholecystectomy can be divided into multiple cutting action units (each time from incision to retraction), multiple hemostasis action units (each time from spot coagulation to cessation), and multiple suturing action units (each time from needle insertion to knot tying). The boundary of each action unit can be automatically determined by detecting the zero velocity point in the instrument pose data and the changes in the contact state between the instrument and tissue in the video frame. The action quality evaluation network in step S32 is a pre-trained deep learning model. Its input is a subset of spatiotemporal features within an action unit, and its output is the sub-action score corresponding to that action unit. The structure of this network can include convolutional layers and comparison modules to compare the motion features of the current action unit with a standard action template. The sub-action score is a numerical value, which can be set from 0 to 100. The higher the score, the closer the cutting, suturing, or hemostasis operation is to an expert level. The Bi-directional Long Short-Term Memory (BiLSTM) network in step S33 is a variant of a recurrent neural network. It contains two independent LSTM layers: one processes the sub-action scoring sequence forward in time, and the other processes the same sequence backward in time. The forward layer captures the evolutionary pattern from beginning to end, such as whether the score gradually decreases; the backward layer captures the dependencies from end to beginning, such as whether subsequent high-scoring actions depend on some preceding preparatory actions. The overall quality evolution trend curve is obtained by concatenating the outputs of the two LSTM layers and passing them through a fully connected layer. The horizontal axis of this curve represents the sequence number or time of the surgical action unit, and the vertical axis represents the predicted cumulative quality index. This curve can show whether the quality of the current surgical stage tends to improve, remain stable, or gradually deteriorate. For example, a large negative slope of the curve indicates a decline in surgical quality.

[0066] Step S31 of this application segments the continuous spatiotemporal feature set into discrete action units, which forms the basis for subsequent fine-grained evaluation. Without this step, only an average score would be output for the entire surgical procedure, which might mask errors in certain critical operations. For example, a surgeon might perform well in most cutting actions but make a significant mistake in a cut near a blood vessel. The overall average score might still be above 80, but the segmented sub-action score for that cut would be below 60. The segmentation operation relies on abrupt changes in instrument pose within the spatiotemporal feature set and changes in contact state in the tissue video frames. This segmentation method based on multimodal data is more accurate than simply using time-uniform partitioning.

[0067] Step S32 calculates a sub-action score independently for each action unit, which reflects the quality details of a single operation. However, the value of a single score is limited because surgical quality depends not only on the quality of individual actions but also on the overall trend of these actions over time. For example, the scores for the first three suturing actions are 85, 82, and 78, which, although within the good range each time, show a downward trend, potentially indicating surgeon fatigue or instrument malfunction. Conversely, a score sequence of 70, 75, and 80 shows an upward trend, suggesting the operator is adapting or improving. The bidirectional long short-term memory network introduced in step S33 is precisely to capture this temporal dependency. The forward LSTM layer can identify trend changes in the score sequence, such as a continuously decreasing slope; the reverse LSTM layer can establish the association between subsequent high-scoring actions and preceding actions, for example, a precise hemostasis action may depend on the tissue edge morphology left by the previous two cutting actions.

[0068] In this embodiment, the output of the bidirectional LSTM is plotted as an overall quality evolution trend curve, making the evaluation result no longer a static score but a dynamic curve. This curve can be updated in real time, and the system can trigger an early warning when the slope of the curve changes from positive to negative for more than three action units. Compared with only outputting the average score or the final score, this trend curve provides richer decision-making information. For example, if the curve shows a continuous decline in quality during the middle of the operation, the surgeon can adjust the operation strategy or pause for rest in time; during postoperative review, the trough area on the curve can be directly located to the specific action unit, which is convenient for teaching and improvement. Thus, steps S31 to S33 decompose the continuous surgical process into a meaningful sequence of action units, perform independent quantitative evaluation of each unit, and then synthesize the global evolution trend through a temporal network, realizing a complete characterization from microscopic operation quality to macroscopic process dynamics.

[0069] In some embodiments of this application, the motion quality assessment network includes a first convolutional module, a second convolutional module, and a comparison module; the first convolutional module is used to extract the velocity and acceleration features of the instrument movement in the current surgical motion unit; the second convolutional module is used to extract the strain rate features of tissue deformation in the same motion unit; the comparison module is used to calculate the first cosine similarity of the velocity and acceleration features relative to the standard motion template, and the second cosine similarity of the strain rate features relative to the standard tissue response template, and to use the weighted sum of the first cosine similarity and the second cosine similarity as the sub-motion score.

[0070] In this embodiment, both the first and second convolutional modules can be implemented using one-dimensional or two-dimensional convolutional layers, with the specific structure depending on the organization of the input data. For example, when velocity and acceleration features are organized as a time-series vector, the first convolutional module can contain two parallel one-dimensional convolutional branches: one branch extracts the velocity variation pattern over time, and the other branch extracts the acceleration variation pattern over time. Velocity and acceleration features reflect the dynamic characteristics of surgical actions. For instance, in cutting operations, excessive speed may lead to uneven cutting edges, while excessively slow speed may cause tissue thermal damage. Sudden acceleration changes indicate possible hand tremors or unstable force application by the operator. The second convolutional module is used to extract the strain rate features of tissue deformation. Strain rate can be understood as the relative degree of tissue deformation per unit time. For example, in hemostasis operations, when an instrument clamps a blood vessel, the change in the strain rate of the vessel wall reflects the magnitude and speed of the clamping force; in suturing operations, the local strain rate when the needle tip pierces the tissue can indicate whether the puncture was clean and precise. Standard action templates and standard tissue response templates are feature vector sets pre-extracted from a large amount of expert surgical data. They can be categorized and stored according to surgical type, surgical stage, and specific action unit category (such as cutting, suturing, hemostasis, and separation). For example, for the cystic duct transection action in cholecystectomy, the standard action template includes the velocity and acceleration curves under ideal conditions; the standard tissue response template includes the strain rate change curve of the cystic duct wall under this ideal operation. Cosine similarity is a measure of the consistency of the directions of two vectors, ranging from -1 to 1. The closer the value is to 1, the more consistent the directions of the two feature vectors are, meaning the higher the contour similarity between the actual operation and the standard template. The comparison module calculates the cosine similarity between the feature vector output by the first convolution module and the corresponding standard action template to obtain the first cosine similarity; it also calculates the cosine similarity between the feature vector output by the second convolution module and the corresponding standard tissue response template to obtain the second cosine similarity. The weighted sum is obtained by multiplying the first cosine similarity by a first weight coefficient and adding the second cosine similarity by a second weight coefficient; the sum of the two weight coefficients is 1. The specific values ​​of the first and second weighting coefficients can be adjusted according to the different requirements of instrument movement precision and tissue protection for different surgical action units. For example, when cutting near important nerves, the precision of tissue processing is more important, and the second weighting coefficient can be set higher in this case.

[0071] In this application, the first and second convolutional modules process two different but complementary information sources. The velocity and acceleration characteristics of the instrument movement describe the operator's active behavior, reflecting the quality of the action execution, such as whether the electrosurgical unit glides across the tissue surface at a uniform speed. The strain rate characteristics of tissue deformation describe the tissue's passive response to external forces, reflecting the quality of the action's outcome, such as whether excessive tensile deformation occurs on both sides of the tissue cutting edge after the electrosurgical unit passes through. Relying solely on velocity and acceleration characteristics is insufficient to determine whether the tissue has been subjected to tensile or compressive stress exceeding its tolerance range, because sometimes, even with normal instrument movement speed, excessive clamping force can lead to tissue tearing; similarly, relying solely on strain rate characteristics is insufficient to determine the efficiency of the action execution, because sometimes, even with small tissue deformation, repeated instrument dwell in the same position can cause unnecessary heat dissipation. The two convolutional modules work in parallel, extracting behavioral and response features respectively, enabling the subsequent comparison module to simultaneously evaluate from both execution and outcome dimensions.

[0072] The comparison module compares the extracted actual features with a standard template using cosine similarity, rather than directly regressing a score. The advantage of this design is that cosine similarity is insensitive to the absolute amplitude of features but sensitive to shape and contour, effectively distinguishing different surgical styles. For example, one surgeon's cutting speed is generally slow but the speed curve is smooth, while another surgeon's cutting speed is standard but exhibits localized jitter. Their speed amplitudes may differ, but their cosine similarities with the standard template objectively reflect whose movement contour is closer to the expert mode. Furthermore, combining similarity calculation with weighted summation allows for flexible adjustment of sub-action scores based on the characteristics of different surgical actions. For example, in delicate neuroanatomical movements, the strain rate characteristics of tissue deformation are more crucial for preventing nerve damage, thus assigning a higher weight to the second cosine similarity; while for actions involving rapid separation of loose tissue, the standardization of instrument movement may be more important, in which case the weight of the first cosine similarity can be increased.

[0073] There is also an interaction between the first and second cosine similarities. When the actual instrument motion characteristics are highly consistent with the standard template, but the tissue response characteristics are abnormal (e.g., the velocity curve is normal but the strain rate curve shows an abnormal peak), it indicates that although the operator's movements are standardized, there may be problems with the instrument model, energy settings, or the pathological state of the tissue itself. In this case, the sub-action score will be lowered due to the lower second cosine similarity, suggesting the need to check instrument parameters or tissue conditions, rather than simply attributing it to the operator's technique. Conversely, if the tissue response characteristics are normal but the instrument motion characteristics deviate from the template, it indicates that the operator may have insufficient skill or hand stability. By fusing the two similarities in a weighted sum, the sub-action score can comprehensively reflect the execution quality of the movement, the tissue protection effect, and the degree of matching between the two, providing more granular input for subsequent stage quality assessments and comprehensive quality assessment indices.

[0074] In some embodiments of this application, the process of generating the surgical operation space reference map in step S1 further includes: extracting multi-scale spatial features of preoperative three-dimensional medical image data using a three-dimensional convolutional layer in a first deep neural network; resampling the multi-scale spatial features using a spatial transformation layer to correct individual pose differences of the target object; and using an upsampling layer and a skip connection structure to output a probability map with the same resolution as the original image, marked with safety boundaries and key avoidance areas, as the surgical operation space reference map.

[0075] In this embodiment, the three-dimensional convolutional layer in the first deep neural network is a convolutional operation unit capable of simultaneously processing three-dimensional spatial information (i.e., length, width, and height directions). Unlike convolutional layers that process ordinary two-dimensional images, the three-dimensional convolutional layer slides the convolution kernel in three dimensions, thereby capturing the three-dimensional morphological features of anatomical structures in three-dimensional space, such as the continuous boundaries of liver tumors in the coronal, sagittal, and transverse planes. Multi-scale spatial features refer to the set of feature maps extracted from the same anatomical structure at different receptive field sizes. For example, for small-diameter vascular structures, small-scale features can preserve their fine contour information; while for larger organ boundaries, large-scale features can provide overall positional distribution information. The spatial transformation layer is a differentiable network module consisting of a localization network, a mesh generator, and a sampler. The localization network learns the mapping from the input feature map to the spatial transformation parameters; the mesh generator generates a sampling mesh based on the transformation parameters; and the sampler resamples the input feature map based on this mesh. The function of this layer is to correct for differences in anatomical posture caused by individualized factors such as patient positioning, respiratory amplitude, and natural organ drooping in three-dimensional medical image data. For example, two patients might have livers tilted at angles within their abdominal cavities that differ by 15 degrees before liver surgery. A spatial transformation layer can learn an affine transformation matrix to rotate the liver images of both patients to a unified reference pose. Individual pose differences refer to the inconsistencies in the position, orientation, and size of the same anatomical structure in the 3D image coordinate system caused by factors such as body size, lesion location, and intraoperative positioning between different target objects. Upsampling layers are used to progressively restore low-resolution feature maps to the resolution of the original image. Common upsampling methods include deconvolution or bilinear interpolation. Skip connection structures are operations that add or stitch together high-resolution feature maps from shallow networks with low-resolution feature maps from deep networks element-wise, compensating for spatial detail lost during upsampling. A safety boundary refers to a safe distance from important anatomical structures (such as major blood vessels, nerves, and bile ducts) during cutting, separation, or other operations, usually measured in millimeters. For example, an area within 5 millimeters of the main portal vein is marked as a high-risk area. Critical avoidance areas refer to anatomical locations where damage would lead to serious complications, such as the bile duct confluence at the porta hepatis or the superior mesenteric artery around the pancreas. Each voxel or pixel in the probability map has a value between 0 and 1, with values ​​closer to 1 indicating a higher confidence that the location belongs to a safe boundary or critical avoidance area. For example, in the output probability map, safe boundary areas can be set to a light or semi-transparent overlay, while critical avoidance areas can be set to a dark or flashing marker.

[0076] In this application, 3D convolutional layers and multi-scale spatial feature extraction together constitute the structured encoding of the original medical images. Using a single-scale convolutional kernel alone can only capture contextual information within a fixed range; small-scale extraction easily loses the overall layout, while large-scale extraction blurs fine boundaries. By extracting multi-scale features in parallel or cascaded manner, the network can simultaneously perceive the location of fine structures such as blood vessels as well as the overall contour of the organ. This multi-scale representation allows subsequent spatial transformation layers to determine the approximate location of organs based on large-scale features and accurately locate local anatomical landmarks based on small-scale features during pose correction, thereby improving the accuracy of resampling.

[0077] The spatial transformation layer and the skip connection structure form a complementary relationship. The spatial transformation layer resamples multi-scale features to eliminate individual pose differences, but the resampling process involves interpolation operations, which inevitably introduces a certain degree of blurring, especially for originally sharp boundary regions. The skip connection structure directly transmits high-resolution feature maps from shallow networks that have not undergone multiple downsampling and spatial transformations to deep networks, where they are fused with the upsampled feature maps. This operation can restore edge detail information lost due to spatial transformation while preserving pose correction effects. The final output probability map has both a uniform anatomical reference pose (facilitating subsequent intraoperative registration) and retains the sharp contours of safety boundaries and key avoidance areas in the original image.

[0078] The combination of upsampling layers and skip connections also addresses the output resolution issue. In typical encoder-decoder structures, the feature map size shrinks significantly after multiple downsampling operations, and directly upsampling back to the original resolution can produce checkerboard artifacts or jagged edges. By introducing skip connections from corresponding encoder layers at different upsampling stages, the network can progressively repair spatial details, ensuring that the final output probability map has the same resolution as the original 3D image. This high-resolution probability map is particularly important for intraoperative guidance, as safety boundaries may be only a few millimeters wide, and low-resolution output cannot provide sufficiently accurate positional cues. Overall, the 3D convolutional layers extract multi-scale features to provide input for spatial transformation, the spatial transformation layers perform pose normalization, and the upsampling layers and skip connection structures work together to restore resolution and output the probability map. These four components are sequentially linked, converting the original 3D image into a baseline reference map that can be directly used for intraoperative dynamic registration and navigation.

[0079] In some embodiments of this application, S7 further includes a visual feedback step: S71: In response to the grading assessment conclusion being lower than the first preset threshold, the spatiotemporal segment corresponding to the low score in the spatiotemporal feature set of the surgical instrument-surgical area tissue interaction is superimposed on the surgical display interface. S72: In the spatiotemporal segment, heatmaps are used to identify the trajectory segments where the instrument pose deviates from the safety boundary in the surgical operation space baseline map, as well as video frames where tissue processing exceeds the critical avoidance area. S73: Generate text prompts that advise surgeons to adjust instrument entry angles or tissue traction in subsequent surgical procedures.

[0080] In this embodiment, a graded assessment conclusion below a first preset threshold can be understood as the comprehensive quality assessment index or phased score failing to reach the minimum acceptable level set by the system. For example, the first preset threshold can be set to 60 points, and visual feedback is triggered when the score is below 60 points. The surgical display interface typically refers to the display area of ​​the intraoperative monitoring screen, head-mounted display device, or the operating console of the surgical robot. Overlay display refers to adding a layer of semi-transparent graphics or annotation information to the original surgical video or navigation image without obscuring the original surgical area view. A low-score spatiotemporal segment refers to a continuous interval extracted from the timeline of the entire surgical procedure, within which the instrument operation standardization score or tissue processing accuracy score is below a certain sub-threshold. For example, in a continuous 5-second suturing action, the score remains below 50 points. A heatmap is a visualization method that uses color depth or warm / cool tones to represent numerical values. For example, red can represent the position with the highest deviation, yellow represents moderate deviation, and blue represents small deviation. The trajectory segment where the instrument's position deviates from the safety boundary in the surgical operation space reference map refers to the portion of the trajectory that enters or approaches the safety boundary area after projecting the actual movement trajectory of the surgical instrument tip onto the reference map. For example, when the instrument tip is less than 3 mm from the portal vein wall, this trajectory segment is marked as a deviation. Video frames where tissue processing exceeds the critical avoidance area refer to frames in the video sequence where the interaction between the instrument and tissue falls within the critical avoidance area, such as the electrosurgical tip touching the bile duct confluence in the Calot's triangle. Text prompts are suggestions presented in natural language and can be displayed in a fixed area on the screen or appear as a pop-up. Correcting the instrument's entry angle refers to changing the puncture or cutting direction of the instrument relative to the tissue surface; adjusting the tissue traction force refers to changing the force applied to the tissue by the traction hook or grasping forceps. For example, the prompt could suggest shifting the grasping forceps 10 degrees to the left and reducing the traction force by approximately 20%. The specific parameters for these recommendations can be automatically calculated and generated by the system based on the degree and direction of the deviation. For example, the required angular offset can be calculated based on the relative orientation of the instrument trajectory and the safety boundary.

[0081] In this application, the overlay display in S71 and the heatmap identification in S72 together construct a multi-layered visual feedback mechanism. Simply overlaying low-scoring spatiotemporal segments only tells the surgeon when the problem occurred, but not the specific location or procedure in which the problem occurred. The heatmap in S72, building on this, locates the problem to spatial coordinates and specific frames, allowing the surgeon to intuitively see at which anatomical location the instrument trajectory deviated from the safety boundary, and in which frame the tissue manipulation intruded into the critical avoidance area. The color intensity of the heatmap also reflects the severity of the deviation; red areas indicate immediate attention, while blue areas indicate minor deviations. This progressive display, from temporal to spatial location, reduces the time cost for doctors to find the problem.

[0082] The heatmap in S72, which provides parallel identification of instrument position deviations and tissue handling exceeding limits, complements the text prompts generated in S73, offering both visual and semantic information. While the heatmap visually presents the distribution of deviations, physicians may not immediately understand the specific corrective actions corresponding to the red areas. For example, seeing a red trajectory segment, the physician needs to determine whether to adjust the instrument angle to the left or right. The text prompts in S73 translate the spatial deviations reflected in the heatmap into specific, actionable instructions. For instance, based on the relative position of the red trajectory segment and the safety boundary, it suggests shifting the instrument tip 2 mm to the right. Simultaneously, the text prompts can generate specific suggestions for adjusting the traction force based on the relative posture of the instrument and tissue in the video frame of tissue handling exceeding limits, such as reducing the traction angle by 15 degrees and lowering the force to below 1 Newton. The visual heatmap provides intuitive spatial perception, while the text prompts offer actionable corrective solutions; the combination of both reduces the cognitive load on physicians in emergency situations.

[0083] The trigger condition set in S71 (the graded assessment conclusion is below the first preset threshold) ensures that visual feedback does not frequently disrupt the surgical process. Feedback is only activated when the score reaches the warning level, avoiding unnecessary screen interference during high-quality surgical procedures. S72 simultaneously identifies both instrument deviation segments and tissue overrun video frames, covering anomalies at both the action execution and result response levels. This allows feedback to address both operator behavioral deviations and adverse tissue effects. For example, if the instrument trajectory is entirely within the safety boundaries, but the tissue handling video frame shows unnecessary traction deformation in the critical avoidance area, the heatmap will mark the deformed area on the video frame, and the text prompt will suggest adjusting the traction point position. This dual-identification mechanism allows feedback to adapt to different types of quality issues, improving the applicability and accuracy of visual feedback.

[0084] Some embodiments of this application also include a model incremental update step: After each surgery is completed, the instrument operation standardization score, tissue processing accuracy score, comprehensive quality assessment index, and corresponding real-time pose data and high-definition video frame sequence generated in this surgery are used as an incremental training sample. Store incremental training samples into the recurrent experience replay buffer; When the number of samples in the loop experience replay buffer reaches the batch threshold, a batch of samples is randomly sampled from the buffer to jointly fine-tune the second deep neural network and the causal inference model.

[0085] In this embodiment, the instrument operation standardization score, tissue processing accuracy score, and comprehensive quality assessment index generated after each surgery, along with the corresponding real-time pose data and high-definition video frame sequence, constitute an incremental training sample. The real-time pose data and high-definition video frame sequence serve as input features to the model, while the three scores (instrument operation standardization score, tissue processing accuracy score, and comprehensive quality assessment index) can serve as labels or auxiliary information in supervised learning. The cyclic experience replay buffer is a storage structure with a fixed capacity. When a new sample is added, if the buffer is full, the oldest stored sample is removed according to the first-in, first-out rule, ensuring that the buffer always retains data from the most recently completed surgeries. For example, the buffer capacity can be set to 500 samples, corresponding to data from the most recent 500 surgeries. The batch threshold is a preset value used to control when the model update operation is triggered. For example, the batch threshold can be set to 32; when the number of accumulated samples in the buffer reaches 32, the system automatically initiates a joint fine-tuning. Random sampling refers to uniformly and randomly selecting a batch of samples from the buffer, rather than selecting the most recent data in chronological order. This avoids the model overfitting the distribution characteristics of recent surgical data. Joint fine-tuning refers to updating the parameters of both the second deep neural network and the causal inference model simultaneously, rather than training them separately. The difference between fine-tuning and training from scratch is that fine-tuning updates the pre-trained model weights using new sample data with small steps of gradient descent, typically employing a small learning rate (e.g., one-tenth of the initial learning rate) to avoid destroying the general features already learned by the model.

[0086] In this application, the three scores generated from the current surgery are bundled with the corresponding input data (pose data and video frame sequences) into an incremental training sample, establishing a complete mapping relationship between input and output. If only the raw data is stored without storing the scores, the subsequent fine-tuning process will lack supervision; conversely, if only the scores are stored, supervised parameter updates of the model are impossible. Storing both in a buffer ensures that each sample contains a complete input-output pair, providing a data foundation for subsequent random sampling and joint fine-tuning. Furthermore, the three scores themselves are calculated under the current model; as the model is continuously fine-tuned, the scores of new samples will gradually change. This self-generated and self-used data stream allows the model to continuously adapt to distribution changes in the actual surgical environment.

[0087] The combination of a recurrent experience replay buffer and random sampling addresses two common problems in incremental learning: catastrophic forgetting and inefficient sampling. If only the latest surgical data is used for each update, the model quickly forgets feature patterns learned from earlier surgeries, leading to a decline in its evaluation ability on older surgical types. The recurrent buffer, by retaining a certain number of historical samples, ensures that each update batch includes both newer and older surgical data (as long as the buffer is not completely replaced). Random sampling further shuffles the temporal order, preventing the model from learning spurious correlations between surgery sequence numbers and scores. For example, if all surgeries performed within a certain period are cholecystectomies, the model might incorrectly associate cholecystectomy features with high scores. Random sampling allows the model to still access other types of surgical samples (such as liver resections and pancreatic surgeries) during updates, thus maintaining the model's generalization ability across different surgical procedures.

[0088] The batch threshold setting and the joint fine-tuning trigger mechanism work together to balance update frequency and computational cost. If updates are performed immediately after each surgery, while the model can quickly respond to changes in surgical style, frequent parameter updates consume significant computational resources, and the large variance in gradient estimation for a single sample can lead to training instability. By setting a batch threshold, the system accumulates a certain number of samples before updating, reducing computational frequency and allowing multiple samples within a batch to average gradient noise, making the update direction more stable. Joint fine-tuning updates both the second deep neural network and the causal inference model simultaneously because the two models are functionally cascaded: the two scores output by the second deep neural network are the inputs to the causal inference model. Updating only one while fixing the other would lead to a mismatch between the models. For example, after a period of incremental updates, the distribution of operation scores by the second deep neural network may change. If the causal inference model still estimates the marginal contribution based on the old score distribution, the calculation results will be biased. Joint fine-tuning allows the two models to evolve collaboratively, maintaining consistency throughout the evaluation chain.

[0089] In some embodiments of this application, the first deep neural network in step S1 and the second deep neural network in step S4 are pre-trained using a contrastive learning approach: constructing positive sample pairs, which include instrument pose subsequences and corresponding tissue video sub-segments under the same surgical operation; constructing negative sample pairs, which include instrument pose subsequences and tissue video sub-segments under different surgical operations or with temporal misalignment; obtaining pre-trained feature extractor weights by minimizing the feature distance of the positive sample pairs and maximizing the feature distance of the negative sample pairs, and initializing at least a portion of the layers of the first deep neural network and the second deep neural network with these weights respectively.

[0090] In this embodiment, a positive sample pair is a data pair consisting of an instrument pose sub-sequence acquired under the same surgical operation and a corresponding tissue video sub-segment. For example, during a cutting action, within a time window from the start to the end of the action, a sequence of instrument pose data and a continuous sequence of tissue video frames from the surgical area are simultaneously captured. These two data points originate from the same spatiotemporal context and are therefore considered a positive sample pair. Negative sample pairs can be constructed in two ways: one is by pairing instrument pose sub-sequences and tissue video sub-segments under different surgical operations, such as pairing the pose data of a suturing action in the first surgery with tissue video segments of different suturing actions in the second surgery; the other is by pairing the same surgical operation but with temporal misalignment, such as pairing the pose data of a certain cutting action with a tissue video segment from another unrelated time period within the same surgery. Feature distance is a measure of the similarity between two samples in the feature space after mapping by a feature extractor, typically using Euclidean distance or cosine distance. Contrastive learning is a self-supervised or semi-supervised learning paradigm that aims to enable the feature extractor to learn discriminative features that distinguish different samples by narrowing the distance between positive sample pairs and widening the distance between negative sample pairs in the feature space. Pre-training refers to the process of initially training the network on large-scale unlabeled or weakly labeled data before formally training it on the target task (such as surgical quality score regression). Feature extractor weights are the trainable parameters of the layers in a deep neural network responsible for converting the raw input data into a high-dimensional feature representation. These typically include the convolutional kernel parameters of convolutional layers and the matrix parameters of fully connected layers. Initialization refers to using the pre-trained feature extractor weights as the initial parameter values ​​for corresponding layers in the first and second deep neural networks, rather than random initialization. At least some layers can be understood as allowing only the parameters of the feature extraction part to be replaced, while retaining the random initialization of the task-specific heads (such as the regression heads). For example, for the first deep neural network, its 3D convolutional layers and spatial transformation layers can be initialized with pre-trained weights; for the second deep neural network, its shared feature extraction backbone can be initialized with pre-trained weights, while the two regression heads still use random initialization.

[0091] In this application, the construction methods of positive and negative sample pairs jointly define the objective of contrastive learning. If only positive sample pairs are constructed without constructing negative sample pairs, the network tends to map all samples to the same point in the feature space, as this minimizes the distance between all sample pairs, but this leads to a loss of discriminative power in the features. The introduction of negative sample pairs forces the network to focus on the differences between different samples, thereby learning meaningful features that can distinguish between different surgical operations or different temporal contexts. Specifically, in the scenario of this application, positive sample pairs require the network to identify the inherent consistency between instrument pose and tissue video, that is, when pose data and video data come from the same operation, their feature representations should be close; negative sample pairs require the network to be able to distinguish between different operations or misaligned pairings, that is, when pose data and video data come from different operations or are temporally mismatched, their feature representations should be far apart. This design enables the pre-trained feature extractor to automatically align the instrument motion modality with the tissue image modality, laying a good foundation for subsequent spatiotemporal registration and scoring calculation.

[0092] There is a synergistic relationship between contrastive learning pre-training and subsequent joint fine-tuning. Without a large amount of labeled scoring data, directly training the first and second deep neural networks can easily lead to overfitting, as surgical quality scoring is expensive expert-annotated information. Contrastive learning utilizes unlabeled surgical data (only needing to know which poses and videos belong to the same operation, not the scoring), which is abundant and easily accessible in surgical records. Through contrastive learning, the network first learns how to correlate instrument movement with tissue deformation across modalities; this ability is fundamental for subsequent tasks (such as dynamic registration and standardized scoring). After pre-training, the feature extractor weights are transferred to the first and second deep neural networks. At this point, the network already possesses good modal alignment capabilities. Fine-tuning with a small amount of labeled data on the regression head and task-specific layers achieves high scoring accuracy. This pre-training plus fine-tuning paradigm reduces dependence on the amount of expert-annotated data and shortens the training time before deployment.

[0093] The minimization and maximization of feature distance are coordinated with the weight initialization method. During the contrastive learning phase, by minimizing the feature distance of positive sample pairs, the network learns to map instrument poses and tissue videos of the same operation to similar locations in the feature space; by maximizing the feature distance of negative sample pairs, the network learns to map different operations or misaligned pairs to distant locations. Through this process, the feature vectors output by the feature extractor already carry information about the operation type, action stage, and modal correspondence. When these weights are used to initialize the first and second deep neural networks, both networks possess better feature representation capabilities from their initial state than with random initialization. For example, the first deep neural network, which would normally need to learn from scratch how to extract anatomical features from 3D images, can directly utilize its learned cross-modal alignment capabilities to aid in understanding the spatial structure in the images by sharing the feature extractor weights obtained through contrastive learning. Simultaneously, the first and second deep neural networks share the same pre-trained feature extractor initialization, making it easier for the two networks to maintain consistency in the feature space during subsequent training, which is beneficial for end-to-end optimization of the entire evaluation process.

[0094] In some embodiments of this application, the formula for calculating the comprehensive quality assessment index E is as follows: E = α•(w1•S_instr + w2•S_tissue) + (1-α)•(w3•ΔR) Wherein, S_instr is the instrument operation standardization score, S_tissue is the tissue processing accuracy score, ΔR is the change in physiological recovery index relative to historical baseline data, α is the balance weight coefficient, which is negatively correlated with the complexity level of the current surgery, and w1, w2, and w3 are the dynamic weight coefficients learned through the causal inference model.

[0095] In this embodiment, the instrument operation standardization score S_instr is a numerical value, ranging from 0 to 100, reflecting the degree to which the surgical instrument conforms to the standard operating procedure in terms of spatial trajectory, speed stability, and entry angle. The tissue processing accuracy score S_tissue is also a numerical value ranging from 0 to 100, reflecting the degree of protection of tissue structures during instrument-tissue interaction, such as the width of thermal damage at the cutting edge and the extent of eschar during hemostasis. The change in physiological recovery indicators relative to historical baseline data ΔR refers to the difference between a physiological indicator measured at a certain postoperative moment (such as serum C-reactive protein concentration, liver function enzyme indicators, etc.) and the patient's preoperative or historical baseline data in healthy individuals. For example, if the normal value of C-reactive protein on postoperative day 1 in historical baseline data is 10 mg / L, while the patient's measured value is 25 mg / L, then ΔR is 15 mg / L. To unify the dimensions, ΔR can be standardized so that its value range is approximately equivalent to S_instr and S_tissue. Historical baseline data can be obtained by retrieving postoperative recovery data from similar cases (same surgical type, similar age, and similar underlying disease status) in the hospital's electronic medical record system. The balancing weight coefficient α is a value between 0 and 1, used to adjust the relative weight of intraoperative procedural scores and postoperative predicted changes in the final composite index. α is negatively correlated with the complexity level of the current surgery, meaning that the more complex the surgery, the smaller the value of α, and the greater the weight (1-α) of the postoperative recovery prediction component in the composite index; the simpler the surgery, the larger the value of α, and the higher the weight of the intraoperative procedural score. The complexity level can be pre-classified based on factors such as surgical type, number of comorbidities, and expected intraoperative blood loss. For example, hepatobiliary and pancreatic surgeries are classified as high complexity (α = 0.3), cholecystectomy as medium complexity (α = 0.6), and superficial tumor resection as low complexity (α = 0.8). The dynamic weight coefficients w1, w2, and w3 are learned by the causal inference model; they are not fixed constants but are automatically adjusted as the model is trained on different surgical types and patient groups. For example, w1, w2, and w3 satisfy the normalization condition, i.e., the sum of the three is 1, or they are independent but scaled by the output of the causal inference model. The dynamic weight coefficients reflect the relative importance of instrument operation standardization, tissue processing precision, and postoperative recovery changes to the final surgical quality in different clinical scenarios.

[0096] In this application, the negative correlation between α and the level of surgical complexity in the calculation formula enables adaptive adjustment of the assessment strategy. For complex surgeries, postoperative recovery is often affected by more uncontrollable factors (such as underlying diseases and surgical trauma stress). Even with highly standardized intraoperative procedures, a good postoperative recovery cannot be fully guaranteed. Therefore, the weight of intraoperative scores is appropriately reduced, while the weight of postoperative predictive changes is increased, making the comprehensive index more reflective of actual clinical outcomes. For simple surgeries, patient recovery is usually smoother, and the changes in postoperative indicators are smaller. In this case, intraoperative procedural scores can more sensitively distinguish differences between different procedural levels, thus increasing the weight of intraoperative scores. This design avoids using the same set of weighting coefficients for all surgical types, making the assessment results more closely aligned with the risk characteristics of different surgeries.

[0097] The coupling relationship between the dynamic weight coefficients w1, w2, and w3 and the causal inference model makes the values ​​of these three coefficients clinically interpretable. Traditional fixed weights (e.g., assuming that instrument operation standardization accounts for 40% of the total score, tissue processing precision accounts for 40%, and postoperative indicators account for 20%) cannot reflect the true impact of different operational dimensions on recovery indicators. By learning dynamic weight coefficients through the causal inference model, the system can automatically identify whether, under the current surgical type and patient characteristics, instrument operation standardization or tissue processing precision has a greater impact on recovery. For example, for a liver resection, the causal inference model may find that the marginal contribution of the tissue processing precision score (reflecting whether the resection margin is neat and whether the blood supply to the remaining liver is good) to postoperative liver function recovery is much greater than that of the instrument operation standardization score; therefore, the learned w2 will be significantly higher than w1. This data-driven weight allocation method is more objective and adaptable than manual pre-setting.

[0098] S_instr and S_tissue are first weighted and summed using their respective dynamic weights w1 and w2, and then fused with ΔR through a balancing coefficient α, forming a two-layer fusion structure. The first layer of fusion (w1·S_instr + w2·S_tissue) combines the scores of the two operational dimensions according to the importance obtained from causal inference to obtain a comprehensive intraoperative operational score. The second layer of fusion then combines the comprehensive intraoperative operational score with the postoperative predicted change according to the complexity of the surgery. This two-layer structure is more flexible than a single-layer weighted sum because the proportions within the intraoperative operational score (the relationship between w1 and w2) and the proportions between intraoperative and postoperative (the relationship between α and 1-α) can be adjusted independently. For example, in highly complex surgeries, α is smaller, meaning it relies more on the postoperative predicted change, but at the same time, w2 may take a larger value because tissue processing is crucial to complex surgeries. This mutually independent adjustment capability allows the comprehensive quality assessment index to finely adapt to different clinical scenarios, improving the accuracy and clinical acceptability of the assessment results.

[0099] A second aspect of this application provides a deep learning-based surgical quality assessment system for executing a corresponding surgical quality assessment method. The system includes: The preoperative planning unit is used to acquire preoperative three-dimensional medical image data of the target object, and based on the preoperative three-dimensional medical image data, it uses a first deep neural network to generate a surgical operation space reference map of the anatomical structure. The intraoperative acquisition unit is used to continuously acquire real-time pose data of surgical instruments and high-definition video frame sequences of surgical area tissues during the operation. The spatiotemporal registration unit is used to dynamically register real-time pose data with high-definition video frame sequences based on the surgical operation space reference map using a spatiotemporal attention mechanism, so as to construct a spatiotemporal feature set of surgical instruments-surgical area tissue interaction. The stage evaluation unit is used to calculate the instrument operation standardization score and tissue processing accuracy score of the current surgical stage based on the spatiotemporal feature set of surgical instrument-surgical area tissue interaction and the second deep neural network. The causal analysis unit is used to obtain at least one physiological recovery indicator of the target subject in the short term after surgery, and input the instrument operation standardization score, tissue processing accuracy score and physiological recovery indicator into the preset causal inference model; through the causal inference model, the marginal contribution of the instrument operation standardization score and tissue processing accuracy score to the physiological recovery indicator is quantified, and a comprehensive quality assessment index is generated. The conclusion output unit is used to output a graded evaluation conclusion corresponding to the surgical operation behavior based on the comparison result between the comprehensive quality assessment index and the preset multi-level quality threshold. And control processors connected to each of the above units, used to coordinate the working timing and data transmission of each unit.

[0100] A third aspect of this application provides an electronic device, including: one or more processors, one or more input devices, one or more output devices, and one or more memories. The processors, input devices, output devices, and memories communicate with each other via a communication bus. The memories store computer programs, including program instructions. The processors execute the program instructions stored in the memories. The processors are configured to invoke the program instructions to execute the aforementioned deep learning-based surgical quality assessment method.

[0101] It should be understood that, in the embodiments of this application, the processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0102] Input devices may include touchpads, fingerprint sensors (for collecting the user's fingerprint information and fingerprint orientation information), microphones, etc., while output devices may include displays (LCDs, etc.), speakers, etc.

[0103] The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store information about the device type.

[0104] In specific implementations, the processor, input device, and output device described in the embodiments of this application can execute the implementation methods described in any embodiment of the deep learning-based surgical quality assessment method provided in the embodiments of this application, or they can execute the implementation methods of the electronic devices described in the embodiments of this application, which will not be repeated here.

[0105] In another embodiment of this application, an electronic device is provided. The electronic device stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the deep learning-based surgical quality assessment method described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in an electronic device, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0106] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0107] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0108] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A surgical quality assessment method based on deep learning, characterized in that, Includes the following steps: S1: Obtain preoperative three-dimensional medical image data of the target object, and based on the preoperative three-dimensional medical image data, generate a surgical operation space reference map of the anatomical structure using a first deep neural network; S2: During the surgical procedure, real-time positional data of surgical instruments and high-definition video frame sequences of the surgical area tissue are collected. S3: Based on the surgical operation space reference map, a spatiotemporal attention mechanism is used to dynamically register the real-time pose data with the high-definition video frame sequence to construct a spatiotemporal feature set of surgical instruments-surgical area tissue interaction. S4: Based on the spatiotemporal feature set of the surgical instrument-surgical area tissue interaction, the instrument operation standardization score and tissue processing accuracy score of the current surgical stage are calculated using the second deep neural network. S5: After the surgery, obtain at least one physiological recovery indicator of the target subject in the short term after the surgery, and input the instrument operation standardization score, tissue processing accuracy score and the physiological recovery indicator into the preset causal inference model; S6: Using the causal inference model, quantify the marginal contribution of the instrument operation standardization score and the tissue processing accuracy score to the physiological recovery indicators, and generate a comprehensive quality assessment index. S7: Based on the comparison result between the comprehensive quality assessment index and the preset multi-level quality threshold, output the graded assessment conclusion corresponding to the operation behavior of the surgery.

2. The method according to claim 1, characterized in that, The dynamic registration in step S3 specifically includes: Based on the high-definition video frame sequence, the first motion trajectory of the instrument tip in the tissue coordinate system is extracted; Based on the real-time pose data, the second motion trajectory of the instrument end effector in the coordinate system of the positioning device is obtained; A differentiable spatial transformation network is used to learn the affine transformation parameters from the positioning device coordinate system to the tissue coordinate system, so as to minimize the dynamic time warping distance between the second motion trajectory and the first motion trajectory.

3. The method according to claim 1, characterized in that, The second deep neural network is a multi-task learning network. The multi-task learning network shares a feature extraction backbone and branches into a first regression head and a second regression head. The first regression head is used to output the instrument operation standardization score, and the second regression head is used to output the tissue processing accuracy score. The loss function of the multi-task learning network is the sum of the mean square error of the instrument operation standardization label, the mean square error of the tissue processing accuracy label, and the maximum mean difference penalty term of the feature vectors of the two regression heads.

4. The method according to claim 1, characterized in that, The causal inference model is a non-parametric model based on structural causal equations. The process of calculating the marginal contribution using the non-parametric model includes: fixing the instrument operation standardization score, changing the tissue treatment accuracy score, and observing the first change in the physiological recovery index; fixing the tissue treatment accuracy score, changing the instrument operation standardization score, and observing the second change in the physiological recovery index; normalizing the first change and the second change, and using them as the marginal contributions of the instrument operation standardization score and the tissue treatment accuracy score, respectively.

5. The method according to claim 1, characterized in that, Step S3 further includes a phased quality assessment step: S31: Divide the spatiotemporal feature set of the surgical instrument-surgical area tissue interaction according to the surgical action unit, and each surgical action unit corresponds to a complete cutting, suturing or hemostasis operation; S32: For each surgical action unit, calculate its sub-action score using a pre-trained motion quality assessment network; S33: Input the scores of each sub-action into a bidirectional long short-term memory network in chronological order, and output the overall quality evolution trend curve of the current surgical stage.

6. The method according to claim 5, characterized in that, The motion quality assessment network includes a first convolutional module, a second convolutional module, and a comparison module. The first convolutional module is used to extract the velocity and acceleration features of the instrument movement in the current surgical motion unit. The second convolutional module is used to extract the strain rate features of tissue deformation in the same motion unit. The comparison module is used to calculate the first cosine similarity of the velocity and acceleration features relative to a standard motion template, and the second cosine similarity of the strain rate features relative to a standard tissue response template, and the weighted sum of the first cosine similarity and the second cosine similarity is used as the sub-motion score.

7. The method according to claim 1, characterized in that, The process of generating the surgical operation space reference map in step S1 further includes: extracting multi-scale spatial features of the preoperative three-dimensional medical image data using the three-dimensional convolutional layer in the first deep neural network; resampling the multi-scale spatial features using a spatial transformation layer to correct individual pose differences of the target object; and outputting a probability map with the same resolution as the original image and marked with safety boundaries and key avoidance areas using an upsampling layer and a skip connection structure, as the surgical operation space reference map.

8. The method according to claim 1, characterized in that, The S7 also includes a visual feedback step: S71: In response to the grading assessment conclusion being lower than the first preset threshold, the spatiotemporal segment corresponding to the low score in the spatiotemporal feature set of the surgical instrument-surgical area tissue interaction is superimposed on the surgical display interface. S72: In the spatiotemporal segment, a heat map is used to identify the trajectory segment where the instrument pose deviates from the safety boundary in the surgical operation space reference map, as well as video frames where tissue processing exceeds the critical avoidance area. S73: Generate text prompts, which are used to advise the surgeon to adjust the entry angle of instruments or adjust tissue traction in subsequent surgical steps.

9. The method according to claim 1, characterized in that, It also includes the incremental model update step: After each surgery is completed, the instrument operation standardization score, tissue processing accuracy score, comprehensive quality assessment index, and corresponding real-time pose data and high-definition video frame sequence generated in this surgery are used as an incremental training sample. The incremental training samples are stored in the cyclic experience replay buffer. When the number of samples in the recurrent experience replay buffer reaches the batch threshold, a batch of samples is randomly sampled from the buffer to jointly fine-tune the second deep neural network and the causal inference model.

10. A deep learning-based surgical quality assessment system for implementing the surgical quality assessment method according to any one of claims 1-9, characterized in that, include: The preoperative planning unit is used to acquire preoperative three-dimensional medical image data of the target object, and based on the preoperative three-dimensional medical image data, to generate a surgical operation space reference map of the anatomical structure using a first deep neural network. The intraoperative acquisition unit is used to continuously acquire real-time pose data of surgical instruments and high-definition video frame sequences of surgical area tissues during the operation. The spatiotemporal registration unit is used to dynamically register the real-time pose data with the high-definition video frame sequence based on the surgical operation space reference map using a spatiotemporal attention mechanism, so as to construct a spatiotemporal feature set of surgical instrument-surgical area tissue interaction. The stage evaluation unit is used to calculate the instrument operation standardization score and tissue processing accuracy score of the current surgical stage based on the spatiotemporal feature set of the surgical instrument-surgical area tissue interaction and using a second deep neural network. The causal analysis unit is used to obtain at least one physiological recovery indicator of the target subject in the short term after surgery, and input the instrument operation standardization score, tissue processing accuracy score and the physiological recovery indicator into a preset causal inference model; through the causal inference model, the marginal contribution of the instrument operation standardization score and the tissue processing accuracy score to the physiological recovery indicator is quantified, and a comprehensive quality assessment index is generated. The conclusion output unit is used to output a graded evaluation conclusion corresponding to the operation behavior of the surgery based on the comparison result between the comprehensive quality evaluation index and the preset multi-level quality threshold. And control processors connected to each of the above units, used to coordinate the working timing and data transmission of each unit.

Images