Intelligent evaluation method for surgical skill based on endoscope video

Abstract

The application provides a kind of intelligent evaluation method of surgical skill based on endoscope video, it is related to artificial intelligence technical field.The application proposes a complete intelligent evaluation process from original endoscope surgery video to quantitative skill score: standardization preprocessing and automatically eliminating irrelevant segment frame;Adopt self-supervised field fine-tuning to obtain frame-level feature representation adapted to different operation types and equipment conditions;On this basis, multi-scale feature pyramid and channel-space attention are used to realize frame-level identification of multiple types of abnormalities such as bleeding, smoke and poor exposure of operation field, and then the timing coding network based on selective state space model is used to timing encode the abnormal process to extract event-level timing features;Finally, the features are jointly modeled with structured metadata including operation type, and the normalized skill score and corresponding grade are output.The application greatly improves the accuracy, generalizability and interpretability of intelligent evaluation of surgical skill through unified quantitative framework and adaptive fusion mechanism.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an intelligent assessment method for surgical skills based on endoscopic video. Background Technology

[0002] The assessment of surgical skills is crucial in medical training and quality control. With the increasing prevalence of minimally invasive surgery, endoscopic video has become an important medium for recording surgical procedures, providing a data foundation for the objective and accurate quantification of surgical skills.

[0003] In the overall skills assessment stage, existing solutions mostly construct regression or classification models around single-type features (such as bleeding-related features or tool motion features). For example, Chinese patent application CN119317972A discloses a video-based surgical skills assessment method using tool tracking. During the operation, the method receives a tool motion trajectory, which includes a sequence of detected tool movements by a surgeon performing surgery using surgical tools. The method then generates a sequence of multi-channel feature matrices to mathematically represent the tool motion trajectory. Next, the method performs a one-dimensional (1D) convolution operation on the sequence of multi-channel feature matrices to generate a sequence of context-aware multi-channel feature representations of the tool motion trajectory, which is then processed by a converter model to generate the skills classification.

[0004] It is evident that the skill assessment of related technologies uses a single modeling approach for multi-source information and technique differences, merely introducing multi-source information and technique differences as scattered additional variables, without forming a unified quantitative and fusion framework. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides an intelligent assessment method for surgical skills based on endoscopic video, which solves the technical problem of the single modeling method for skill scoring based on multi-source information and surgical procedure differences.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for intelligent assessment of surgical skills based on endoscopic video, comprising:

[0010] Acquire and standardize preprocessed endoscopic surgical videos, and smoothly identify effective frame sequences by combining a discriminative model with a time window; and acquire structured metadata that includes at least the surgical procedure type;

[0011] Based on each valid frame and its timestamp, the feature extraction network is fine-tuned in a self-supervised domain until convergence; the feature extraction network is a deep neural network with multi-level feature representation capabilities.

[0012] Each valid frame is used as input to the converged feature extraction network, and several feature maps with different resolutions are extracted from different layers of the network. The feature maps are used as input to a multi-channel discriminator that combines a multi-scale feature pyramid and a channel-space attention mechanism to obtain the corresponding multi-scale integrated features and frame-level anomaly probabilities.

[0013] Multi-scale integrated features, frame-level anomaly probabilities, probability differences, and normalized time of each effective frame are spliced ​​together to obtain a fusion vector that corresponds one-to-one with each anomaly category; and the fusion vector is used as the input of a temporal coding network based on a selective state space model, and after context correction, event-level temporal features are extracted in each anomaly interval.

[0014] Based on the event-level temporal features of each anomaly category, anomaly statistical features are obtained, and a comprehensive feature vector is constructed in conjunction with structured metadata. The comprehensive feature vector is then used as the input to each base learner, and the outputs of each base learner are weighted and fused through a gating network with technique type as input to obtain a normalized skill score and the corresponding skill level.

[0015] Preferably, the step of smoothly identifying the effective frame sequence by combining a discriminative model with a time window includes:

[0016] The preprocessed normalized frames are converted into luminance maps, and the global luminance mean, luminance variance, average gradient intensity, and edge pixel ratio are defined based on the luminance maps to construct frame-level appearance feature vectors.

[0017] The frame-level appearance feature vectors of each standardized frame are used as input to the discrimination model to predict the probability that each standardized frame belongs to an irrelevant frame.

[0018] Smooth the predicted irrelevant frame probabilities over a time window, calculate the smoothed irrelevant frame probabilities, and define the corresponding validity mask.

[0019] Based on each validity mask, obtain the set of valid frame indices and sort the set in chronological order to define the valid frame sequence.

[0020] Preferably, the step of performing self-supervised domain fine-tuning of the feature extraction network until convergence based on each valid frame and its timestamp includes:

[0021] The feature extraction network to be fine-tuned is used as the student network, and a teacher network with the same structure is constructed.

[0022] For each valid frame, two different augmented views are generated and used as inputs to the student network and teacher network, respectively, to obtain the corresponding embedding representations;

[0023] Temporal weights are constructed using the timestamps of each valid frame, and the embedded representation output by the teacher network is used as a pseudo-label to construct a distribution consistency loss. A temporal smoothing regularization term is constructed based on the change magnitude of the embedded representation output by the student network in adjacent frames. An alignment loss is constructed based on the activity criteria of each valid frame. Finally, the total loss for self-supervised training is constructed based on the distribution consistency loss, the temporal smoothing regularization term, and the alignment loss.

[0024] The total loss is used to fine-tune the student network under self-supervision until convergence.

[0025] Preferably, the step of using the feature map as input to a multi-channel discriminator combining a multi-scale feature pyramid and a channel-space attention mechanism to obtain the corresponding multi-scale integrated features and frame-level anomaly probabilities includes:

[0026] Based on the extracted feature maps of different resolutions, a multi-scale feature pyramid structure is constructed, and low-resolution features are fused to high-resolution features step by step to obtain the corresponding scale feature maps.

[0027] For feature maps at each scale, a channel-spatial attention mechanism is introduced to obtain the corresponding scale-level feature vectors; and all scale-level feature vectors are concatenated in the feature dimension to obtain the multi-scale comprehensive feature representation of each effective frame.

[0028] A multi-class anomaly detection head is constructed on multi-scale comprehensive feature representation. The frame-level anomaly probability of effective frames in each anomaly category is obtained through linear mapping and Sigmoid activation function transformation. The multi-class anomaly detection head constructs an optimization objective during the training phase through weighted binary cross-entropy loss and total variation regularization term.

[0029] Preferably, the fused vector is used as input to a temporal coding network based on a selective state-space model to obtain the temporal representations of all valid frames for each anomaly category, including:

[0030] For each anomaly category, a corresponding discrete-time state-space model is constructed;

[0031] The corresponding gating vector is calculated based on the fusion vector of the current frame, and the parameters of the discrete-time state-space model are updated based on the gating vector.

[0032] The updated model is used to mix the hidden state and the fusion vector of the current valid frame, and the hidden state is adaptively updated to obtain the temporal representation of the next valid frame.

[0033] Traverse the valid frame sequence to obtain the temporal representation of all valid frames for each anomaly category.

[0034] Preferably, the extraction of event-level temporal features within each abnormal interval after context correction includes:

[0035] The anomaly intensity after context correction is calculated based on the temporal representation of the effective frames output by the temporal coding network for each anomaly category.

[0036] Binary judgment is performed on each valid frame based on the abnormal intensity to filter out several abnormal intervals;

[0037] Within each anomalous interval, interval-level time-series features are calculated based on anomalous intensity, timestamp, and time-series representation, and average pooling is performed on the time-series representation to obtain an event-level comprehensive representation; among which, the interval-level time-series features are duration, intensity integral, evolution trend slope, peak intensity, and peak relative position;

[0038] By cascading interval-level and event-level time series features on the feature dimension, event-level time series features are obtained.

[0039] Preferably, the structured metadata also includes the total surgical time and the surgeon's qualifications; the step of obtaining abnormal statistical features based on the event-level time-series features of each abnormality category, and constructing a comprehensive feature vector in conjunction with the structured metadata, includes:

[0040] Based on the duration and intensity integral of each abnormality category across all abnormal intervals, and combined with the total surgical time, the percentage of abnormal time and the percentage of abnormal intensity integral are calculated respectively.

[0041] Based on the evolution trend slope, peak intensity, and peak relative position of each anomaly category across all anomaly intervals, the average evolution trend, maximum peak intensity, and average peak relative position are calculated respectively.

[0042] Record the number of events and calculate the average duration based on the duration of each anomaly category across all anomaly intervals;

[0043] The overlap duration of abnormal intervals of different abnormal categories is compared pairwise, and the abnormal overlap ratio is calculated by combining the total operation time.

[0044] The event-level comprehensive representation of each anomaly category is averaged and pooled to obtain the category-level representation;

[0045] The surgical procedure type, total surgical time, and surgeon qualifications were vectorized separately.

[0046] The system summarizes the percentage of abnormal time, percentage of abnormal intensity integral, average evolution trend, maximum peak intensity, average peak relative position, number of events, average duration, abnormal overlap ratio, and category-level representation, and combines the vectorized results of surgical procedure type, total operation time, and surgeon qualifications to construct a comprehensive feature vector.

[0047] An intelligent surgical skills assessment system based on endoscopic video includes:

[0048] The acquisition and preprocessing module is used to acquire and standardize preprocessed endoscopic surgical videos, and smoothly identify effective frame sequences by combining a discriminative model with a time window; and to acquire structured metadata that includes at least the surgical procedure type.

[0049] The fine-tuning module is used to perform self-supervised domain fine-tuning of the feature extraction network until convergence based on each valid frame and its timestamp; the feature extraction network is a deep neural network with multi-level feature representation capabilities;

[0050] The recognition module is used to take each valid frame as input to the converged feature extraction network and extract several feature maps with different resolutions from different layers of the network; and to take the feature maps as input to a multi-channel discriminator that combines a multi-scale feature pyramid and a channel-space attention mechanism to obtain the corresponding multi-scale comprehensive features and frame-level anomaly probability.

[0051] The analysis module is used to splice the multi-scale integrated features, frame-level anomaly probabilities, probability differences and normalized time of each effective frame to obtain the fusion vector corresponding to each anomaly category; and to use the fusion vector as the input of a temporal coding network based on a selective state space model, and extract event-level temporal features in each anomaly interval after context correction.

[0052] The evaluation module is used to obtain statistical features of anomalies based on event-level time-series features of each anomaly category, and to construct a comprehensive feature vector in conjunction with structured metadata. The comprehensive feature vector is used as the input of each base learner, and the output of each base learner is weighted and fused through a gating network with technique type as input to obtain a normalized skill score and the corresponding skill level.

[0053] A storage medium storing a computer program, wherein the computer program causes a computer to perform the surgical skills intelligent assessment method as described above.

[0054] An electronic device, comprising:

[0055] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the intelligent assessment of surgical skills as described above.

[0056] (III) Beneficial Effects

[0057] This invention provides an intelligent assessment method for surgical skills based on endoscopic video. Compared with existing technologies, it has the following advantages:

[0058] This invention proposes a complete intelligent assessment process from raw endoscopic surgical videos to quantitative skill scores: Standardized preprocessing and automatic removal of irrelevant frames such as lens cleaning, instrument changes, and lens withdrawal from the body cavity; self-supervised domain fine-tuning to obtain frame-level feature representations adapted to different surgical procedures and equipment conditions; based on this, multi-scale feature pyramids and channel-space attention are used to achieve frame-level identification of various anomalies such as bleeding, smoke, and poor surgical field exposure; then, a temporal coding network based on a selective state-space model is used to temporally encode the abnormal processes to extract event-level temporal features; finally, the features are jointly modeled with structured metadata including surgical procedure types to output a normalized skill score and corresponding level. This invention overcomes the fragmented processing methods of related technologies for multi-source information and surgical procedure differences, greatly improving the accuracy, generalization, and interpretability of intelligent surgical skill assessment through a unified quantitative framework and adaptive fusion mechanism. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 A block diagram illustrating an intelligent surgical skill assessment method based on endoscopic video, provided in an embodiment of the present invention.

[0061] Figure 2 A flowchart illustrating an intelligent surgical skill assessment method based on endoscopic video, provided as an embodiment of the present invention.

[0062] Figure 3 This is a structural block diagram of a multi-channel discriminant head that combines a multi-scale feature pyramid with a channel-space attention mechanism, provided for an embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0065] This invention proposes an intelligent assessment method for surgical skills based on endoscopic video, which improves upon existing technologies in three aspects: input preprocessing, abnormal temporal sequence modeling, and overall scoring. In short:

[0066] At the input end, by unifying temporal sampling and spatial scale, and based on frame-level features such as brightness, texture, and edge, as well as the trained discrimination model, irrelevant fragment frames such as wiping the lens, changing instruments, and the lens exiting the body cavity are automatically identified and removed. Furthermore, self-supervised domain fine-tuning is adopted to learn frame-level features on a large number of unlabeled surgical videos, so that the representation can adapt to different surgical procedures and equipment conditions.

[0067] At the anomaly analysis end, frame-level features obtained through self-supervision are used, combined with multi-scale feature pyramids and attention mechanisms to obtain frame-level probabilities of multiple anomalies. Then, the multi-scale features and anomaly probabilities are temporally encoded through a selective state-space model to quantify the start and end times, duration, intensity integrals, and trend of anomalies such as bleeding, smoke, and poor surgical field exposure.

[0068] On the skill scoring side, the interval features of multiple abnormal events are modeled in a unified manner with the total operation time, operation type, surgeon information and abnormal co-occurrence relationship. An evaluation model composed of multiple base learners is constructed, and the output of each base learner is weighted and combined using a gating network with operation type as input to obtain a normalized skill score and corresponding skill level, which is used for objective comparison of surgical operation quality under different operation types and case complexity conditions.

[0069] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0070] Example 1:

[0071] like Figure 1 As shown, this embodiment of the invention provides an intelligent assessment method for surgical skills based on endoscopic video, including:

[0072] S1. Acquire and standardize preprocessed endoscopic surgical videos, and smoothly identify effective frame sequences by combining a discriminant model with a time window; and acquire structured metadata that includes at least the surgical procedure type;

[0073] S2. Based on each valid frame and its timestamp, the feature extraction network is fine-tuned in a self-supervised domain until convergence; where the feature extraction network is a deep neural network with multi-level feature representation capabilities;

[0074] S3. Each valid frame is used as input to the converged feature extraction network, and several feature maps with different resolutions are extracted from different layers of the network. The feature maps are used as input to a multi-channel discriminator that combines a multi-scale feature pyramid and a channel-space attention mechanism to obtain the corresponding multi-scale integrated features and frame-level anomaly probabilities.

[0075] S4. Concatenate the multi-scale integrated features, frame-level anomaly probabilities, probability differences and normalized time of each effective frame to obtain a fusion vector that corresponds one-to-one with each anomaly category; and use the fusion vector as the input of a temporal coding network based on a selective state space model, and extract event-level temporal features in each anomaly interval after context correction.

[0076] S5. Based on the event-level time-series features of each anomaly category, obtain anomaly statistical features and construct a comprehensive feature vector in conjunction with structured metadata; and use the comprehensive feature vector as the input of each base learner, and perform weighted fusion of the outputs of each base learner through a gating network with technique type as input to obtain a normalized skill score and the corresponding skill level.

[0077] The embodiments of this invention break through the fragmented processing methods of related technologies for multi-source information and surgical procedure differences. By using a unified quantitative framework and an adaptive fusion mechanism, it greatly improves the accuracy, generalization and interpretability of intelligent assessment of surgical skills.

[0078] like Figure 2 As shown, Figure 2 A flowchart of an intelligent surgical skill assessment method based on endoscopic video is disclosed below. Figure 2 Please describe in detail each step of the above scheme:

[0079] In step S1, preprocessed endoscopic surgical videos are acquired and standardized, and valid frame sequences are smoothly identified by combining a discriminative model with a time window; and structured metadata including at least the surgical procedure type is acquired.

[0080] In this step, after completing the ethical approval of the medical institution and data de-identification, laparoscopic and other endoscopic surgical videos are collected in accordance with relevant medical industry standards, and structured metadata such as surgical procedure type, surgeon qualifications, and instrument information are recorded simultaneously.

[0081] For example, taking a single surgical case as an example, the original video is regarded as a sequence of frames arranged in chronological order, and the j-th frame is denoted as I. j The frame index is j=0,1,…,N raw -1, where N raw This represents the total number of video frames for this surgical procedure.

[0082] Optional, standardized preprocessing for the raw video includes using uniform temporal sampling and spatial scaling.

[0083] Specifically:

[0084] To establish a unified sampling standard across the timeline, a target sampling frequency f is set. target Corresponding time step And define the timestamp of the j-th frame as... =j* The total operation time is expressed as T. total =N raw * .

[0085] At the spatial scale, for each frame image I j By scaling proportionally and centering the image, its geometry is transformed to a uniform size H×W to obtain the aligned image. H and W represent the height and width of the image, respectively. During scaling and padding, parameters such as scaling factors and padding values ​​are recorded for coordinate mapping when needed. Subsequently, [the following is performed / processed]... Normalization is performed by subtracting the mean and dividing by the standard deviation for each channel to obtain the corresponding standardized frame. =Normalize( Normalize represents the aforementioned normalization operation.

[0086] Furthermore, unlike related technologies that often directly model raw or simply downsampled surgical videos, typically only performing operations such as resolution unification, this invention addresses irrelevant fragment frames—such as lens wiping, instrument changing, lens removal from the body cavity, and image occlusion—that are weakly related to surgical skills. These fragments are then explicitly characterized by frame-level features based on brightness, texture, and edge proportions, and a discriminative model is trained to identify and remove them, rather than treating them as part of the overall effective process.

[0087] In an optional implementation, the step of smoothly identifying the valid frame sequence by combining a discriminative model with a time window includes:

[0088] S10. Convert the preprocessed standardized frame into a luminance map, and define the global luminance mean, luminance variance, gradient intensity average and edge pixel ratio based on the luminance map to construct a frame-level appearance feature vector.

[0089] S20. Use the frame-level appearance feature vector of each standardized frame as the input of the discrimination model to predict the probability that each standardized frame belongs to an irrelevant frame.

[0090] S30. Perform time window smoothing on the predicted irrelevant frame probability, calculate the smoothed irrelevant frame probability, and define the corresponding validity mask.

[0091] S40. Based on each validity mask, obtain the set of valid frame indices and sort the set in chronological order to define the valid frame sequence.

[0092] Continuing with the example above, the specific steps for identifying valid frame sequences are as follows:

[0093] The j-th normalized frame Convert to brightness map Y j (p), where p is the pixel coordinate, defined as:

[0094] Global average brightness ;

[0095] Luminance variance ;

[0096] Brightness gradient used to depict texture and edge information ;

[0097] Gradient strength average ;

[0098] and edge pixel ratio .

[0099] in, Let L2 be the norm, and γ be the gradient magnitude threshold. This is an indicator function.

[0100] Based on this, construct the frame-level appearance feature vector corresponding to the j-th standardized frame. The superscript T indicates transpose.

[0101] In a small number of manually labeled samples, frames such as lens wiping and lens exiting the body cavity are marked as irrelevant frames (corresponding to the label of the j-th normalized frame). The actual operation frame is marked as the relevant frame (the label corresponding to the j-th normalized frame). These samples are used to train a frame-independent segment discrimination model. Taking logistic regression as an example, the discriminant function is defined as follows:

[0102]

[0103] Among them, w irrel b irrel The parameters to be learned (weights and biases), q j Let be the predicted probability that the j-th standardized frame belongs to an irrelevant frame. During training, minimize the weighted binary cross-entropy loss:

[0104]

[0105] Here, parameter α is used to balance the difference in the number of irrelevant frames and relevant frames, and log is a logarithmic function.

[0106] In practical applications, to suppress occasional noise and consider temporal continuity, temporal window smoothing is performed based on the probability q j . A window radius R is selected, and a window index set is defined to calculate the probability of irrelevant frames after smoothing ; where represents a natural number.

[0107] Next, let the decision threshold for irrelevant frames be τ irrel , which can be determined from the validation set. A validity mask is defined for each frame, and from this, an effective frame index set is obtained. The set is sorted in chronological order and denoted as j(0) < j(1) < < j(N - 1), and an effective frame sequence is defined.

[0108] Among them, N is the total number of effective frames, X i is the i-th effective frame, and t i is its corresponding timestamp. The total surgical duration T total is still determined by the original frame sequence and does not change with the elimination of irrelevant frames.

[0109] In step S2, based on each effective frame and its timestamp, self-supervised domain fine-tuning of the feature extraction network is performed until convergence; where the feature extraction network is a deep neural network with multi-level feature representation capabilities.

[0110] In this step, as Figure 2 shown, using the effective frame X i obtained in step S1 and its timestamp t i , self-supervised domain fine-tuning of the feature extraction network is performed without the need for manual labels, enabling the network to automatically learn visual representations that conform to the characteristics of the endoscopic surgery scenario.

[0111] It should be noted that the feature extraction network is a deep neural network with multi-level feature representation capabilities, which can output feature maps with multiple spatial resolutions from different network depths to support the construction of a multi-scale feature pyramid in subsequent steps. Specifically, the feature extraction network includes, but is not limited to, a convolutional neural network (such as ResNet), a vision Transformer network (such as a sliding window Transformer), or a hybrid architecture network of convolution and Transformer.

[0112] In an optional implementation manner, this step includes:

[0113] S21. Use the feature extraction network to be fine-tuned as the student network and construct a teacher network with the same structure;

[0114] S22. For each valid frame, generate two different enhanced views, which are used as the inputs of the student network and the teacher network respectively, and obtain the corresponding embedding representations;

[0115] S23. Use the timestamps of each valid frame to construct time weights, and use the embedding representation output by the teacher network as pseudo labels to construct a distribution consistency loss; construct a temporal smoothing regularization term based on the change amplitude of the embedding representation output by the student network for adjacent frames; construct an alignment loss based on the activity criteria of each valid frame; and construct a total loss for self-supervised training based on the distribution consistency loss, the temporal smoothing regularization term, and the alignment loss;

[0116] S24. Use the total loss to perform self-supervised domain fine-tuning on the student network until convergence.

[0117] Continuing the above example, the specific steps for fine-tuning the feature extraction network are as follows:

[0118] Let the feature extraction network be f enc ( ; θ s ), which serves as the student network. At the same time, construct a teacher network f enc ( ; θ t ), where θ s and θ t are the parameters of the student network and the teacher network respectively.

[0119] For the valid frame X i , generate two different enhanced views Aug1(X i ) and Aug2(X i ) through methods such as random scale cropping, color perturbation, and local occlusion, and input them into the student network and the teacher network respectively to obtain the embedding representations:

[0120]

[0121] Among them, and are the embedding representations output by the student network and the teacher network respectively with the i-th valid frame X i as the input.

[0122] Since the output of the teacher network needs to be used as pseudo labels in the training process of the student network, the parameters of the teacher network can be updated by the exponential moving average method of the parameters of the student network:

[0123]

[0124] Among them, 0 < m < 1 is the momentum coefficient.

[0125] Furthermore, to introduce a temporal prior to the surgical procedure, a timestamp t is used. i and t j Constructing time weights ,in >0 is the time scale hyperparameter, which makes frames with similar times have higher weights in the self-supervised loss, and exp is the exponential function.

[0126] Let Consistency ( , If ) represents the distributional consistency loss between embedded representations, then the consistency term can be written as .

[0127] Considering the temporal continuity of surgical videos, a temporal smoothing regularization term is introduced to constrain the variation in student embeddings between adjacent frames. ,in With the (i+1)th valid frame X i+1 The input is the embedded representation of the student network output.

[0128] When there are activity labels such as surgeon, instrument, manipulation action, and target tissue, alignment loss can be defined. This enhances the clustering of frames with similar activity semantics in the embedding space.

[0129] Therefore, the total loss for self-supervised training can be constructed as:

[0130]

[0131] in, , These are the weight coefficients for the temporal smoothing regularization term and the alignment loss, respectively.

[0132] Understandably, the convergent feature extraction network is obtained through training on a large number of unlabeled endoscopic surgical videos. Its network parameters are automatically determined by data and are adaptable to different surgical procedures and equipment conditions.

[0133] In step S3, each valid frame is used as input to the converged feature extraction network, and several feature maps with different resolutions are extracted from different layers of the network. The feature maps are used as input to a multi-channel discriminator that combines a multi-scale feature pyramid and a channel-space attention mechanism to obtain the corresponding multi-scale integrated features and frame-level anomaly probabilities.

[0134] In this step, such as Figure 2 As shown, the feature extraction network obtained in step S2 is used. By combining a multi-scale feature pyramid with a channel-space attention mechanism, a multi-scale comprehensive feature vector z is extracted for each valid frame. iBased on this, the frame-level probabilities of various skill-related anomalies are output to achieve frame-level anomaly identification.

[0135] In an optional implementation, the step of using the feature map as input to a multi-channel discriminator combining a multi-scale feature pyramid and a channel-space attention mechanism to obtain the corresponding multi-scale integrated features and frame-level anomaly probabilities includes:

[0136] S100. Based on the extracted feature maps of different resolutions, a multi-scale feature pyramid structure is constructed, and the low-resolution features are fused to the high-resolution features step by step to obtain the corresponding scale feature maps.

[0137] S200. For feature maps at each scale, a channel-space attention mechanism is introduced to obtain the corresponding scale-level feature vectors; and all scale-level feature vectors are concatenated in the feature dimension to obtain the multi-scale comprehensive feature representation of each effective frame.

[0138] S300. Construct a multi-class anomaly detection head based on multi-scale comprehensive feature representation. Obtain the frame-level anomaly probability of effective frames in each anomaly category through linear mapping and Sigmoid activation function transformation. The multi-class anomaly detection head constructs an optimization objective during the training phase through weighted binary cross-entropy loss and total variation regularization term.

[0139] Unlike related technologies that primarily design features around a few explicit indicators such as bleeding volume and instrument trajectory, and rely mainly on labeled samples for supervised training, this invention utilizes mechanisms such as temporal information weighting and self-supervised consistency constraints to perform domain adaptation on a large number of unlabeled surgical videos. This significantly improves the coverage and adaptability of frame-level representation to different surgical procedures and equipment conditions.

[0140] like Figure 3 As shown, Figure 3 A schematic diagram of a multi-channel discriminant head combining a multi-scale feature pyramid and a channel-spatial attention mechanism is disclosed, including a multi-scale feature pyramid, a channel-spatial attention channel, and a multi-type anomaly detection head.

[0141] It should be noted that Figure 3 The three feature maps with different resolutions shown are for illustrative purposes only, and those skilled in the art can adjust them according to actual needs.

[0142] Continuing with the example above, refer to... Figure 3 The specific steps for obtaining the corresponding multi-scale integrated features and frame-level anomaly probabilities are as follows:

[0143] valid frame X i enter Three feature maps with different resolutions are extracted from different layers of the network. .in, It has high spatial resolution, reflecting detailed information such as instrument tips and local bleeding points. Reflecting the local surgical field structure, It reflects the overall layout of the surgical area.

[0144] Based on the above three feature maps, a multi-scale feature pyramid structure is constructed, which fuses features from low resolution to high resolution step by step, including:

[0145] A linear transformation is performed on the low-resolution features to obtain the third-scale feature map. ,in This is a convolution or 1×1 transform operator; * indicates convolution. It transforms the medium-resolution features and combines them with the upsampled features. The sum is used to obtain the second-scale feature map. Similar processing is applied to the high-resolution features to obtain the first-scale feature map. Up ( () is the upsampling operator. Thus, the three-scale feature map is obtained. , , .

[0146] At each scale, a channel-space joint attention mechanism is introduced. Taking scale l∈{1,2,3} as an example, for... Global average pooling (spatial GAP) and global max pooling (spatial GMP) are performed in the spatial dimension to obtain two channel statistical vectors. These vectors are then transformed by a shared multilayer perceptron (MLP) nonlinear transformation and passed through a sigmoid function to obtain the corresponding channel attention weight vectors. After channel weighting, the average and maximum values ​​of the weighted features are calculated along the channel dimension, and then convolution and sigmoid are applied to obtain the corresponding spatial attention map. Attention-enhanced feature map is ,in This indicates element-wise multiplication.

[0147] Global average pooling (GAP) is performed on the attention-enhanced feature maps at each scale to obtain scale-level feature vectors. .

[0148] The features are concatenated along the feature dimension to form a multi-scale comprehensive feature representation of the i-th valid frame. .

[0149] For a pre-defined set of abnormality categories (such as bleeding, smoke, poor surgical field exposure, etc.), in z iA multi-class anomaly detection head is constructed. For any anomaly category c, the discrimination score of the frame in that category is obtained through a linear layer linear mapping. , where w c and b c For class c, the parameters are used. Furthermore, the scores are converted into frame-level anomaly probabilities using a Sigmoid activation function. .

[0150] On frames with manual annotations, the anomaly label is y. i,c Construct a weighted binary cross-entropy loss ∈{0,1}. This is to balance the differences in the number of samples from different anomaly categories.

[0151] To suppress excessive jitter in predictions over time, a total variation regularization term is introduced. .

[0152] The optimization objective of the frame-level anomaly detection stage is:

[0153]

[0154] in, These are the weighting coefficients. Through training with labeled anomaly samples, the multi-scale fusion parameters, attention weights, and classification head parameters are automatically learned from the data, outputting the frame-level anomaly probability P. i,c With feature z i Used for subsequent time series modeling and event-level time series feature extraction.

[0155] In step S4, the multi-scale integrated features, frame-level anomaly probabilities, probability differences, and normalized time of each valid frame are concatenated to obtain a fusion vector that corresponds one-to-one with each anomaly category; and the fusion vector is used as the input of a temporal coding network based on a selective state-space model, and after context correction, event-level temporal features are extracted in each anomaly interval.

[0156] In this step, such as Figure 2 As shown, the multi-scale frame-level feature z obtained in step S3 is used. i and frame-level anomaly probability P i,c Combined with timestamp t i and time step We construct a time-series coding network based on the Selective StateSpace Model (Selective SSM) to model the occurrence and evolution of each type of anomaly on the time axis and extract the interval-level time-series features of the anomaly events to achieve anomaly time-series analysis.

[0157] In an optional implementation, the step of using the fused vector as input to a temporal coding network based on a selective state-space model to obtain the temporal representation of all valid frames for each anomaly category, and extracting event-level temporal features within each anomaly interval after context correction, includes:

[0158] S1000. For each anomaly category, construct the corresponding discrete-time state-space model;

[0159] S2000: Calculate the corresponding gate vector based on the fusion vector of the current frame, and update the parameters of the discrete-time state-space model based on the gate vector;

[0160] S3000: The updated model is used to mix the hidden state and fusion vector of the current valid frame, and the hidden state is adaptively updated to obtain the temporal representation of the next valid frame.

[0161] S4000: Traverse the valid frame sequence and obtain the temporal representation of all valid frames in each anomaly category;

[0162] S5000: Based on the temporal representation of the effective frames output by the temporal coding network in each anomaly category, calculate the anomaly intensity after context correction;

[0163] S6000: Perform binary judgment on each valid frame based on the abnormal intensity to filter out several abnormal intervals;

[0164] S7000. Within each anomalous interval, interval-level time series features are calculated based on anomalous intensity, timestamp, and time series representation. The time series representation is then averaged and pooled to obtain an event-level comprehensive representation. The interval-level time series features include duration, intensity integral, evolution trend slope, peak intensity, and peak relative position.

[0165] S8000: The interval-level time series features and the event-level integrated representation are cascaded on the feature dimension to obtain the event-level time series features.

[0166] Related technologies provide incomplete descriptions of the temporal structure of abnormal processes. Unlike existing methods that typically output skill-related classification or regression results directly within a single frame or fixed-length time window, and then obtain the overall surgical indicators through simple averaging, weighted summation, or extreme value extraction, the algorithm proposed in this invention explicitly calculates the interval features of various anomalies on the time axis, including the start frame, end frame, duration, probability integral, intensity change slope, and peak position. It also incorporates a mechanism for sequence modeling and smoothing correction of frame-level anomaly probabilities based on the complete surgical process. This allows for the differentiation of different temporal patterns, such as transient fluctuations and persistent anomalies, at the feature level, providing a structured and quantitative description of the occurrence and evolution of abnormal events over time.

[0167] Continuing with the example above, the specific steps for extracting event-level time series features are as follows:

[0168] To characterize the local variation of anomaly probabilities over time, a first-order difference is defined for the probability sequence of each anomaly category c. When i=0, let .

[0169] To reflect the relative progress of the surgery, normalized time is introduced. .

[0170] At each time i, the multi-scale features, anomaly probability, probability difference, and relative time are concatenated along the feature dimension to obtain a fusion vector. .

[0171] For each anomaly category c, a discrete-time state-space model is constructed, the basic form of which is:

[0172]

[0173] Among them, s i,c Let m be the hidden state of category c in the i-th frame. i,c For the corresponding timing output representation, A c B c Let C be the state transition matrix to be learned. c D c These are the input and output transformation matrices, respectively.

[0174] To enable the model to adaptively adjust its dependence on historical states and current inputs based on the signal, a selective gating mechanism is introduced. For each frame i and class c, based on the fusion vector v... i,c Calculate the gate vector W g,c b g,c These are the gating parameters (weights and biases). G is the sigmoid function. i,c Each dimension takes values ​​in the interval (0,1). Let Diag( Let be an operator that generates a diagonal matrix from a vector, defined as follows: .

[0175] Based on this, the state update can be written as That is, at each time step, G i,c Control "historical status items" "and "current input item "Relative weights in different dimensions."

[0176] Finally, linear readout is used at the output. .

[0177] It is understandable that the above {A} c B c C c D c W g,c ,b g,c The parameters of} were all obtained through training on surgical video sequences with anomaly annotations.

[0178] Based on the temporal representation m of category c in the i-th frame i,c Calculate the context-corrected anomaly intensity ,in and The time-series regression parameters (weights and biases) for this category are determined through training.

[0179] according to With the preset threshold τ of category c c The relationship is determined by performing a binary judgment on each frame, when... ≥τ c season =1, otherwise let =0. On the time axis, adjacent elements that satisfy... Frames with a duration of 1 are joined into a continuous segment, and the minimum duration D is determined. min Filter out segments with durations shorter than a threshold. The k-th (valid) abnormal interval is determined by the starting frame index s. k,c and end frame index e k,c Confirmed, the corresponding time interval is ,in , These are the start and end times of the k-th valid outlier interval for category c, respectively. , The starting frame index s k,c and end frame index e k,c The corresponding timestamp.

[0180] Within each anomalous interval, the anomalous intensity of category c in the i-th frame is used. timestamp t i And timing representation m i,c Calculate the interval-level time series features. Where:

[0181] Duration is D k,c = -

[0182] Anomaly intensity integral is ;

[0183] To characterize the changing trend of anomaly intensity within the interval, the point set (t) is analyzed. i , Perform least-squares linear fitting Where the slope S k,c =a k,c This indicates the overall trend of an anomaly strengthening or weakening over time, b k,c Slope distance;

[0184] The peak intensity within the interval is ;

[0185] The peak corresponds to time t max,k,c The relative position is ;

[0186] The time sequence representation m within this interval i,c Perform average pooling to obtain the event-level comprehensive representation of the k-th valid outlier interval of category c. .

[0187] Finally, the duration D k,c Intensity integral AUC k,c Evolutionary trend slope S k,c Peak intensity P max,k,c Peak relative position R max,k,c and event-level comprehensive representation By cascading along the feature dimension, an event-level temporal feature of the c-th type of abnormal event within the k-th abnormal interval is formed. .

[0188] In step S5, based on the event-level temporal features of each anomaly category, anomaly statistical features are obtained, and a comprehensive feature vector is constructed in conjunction with structured metadata; the comprehensive feature vector is used as the input of each base learner, and the outputs of each base learner are weighted and fused through a gating network with technique type as input to obtain a normalized skill score and the corresponding skill level.

[0189] In this step, such as Figure 2 As shown, the abnormal event feature φ obtained in step S4 is used. k,c Combined with the surgical procedure type and total operation time T obtained in step S1 total Structured metadata, such as surgeon information, is used to intelligently quantify and evaluate the surgical skills involved in the entire procedure.

[0190] In an optional implementation, the step of obtaining anomaly statistical features based on the event-level time-series features of each anomaly category and constructing a comprehensive feature vector in conjunction with structured metadata includes:

[0191] S10000: Based on the duration and intensity integral of each abnormal category across all abnormal intervals, and combined with the total surgical time, calculate the percentage of abnormal time and the percentage of abnormal intensity integral, respectively.

[0192] S20000: Based on the evolution trend slope, peak intensity, and peak relative position of each anomaly category in all anomaly intervals, calculate the average evolution trend, maximum peak intensity, and average peak relative position respectively.

[0193] S30000: Record the number of events and calculate the average duration based on the duration of each anomaly category across all anomaly intervals;

[0194] S40000: Compare the overlap duration of abnormal intervals of different abnormal categories pairwise, and calculate the abnormal overlap ratio in combination with the total operation time;

[0195] S50000: Average pooling is performed on the event-level comprehensive representation of each anomaly category to obtain the category-level representation;

[0196] S60000, respectively vectorizes surgical procedure type, total surgical time and surgeon qualifications;

[0197] S70000 summarizes the percentage of abnormal time, percentage of abnormal intensity integral, average evolution trend, maximum peak intensity, average peak relative position, number of events, average duration, abnormal overlap ratio and category-level representation, and combines the vectorized results of surgical procedure type, total operation time and surgeon qualifications to construct a comprehensive feature vector.

[0198] Continuing with the example above, the specific steps for constructing the comprehensive feature vector are as follows:

[0199] For each abnormality category c, calculate the percentage of abnormal time in the entire operation based on the duration of all abnormal events. .

[0200] Calculate the proportion of abnormal intensity integrals based on the intensity integrals of each event. .

[0201] Select the highest peak intensity from all anomalous events in this category. .

[0202] Number of recorded events M c and average duration .

[0203] Calculate the average evolution trend of this category of anomalies based on the slope of the evolution trend of each event. ,in A value greater than 0 indicates that the overall trend of this type of anomaly is worsening. <0 indicates that the overall abnormality is under effective control.

[0204] Calculate the average relative peak position of this category of anomalies based on the relative peak positions of each event. ,in A value close to 0 indicates that abnormal peaks mostly occur in the early stages of an event, while a value close to 1 indicates that peaks mostly occur in the later stages of an event.

[0205] For different categories c1 and c2, based on their respective abnormal time intervals, the overlap duration of the abnormal intervals on the time axis can be calculated and compared with T. total Normalization is performed to obtain the abnormal overlap ratio. This is used to describe the co-occurrence of multiple types of anomalies.

[0206] Comprehensive representation of event levels within each category By performing average pooling, we can obtain the category-level representation of category c. .

[0207] The above-mentioned abnormal statistical features are summarized into a vector. , where C is the total number of anomaly categories.

[0208] After obtaining the above-mentioned statistical features and category-level representations of anomalies, these are combined with metadata such as surgical procedure type and surgeon qualifications to construct a comprehensive feature vector F representing the entire surgical case. surgery .

[0209] Optionally, embodiments of the present invention limit the structured metadata to include at least the following three types of information:

[0210] (a) Procedure type: Represented using one-hot coding, denoted as ,in Total number of surgical procedure categories;

[0211] (b) Total operation time: T as defined in step (1) total And perform normalization, denoted as ,in For reference duration;

[0212] (c) Practitioner's qualifications: Represented by discrete level coding, denoted as ,in This refers to the seniority level.

[0213] The comprehensive feature vector is obtained by concatenating anomaly statistical features and metadata along the feature dimension. .

[0214] Furthermore, unlike related technologies where the importance of various features generally remains fixed within the model under different surgical procedures and operational difficulties, this invention introduces a mechanism that adaptively adjusts feature weights based on surgical procedure type or case complexity. This ensures that the same scoring model is no longer limited in its ability to reflect differences in case complexity and operation across various surgical scenarios.

[0215] Continuing with the example above, the specific steps to obtain the normalized skill score and corresponding skill level are as follows:

[0216] The surgical procedure type uses a single-hot vector V. type This indicates that the input to the gated network Gate ( In the process, a set of unnormalized weights α is obtained, and the weights w of each base learner are obtained through Softmax operation. m =Softmax(α) m m=1,…,M

[0217] Comprehensive feature vector F surgery Input multiple base learners f respectively m ( ), to obtain the intermediate score s m =f m (F surgery The original skill scores are obtained by linearly weighting the scores using weights. .

[0218] On surgical samples with expert ratings, the gating network parameters and the parameters of each base learner are trained to enable the model to learn the correspondence between abnormal features and skill scores under different surgical procedures and case complexities. For S... raw Perform a linear transformation and truncate to the interval [0,1] to obtain the normalized skill score S. final =clip(aS raw +b,0,1), where a and b are the scaling factor and offset term determined by the calibration sample, respectively, and clip(x,0,1) is the truncation function used to restrict the value x to the range [0,1].

[0219] Finally, based on the pre-set thresholds η2 and η3 (satisfying 0 < η2 < η3 < 1), S is... final Divided into different skill levels:

[0220]

[0221] Where g represents the corresponding discrete skill level.

[0222] Thus, this embodiment of the invention completes all the steps of the intelligent assessment method for surgical skills based on endoscopic video.

[0223] Example 2:

[0224] This invention provides an intelligent surgical skill assessment system based on endoscopic video, comprising:

[0225] The acquisition and preprocessing module is used to acquire and standardize preprocessed endoscopic surgical videos, and smoothly identify effective frame sequences by combining a discriminative model with a time window; and to acquire structured metadata that includes at least the surgical procedure type.

[0226] The fine-tuning module is used to perform self-supervised domain fine-tuning of the feature extraction network until convergence based on each valid frame and its timestamp; the feature extraction network is a deep neural network with multi-level feature representation capabilities;

[0227] The recognition module is used to take each valid frame as input to the converged feature extraction network and extract several feature maps with different resolutions from different layers of the network; and to take the feature maps as input to a multi-channel discriminator that combines a multi-scale feature pyramid and a channel-space attention mechanism to obtain the corresponding multi-scale comprehensive features and frame-level anomaly probability.

[0228] The analysis module is used to splice the multi-scale integrated features, frame-level anomaly probabilities, probability differences and normalized time of each effective frame to obtain the fusion vector corresponding to each anomaly category; and to use the fusion vector as the input of a temporal coding network based on a selective state space model, and extract event-level temporal features in each anomaly interval after context correction.

[0229] The evaluation module is used to obtain statistical features of anomalies based on event-level time-series features of each anomaly category, and to construct a comprehensive feature vector in conjunction with structured metadata. The comprehensive feature vector is used as the input of each base learner, and the output of each base learner is weighted and fused through a gating network with technique type as input to obtain a normalized skill score and the corresponding skill level.

[0230] Example 3:

[0231] This invention provides a storage medium storing a computer program, wherein the computer program causes a computer to execute the intelligent surgical skill assessment method as described in Embodiment 1.

[0232] Example 4:

[0233] This invention provides an electronic device, comprising:

[0234] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the intelligent surgical skills assessment method as described in Example 1.

[0235] It is understood that the intelligent surgical skill assessment system, storage medium and electronic device based on endoscopic video provided in the embodiments of the present invention correspond to the intelligent surgical skill assessment method based on endoscopic video provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can be referred to the corresponding parts of the method, and will not be repeated here.

[0236] In summary, compared with existing technologies, it has the following beneficial effects:

[0237] 1. This invention employs standardized temporal sampling and spatial scale at the input end to standardize endoscopic surgical videos. It trains an irrelevant segment discrimination model using frame-level features such as brightness, texture, and edges, and automatically identifies irrelevant segments such as lens wiping, instrument changes, and lens withdrawal from the body cavity using temporal window smoothing, removing them from subsequent analysis. Subsequently, through student-teacher self-supervised domain fine-tuning with temporal weights and temporal smoothing constraints, frame-level visual representations are learned on a large number of unlabeled surgical videos. This allows the features to adapt to different surgical procedures and equipment conditions, while reducing the interference of irrelevant images on the evaluation results and decreasing reliance on large-scale manual annotation and single explicit features.

[0238] 2. In this embodiment of the invention, after obtaining a unified frame-level feature representation, a multi-scale feature pyramid and a channel-space attention mechanism are introduced to provide frame-level probability predictions for various anomalies such as bleeding, smoke, and poor surgical field exposure. Then, a selective state-space model is constructed using a fusion vector composed of multi-scale features, anomaly probabilities, probability differences, and temporal location. This model performs temporal encoding on the anomaly sequences, transforming discrete frame-level results into event-level temporal features with indicators such as start and end times, duration, intensity integral, trend, peak value, and relative position. This process achieves automatic extraction of skill-related temporal structure features from endoscopic videos, reducing the workload of manual frame-by-frame review and reliance on subjective experience.

[0239] 3. Based on event-level temporal features, this embodiment of the invention calculates the time proportion, intensity proportion, number of events, and average duration of each abnormal category, and describes the temporal overlap between different abnormal categories. These statistics are combined with structured data such as total surgical time, surgical procedure type, and surgeon qualifications to form a comprehensive feature vector for the entire surgical case. The evaluation model consists of multiple base learners. The surgical procedure type is mapped to the weights of each base learner via a gating network. The outputs are weighted and fused to obtain a normalized skill score and corresponding skill level. This allows for quantitative comparison of surgical operation quality under different surgical procedures and case complexity conditions on the same scale, and can serve as a reference for clinical quality control and teaching feedback.

[0240] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0241] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent assessment of surgical skills based on endoscopic video, characterized in that, include: Acquire and standardize preprocessed endoscopic surgical videos, and smoothly identify effective frame sequences by combining a discriminant model with a time window. And obtain structured metadata that includes at least the technique type; Based on each valid frame and its timestamp, the feature extraction network is fine-tuned in a self-supervised domain until convergence; the feature extraction network is a deep neural network with multi-level feature representation capabilities. Each valid frame is used as input to the converged feature extraction network, and several feature maps with different resolutions are extracted from different layers of the network. The feature maps are used as input to a multi-channel discriminator that combines a multi-scale feature pyramid and a channel-space attention mechanism to obtain the multi-scale comprehensive features of each valid frame and the frame-level anomaly probability for multiple preset anomaly categories. Calculate the first-order difference of the frame-level anomaly probability for the same anomaly category for adjacent valid frames, and obtain the frame-level anomaly probability difference for each valid frame for each anomaly category; concatenate the multi-scale integrated features, normalized time, frame-level anomaly probability for each anomaly category and frame-level anomaly probability difference for each valid frame to obtain a fusion vector corresponding to each anomaly category; and use the fusion vector as the input of a temporal coding network based on a selective state-space model, and extract event-level temporal features in each anomaly interval after context correction. Based on the event-level temporal features of each anomaly category, anomaly statistical features are obtained, and a comprehensive feature vector is constructed in conjunction with structured metadata. The comprehensive feature vector is then used as the input to each base learner, and the outputs of each base learner are weighted and fused through a gating network with technique type as input to obtain a normalized skill score and the corresponding skill level.

2. The intelligent surgical skill assessment method as described in claim 1, characterized in that, The method of smoothly identifying valid frame sequences by combining a discriminative model with a time window includes: The preprocessed normalized frames are converted into luminance maps, and the global luminance mean, luminance variance, average gradient intensity, and edge pixel ratio are defined based on the luminance maps to construct frame-level appearance feature vectors. The frame-level appearance feature vectors of each standardized frame are used as input to the discrimination model to predict the probability that each standardized frame belongs to an irrelevant frame. Smooth the predicted irrelevant frame probabilities over a time window, calculate the smoothed irrelevant frame probabilities, and define the corresponding validity mask. Based on each validity mask, obtain the set of valid frame indices and sort the set in chronological order to define the valid frame sequence.

3. The intelligent surgical skill assessment method as described in claim 1, characterized in that, The step of performing self-supervised neighborhood fine-tuning of the feature extraction network until convergence based on each valid frame and its timestamp includes: The feature extraction network to be fine-tuned is used as the student network, and a teacher network with the same structure is constructed. For each valid frame, two different augmented views are generated and used as inputs to the student network and teacher network, respectively, to obtain the corresponding embedding representations; Temporal weights are constructed using the timestamps of each valid frame, and the embedded representation output by the teacher network is used as a pseudo-label to construct a distribution consistency loss. A temporal smoothing regularization term is constructed based on the change magnitude of the embedded representation output by the student network in adjacent frames. An alignment loss is constructed based on the activity criteria of each valid frame. Finally, the total loss for self-supervised training is constructed based on the distribution consistency loss, the temporal smoothing regularization term, and the alignment loss. The total loss is used to fine-tune the student network under self-supervision until convergence.

4. The intelligent surgical skill assessment method as described in claim 1, characterized in that, The step of using the feature map as input to a multi-channel discriminator combining a multi-scale feature pyramid and a channel-space attention mechanism to obtain multi-scale integrated features of each effective frame and frame-level anomaly probabilities for multiple preset anomaly categories includes: Based on the extracted feature maps of different resolutions, a multi-scale feature pyramid structure is constructed, and low-resolution features are fused to high-resolution features step by step to obtain the corresponding scale feature maps. For feature maps at each scale, a channel-spatial attention mechanism is introduced to obtain the corresponding scale-level feature vectors; and all scale-level feature vectors are concatenated in the feature dimension to obtain the multi-scale comprehensive feature representation of each effective frame. A multi-class anomaly detection head is constructed on multi-scale comprehensive feature representation. The frame-level anomaly probability of effective frames in each anomaly category is obtained through linear mapping and Sigmoid activation function transformation. The multi-class anomaly detection head constructs an optimization objective during the training phase through weighted binary cross-entropy loss and total variation regularization term.

5. The intelligent surgical skill assessment method as described in claim 1, characterized in that, The fused vector is used as input to a temporal coding network based on a selective state-space model to obtain the temporal representations of all valid frames for each anomaly category, including: For each anomaly category, a corresponding discrete-time state-space model is constructed; The corresponding gating vector is calculated based on the fusion vector of the current frame, and the parameters of the discrete-time state-space model are updated based on the gating vector. The updated model is used to mix the hidden state and the fusion vector of the current valid frame, and the hidden state is adaptively updated to obtain the temporal representation of the next valid frame. Traverse the valid frame sequence to obtain the temporal representation of all valid frames for each anomaly category.

6. The intelligent surgical skill assessment method as described in claim 1, characterized in that, The extraction of event-level temporal features within each anomalous interval after context correction includes: The anomaly intensity after context correction is calculated based on the temporal representation of the effective frames output by the temporal coding network for each anomaly category. Binary judgment is performed on each valid frame based on the abnormal intensity to filter out several abnormal intervals; Within each anomalous interval, interval-level time-series features are calculated based on anomalous intensity, timestamp, and time-series representation, and average pooling is performed on the time-series representation to obtain an event-level comprehensive representation; among which, the interval-level time-series features are duration, intensity integral, evolution trend slope, peak intensity, and peak relative position; By cascading interval-level and event-level time series features on the feature dimension, event-level time series features are obtained.

7. The intelligent surgical skill assessment method as described in claim 6, characterized in that, The structured metadata also includes the total surgical time and the surgeon's qualifications; the acquisition of abnormal statistical features based on event-level time-series features of each abnormality category, and the construction of a comprehensive feature vector in conjunction with the structured metadata, includes: Based on the duration and intensity integral of each abnormality category across all abnormal intervals, and combined with the total surgical time, the percentage of abnormal time and the percentage of abnormal intensity integral are calculated respectively. Based on the evolution trend slope, peak intensity, and peak relative position of each anomaly category across all anomaly intervals, the average evolution trend, maximum peak intensity, and average peak relative position are calculated respectively. Record the number of events and calculate the average duration based on the duration of each anomaly category across all anomaly intervals; The overlap duration of abnormal intervals of different abnormal categories is compared pairwise, and the abnormal overlap ratio is calculated by combining the total operation time. The event-level comprehensive representation of each anomaly category is averaged and pooled to obtain the category-level representation; The surgical procedure type, total surgical time, and surgeon qualifications were vectorized separately. The system summarizes the percentage of abnormal time, percentage of abnormal intensity integral, average evolution trend, maximum peak intensity, average peak relative position, number of events, average duration, abnormal overlap ratio, and category-level representation, and combines the vectorized results of surgical procedure type, total operation time, and surgeon qualifications to construct a comprehensive feature vector.

8. A surgical skill intelligent assessment system based on endoscopic video, characterized in that, include: The acquisition and preprocessing module is used to acquire and standardize preprocessed endoscopic surgical videos, and smoothly identify effective frame sequences by combining a discriminant model with a time window. And obtain structured metadata that includes at least the technique type; The fine-tuning module is used to perform self-supervised domain fine-tuning of the feature extraction network until convergence based on each valid frame and its timestamp; the feature extraction network is a deep neural network with multi-level feature representation capabilities; The recognition module is used to take each valid frame as input to the converged feature extraction network and extract several feature maps with different resolutions from different layers of the network; and to take the feature maps as input to a multi-channel discriminator that combines a multi-scale feature pyramid and a channel-space attention mechanism to obtain the multi-scale comprehensive features of each valid frame and the frame-level anomaly probability for multiple preset anomaly categories. The analysis module is used to calculate the first-order difference of the frame-level anomaly probability of adjacent valid frames for the same anomaly category, obtain the frame-level anomaly probability difference of each valid frame for each anomaly category; concatenate the multi-scale integrated features, normalized time, frame-level anomaly probability and frame-level anomaly probability difference of each valid frame to obtain a fusion vector corresponding to each anomaly category; and use the fusion vector as the input of a temporal coding network based on a selective state-space model, and extract event-level temporal features in each anomaly interval after context correction. The evaluation module is used to obtain statistical features of anomalies based on event-level time-series features of each anomaly category, and to construct a comprehensive feature vector in conjunction with structured metadata. The comprehensive feature vector is used as the input of each base learner, and the output of each base learner is weighted and fused through a gating network with technique type as input to obtain a normalized skill score and the corresponding skill level.

9. A storage medium, characterized in that, It stores a computer program, wherein the computer program causes the computer to perform the intelligent surgical skills assessment method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the intelligent assessment method for surgical skills as described in any one of claims 1 to 7.

Images