A clinical operation video scoring method and device based on contrast learning, equipment and storage medium
By employing a contrastive learning-based approach, utilizing a spatiotemporal feature extraction network and support set-optimized feature representation, the instability and high-cost annotation issues of existing clinical skills assessments are addressed, enabling automatic and objective scoring of clinical operation videos with limited data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- THE FIRST AFFILIATED HOSPITAL OF WENZHOU MEDICAL UNIV
- Filing Date
- 2026-01-21
- Publication Date
- 2026-06-16
AI Technical Summary
Existing clinical skills assessment methods rely on the subjective judgment of senior experts, which is unstable and requires costly manual annotation. They are difficult to fully capture subtle differences in skills, and the assessment models lack fine-grained discrimination capabilities.
A contrastive learning-based approach is adopted, which uses a spatiotemporal feature extraction network and a support set to optimize the feature extraction network with intra-class contrastive loss and cross-set contrastive loss. Combined with global pooling and L2 norm normalization, automatic scoring of clinical operation videos is achieved.
By effectively training the model with limited labeled data, we can learn feature representations that are sensitive to skill levels and have good discriminative power, thereby achieving objective skill assessment and improving the stability and efficiency of the assessment.
Smart Images
Figure CN122223770A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical clinical operation video processing technology, and in particular to a clinical operation video scoring method, apparatus, device, and storage medium based on contrastive learning. Background Technology
[0002] Accurate assessment of clinical skills plays a crucial role in the training and professional development of surgeons, directly impacting training effectiveness, professional certification, and continuous improvement in medical quality. Currently, clinical skills assessment in practice largely relies on the subjective judgment of senior surgeons. However, this traditional method has several limitations. First, assessment results are easily influenced by the assessor's personal preferences, leading to a lack of stability and consistency. Second, experts need to invest a significant amount of valuable time reviewing and scoring clinical procedures, a cumbersome and inefficient process.
[0003] Current clinical procedures are increasingly being recorded from various angles by high-definition video systems, generating massive amounts of video data. This creates crucial conditions for applying computer vision and artificial intelligence technologies to achieve automated assessment of clinical skills. Some beneficial explorations have already been conducted in academia and industry, such as developing methods based on surgical instrument motion trajectory analysis and technologies based on surgical action recognition.
[0004] Despite the progress made by the aforementioned computer-aided assessment methods, existing technologies still face significant challenges in practical applications. First, clinical skills are inherently complex, involving rich details and intricate judgments; existing simple kinematic indicators or basic movement recognition methods often fail to fully capture and represent the subtle differences in skill levels. Second, many supervised learning-based methods heavily rely on large amounts of clinical operation video data with precise expert rating labels; however, acquiring such high-quality, large-scale labeled datasets is not only costly but also extremely time-consuming, constituting one of the main bottlenecks to the widespread application of these technologies. Furthermore, the specific details of score deductions in videos with the same rating can vary greatly; accurately distinguishing adjacent skill levels is often more difficult than distinguishing levels with significant differences, placing higher demands on the fine-grained discriminative capabilities of assessment models.
[0005] In conclusion, there is an urgent need to develop a new clinical skills assessment method that can effectively identify high-level feature differences between videos with the same or different ratings, reduce reliance on large-scale manually labeled data, and overcome the shortcomings of existing technologies. Summary of the Invention
[0006] Therefore, it is necessary to provide a clinical operation video scoring method based on comparative learning to address the above-mentioned technical problems.
[0007] The present invention adopts the following technical solution: This invention provides a clinical operation video scoring method based on contrastive learning, comprising: Acquire video data containing clinical procedure processes and corresponding expert skill rating tags; The video data is divided into a training set, a support set, and a test set, wherein the support set contains several representative video samples for each skill rating level; The video data in the training set, support set, and test set are preprocessed to obtain preprocessed video data. A spatiotemporal feature extraction network is used to process the preprocessed video data and extract spatiotemporal features; Based on the support set, a contrastive learning framework is used to train the spatiotemporal feature extraction network. The contrastive learning framework is optimized using the spatiotemporal features of the training set and the support set and the corresponding skill score labels. For videos from the test set or new videos to be evaluated, their spatiotemporal features are extracted, and the skill rating level of the video to be evaluated is predicted based on the similarity between the spatiotemporal features of the video to be evaluated and the spatiotemporal features of samples in the support set. The beneficial effect is that a complete automatic scoring process for clinical operation videos is established. By introducing a support set and a contrastive learning framework, effective training can be performed with limited labeled data, achieving objective skill assessment.
[0008] Preferably, the process of training the spatiotemporal feature extraction network using a contrastive learning framework includes the following steps: Calculate the intra-class contrast loss between samples within the training set. Calculate the intra-class contrast loss among samples within the support set. Calculate the cross-set contrastive loss between the training set samples and the support set samples. The intra-class contrast loss and the cross-set contrast loss are combined according to preset weighting coefficients to obtain the total loss function. Based on the total loss function, the trainable parameters of the spatiotemporal feature extraction network are updated using an optimization algorithm. The beneficial effect is that by combining contrastive losses from different sources, the feature space is optimized more comprehensively, enhancing the model's ability to learn discriminative features.
[0009] Preferably, the calculation process of the intra-class contrast loss includes the following steps: For a batch of samples in the same subset of data, calculate the similarity score between features of every two samples. This score is obtained by multiplying the two features by the dot product and dividing by the temperature hyperparameter. Calculate a separate loss value for each sample in the set. The calculation process uses a positive sample mask. Only when two samples have the same class label and are not the same sample are they marked as a positive sample pair. The mask value is defined as 1, otherwise it is 0. When calculating the loss of a sample, only its relationship with all positive sample pairs is considered. For each positive sample pair, a logarithmic term is subtracted from their similarity scores. This logarithmic term is the exponential sum of the similarity scores of the current sample with all other samples in the set except itself, and then the natural logarithm is taken. The summation of the above differences for all positive sample pairs, divided by the total number of positive sample pairs, is averaged to obtain the intra-class contrast loss value for that sample.
[0010] Preferably, the calculation process of the cross-set contrast loss includes the following steps: For each sample feature from the training set and all sample features from the support set, a similarity score is calculated between them. This score is obtained by multiplying the training set sample features by the support set sample features and dividing by the temperature hyperparameter. For each training set sample, calculate its individual loss value. The calculation process is as follows: First, establish a cross-set positive sample mask matrix. Only when the class label of a training set sample is exactly the same as the class label of a support set sample are they considered as positive sample pairs and marked as 1 at the corresponding position in the mask matrix; otherwise, they are marked as 0. When calculating the loss, only combinations of pairs labeled as positive samples are considered. For the current training sample, the similarity score calculated with all positive sample support set features is subtracted by a logarithmic term. This logarithmic term is the logarithm of the exponential sum of the similarity scores between the current training sample and all samples in the support set. Finally, this difference result for all positive sample pairs is summed and normalized by dividing by the total number of positive sample pairs, thus obtaining the cross-set contrastive loss value for the training sample. Its beneficial effect is that it clarifies the specific calculation methods for intra-class loss and cross-set loss. Intra-class loss aims to bring similar samples closer together and push away dissimilar samples, while cross-set loss uses the support set to guide the learning of training set samples, jointly improving the discriminative power of the features.
[0011] Preferably, after extracting the spatiotemporal features, the method further includes processing the spatiotemporal features. The post-processing steps include applying global pooling and L2 norm normalization. The normalization operation is calculated as follows: the squares of each dimension of the original feature vector are summed, and then the square root of this sum is taken to obtain the L2 norm of the vector. Finally, each dimension of the original feature vector is divided by this calculated L2 norm, resulting in a new normalized feature vector. The beneficial effect is that global pooling and L2 normalization map the feature vector onto a unit hypersphere, giving the features a fixed scale, which facilitates subsequent comparisons based on cosine similarity.
[0012] Preferably, the process of predicting the skill rating level of the video to be evaluated based on the similarity between the spatiotemporal features of the video to be evaluated and the spatiotemporal features of the support set samples includes the following steps: Calculate the similarity score between the features of the video to be evaluated and the features of each sample in the support set; From all the calculated similarity scores, find the one with the largest value and determine its corresponding support set samples; The labels of the sample are directly used as the predicted rating level of the video to be evaluated. Its advantage is that it provides a simple and effective prediction method based on the nearest neighbor concept, utilizing learned features and support set information to directly rate new videos, making it easy to implement and understand.
[0013] A clinical procedure video scoring device based on contrastive learning, comprising: The acquisition unit acquires video data containing clinical operation procedures and corresponding expert skill rating labels; The preprocessing unit divides the video data into a training set, a support set, and a test set, wherein the support set contains several representative video samples for each skill rating level; and preprocesses the video data in the training set, support set, and test set to obtain preprocessed video data. The spatiotemporal feature extraction unit uses a spatiotemporal feature extraction network to process the preprocessed video data and extract spatiotemporal features; The training unit trains the spatiotemporal feature extraction network using a contrastive learning framework based on the support set. The contrastive learning framework is optimized using the spatiotemporal features of the training set and the support set, as well as the corresponding skill rating labels. The prediction and evaluation unit extracts the spatiotemporal features of the video to be evaluated from the test set or a new video to be evaluated, and predicts the skill rating level of the video to be evaluated based on the similarity between the spatiotemporal features of the video to be evaluated and the spatiotemporal features of the samples in the support set.
[0014] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the clinical operation video scoring method based on contrastive learning as described above.
[0015] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the clinical operation video scoring method based on contrastive learning as described above.
[0016] This invention introduces a support set and a contrastive learning framework, which enables effective training of the model with limited labeled data. The model learns feature representations that are sensitive to skill level and have good discriminative power, thereby achieving objective skill scoring of clinical operation videos. Attached Figure Description
[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating a clinical operation video scoring method based on contrastive learning provided in this invention. Figure 2 This invention provides a specific OSATS quantitative skill scoring standard for the dressing change video. Figure 3 This is a schematic diagram illustrating the comparative learning method for a clinical operation video scoring method based on comparative learning, which is provided by the present invention. Detailed Implementation
[0018] Specific details are set forth in the following description in conjunction with the accompanying drawings to provide a full understanding of the invention. The invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0019] In this invention, the spatiotemporal feature extraction network refers to a deep learning network that can simultaneously process the spatial information of video images and the temporal dynamic information between frames. The spatiotemporal transformation module (TSM) is used in this invention.
[0020] Skill rating refers to the quantitative level or score given by senior experts based on the Standardized Assessment Scale (OSATS) for the operator's skills demonstrated in a clinical procedure video. In this invention, the skill rating is the overall quantitative score for each procedure in the clinical procedure video within the OSATS.
[0021] In this invention, the clinical operation videos that require scoring commonly include clinical surgical procedures, dressing change procedures, basic nursing procedures, invasive diagnostic and treatment procedures, and procedures specific to each department. Specific quantitative scoring can directly adopt the scoring sheets for standardized residency training in various regions, such as the "Scoring Sheet for Each Examination Station of the Standardized Residency Training Clinical Practice Ability Graduation Examination in Zhejiang Province".
[0022] Support set: refers to a set of representative video features containing a small number of known labels for each skill rating level, used as a benchmark for classification or measurement.
[0023] Figure 1 This is a flowchart illustrating a clinical operation video scoring method based on contrastive learning provided in this invention, comprising the following steps: 1. Acquisition and preparation of clinical operation video data; 2. Preprocessing of clinical operation videos; 3. Obtaining clinical operation video features using a spatiotemporal feature extraction network; 4. Training using contrastive learning based on support sets; 5. Predicting video skill scores and evaluating performance.
[0024] I. Specifically, the main function of the clinical operation video data acquisition and preparation steps is to read the videos and obtain the corresponding quantitative skill scores from OSATS, which serve as benchmark labels for training and assessment. Figure 2 Taking the specific quantitative scoring of the dressing change technique as an example, the total score is divided into eight levels. The collected labeled video data is divided into training set, support set, and test set.
[0025] Preferably, this step involves selecting five representative video samples for each different skill rating level. These samples serve as reference standards in subsequent comparative learning training and the final rating prediction stages.
[0026] II. Specifically, the clinical operation video preprocessing mainly includes the following three steps: 1. Extract a series of frames or consecutive video segments from a video stream. Set the number of frames in each video segment, the time interval between sampled frames, and the total number of segments sampled from each complete video.
[0027] 2. Decode the selected video frames from the original compressed format H.264 into a processable image data format.
[0028] 3. Perform spatial transformation and data augmentation on the decoded single-frame images. This mainly includes: size transformation: using bilinear interpolation to adjust the frame image to the standard 224x224 pixel resolution required for model input; data augmentation: applying spatial transformations such as multi-scale cropping and random horizontal flipping to improve the model's generalization ability; formatting: integrating the processed multi-frame image data and converting it into tensor format, specifying the channel order as batch size, number of channels, height, and width.
[0029] III. Specifically, obtaining clinical operation video features using a spatiotemporal feature extraction network mainly involves the following three steps: 1. The TSM network was selected as the backbone network. Model weights pre-trained on the large-scale general video action recognition dataset Something-Something V2 were used. To initialize network parameters .
[0030] 2. Convert the preprocessed video tensor Input into the selected spatiotemporal feature extraction network In the middle, the network performs forward propagation computation, outputting feature maps or feature vectors that represent the high-level semantic information of the input video segment. : 3. To better adapt the model to specific clinical skills assessment tasks, the parameters of the pre-trained model were adjusted. Fine-tuning is performed. The specific strategy is to freeze earlier layers of the network. The parameters are set so that they remain constant during training, while only the parameters of the deeper layers of the network are trained. .
[0031] IV. Specifically, a schematic diagram of the contrastive learning training steps based on support sets can be seen. Figure 3 It mainly includes the following three steps: 1. Raw features output from the feature extraction network Further feature mapping is needed for contrastive loss calculation. Global adaptive average pooling is then applied. Compressing the spatial dimension of the feature map yields a fixed-length feature vector. Subsequently, the eigenvectors are normalized using the L2 norm, calculated as follows: in, It is an eigenvector The Dimensional element. Normalized features. Located on the unit hypersphere.
[0032] 2. The core of the method of this invention lies in using a contrastive loss function to guide feature learning. Preferably, it combines two contrastive losses: (a) intra-class contrastive loss and (b) cross-set contrastive loss. (a) Intra-class contrastive loss aims to enhance the intra-class compactness and inter-class separability of features. For a batch of samples from the same subset of data... Normalized features and their corresponding skill rating tags First, calculate the original similarity score matrix. ,in , The temperature hyperparameter is set to 0.07. To avoid comparing samples with themselves and to perform numerical stabilization, a mask is defined with negative infinity on the diagonal and zeros at other positions. Calculate the effective index score. ,in It is the Kronecker function. The value is 1 if the condition is met, and 0 otherwise. Then, a positive sample mask is defined. ,in If and only if and Otherwise, it is 0. Calculate each sample. Intra-class contrast loss for: Final intra-class loss It is the average of the losses of all samples: ; (b) Cross-set comparison loss: Using the support set as a reference, the feature learning of the training set samples is guided. Assume the training set batch contains... Features and tags Supported sets include Features and tags Calculate the original similarity score matrix between the features in the training set and the features in all support sets. ,in Define a cross-set positive sample mask. ,in If and only if Otherwise, it is 0. Calculate the result for each training sample. Cross-set contrast loss for: Final cross-set loss It is the average loss of all training samples: .
[0033] 3. Combine the multiple loss terms calculated above into a final total loss function by weighted summation. : in, , and The preset weights are all set to 0.5, used to adjust the contribution of various losses to the overall optimization. The Adam optimizer is used to calculate the gradient of the total loss with respect to the trainable parameters of the model; this training process is repeated for 200 epochs.
[0034] V. Specifically, the video skills score prediction and performance evaluation process mainly includes the following two steps: 1. After model training, the learned feature representations and support set information are used to predict skill scores for unseen clinical operation videos. Specifically, for each support set video, the final, post-processed feature vector is saved, and these feature vectors are... and their corresponding known skill rating tags These features are stored to form a reference feature library. Then, for a new video requiring skill level assessment, the same preprocessing and feature extraction procedures as steps two and three are performed to obtain its normalized feature vector. Calculate the feature vector of the video to be evaluated. Compared with all feature vectors in the support set feature library The similarity score between them. Preferably, cosine similarity is used as the similarity measure to obtain a similarity vector. ,in .
[0035] 2. Find the similarity vector The index corresponding to the maximum value in the middle This will support centralized first Labels of each sample Predicted score for the video to be evaluated .
[0036] 3. To objectively measure the effectiveness of the method of this invention, an evaluation index system is adopted, consisting of (a) precision accuracy, (b) tolerance accuracy, and (c) the area under the macro-average receiver operating characteristic curve. (a) Predictive score calculation based on accuracy Compared with the actual scores given by experts The percentage of completely identical test samples: in, It is the number of test samples. It is an indicator function (1 if the condition is true, 0 otherwise).
[0037] (b) Tolerance accuracy calculation: The absolute value of the difference between the predicted score and the actual score is less than or equal to a certain preset threshold. The percentage of test samples: (c) The area under the macro-average receiver operating characteristic curve, and the scores of each category obtained by the category maximum similarity scoring method. As a model for each sample Category The prediction confidence level. For each skill rating level Treat it as a binary classification problem and calculate the area under the corresponding ROC curve. Then, calculate all Arithmetic mean of AUC values at each level: In summary, this invention proposes a clinical operation video scoring method based on contrastive learning. Its core technological contributions are: designing and implementing a data flow capable of effectively processing clinical operation video data; employing a powerful spatiotemporal feature extraction network (TSM) to capture key spatiotemporal information of clinical operations; creatively introducing a contrastive learning framework that incorporates a support set, learning feature representations that are highly sensitive to skill levels and possess good discriminative power by simultaneously optimizing intra-class and cross-set contrastive losses; and finally, predicting the skill score based on the feature similarity with the support set samples.
Claims
1. A clinical operation video scoring method based on contrastive learning, characterized in that, Includes the following steps: Acquire video data containing clinical procedure processes and corresponding expert skill rating tags; The video data is divided into a training set, a support set, and a test set, wherein the support set contains several representative video samples for each skill rating level; The video data in the training set, support set, and test set are preprocessed to obtain preprocessed video data. The preprocessed video data is processed using a spatiotemporal feature extraction network to extract spatiotemporal features; Based on the support set, the spatiotemporal feature extraction network is trained using a contrastive learning framework, which is optimized using the spatiotemporal features of the training set and the support set and the corresponding skill rating labels. For any video from the test set or a new video to be evaluated, its spatiotemporal features are extracted, and the skill rating level of the video to be evaluated is predicted based on the similarity between the spatiotemporal features of the video to be evaluated and the spatiotemporal features of the samples in the support set.
2. The method according to claim 1, characterized in that, The process of training the spatiotemporal feature extraction network using a contrastive learning framework includes the following steps: Calculate the intra-class contrast loss between samples within the training set; Calculate the intra-class contrast loss between samples within the support set; Calculate the cross-set contrast loss between the training set samples and the support set samples; The intra-class contrast loss and the cross-set contrast loss are combined according to preset weighting coefficients to obtain the total loss function: Based on the total loss function, the trainable parameters of the spatiotemporal feature extraction network are updated using an optimization algorithm.
3. The method according to claim 2, characterized in that, The calculation process of the intra-class contrast loss includes the following steps: For a batch of samples in the same subset of data, calculate the similarity score between features of every two samples. This score is obtained by multiplying the two features by the dot product and dividing by the temperature hyperparameter. Calculate a separate loss value for each sample in the set. The calculation process uses a positive sample mask. Only when two samples have the same class label and are not the same sample are they marked as a positive sample pair. The mask value is defined as 1, otherwise it is 0. When calculating the loss of a sample, only its relationship with all positive sample pairs is considered. For each positive sample pair, a logarithmic term is subtracted from their similarity scores. This logarithmic term is the exponential sum of the similarity scores of the current sample with all other samples in the set except itself, and then the natural logarithm is taken. The summation of the above differences for all positive sample pairs, divided by the total number of positive sample pairs, is averaged to obtain the intra-class contrast loss value for that sample.
4. The method according to claim 2, characterized in that, The calculation process for the cross-set comparison loss includes the following steps: For each sample feature from the training set and all sample features from the support set, a similarity score is calculated between them. This score is obtained by multiplying the training set sample features by the support set sample features and dividing by the temperature hyperparameter. For each training set sample, calculate its individual loss value. The calculation process is as follows: First, establish a cross-set positive sample mask matrix. Only when the class label of a training set sample is exactly the same as the class label of a support set sample are they considered as positive sample pairs and marked as 1 at the corresponding position in the mask matrix; otherwise, they are marked as 0. When calculating the loss, only combinations of pairs labeled as positive are considered. For the current training sample, the similarity score calculated with all positive support set features is subtracted by a logarithmic term. This logarithmic term is the logarithm of the sum of the exponents of the similarity scores between the current training sample and all samples in the support set. Finally, this difference for all positive pairs is summed and normalized by dividing by the total number of positive pairs, thus obtaining the cross-set contrastive loss value for that training sample.
5. The method according to claim 1, characterized in that, After extracting the spatiotemporal features, the process also includes processing the spatiotemporal features. The post-processing steps include applying global pooling and L2 norm normalization. The normalization operation is calculated through the following process: summing the squares of the values of each dimension of the original feature vector, then taking the square root of the summation result to obtain the L2 norm of the vector; finally, dividing each dimension value of the original feature vector by the calculated L2 norm to generate a new normalized feature vector.
6. The method according to claim 1, characterized in that, The process of predicting the skill rating level of the video to be evaluated based on the similarity between the spatiotemporal features of the video to be evaluated and the spatiotemporal features of the support set samples includes the following steps: Calculate the similarity score between the features of the video to be evaluated and the features of each sample in the support set; From all the calculated similarity scores, find the one with the largest value and determine its corresponding support set samples; The label of this sample is directly used as the predicted rating level of the video to be evaluated.
7. A clinical operation video scoring device based on contrastive learning, characterized in that, include: The acquisition unit acquires video data containing clinical operation procedures and corresponding expert skill rating labels; The preprocessing unit divides the video data into a training set, a support set, and a test set, wherein the support set contains several representative video samples for each skill rating level. The video data in the training set, support set, and test set are preprocessed to obtain preprocessed video data. The spatiotemporal feature extraction unit uses a spatiotemporal feature extraction network to process the preprocessed video data and extract spatiotemporal features; The training unit trains the spatiotemporal feature extraction network using a contrastive learning framework based on the support set. The contrastive learning framework is optimized using the spatiotemporal features of the training set and the support set, as well as the corresponding skill rating labels. The prediction and evaluation unit extracts the spatiotemporal features of the video to be evaluated from the test set or a new video to be evaluated, and predicts the skill rating level of the video to be evaluated based on the similarity between the spatiotemporal features of the video to be evaluated and the spatiotemporal features of the samples in the support set.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the clinical operation video scoring method based on contrastive learning as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the clinical operation video scoring method based on contrastive learning as described in any one of claims 1 to 6.