Method and system for automatic assessment and description generation of surgical skills
By using a multimodal fusion method of video and text features, the subjectivity and time-consuming nature of traditional surgical skill assessment are solved, enabling multi-dimensional automated assessment and natural language description of surgical skills, thus improving the accuracy and efficiency of the assessment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2026-03-23
- Publication Date
- 2026-07-10
Smart Images
Figure CN122369830A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence, computer vision, natural language processing, and medical auxiliary assessment technology. Specifically, it relates to a method and system for automatic assessment and description generation of surgical skills, and more specifically, it relates to a method and system for automatic assessment and description generation of surgical skills based on video and text multimodal interaction. Background Technology
[0002] Surgical skills assessment is a crucial component of surgical training and evaluation. With the rapid development of robot-assisted surgery, the demand for surgical skills training has increased significantly, meaning that accurate surgical skills assessment will be key to ensuring training quality and patient safety. Traditional surgical skills assessments primarily employ intraoperative and postoperative evaluations. Intraoperative evaluation involves direct observation and scoring of surgical procedures during the operation, while postoperative evaluation involves observing and scoring the surgical procedures after the operation by replaying the surgical video. However, traditional surgical skills assessments suffer from several problems: they rely heavily on the surgeon's professional skill and personal experience, resulting in subjective assessments and are extremely time-consuming, hindering their widespread adoption.
[0003] In recent years, surgical skills assessment has gradually shifted from manual to automated assessment, reducing its reliance on human resources by utilizing computer resources. However, existing automated assessment methods are mostly based on a single modality, such as assessing movement quality solely through video or generating only video descriptions. They lack a multi-dimensional comprehensive evaluation of the surgical process, making it difficult to simultaneously provide professional scores and natural language descriptions, which is detrimental to teaching feedback and skills improvement.
[0004] Therefore, based on the difficulties and challenges in the field of surgical skill assessment, this invention utilizes multimodal fusion of video and text features to automatically assess surgical operations and generate textual descriptions of the surgical operations, providing a multi-dimensional comprehensive evaluation for surgical skill assessment.
[0005] This invention proposes a method for surgical skill assessment and surgical procedure text description generation that utilizes multimodal fusion to achieve dual-task output. Specifically, this invention transforms scoring information into text form through learnable prompts and utilizes a context-aware and score-aware Transformer to achieve bidirectional interaction between video and text modalities, enhancing the accurate fusion of assessment information. In text generation, a template containing fixed assessment information such as action categories and scores is introduced, and learnable lexical units are inserted to enhance the flexibility and professionalism of language expression, achieving natural embedding of assessment information. Continuous scores are converted into discrete intervals, and categorized text prompts are constructed, transforming the score prediction task into a video. The text matching problem is addressed to improve the accuracy of scoring information in the generated text. Supervision is implemented by combining cross-entropy loss and mean squared error loss, and a ternary attention mask is used to guide the model to focus on key evaluation information and video content during the generation process. Based on the existing surgical video dataset, narrative text containing professional terminology and multi-dimensional evaluation information is generated through cue engineering and a large language model, and verified by domain experts to improve data quality and applicability. Summary of the Invention
[0006] In view of the deficiencies in the prior art, the purpose of this invention is to provide a method and system for automatically assessing and describing surgical skills.
[0007] A method for automatically assessing and generating surgical skills according to the present invention includes: Step S1: Obtain the surgical video dataset, and annotate the obtained surgical video dataset to obtain the corresponding text description and video real score; Step S2: Preprocess the surgical video, the corresponding text description, and the video's true score to obtain the preprocessed surgical video, the corresponding text description, and the video's true score. Step S3: Train the visual-language model based on the preprocessed surgical video, the corresponding text description, and the video's real score to obtain the trained visual-language model; Step S4: Acquire the target surgical video, preprocess the acquired target surgical video, use the trained vision-language model to score the preprocessed target surgical video, and generate video description text.
[0008] Preferably, the preprocessing of the surgical video in step S2 includes: Step S2.1: Decode the surgical video to obtain a frame sequence; perform frame sampling according to a preset sampling strategy, wherein the entire video is uniformly sampled in the time dimension under a uniform sampling strategy, and the sampled frame sequence is used for subsequent processing; Step S2.2: Standardize the format of the current frame data; when the frame data is empty, use a zero-value frame sequence of a preset size as a fallback in the dataset augmentation process; when the frame data is in tensor format, convert it to NumPy format and rearrange the dimensions, from (T,C,H,W) to (T,H,W,C), where T is the frame sequence length, C is the number of channels, H is the image height, and W is the image width; then convert each frame to PIL.Image and form a frame list; Step S2.3: Perform data augmentation processing on all frame images in the frame sequence, including resizing and random cropping, then convert the PIL.Image format into a tensor format that PyTorch can process, and finally perform normalization processing to obtain the processed video data.
[0009] Preferably, the preprocessing of the text description in step S2 includes: Step S2.4: Use a pre-trained word segmentation model to segment the text description to obtain a word segmentation sequence; Step S2.5: For the obtained word segmentation sequence, perform structural normalization processing, add preset special markers at the beginning and end of the word segmentation sequence to distinguish sentence boundaries and functional nodes; for scenarios with additional text, complete the orderly splicing of multiple text segments and paragraph differentiation to construct a recognizable structured sequence. Step S2.6: Apply a multi-strategy dynamic masking mechanism to the identifiable structured sequence to obtain the processed text description.
[0010] Preferably, the preprocessing of the total video score in step S2 includes: Based on the statistical distribution of actual scores in the surgical video dataset, and divided into multiple score intervals according to a preset number, each score interval is converted into a corresponding score prompt text. The score prompt text is then encoded using a text encoder to obtain the actual score prompt feature sequence.
[0011] Preferably, step S3 includes: Step S3.1: Use the Video Swin Transformer model in the vision-language model to extract video features from the preprocessed surgical video to obtain a video feature sequence; Step S3.2: Use the score prediction model in the visual-language model to predict the surgical skill score based on the video feature sequence, and convert the obtained predicted score into score text; Step S3.3: Construct a text description sequence based on the preprocessed text description and the preset prompt template, write the fractional text back into the text description sequence, and obtain the fused text features; Step S3.4: Using a bidirectional cross-attention fusion mechanism, the real rating prompt feature sequence is used as the query and the video feature sequence is used as the key and value. Video condition enhancement is performed on the real rating prompt feature sequence to obtain the enhanced real rating prompt feature sequence. The video feature sequence is used as the query and the enhanced real rating prompt feature sequence is used as the key and value to perform prompt semantic enhancement on the video feature, thereby obtaining the video enhanced feature that integrates score semantic information. Step S3.5: Jointly model and train BERT based on the fused text features and the video enhancement features with fused score semantic information to obtain the trained visual-language model.
[0012] Preferably, step S3.4 includes:
[0013] in, Enhanced real score hint feature sequence; This refers to the Multi-head CrossAttention mechanism. Provide feature sequences for accurate scoring; For video feature sequences; The learnable coefficient;
[0014] in, Enhanced video features that incorporate fractional semantic information; The learnable coefficient.
[0015] Preferably, in the process of jointly modeling and training BERT based on the fused text features and the fused score semantic information of the video enhancement features, the total loss function is:
[0016] in, , , , This is the loss coefficient; The mean squared error loss for score regression; Cross-entropy is used for classification loss; Cross-entropy loss generated for text; The loss is for sparse mask regularization; The sparse mask regularization loss is:
[0017] in, For regularization hyperparameters, Indicates the first The video and the first The activation values of the attention mask between videos.
[0018] Preferably, step S4 includes: Step S4.1: Use the Video Swin Transformer model in the trained vision-language model to extract video features from the target surgical video to obtain the target video feature sequence; Step S4.2: Using the score prediction model in the trained visual-language model, the surgical skill score is predicted based on the feature sequence of the target video to obtain the predicted score of the target video, and the predicted score of the target video is converted into predicted score text; Step S4.3: Construct an initial text description sequence based on the manually preset template and the MASK placeholder sequence, write the predicted score text back into the initial text description sequence, and obtain the fused target text features; Step S4.4: Using a bidirectional cross-attention fusion mechanism, the real rating prompt feature sequence is used as the query and the target video feature sequence is used as the key and value. Video condition enhancement is performed on the real rating prompt feature sequence to obtain the enhanced real rating prompt feature sequence. The target video feature sequence is used as the query and the enhanced real rating prompt feature sequence is used as the key and value. The prompt semantic enhancement is performed on the target video feature to obtain the target video enhanced feature that integrates score semantic information. Step S4.5: Input the target video enhancement features with fused semantic information and the fused target text features into the BERT caption generation model in the trained visual-language model to generate descriptive captions representing the surgical skill assessment results.
[0019] An automatic surgical skills assessment and description generation system according to the present invention includes: Module M1: Acquires surgical video datasets, annotates the surgical video datasets, and obtains corresponding text descriptions and video real scores for the surgical video datasets; Module M2: Preprocesses the surgical video, corresponding text description, and video real score to obtain the preprocessed surgical video, corresponding text description, and video real score. Module M3: The visual-language model is trained based on the preprocessed surgical video, the corresponding text description, and the video's actual score to obtain the trained visual-language model; Module M4: Acquires the target surgical video, preprocesses the acquired target surgical video, scores the preprocessed target surgical video using the trained vision-language model, and generates video description text.
[0020] Preferably, the preprocessing of the total video score in module M2 includes: Based on the statistical distribution of actual scores of surgical videos in the surgical video dataset, and divided into multiple score intervals according to a preset number, each score interval is converted into a corresponding score prompt text, and the score prompt text is encoded using a text encoder to obtain the actual score prompt feature sequence. The module M3 includes: Module M3.1: Uses the Video Swing Transformer model in the vision-language model to extract video features from the preprocessed surgical video to obtain a video feature sequence; Module M3.2: Utilizes the score prediction model in the visual-language model to predict the surgical skill score based on the video feature sequence, and converts the obtained predicted score into score text; Module M3.3: Constructs a text description sequence based on the preprocessed text description and preset prompt template, writes the fractional text back into the text description sequence, and obtains the fused text features; Module M3.4: Through a bidirectional cross-attention fusion mechanism, the real score suggestion feature sequence is used as the query and the video feature sequence is used as the key and value. Video condition enhancement is performed on the real score suggestion feature sequence to obtain the enhanced real score suggestion feature sequence. The video feature sequence is used as the query and the enhanced real score suggestion feature sequence is used as the key and value to perform suggestion semantic enhancement on the video features, thereby obtaining video enhanced features that fuse score semantic information. Module M3.5: Jointly model and train BERT based on fused text features and video enhancement features with fused semantic information to obtain a trained visual-language model; The module M4 includes: Module M4.1: Utilizes the Video Swin Transformer model in the trained vision-language model to extract video features from the target surgical video, obtaining the target video feature sequence; Module M4.2: Utilizes the score prediction model in the trained visual-language model to predict the surgical skill score based on the feature sequence of the target video, and converts the predicted score of the target video into predicted score text; Module M4.3: Constructs an initial text description sequence based on manually preset templates and MASK placeholder sequences, writes the predicted score text back into the initial text description sequence, and obtains the fused target text features; Module M4.4: Through a bidirectional cross-attention fusion mechanism, the real score suggestion feature sequence is used as the query and the target video feature sequence is used as the key and value. Video condition enhancement is performed on the real score suggestion feature sequence to obtain the enhanced real score suggestion feature sequence. The target video feature sequence is used as the query and the enhanced real score suggestion feature sequence is used as the key and value to perform suggestion semantic enhancement on the target video features, thereby obtaining the target video enhanced features that fuse score semantic information. Module M4.5: Input the target video enhancement features with fused semantic information and the fused target text features into the BERT caption generation model in the trained visual-language model to generate descriptive captions representing the surgical skill assessment results.
[0021] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention extracts video features and text features, and uses a cross-attention mechanism to achieve dual-modal feature fusion, realizing dual-task collaborative learning of score prediction and text generation. Finally, it outputs surgical skill scoring results and surgical operation text descriptions through dual-task branches. 2. This invention simultaneously processes video and text features to achieve deep bimodal fusion, promoting the learning of both "rating prediction" and "text generation" tasks. Furthermore, to enhance the model's understanding of scores, learnable prompt templates are used to convert rating information into text form, and a cross-attention mechanism is employed to achieve bidirectional interactive fusion of video and text features. 3. The multi-task joint loss function in this invention includes rating regression loss, action classification loss, text generation loss and sparse attention regularization loss, and achieves multi-task collaborative optimization through weighted combination; Attached Figure Description Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a visual-language model training method.
[0022] Figure 2 A flowchart for generating automatic assessment and description methods for surgical skills. Detailed Implementation
[0023] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0024] Example 1 The present invention provides an automatic surgical skill assessment and description generation method, such as... Figures 1 to 2 As shown, it includes: Step S1: Obtain the surgical video dataset, annotate the surgical video dataset, and obtain the corresponding text description and video real score of the surgical video dataset; Step S2: Preprocess the surgical video, the corresponding text description, and the video's true score to obtain the preprocessed surgical video, the corresponding text description, and the video's true score. Specifically, the preprocessing of the surgical video includes: Step S2.1: Decode the surgical video to obtain a frame sequence; perform frame sampling according to a preset sampling strategy, wherein the entire video is uniformly sampled in the time dimension under a uniform sampling strategy, and the sampled frame sequence is used for subsequent processing; Step S2.2: Standardize the format of the current frame data; when the frame data is empty, use a zero-value frame sequence of a preset size as a fallback in the dataset augmentation process; when the frame data is in tensor format, convert it to NumPy format and rearrange the dimensions, from (T,C,H,W) to (T,H,W,C), where T is the frame sequence length, C is the number of channels, H is the image height, and W is the image width; then convert each frame to PIL.Image and form a frame list; Step S2.3: Perform data augmentation processing on all frame images in the frame sequence, including resizing and random cropping, then convert the PIL.Image format into a tensor format that PyTorch can process, and finally perform normalization processing to obtain the processed video data.
[0025] The preprocessing of the text description includes: Step S2.4: Use a pre-trained word segmentation model to segment the text description to obtain a word segmentation sequence; Step S2.5: For the obtained word segmentation sequence, perform structural normalization processing, add preset special markers at the beginning and end of the word segmentation sequence to distinguish sentence boundaries and functional nodes; for scenarios with additional text, complete the orderly splicing of multiple text segments and paragraph differentiation to construct a recognizable structured sequence. Step S2.6: Apply a dynamic masking mechanism that supports multiple strategies to the identifiable structured sequence to obtain the processed text description.
[0026] The preprocessing of the total video score includes: Based on the statistical distribution of actual scores in the surgical video dataset, and divided into multiple score intervals according to a preset number, each score interval is converted into a corresponding score prompt text. The score prompt text is then encoded using a text encoder to obtain the actual score prompt feature sequence.
[0027] Step S3: Train the visual-language model based on the preprocessed surgical video, the corresponding text description, and the video's real score to obtain the trained visual-language model; Specifically, step S3 includes: Step S3.1: Use the Video Swin Transformer model in the vision-language model to extract video features from the surgical video to obtain a video feature sequence; Step S3.2: Use the score prediction model in the visual-language model to predict the surgical skill score based on the video feature sequence, and convert the obtained predicted score into score text; Step S3.3: Construct a text description sequence based on the preprocessed text description and the preset prompt template, write the fractional text back into the text description sequence, and obtain the fused text features; Step S3.4: Using a bidirectional cross-attention fusion mechanism, the real rating prompt feature sequence is used as the query and the video feature sequence is used as the key and value. Video condition enhancement is performed on the real rating prompt feature sequence to obtain the enhanced real rating prompt feature sequence. The video feature sequence is used as the query and the enhanced real rating prompt feature sequence is used as the key and value to perform prompt semantic enhancement on the video feature, thereby obtaining the video enhanced feature that integrates score semantic information. More specifically,
[0028] in, Enhanced real score hint feature sequence; This refers to the Multi-head CrossAttention mechanism. Provide feature sequences for accurate scoring; For video feature sequences; The learnable coefficient;
[0029] in, Enhanced video features that incorporate fractional semantic information; The learnable coefficient.
[0030] Step S3.5: Jointly model and train BERT based on the fused text features and the video enhancement features with fused score semantic information to obtain the trained visual-language model.
[0031] Specifically, in the joint modeling and training process of BERT based on the fused text features and the fused semantic information of the video enhancement features, the total loss function is:
[0032] in, , , , This is the loss coefficient; The mean squared error loss for score regression; Cross-entropy is used for classification loss; Cross-entropy loss generated for text; The loss is for sparse mask regularization; The sparse mask regularization loss is:
[0033] in, For regularization hyperparameters, Indicates the first The video and the first The activation values of the attention mask between videos.
[0034] Step S4: Acquire the target surgical video, preprocess the acquired target surgical video, use the trained vision-language model to score the preprocessed target surgical video, and generate video description text.
[0035] Specifically, step S4 includes: Step S4.1: Use the Video Swin Transformer model in the trained vision-language model to extract video features from the target surgical video to obtain the target video feature sequence; Step S4.2: Using the score prediction model in the trained visual-language model, the surgical skill score is predicted based on the feature sequence of the target video to obtain the predicted score of the target video, and the predicted score of the target video is converted into predicted score text; Step S4.3: Construct an initial text description sequence based on the manually preset template and the MASK placeholder sequence, write the predicted score text back into the initial text description sequence, and obtain the fused target text features; Step S4.4: Using a bidirectional cross-attention fusion mechanism, the real rating prompt feature sequence is used as the query and the target video feature sequence is used as the key and value. Video condition enhancement is performed on the real rating prompt feature sequence to obtain the enhanced real rating prompt feature sequence. The target video feature sequence is used as the query and the enhanced real rating prompt feature sequence is used as the key and value. The prompt semantic enhancement is performed on the target video feature to obtain the target video enhanced feature that integrates score semantic information. Step S4.5: Input the target video enhancement features with fused semantic information and the fused target text features into the BERT caption generation model in the trained visual-language model to generate descriptive captions representing the surgical skill assessment results.
[0036] The present invention also provides an automatic surgical skill assessment and description generation system, which can be implemented by executing the process steps of the automatic surgical skill assessment and description generation method. That is, those skilled in the art can understand the automatic surgical skill assessment and description generation method as a preferred embodiment of the automatic surgical skill assessment and description generation system.
[0037] This invention extracts video and text features and uses a cross-attention mechanism to achieve dual-modal feature fusion, enabling collaborative learning of both score prediction and text generation. Finally, it outputs surgical skill scoring results and surgical operation text descriptions through dual-task branches.
[0038] Example 2 Example 2 is a preferred example of Example 1. A method for automatically assessing and generating surgical skills according to the present invention includes: Step 1: Surgical procedure video acquisition and high-quality tag construction; Specifically, the high-quality tag construction in step 1 includes: Step 1.1: Obtain the surgical video dataset, annotate the surgical video dataset, and obtain the corresponding text descriptions and video real scores for the surgical video dataset.
[0039] Step 2: Preprocess the video, text description, and total video score separately; Specifically, step 2 includes: Video preprocessing includes: Step 2.1.1: First, perform online decoding and video pre-frame extraction on the video file. Video pre-frame extraction is achieved through grouped random sampling. Then, downsample the video at equal intervals and adjust the image resolution.
[0040] Step 2.1.2: Detect the original format of the input frame data. If the input is empty, return a zero tensor. When the input is detected to be in torch tensor format, perform format conversion to uniformly convert the data to NumPy data format and rearrange the dimensions. Convert to The format is as follows: T is the time dimension, representing the length of the video frame sequence; C is the channel dimension, representing the RGB three-channel color information; H is the height dimension, representing the number of vertical pixels of the image; W is the width dimension, representing the number of horizontal pixels of the image; then each frame is converted into PIL.Image format and collected into a frame sequence.
[0041] When in training mode, the video is resized, then randomly cropped, then the PIL image is converted into a tensor format that PyTorch can process, and finally normalized. When the video data is the target surgical video, the target surgical video is resized and then cropped in the center during model-based inference. Then, the PIL image is converted into a tensor format that PyTorch can process, and finally, it is standardized.
[0042] Specifically, the PIL image sequence is converted into a float tensor, and the pixels are scaled to [0,1] to produce... Format to suit video processing.
[0043] Standardization implementation process: Standardization is performed on each channel separately, using the following formula:
[0044] in, The value after standardization. The original value, The mean for each channel, The standard deviation for each channel.
[0045] Preprocessing for text descriptions includes: Step 2.2.1: Convert the text into sub-words. During the model training process, real word segmentation technology is used, then part-of-speech tagging verification is performed, and finally attention weight alignment detection is performed. When performing inference based on the model, a masked tag sequence is generated first, a fixed-length prediction template is constructed, and then a standardized input format is provided for the text generation task.
[0046] Step 2.2.2: Add special markers [CLS] and [SEP] to the beginning and end of the sequence, respectively. [CLS] is commonly used for sentence-level representation and also serves as the starting identifier for generation / labeling tasks. [SEP] indicates the end of a sentence or sentence separation, helping the model distinguish sentence boundaries from subsequent labeled segments. If there is additional text, it is concatenated after the original text and distinguished using different paragraph markers.
[0047] Step 2.2.3: To achieve masked language model training, this embodiment adopts a multi-strategy masking mechanism: Masking position selection strategies: There are four position selection strategies: random masking, which randomly selects positions in the text for masking with a preset probability; part-of-speech tagging-based masking, which prioritizes masking content words such as nouns and verbs while preserving function words; attention weight-based masking, which selects important words for masking based on pre-computed language model attention weights; and dynamic masking, which masks the entire text sequence during the inference stage for text generation tasks.
[0048] Mask content replacement strategy: For selected mask positions, replace them with a special mask marker [MASK], a random word from the vocabulary, or leave the original word unchanged.
[0049] Preprocessing for the total video score includes: Based on the statistical distribution of actual scores in the surgical video dataset, and divided into multiple score intervals according to a preset number, each score interval is converted into a corresponding score prompt text. The score prompt text is then encoded using a text encoder to obtain the actual score prompt feature sequence.
[0050] Step 3: Train the visual-language model based on the preprocessed surgical video, the corresponding text description, and the video's real score to obtain the trained visual-language model; Specifically, step 3 includes: Step 3.1: Use the Video Swin Transformer model in the vision-language model to extract video features from the surgical video to obtain a video feature sequence; Specifically, step 3.1 includes: In this embodiment, the Swin model is used to extract features from the input video sequence to obtain the corresponding spatiotemporal feature embeddings; the input video is denoted as:
[0051] in, For batch size, For time frames, For the number of channels, For the height of the frame, The width of the frame.
[0052] The video feature embedding is obtained through the video encoder (VideoEncoder):
[0053] Preferred:
[0054] in, The dimension of the feature vector for each frame.
[0055] More specifically, step S3.1 includes: Step 3.1.1: Data reading and format standardization; Read the raw video frame tensor from the keyword parameter dictionary; its shape can be: or
[0056] They are uniformly organized into a form that separates the time dimension and the batch dimension: First, unify them into shapes. ; Then expand the time dimension to the batch dimension to construct a batch of images frame by frame:
[0057] in, This represents an image batch consisting of all video frames, which is then fed into the Swing Transformer model.
[0058] Step 3.1.2: Data aggregation and batch processing input; The data loader aggregates the frame tensors corresponding to multiple video samples into batch data and packages them into the following shape:
[0059] As input to the forward inference of the neural network.
[0060] Step 3.1.3: Spatial feature extraction; Spatial feature extraction is performed on each frame of the image using the Swin Transformer model, including the following sub-steps: Step 3.1.3.1: Image patch embedding; For any frame image This performs an image-to-block embedding transformation. Non-overlapping image block partitioning and linear projection are achieved through 2D convolution. Let the block size be... Using convolution kernels Size is Step length for Convolution operation:
[0061] The output shape is:
[0062]
[0063]
[0064] in, This is the embedding dimension. The feature map is then flattened into a sequence of patches, and an optional normalization operation is performed:
[0065]
[0066] Here, LN represents the LayerNorm operation.
[0067] Step 3.1.3.2: Window partitioning and window-based multi-head self-attention; feature map By window size Divide the window into several non-overlapping windows to obtain the window tensor. :
[0068] in This represents the number of windows in each feature map.
[0069] Perform Window Multi-head SelfAttention (WMSA) on the token sequence within each window:
[0070] in, These are query, key, and value matrices, respectively. For single-head dimension, This is a relative position offset.
[0071] Step 3.1.4: Layered construction and downsampling. Swin Transformer constructs layered feature representations by stacking multiple levels of BasicLayer, and inserts Patch Merging modules between adjacent stages to achieve downsampling and increase the number of channels; For the input feature map resolution is Patch Merging concatenates adjacent 2×2 blocks along the channel dimension and reduces the dimension through linear layers:
[0072] Multi-scale feature maps with decreasing resolution and enhanced semantic information are generated layer by layer.
[0073] Step 3.1.5: Displacement window attention; To overcome the locality limitation of fixed windows, a shifted window multi-head self-attention mechanism (SWMSA) is introduced: Perform a cyclic translation of the feature map in the spatial dimension, with a translation size of shift_size (usually half the window size):
[0074] Windowing and SWMSA computation are performed on the translated features to achieve cross-window information interaction. Inverse translation is performed on the attention output to restore the original spatial alignment. By alternating between WMSA and SWMSA across multiple layers, both local modeling and global context are considered.
[0075] Step 3.1.6: Feature Aggregation and Video-Level Temporal Feature Output: After completing multi-stage attention and hierarchical sampling, the Swin Transformer outputs the spatial feature map of the final layer. The final feature map corresponding to any frame is denoted as... .
[0076] Perform global average pooling on the feature map in the spatial dimension to obtain the feature vector of the frame:
[0077] in, , t is the spatial dimension of the final feature map; b is the batch dimension index, i.e., the number of the currently processed video sample in the batch; t is the frame index, i.e., which frame in the video; i is the index in the spatial height direction, corresponding to the row position of the final feature map; j is the index in the spatial width direction, corresponding to the column position of the final feature map. The temporal feature embedding of a video is constructed by stacking the features of each frame in chronological order.
[0078] in, Alternatively, it can be mapped to a preset dimension through a linear transformation.
[0079] Step S3.2: Use the score prediction model in the visual-language model to predict the surgical skill score based on the video feature sequence, and convert the obtained predicted score into score text; Step S3.3: Construct a text description sequence based on the preprocessed text description and the preset prompt template, write the fractional text back into the text description sequence, and obtain the fused text features; Step S3.4: Using a bidirectional cross-attention fusion mechanism, the real rating prompt feature sequence is used as the query and the video feature sequence is used as the key and value. Video condition enhancement is performed on the real rating prompt feature sequence to obtain the enhanced real rating prompt feature sequence. The video feature sequence is used as the query and the enhanced real rating prompt feature sequence is used as the key and value to perform prompt semantic enhancement on the video feature, thereby obtaining the video enhanced feature that integrates score semantic information. More specifically, a prompt-guided bimodal fusion mechanism is adopted, which consists of two stages: Enhanced prompt embedding based on video information Build learnable tips that include rating information Interacting with video features through cross-attention mechanisms:
[0080] in This is a multi-head cross-attention mechanism. For prompt embedding For video feature embedding, For the enhanced prompt embedding, The learnable coefficient.
[0081] Video Feature Embedding Enhancement Based on Text Information Use enhanced prompts to embed guiding video features to perceive scoring information:
[0082] in Video feature embedding to integrate rating information, The learnable coefficient.
[0083] Since the implementation logic of the two stages is similar, the following explanation will take video-based information embedding enhancement as an example: Data input preparation involves embedding video features as Key / Value roles, where the Value uses the same input as the Key; and embedding prompts as Query roles. Key / Value sequences and query sequences are then constructed.
[0084] Calculate attention score:
[0085] in, Attention score It is a single-head vector dimension.
[0086] If an additive attention mask is provided, it is added to mask areas that should not be noticed:
[0087] in, This represents the attention score.
[0088] Calculate attention probability: Apply softmax along the last dimension:
[0089] in, This represents the attention probability.
[0090] Then random deactivation is applied to the attention probability. If a head mask is provided, the head mask is multiplied back by the probability.
[0091] A weighted summation is performed, multiplying the resulting attention probabilities by the Values from the video feature embeddings and summing the results to generate the final output. This output incorporates the information from the video feature embeddings that is most relevant to the Query in the cue embedding.
[0092] Step S3.5: Jointly model and train BERT based on the fused text features and the video enhancement features with fused score semantic information to obtain the trained visual-language model.
[0093] The total loss function is:
[0094] in: The mean squared error loss for score regression; Cross-entropy loss for action classification; Cross-entropy loss generated for text; The loss is for sparse mask regularization; , , , This is the loss coefficient, used to balance various types of losses.
[0095] Among them, calculation Mean squared error loss for rating regression: The fused video features are averaged over time within the model, and continuous score predictions are obtained through a multi-layer fully connected layer. The samples are then weighted according to the difficulty coefficient.
[0096] calculate Cross-entropy loss for action classification: The model performs a temporal average of the fused video features, maps the true scores to categories according to the score intervals, obtains the classification prediction through the classification head, and then calculates the cross-entropy loss.
[0097] calculate Cross-entropy loss in text production: The Transformer encoder takes the fused visual features and the input word IDs, outputs a vector for each position in the forward pass, and calculates the cross-entropy for the predicted word positions.
[0098] calculate Sparse masking regularization loss: The calculation formula is as follows:
[0099] in For regularization hyperparameters, This represents the activation value of the attention mask.
[0100] If learnable attention masks are enabled, sparse regularization will be computed, and mask soft and hard transformations can be enabled.
[0101] Step 4: Acquire the target surgical video, preprocess the acquired target surgical video, use the trained vision-language model to score the preprocessed target surgical video, and generate video description text.
[0102] In this embodiment, the output using the dual-task branch includes: The scoring prediction branch regresses and classifies the fused spatiotemporal visual features to obtain the action quality score and difficulty category.
[0103] Text generation branch: Based on the above features and motion quality-related prompts, generate video description text.
[0104] Example 3 Example 3 is a preferred example of Example 1 and / or Example 2. A method for automatically assessing and generating surgical skills according to the present invention includes: Input and visual feature extraction steps: Read video frame tensors from the batch processing script, rearrange them to a uniform number of frames and spatial resolution, and then input them into the video Swin Transformer backbone network to encode the video into a spatiotemporal token sequence (sequence length L), and project it onto the preset hidden feature dimension through a linear transformation.
[0105] Action quality assessment steps: Perform global mean pooling on the above spatiotemporal token sequence to obtain the video global feature vector representing the entire action; input the vector into the scoring regression head (multilayer perceptron) to output a continuous prediction score, and calculate the mean squared error loss based on the labeled score; at the same time, retain the continuous prediction score as the first output of the action quality assessment.
[0106] Text prompt and score / difficulty embedding construction steps: Divide the score distribution of the training set into multiple score intervals, construct interval prompt text and obtain the shared prompt embedding through CLIP text encoder; During inference / training, convert the sample predicted score into a token and write it into the preset template position; Two-way cross-attention fusion steps: In the unified multimodal coding layer, bidirectional cross-attention fusion is performed using visual feature sequences and cue embedding sequences as two interacting modal channels: (1) Using the cue embedding sequence as the query and the spatiotemporal visual feature sequence as the key and value, obtain the cue representation after visual content modulation, and add the representation back to the original cue embedding according to the learnable weights to obtain the updated cue vector sequence, and output the corresponding cue weight parameters. (2) Using the spatiotemporal visual feature sequence as the query and the prompt embedding sequence as the key and value, the visual feature representation modulated by the action quality information is obtained, and the learned weight is added to the original visual feature to obtain the updated spatiotemporal visual feature sequence, and the corresponding visual weight parameters are output.
[0107] Action quality assessment steps: Based on the updated spatiotemporal visual feature sequence, temporal aggregation is performed again to obtain the action representation vector for the classification task; this vector is input into the classification head, and the score interval category is output (obtained by mapping the real score according to the preset interval), and the cross-entropy loss is calculated according to the labeled category; finally, the second output (difficulty category) of the action quality assessment branch and its classification loss are formed.
[0108] The action quality assessment prompts are injected into the text generation input steps as follows: Based on the word sequence described in natural language, prompt tokens constructed from score and difficulty information are inserted or replaced at preset positions. The embeddings corresponding to the prompt tokens are aligned with the prompt embeddings, so that action quality-related information is explicitly written into the input word ID sequence, forming an enhanced text input for generation. Multimodal Transformer text generation steps: The updated spatiotemporal visual feature sequence and the updated text input sequence are input together into the multimodal Transformer, i.e., a BERT-based video-text joint encoding model. During the training phase, vocabulary classification is performed on the positions of the masked words, and the loss of the masking language model is calculated to achieve subtitle learning based on multimodal conditions. During the inference phase, by setting the decoding mode and beam search parameters, and using the starting token as the initial input, a text token sequence is generated step by step, and the video subtitles and their corresponding confidence scores are output.
[0109] Those skilled in the art will understand that, besides implementing the system, apparatus, and modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.
[0110] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A method for automatically assessing and generating descriptions of surgical skills, characterized in that, include: Step S1: Obtain the surgical video dataset, and annotate the obtained surgical video dataset to obtain the corresponding text description and video real score; Step S2: Preprocess the surgical video, the corresponding text description, and the video's true score to obtain the preprocessed surgical video, the corresponding text description, and the video's true score. Step S3: Train the visual-language model based on the preprocessed surgical video, the corresponding text description, and the video's real score to obtain the trained visual-language model; Step S4: Acquire the target surgical video, preprocess the acquired target surgical video, use the trained vision-language model to score the preprocessed target surgical video, and generate video description text.
2. The method for automatically assessing and generating descriptions of surgical skills according to claim 1, characterized in that, The preprocessing of the surgical video in step S2 includes: Step S2.1: Decode the surgical video to obtain a frame sequence; perform frame sampling according to a preset sampling strategy, wherein the entire video is uniformly sampled in the time dimension under a uniform sampling strategy, and the sampled frame sequence is used for subsequent processing; Step S2.2: Standardize the format of the current frame data; when the frame data is empty, use a zero-value frame sequence of a preset size as a fallback in the dataset augmentation process; when the frame data is in tensor format, convert it to NumPy format and rearrange the dimensions, from (T,C,H,W) to (T,H,W,C), where T is the frame sequence length, C is the number of channels, H is the image height, and W is the image width; then convert each frame to PIL.Image and form a frame list; Step S2.3: Perform data augmentation processing on all frame images in the frame sequence, including resizing and random cropping, then convert the PIL.Image format into a tensor format that PyTorch can process, and finally perform normalization processing to obtain the processed video data.
3. The method for automatically assessing and describing surgical skills according to claim 1, characterized in that, The preprocessing of the text description in step S2 includes: Step S2.4: Use a pre-trained word segmentation model to segment the text description to obtain a word segmentation sequence; Step S2.5: For the obtained word segmentation sequence, perform structural normalization processing, add preset special markers at the beginning and end of the word segmentation sequence to distinguish sentence boundaries and functional nodes; for scenarios with additional text, complete the orderly splicing of multiple text segments and paragraph differentiation to construct a recognizable structured sequence. Step S2.6: Apply a multi-strategy dynamic masking mechanism to the identifiable structured sequence to obtain the processed text description.
4. The method for automatically assessing and describing surgical skills according to claim 1, characterized in that, The preprocessing of the total video score in step S2 includes: Based on the statistical distribution of actual scores in the surgical video dataset, and divided into multiple score intervals according to a preset number, each score interval is converted into a corresponding score prompt text. The score prompt text is then encoded using a text encoder to obtain the actual score prompt feature sequence.
5. The method for automatically assessing and describing surgical skills according to claim 4, characterized in that, Step S3 includes: Step S3.1: Use the Video Swin Transformer model in the vision-language model to extract video features from the preprocessed surgical video to obtain a video feature sequence; Step S3.2: Use the score prediction model in the visual-language model to predict the surgical skill score based on the video feature sequence, and convert the obtained predicted score into score text; Step S3.3: Construct a text description sequence based on the preprocessed text description and the preset prompt template, write the fractional text back into the text description sequence, and obtain the fused text features; Step S3.4: Using a bidirectional cross-attention fusion mechanism, the real rating prompt feature sequence is used as the query and the video feature sequence is used as the key and value. Video condition enhancement is performed on the real rating prompt feature sequence to obtain the enhanced real rating prompt feature sequence. The video feature sequence is used as the query and the enhanced real rating prompt feature sequence is used as the key and value to perform prompt semantic enhancement on the video feature, thereby obtaining the video enhanced feature that integrates score semantic information. Step S3.5: Jointly model and train BERT based on the fused text features and the video enhancement features with fused score semantic information to obtain the trained visual-language model.
6. The method for automatically assessing and generating descriptions of surgical skills according to claim 5, characterized in that, Step S3.4 includes: in, Enhanced true rating prompt feature sequence; This refers to the Multi-head Cross Attention mechanism. Provide feature sequences for accurate scoring; For video feature sequences; The learnable coefficient; in, Enhanced video features that incorporate fractional semantic information; The learnable coefficient.
7. The method for automatically assessing and generating surgical skills according to claim 5, characterized in that, In the joint modeling and training process of BERT based on the fused text features and the fused semantic information of the video enhancement features, the total loss function is: in, , , , This is the loss coefficient; The mean squared error loss for score regression; Cross-entropy is used for classification loss; Cross-entropy loss generated for text; The loss is for sparse mask regularization; The sparse mask regularization loss is: in, For regularization hyperparameters, Indicates the first The video and the first The activation values of the attention mask between videos.
8. The method for automatically assessing and describing surgical skills according to claim 4, characterized in that, Step S4 includes: Step S4.1: Use the Video Swin Transformer model in the trained vision-language model to extract video features from the target surgical video to obtain the target video feature sequence; Step S4.2: Using the score prediction model in the trained visual-language model, the surgical skill score is predicted based on the feature sequence of the target video to obtain the predicted score of the target video, and the predicted score of the target video is converted into predicted score text; Step S4.3: Construct an initial text description sequence based on the manually preset template and the MASK placeholder sequence, write the predicted score text back into the initial text description sequence, and obtain the fused target text features; Step S4.4: Using a bidirectional cross-attention fusion mechanism, the real rating prompt feature sequence is used as the query and the target video feature sequence is used as the key and value. Video condition enhancement is performed on the real rating prompt feature sequence to obtain the enhanced real rating prompt feature sequence. The target video feature sequence is used as the query and the enhanced real rating prompt feature sequence is used as the key and value. The prompt semantic enhancement is performed on the target video feature to obtain the target video enhanced feature that integrates score semantic information. Step S4.5: Input the target video enhancement features with fused semantic information and the fused target text features into the BERT caption generation model in the trained visual-language model to generate descriptive captions representing the surgical skill assessment results.
9. A surgical skills automatic assessment and description generation system, characterized in that, include: Module M1: Acquires surgical video datasets, annotates the surgical video datasets, and obtains corresponding text descriptions and video real scores for the surgical video datasets; Module M2: Preprocesses the surgical video, corresponding text description, and video real score to obtain the preprocessed surgical video, corresponding text description, and video real score. Module M3: The visual-language model is trained based on the preprocessed surgical video, the corresponding text description, and the video's actual score to obtain the trained visual-language model; Module M4: Acquires the target surgical video, preprocesses the acquired target surgical video, scores the preprocessed target surgical video using the trained vision-language model, and generates video description text.
10. The automatic surgical skill assessment and description generation system according to claim 9, characterized in that, The preprocessing of the total video score in module M2 includes: Based on the statistical distribution of actual scores of surgical videos in the surgical video dataset, and divided into multiple score intervals according to a preset number, each score interval is converted into a corresponding score prompt text, and the score prompt text is encoded using a text encoder to obtain the actual score prompt feature sequence. The module M3 includes: Module M3.1: Uses the Video Swing Transformer model in the vision-language model to extract video features from the preprocessed surgical video to obtain a video feature sequence; Module M3.2: Utilizes the score prediction model in the visual-language model to predict the surgical skill score based on the video feature sequence, and converts the obtained predicted score into score text; Module M3.3: Constructs a text description sequence based on the preprocessed text description and preset prompt template, writes the fractional text back into the text description sequence, and obtains the fused text features; Module M3.4: Through a bidirectional cross-attention fusion mechanism, the real score suggestion feature sequence is used as the query and the video feature sequence is used as the key and value. Video condition enhancement is performed on the real score suggestion feature sequence to obtain the enhanced real score suggestion feature sequence. The video feature sequence is used as the query and the enhanced real score suggestion feature sequence is used as the key and value to perform suggestion semantic enhancement on the video features, thereby obtaining video enhanced features that fuse score semantic information. Module M3.5: Jointly model and train BERT based on fused text features and video enhancement features with fused semantic information to obtain a trained visual-language model; The module M4 includes: Module M4.1: Utilizes the Video Swin Transformer model in the trained vision-language model to extract video features from the target surgical video, obtaining the target video feature sequence; Module M4.2: Utilizes the score prediction model in the trained visual-language model to predict the surgical skill score based on the feature sequence of the target video, and converts the predicted score of the target video into predicted score text; Module M4.3: Constructs an initial text description sequence based on manually preset templates and MASK placeholder sequences, writes the predicted score text back into the initial text description sequence, and obtains the fused target text features; Module M4.4: Through a bidirectional cross-attention fusion mechanism, the real score suggestion feature sequence is used as the query and the target video feature sequence is used as the key and value. Video condition enhancement is performed on the real score suggestion feature sequence to obtain the enhanced real score suggestion feature sequence. The target video feature sequence is used as the query and the enhanced real score suggestion feature sequence is used as the key and value to perform suggestion semantic enhancement on the target video features, thereby obtaining the target video enhanced features that fuse score semantic information. Module M4.5: Input the target video enhancement features with fused semantic information and the fused target text features into the BERT caption generation model in the trained visual-language model to generate descriptive captions representing the surgical skill assessment results.