Surgical evaluation model training method and surgical evaluation method based on same

Abstract

The application relates to the technical field of artificial intelligence, and discloses a surgical evaluation model training method and a surgical evaluation method based on the same. The method comprises the following steps: acquiring a surgical video, performing cutting on the surgical video to obtain a surgical segment, and performing scoring and description on the surgical segment to obtain a surgical score and a surgical description; a multimodal large model is constructed, the surgical segment, the surgical score and the surgical description are used as training samples, the multimodal large model is trained by using a composite loss function constructed by combining a text generation loss and a score loss, and a trained multimodal large model is obtained; the trained multimodal large model is verified, the trained multimodal large model is updated according to a verification result, and a surgical evaluation model is obtained. The technical scheme provided by the application can realize accurate and interpretable surgical quality evaluation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a surgical evaluation model training method and a surgical evaluation method based thereon. Background Technology

[0002] The teaching and quality evaluation of gynecological pelvic surgery is at a critical stage of transition from the traditional "experience-driven" model to an intelligent one. For a long time, technical training for complex surgeries has relied heavily on the apprenticeship system. This model suffers from problems such as strong subjectivity, low standardization, and significant time and space limitations, making it difficult to achieve standardized and homogeneous dissemination of top-tier surgical techniques. Surgical quality varies widely in grassroots hospitals, and young physicians lack objective and timely intraoperative feedback for their professional development. Although related research has made breakthroughs in preoperative planning, a complete solution that effectively integrates anatomical identification, instrument tracking, and skill quantification has yet to be developed in the field of surgical analysis and evaluation.

[0003] When processing surgical videos, related technologies generally suffer from fundamental flaws such as functional separation and superficial understanding, failing to simultaneously output high-quality surgical descriptions and scores. Specifically, machine learning-based automated assessment systems can output objective skill scores, but lack structured semantic descriptions of specific operational procedures. This gap in understanding the deep logic of surgery prevents these technologies from generating accurate descriptions that conform to medical facts, and also from providing reliable logical basis for skill assessment. Ultimately, this results in a disconnect between the two major tasks of scoring and description, making it difficult to meet the dual needs of clinical teaching and assessment for process explanation and quantitative analysis.

[0004] Therefore, how to achieve accurate and interpretable surgical quality assessment has become a problem that needs to be solved. Summary of the Invention

[0005] This application provides a surgical evaluation model training method and a surgical evaluation method based thereon, which solves the problem of how to achieve accurate and interpretable surgical quality assessment.

[0006] To achieve the above objectives, the main technical solutions adopted in this application include: In a first aspect, embodiments of this application provide a method for training a surgical evaluation model, the method comprising: The surgical video is acquired, and the video is cropped to obtain surgical segments. The surgical segments are then scored and described to obtain a surgical score and a surgical description, respectively. A multimodal large model is constructed using the surgical segment, the surgical score, and the surgical description as training samples. The multimodal large model is trained using a composite loss function constructed by combining text generation loss and rating loss to obtain the trained multimodal large model. The text generation loss is used to characterize the difference between the surgical description and the predicted operation description output by the multimodal large model, and the rating loss is used to characterize the difference between the surgical score and the predicted score output by the multimodal large model. The trained multimodal large model is validated, and the trained multimodal large model is updated based on the validation results to obtain a surgical evaluation model.

[0007] The surgical automatic evaluation model training method provided in this embodiment first crops the acquired surgical video to obtain key surgical segments, and obtains surgical scores and descriptions. Then, it uses this data to train a multimodal large-scale model, and optimizes it through a composite loss function combining text generation loss and scoring loss. This ensures that the trained multimodal large-scale model can not only accurately output predicted scores but also simultaneously generate logically consistent predicted operation descriptions. Finally, through validation and iterative updates, a surgical evaluation model capable of automatically analyzing and evaluating surgical videos is obtained. The surgical evaluation model ensures the accuracy of the output scores and provides clear explanations through the generated text descriptions, guaranteeing accurate quantification and interpretable assessment of surgical quality, thereby achieving accurate and interpretable surgical quality evaluation.

[0008] In one implementation, the step of cropping the surgical video to obtain surgical segments includes: The surgical video is decoded to obtain pixel information for each frame; The pixel differences between adjacent frames are obtained based on the pixel information of each frame; Based on the pixel information of each frame and the pixel differences between adjacent frames, the surgical video is cropped to obtain surgical segments.

[0009] This embodiment decodes the surgical video to obtain the pixel information of each frame, calculates the pixel differences between adjacent frames, and uses threshold or adaptive methods to identify key turning points in the surgical video (such as stage switching, instrument entry and exit), thereby cutting the surgical video into several surgical segments of different operations. This effectively removes redundant content, highlights key operations, and facilitates subsequent analysis, storage, and teaching applications.

[0010] In one implementation, cropping the surgical video to obtain surgical segments based on the pixel information of each frame and the pixel differences between adjacent frames includes: The cumulative motion intensity is obtained by smoothing the pixel differences between adjacent frames using a sliding time window: Based on the cumulative motion intensity and the preset threshold, the starting frame of each surgical segment is determined: The end frame of each surgical segment is determined based on a preset continuous low-motion state: The surgical video is cropped based on the start frame and the end frame to obtain multiple surgical segments.

[0011] This embodiment smooths pixel differences between adjacent frames using a sliding time window to obtain cumulative motion intensity, thereby eliminating transient noise interference and highlighting continuous operation signals. Based on this, a preset threshold is set; when the cumulative motion intensity first exceeds the preset threshold, it is marked as the start frame of the surgical segment. In the end frame detection, a duration constraint of continuous low motion states is introduced, and when the cumulative motion intensity is below the end threshold and the continuous low motion states meet a certain number of frames, the end frame of the surgical segment is determined. This embodiment effectively smooths noise, clearly defines surgical segments, and avoids missegmentation caused by short pauses in instruments, thus improving the accuracy and semantic integrity of surgical segment trimming.

[0012] In one implementation, the trained multimodal large model is obtained through the following method: Construct a large multimodal model that includes visual encoding units, text encoding units, and multimodal fusion units; The surgical segment is input into the visual encoding unit to obtain video segment features; The surgical description is encoded using the text encoding unit to obtain the operation description text; The video segment features and the operation description text are input into the multimodal fusion unit to obtain a multimodal joint feature representation; Based on the multimodal joint feature representation, a prediction operation description and a prediction score are generated respectively; Based on the text generation loss constructed according to the prediction operation description and the rating loss constructed according to the prediction rating, a composite loss function is obtained, and the multimodal large model is trained using the composite loss function to obtain the trained multimodal large model.

[0013] This embodiment achieves efficient joint processing of surgical segments and surgical descriptions by constructing a large multimodal model comprising a visual encoding unit, a text encoding unit, and a multimodal fusion unit. First, the surgical segment is input into the visual encoding unit to extract video segment features. Simultaneously, the surgical description is encoded by the text encoding unit to obtain the operation description text. Then, the two are deeply fused in the multimodal fusion unit to obtain a semantically rich multimodal joint feature representation. Based on this multimodal joint feature representation, predicted operation descriptions and predicted scores are generated in parallel. Text generation loss and scoring loss are constructed separately and jointly optimized using a composite loss function to finally obtain a trained large multimodal model. This trained model can more accurately understand the surgical process, generate predicted operation descriptions that conform to the surgical description, and provide predicted scores, thereby effectively assisting surgical teaching, skills assessment, and clinical decision support, significantly improving the automated and intelligent level of surgical video analysis.

[0014] In one implementation, inputting the surgical segment into the visual encoding unit to obtain video segment features includes: Visual feature vectors are extracted from the surgical segment, and a frame-level feature sequence is constructed based on the visual feature vectors; A new input sequence is obtained by adding a preset learning label before the frame-level feature sequence; Temporal modeling is performed on the new input sequence, and the feature vector corresponding to the preset learning label is determined as the video segment feature of the surgical segment.

[0015] This embodiment encodes variable-length surgical segments into fixed-length video segment features using pre-set learning labels. First, frame-level feature sequences are extracted from the surgical segments. These sequences are then prefixed with pre-set learning labels to obtain a new input sequence, which is then used for temporal modeling. Through a self-attention mechanism, the pre-set learning labels aggregate the spatiotemporal information of the entire surgical segment, ultimately outputting a fixed-length video segment feature. This embodiment effectively solves the dimensionality mismatch problem caused by varying input lengths, enabling the visual encoding unit to flexibly handle surgical segments of arbitrary lengths while preserving complete global contextual information, providing a unified input foundation for subsequent multimodal fusion units.

[0016] In one implementation, the composite loss function is constructed in the following ways: The text generation loss is constructed based on the correct prediction probability of each word in the surgical description by the multimodal large model; The score loss is constructed based on the numerical difference between the surgical score and the predicted score; The composite loss function is constructed by weighted summing of the text generation loss and the rating loss.

[0017] This embodiment constructs a composite loss function to jointly optimize the multimodal large model. It constructs a text generation loss based on the correct prediction probability of each word in the surgical description, and then constructs a scoring loss based on the numerical difference between the predicted score and the surgical score. The text generation loss and the scoring loss are weighted and summed to form a composite loss function, which enables the multimodal large model to improve simultaneously in the two tasks of generating predicted operation descriptions and predicting scores. The generated predicted operation descriptions are more consistent with the surgical descriptions, and the predicted scores are closer to the surgical scores, thus possessing greater professionalism and practicality.

[0018] Secondly, embodiments of this application provide a surgical evaluation method, the method comprising: Obtain the surgical video to be evaluated, input the surgical video to be evaluated into the surgical evaluation model obtained by any of the above-mentioned training methods, and obtain the target prediction operation description and target prediction score for each surgical segment in the surgical video to be evaluated.

[0019] Thirdly, embodiments of this application provide a surgical evaluation model training device, the device comprising: The surgical evaluation unit is used to acquire surgical videos, crop the surgical videos to obtain surgical segments, and score and describe the surgical segments to obtain surgical scores and surgical descriptions, respectively. The model acquisition unit is used to construct a multimodal large model. Using the surgical segment, the surgical score, and the surgical description as training samples, it trains the multimodal large model using a composite loss function constructed by combining text generation loss and rating loss, thereby obtaining the trained multimodal large model. The text generation loss characterizes the difference between the surgical description and the predicted operational description output by the multimodal large model, and the rating loss characterizes the difference between the surgical score and the predicted score output by the multimodal large model. The verification unit is used to verify the trained multimodal large model and update the trained multimodal large model according to the verification results to obtain the surgical evaluation model.

[0020] Fourthly, embodiments of this application provide a computer device, including: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes these computer instructions to perform the surgical evaluation model training method or the surgical evaluation method described above.

[0021] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which are used to cause a computer to execute the surgical evaluation model training method or the surgical evaluation method described above. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating a surgical evaluation model training method provided in this application embodiment; Figure 2 A flowchart of step S11 provided in an embodiment of this application; Figure 3 A flowchart of step S151 provided in an embodiment of this application; Figure 4 A flowchart of step S311 provided in an embodiment of this application; Figure 5 A flowchart of step S3121 provided in the embodiments of this application; Figure 6 A flowchart of step S331 provided in an embodiment of this application; Figure 7 This application provides a surgical evaluation model training device. Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The teaching system and quality evaluation of gynecological pelvic surgery are undergoing a transformation from the traditional "experience-driven" model to an intelligent, data-driven paradigm. For a long time, the transmission and training of complex surgical skills have heavily relied on the "apprenticeship" model. While this model has played an important role in clinical practice, its inherent shortcomings—high subjectivity, low standardization, and significant time and space limitations—have made it difficult to achieve standardized and homogeneous dissemination of top-tier surgical techniques. This situation not only results in inconsistent surgical quality in primary healthcare institutions but also hinders the development of young physicians by lacking objective and immediate intraoperative feedback mechanisms, thus restricting the rapid improvement of clinical skills.

[0026] Currently, surgical video analysis technologies generally suffer from fundamental flaws such as functional separation and superficial understanding, making it difficult to simultaneously output high-quality surgical procedure descriptions and objective skill scores within the same framework. Specifically, while machine learning-based automated assessment systems can provide quantitative operational scores, their outputs often lack structured semantic descriptions of specific operational processes, making it difficult to reconstruct the internal logic and operational details of surgical steps. This gap in understanding the deep logic of surgery prevents related technologies from generating accurate textual descriptions that conform to anatomical and surgical facts, and also from providing reliable logical basis and clinical interpretation for skill assessment. Ultimately, this results in the two major tasks of skill scoring and procedure description being isolated and independent, failing to meet the dual requirements of process interpretability and result quantifiability in clinical teaching and quality evaluation systems. Therefore, constructing an intelligent analysis system capable of simultaneously achieving semantic understanding of surgical procedures and quantitative evaluation of operational skills has become a key breakthrough in promoting the standardization and homogenization of gynecological pelvic surgery teaching.

[0027] In conclusion, achieving accurate and interpretable surgical quality assessment has become a technical problem that needs to be solved.

[0028] To address the aforementioned technical problems, this application provides a method for training a surgical evaluation model. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system containing a set of computer-executable instructions. Furthermore, although the logical order of each step is shown in the flowchart, in some cases, these steps can be executed in a different order than those shown or described herein.

[0029] This embodiment provides a method for training a surgical evaluation model. Figure 1 A flowchart of a surgical evaluation model training method provided in this application embodiment is shown below. Figure 1 As shown, the process includes the following steps: Step S1: Obtain the surgical video, crop the surgical video to obtain surgical segments, and score and describe the surgical segments to obtain surgical scores and surgical descriptions respectively.

[0030] Specifically, at least 200 hours of laparoscopic gynecological surgery videos were collected, covering representative core procedures such as total hysterectomy, sacrovaginal fixation, and sacroligament suspension. The surgical videos were uniformly set to a resolution of 1920×1080 pixels and a frame rate of 25fps. After the surgical videos were collected, they were cropped to extract surgical segments with analytical value; these segments were then... The gynecological pelvic cavity, modified based on the principles of membranous anatomy, was manually annotated by experienced gynecological surgeons, resulting in: in, Experts indicated their opinion on the first To ensure the objectivity and accuracy of the surgical descriptions provided for each surgical segment, a labeling team of three or more experts with the title of associate chief physician or above was formed. Based on the theory of membrane anatomy, standardized descriptions were made of key operations such as instrument use and anatomical layer identification during the operation (e.g., "blunt dissection along the vesicouterine reflection with an ultrasonic scalpel, clear layers, and no bleeding") to obtain the surgical description. Indicates the number of surgical fragments; Experts indicated their opinion on the first The surgical segment is scored on a scale of 1 to 10, with scores ranging from 1 to 10. A weighted average is applied based on three dimensions: anatomical accuracy (40%), operative fluency (30%), and bleeding control (30%). All experts score independently without interference. To ensure consistency, the Cohen-Kappa coefficient is used for consistency testing, requiring a coefficient value of at least 0.8. Disputed surgical segments are reviewed collectively by the team to reach a consensus.

[0031] Surgical scoring can also be achieved by having multiple experts independently score each surgical segment and calculating the average as the surgical score for that segment. The score is expressed as follows: in Indicates the first Surgical scores for each surgical segment, The number of experts who participated in the scoring. Indicates the first Experts on the first The scores for each surgical segment were calculated. By using the mean of multiple expert scores to reduce the subjective bias of individual experts, a relatively stable reference standard was formed for subsequent comparison with the predicted scores output by the multimodal large model.

[0032] Step S3: Construct a multimodal large model. Use surgical fragments, surgical scores, and surgical descriptions as training samples. Train the multimodal large model using a composite loss function constructed by combining text generation loss and scoring loss to obtain the trained multimodal large model. Among them, the text generation loss is used to characterize the difference between the surgical description and the predicted operation description output by the multimodal large model, and the scoring loss is used to characterize the difference between the surgical score and the predicted score output by the multimodal large model.

[0033] Specifically, in constructing the multimodal large model, surgical fragments, surgical scores, and surgical descriptions are used as training samples. End-to-end training of the multimodal large model is driven by a composite loss function combining text generation loss and scoring loss. The text generation loss quantifies the semantic and sequence differences between the predicted operational descriptions generated by the multimodal large model and the surgical descriptions, ensuring that the multimodal large model accurately understands visual content and generates predicted operational descriptions that conform to clinical logic. The scoring loss measures the numerical deviation between the predicted scores output by the multimodal large model and the surgical scores, enabling the multimodal large model to learn its ability to assess surgical quality or operational standardization from surgical scores and descriptions. Through this multi-task learning paradigm, the trained multimodal large model can not only generate coherent and accurate predicted operational descriptions based on input surgical fragments but also output predicted scores with clinical reference value, thus playing a crucial role in surgical teaching, automated assessment, and surgical procedure recording.

[0034] Step S5: Validate the trained multimodal large model and update the trained multimodal large model based on the validation results to obtain the surgical evaluation model.

[0035] Specifically, the verification method can be to introduce an algorithm performance verification unit: using the intragroup correlation coefficient (ICC) as a consistency evaluation index to judge the similarity between the predicted score and the surgical score output by the well-trained multimodal large model, which is expressed as follows: The between-group variation describes the numerical difference in predicted scores for different surgical segments, reflecting the quality differences within the surgical segments themselves, and is an effective signal that the trained multimodal model should capture. The within-group variation describes the difference between the predicted score and the actual surgical score for the same surgical segment, reflecting the inconsistency between the trained multimodal model and the expert's surgical score, and is considered measurement error. At this point, a surgical evaluation model is generally considered to be obtained when the consistency between the well-trained multimodal large model and the surgical score given by the expert reaches a level of "good" or "high consistency". If the current training data is insufficient or of poor quality, the trained multimodal model fails to stably learn an effective evaluation pattern. In this case, new training data needs to be added, the model retrained, and the ICC value recalculated for verification. This process is repeated until the ICC reaches or exceeds the threshold of 0.75, thus obtaining the surgical evaluation model. The method provided in this embodiment statistically ensures that the output of the surgical evaluation model closely matches the surgical scores given by experts, thereby avoiding erroneous training feedback or misleading decisions due to errors.

[0036] The validation method can also introduce a clinical outcome-related validation unit to determine whether the trained multimodal model has actual clinical significance, rather than just the consistency between the predicted score and the surgical score. The following regression model is constructed: in Indicates clinical outcomes (e.g., operation time, blood loss, or complication rate); Indicates the predicted score; Represents the random error term; This represents the intercept of the regression model; This represents the influence coefficient of a trained multimodal large model. A significant value of 0 indicates that the predicted score is unrelated to the clinical outcome. Therefore, this score is merely a "formal score" and lacks clinical application value. In such cases, the trained multimodal model needs to be updated and new real-world clinical datasets need to be introduced. And fine-tune the parameters of the trained multimodal large model (i.e. ) By fine-tuning the system using newly added real clinical datasets, potential biases in the trained multimodal model can be corrected in a timely manner, effectively maintaining the system's accuracy and reliability, and thus obtaining a surgical evaluation model. If the value is not zero, it indicates a significant correlation between the predicted score and the clinical outcome, proving that the trained multimodal model has a statistically significant correlation with the clinical outcome variable. Therefore, the predicted score and the clinical outcome are objectively related, confirming the clinical application value of the trained multimodal model, and ultimately yielding a surgical evaluation model.

[0037] Another validation method is to introduce a prospective training effectiveness validation unit to determine whether the trained multimodal large model improves the learning rate in a real training environment. An exponential learning curve is used to describe the learning rate improvement process. in This represents the trainee's skill level during the t-th surgery; This represents the theoretically achievable optimal level of stability. This represents the gap between initial and final abilities. This represents the learning rate parameter; is a natural constant. The exponential learning curve reflects the characteristics of a learner's skill improvement: rapid initial improvement followed by a gradual stabilization. This is achieved by comparing the performance of learners using a pre-trained multimodal large model with those not using a pre-trained multimodal large model. The value quantifies the acceleration effect of a well-trained multimodal large model on the learning curve. If the trained multimodal large model... Significantly higher than large multimodal models that did not use a pre-trained model. This indicates that the skill maturation cycle can be shortened, training efficiency improved, and thus a surgical evaluation model obtained. If the well-trained multimodal large model... No significantly higher performance than untrained multimodal large models. At this point, new training data needs to be added, the training process needs to be repeated, and the calculations need to be performed again. The value is then validated. This process is repeated until the trained multimodal large model is validated. No significantly higher performance than untrained multimodal large models. Ultimately, a surgical evaluation model was obtained.

[0038] The surgical automatic evaluation model training method provided in this embodiment first crops the acquired surgical video to obtain key surgical segments, and obtains surgical scores and descriptions. Then, it uses this data to train a multimodal large-scale model, and optimizes it through a composite loss function combining text generation loss and scoring loss. This ensures that the trained multimodal large-scale model can not only accurately output predicted scores but also simultaneously generate logically consistent predicted operation descriptions. Finally, through validation and iterative updates, a surgical evaluation model capable of automatically analyzing and evaluating surgical videos is obtained. The surgical evaluation model ensures the accuracy of the output scores and provides clear explanations through the generated text descriptions, guaranteeing accurate quantification and interpretable assessment of surgical quality, thereby achieving accurate and interpretable surgical quality evaluation.

[0039] Figure 2 The flowchart for obtaining surgical segments by cropping from surgical videos provided in this application embodiment may include the following steps: Step S11: Decode the surgical video to obtain pixel information for each frame.

[0040] Specifically, surgical videos (such as laparoscopic or robot-assisted surgical recordings) are decoded into a continuous frame sequence, which is expressed as follows: in, Represented as pixel information of the t-th frame, with resolution determined by Decision (e.g., 1080p or 4K); This represents the total number of frames in the surgical video, which is related to the duration and frame rate of the surgical video.

[0041] Step S13: Obtain the pixel differences between adjacent frames based on the pixel information of each frame.

[0042] Specifically, by obtaining the pixel differences between adjacent frames based on the pixel information of each frame, the degree of change between pixel information in each frame of the surgical video is quantified, providing a basis for subsequent surgical video cropping. For the first... Frame Image With the previous frame The pixel difference between adjacent frames is calculated and expressed as follows: in Indicates the first Frame and the Pixel differences between frames; in The Frobenius norm of a matrix can effectively measure the overall variation in pixel differences between adjacent frames.

[0043] Step S15: Based on the pixel information of each frame and the pixel differences between adjacent frames, the surgical video is cropped to obtain surgical segments.

[0044] Specifically, surgical segments are obtained by cropping the surgical video using the pixel information of each frame and the pixel differences between adjacent frames. For example, key turning points in the surgical process (such as changes in surgical stages, instrument insertion and removal, etc.) can be identified by setting reasonable thresholds or using adaptive methods to crop the surgical video. For example, when When the intensity exceeds a preset threshold, it can be considered that a significant scene change has occurred and is marked as the boundary of the surgical segment. At the same time, continuous low-difference areas are merged into the same stage. The surgical segment obtained by the final trimming can effectively remove redundant content, highlight the key surgical operations, and facilitate subsequent analysis, storage or teaching applications.

[0045] This embodiment decodes the surgical video to obtain the pixel information of each frame, calculates the pixel differences between adjacent frames, and uses threshold or adaptive methods to identify key turning points in the surgical video (such as stage switching, instrument entry and exit), thereby cutting the surgical video into several surgical segments of different operations. This effectively removes redundant content, highlights key operations, and facilitates subsequent analysis, storage, and teaching applications.

[0046] Figure 3 The flowchart provided in this application describes how to crop surgical videos to obtain surgical segments based on pixel information of each frame and pixel differences between adjacent frames. This process may include the following steps: Step S151: A sliding time window is used to smooth the pixel differences between adjacent frames to obtain the cumulative motion intensity.

[0047] Specifically, a sliding time window is used to smooth pixel differences between adjacent frames. This is achieved by defining a sliding time window with frame t as the endpoint and a preset time range τ. The cumulative motion intensity is obtained by summing the pixel differences between adjacent frames within the sliding time window. Its representation is as follows: The sliding time window can effectively reflect the cumulative effect of motion in the time dimension, filtering out misjudgments caused by brief fluctuations, eliminating the interference of instantaneous noise, and enhancing continuous motion signals, thus providing a basis for the start and end frames of each subsequent surgical segment.

[0048] Step S153: Determine the starting frame of each surgical segment based on the cumulative motion intensity and a preset threshold.

[0049] Specifically, in surgical video analysis, to automatically segment continuous surgical videos into surgical segments with clear semantics, it is necessary to accurately identify the starting frame of each surgical segment. This is achieved using a method based on cumulative motion intensity. This goal is achieved using a preset threshold method. A preset threshold is set... When the cumulative motion intensity first exceeds a preset threshold, the end of the sliding time window is considered the start frame of the surgical segment. The starting frame of a surgical segment The expression form is: By combining the persistence judgment derived from cumulative motion intensity, the starting frame of the surgical segment can be automatically and robustly detected, providing a reliable basis for subsequent segment analysis.

[0050] Step S155: Determine the end frame of each surgical segment based on the preset continuous low motion state.

[0051] Specifically, after determining the starting frame of each surgical segment, based on the starting frame, the cumulative motion intensity... Below the end threshold And the continuous low motion state satisfies The surgical segment is only considered to have ended at frame 1. End frame of a surgical segment Its expression form is: in For the first The starting frame index of each surgical segment. The end threshold, This is the preset minimum number of frames for continuous low-motion states. By introducing continuous low-motion states, erroneous segmentation caused by brief pauses in the instrument or instantaneous fluctuations in motion intensity is effectively avoided, ensuring that the segmented surgical fragments have true semantic integrity and providing reliable fragment boundaries for subsequent evaluation or analysis of surgical fragments.

[0052] Step S157: Based on the start frame and the end frame, the surgical video is cropped to obtain multiple surgical segments.

[0053] Specifically, based on the aforementioned start and end frame indices, the original video is cropped to obtain a set of surgical video clips: in Indicates the first A surgical fragment obtained by cropping. This represents the total number of surgical segments obtained from the surgical video. For example, you can set the following parameters: Preset Threshold. End threshold At the same time, a minimum number of frames for continuous low motion states is introduced. To avoid missegmentation caused by brief pauses in instrumentation, these parameters are optimized to ensure that each segmented surgical fragment contains a complete operation (such as tissue dissection or hemostasis) and automatically controls the fragment duration within a reasonable range of 3 to 15 seconds, conforming to the natural rhythm of surgical procedures. Finally, the algorithm outputs... These surgical segments provide a video basis for subsequent surgical descriptions and scoring.

[0054] This embodiment smooths pixel differences between adjacent frames using a sliding time window to obtain cumulative motion intensity, thereby eliminating transient noise interference and highlighting continuous operation signals. Based on this, a preset threshold is set; when the cumulative motion intensity first exceeds the preset threshold, it is marked as the start frame of the surgical segment. In the end frame detection, a duration constraint of continuous low motion states is introduced, and when the cumulative motion intensity is below the end threshold and the continuous low motion states meet a certain number of frames, the surgical segment is determined to have ended. This embodiment effectively smooths noise, clearly defines surgical segments, and avoids missegmentation caused by short pauses in instruments, thus improving the accuracy and semantic integrity of surgical segment trimming.

[0055] Figure 4 A flowchart illustrating the method for obtaining a trained multimodal large model as provided in this application embodiment, the process may include the following steps: Step S311: Construct a large multimodal model that includes visual coding units, text coding units, and multimodal fusion units.

[0056] Specifically, the visual encoding unit can use a Vision Transformer (ViT) or ResNet-50 to read surgical videos to obtain video segment features, while the text encoding unit encodes surgical descriptions based on pre-trained language models (such as bidirectional encoders representing transformations from BERT, or Large Language Model Meta-Artificial Intelligence (LLaMA)) to obtain operation description text. Finally, the multimodal fusion unit achieves alignment and interaction between video segment features and operation description text through a cross-attention mechanism, outputting a fused multimodal joint feature representation.

[0057] Step S312: Input the surgical segment into the visual coding unit to obtain video segment features.

[0058] Specifically, the surgical segment is input into the visual encoding unit, which is usually composed of a deep convolutional neural network (such as ResNet-50) or a visual transformer (ViT) based on a self-attention mechanism. This unit is used to effectively capture the content in the surgical segment, such as instrument position, tissue state, and operation trajectory. Finally, the surgical segment is encoded and converted into video segment features.

[0059] Step S313: Encode the surgical description using a text encoding unit to obtain the operation description text.

[0060] Specifically, by describing the surgery Input text encoding unit to translate surgical description Mapped to operation description text Its form of expression is: in, For text encoding units, language models such as bidirectional encoders (BERT) and Large Language Model Meta-Artificial Intelligence (LLaMA) can be used. This process aims to capture key information contained in the surgical description, such as operational semantics, action types, and interactive objects, and encode them into the operational description text. This provides a semantic alignment basis for subsequent alignment and fusion with visual features.

[0061] Step S314: Input the video segment features and operation description text into the multimodal fusion unit to obtain a multimodal joint feature representation.

[0062] Specifically, the video segment features and operation description text are input into the multimodal fusion unit, which is represented as follows: Features of video segments To describe the operation text, Multimodal joint feature representation By using a multimodal fusion unit, video segment features and operational description text are no longer treated in isolation. Instead, their interaction is calculated through a cross-attention mechanism, ultimately outputting a multimodal joint feature representation. It not only preserves the information in the surgical fragments, but also incorporates semantic guidance for the surgical description, thus more accurately reflecting the high-level semantics of "description in the current scene".

[0063] Step S315: Based on the multimodal joint feature representation, generate the prediction operation description and prediction score respectively.

[0064] Specifically, multimodal joint feature representation The inputs are fed into two parallel branches: a text generation branch and a rating regression branch. In the text generation branch, an autoregressive decoder decodes the multimodal joint feature representation to generate a prediction operation description, which is represented as follows: In the rating regression branch, the rating regression branch is designed as a three-layer MLP (hidden layer dimensions 512→256→1), through a linear mapping (weight matrix) and bias Multimodal joint feature representation The process is performed to obtain the predicted score, which is represented as follows: in, For predictive procedure descriptions (e.g., "blunt dissection along the vesicouterine reflection with clear layers"), The predicted score (e.g., 8.5 / 10) reflects the standardization of operations.

[0065] Step S316: Obtain a composite loss function by constructing the text generation loss based on the prediction operation description and the rating loss based on the prediction rating, and use the composite loss function to train a multimodal large model to obtain the trained multimodal large model.

[0066] Specifically, firstly, a text generation loss is constructed based on the predicted operation description output by the text generation branch to measure the difference between the predicted operation description and the surgical description. Simultaneously, a rating loss is constructed based on the predicted score output by the rating regression branch to measure the difference between the predicted score and the surgical score. The text generation loss and the rating loss are then weighted and summed to obtain a composite loss function. This composite loss function is used to train the multimodal large model end-to-end. The parameters of the multimodal large model are optimized using the backpropagation algorithm, ultimately obtaining a well-trained multimodal large model capable of simultaneously and accurately outputting both the predicted operation description and the predicted score.

[0067] This embodiment achieves efficient joint processing of surgical segments and surgical descriptions by constructing a large multimodal model comprising a visual encoding unit, a text encoding unit, and a multimodal fusion unit. First, the surgical segment is input into the visual encoding unit to extract video segment features. Simultaneously, the surgical description is encoded by the text encoding unit to obtain the operation description text. Then, the two are deeply fused in the multimodal fusion unit to obtain a semantically rich multimodal joint feature representation. Based on this multimodal joint feature representation, predicted operation descriptions and predicted scores are generated in parallel. Text generation loss and scoring loss are constructed separately and jointly optimized using a composite loss function to finally obtain a trained large multimodal model. This trained model can more accurately understand the surgical process, generate predicted operation descriptions that conform to the surgical description, and provide predicted scores, thereby effectively assisting surgical teaching, skills assessment, and clinical decision support, significantly improving the automated and intelligent level of surgical video analysis.

[0068] Figure 5 The flowchart provided in this application embodiment, which describes inputting surgical segments into a visual encoding unit to obtain video segment features, may include the following steps: Step S3121: Extract visual feature vectors from the surgical fragments and construct frame-level feature sequences based on the visual feature vectors.

[0069] Specifically, surgical fragments It contains Frame images, denoted as image sequence. First, the visual encoder within the visual coding unit extracts features from each frame of the image, obtaining the corresponding frame-level feature sequence: in Represents a visual encoder. For the first The visual feature vectors of the frames can be used to obtain the frame-level feature sequence of the entire surgical segment: in, It is a frame-level feature sequence.

[0070] Step S3123: Add a preset learning label before the frame-level feature sequence to obtain a new input sequence.

[0071] Specifically, in order to obtain the global semantic representation of the entire surgical segment, a pre-set learned label (CLS Token) is added before the frame-level feature sequence, denoted as... .Will Concatenated with the frame feature sequence, a new input sequence is formed: Step S3125: Perform time-series modeling on the new input sequence and determine the feature vector corresponding to the preset learning label as the video segment feature of the surgical segment.

[0072] Specifically, the new input sequence The input information extractor (Transformer encoder) performs time series modeling, and its computation process can be represented as follows: The output sequence is: in The feature vector corresponding to the preset learning label is used. Since the self-attention mechanism of the Transformer encoder can model the global information of the entire sequence, the CLS Token will aggregate the semantic information of all video frames during the encoding process to obtain the global representation of the video segment. The feature vector corresponding to the preset learning label is determined as the video segment feature of the surgical segment.

[0073] This embodiment encodes variable-length surgical segments into fixed-length video segment features using pre-set learning labels. First, frame-level feature sequences are extracted from the surgical segments. These sequences are then prefixed with pre-set learning labels to obtain a new input sequence, which is then used for temporal modeling. Through a self-attention mechanism, the pre-set learning labels aggregate the spatiotemporal information of the entire surgical segment, ultimately outputting a fixed-length video segment feature. This embodiment effectively solves the dimensionality mismatch problem caused by varying input lengths, enabling the visual encoding unit to flexibly handle surgical segments of arbitrary lengths while preserving complete global contextual information, providing a unified input foundation for subsequent multimodal fusion units.

[0074] Figure 6 A flowchart illustrating the construction method of the composite loss function provided in the embodiments of this application is shown. The process may include the following steps: Step S331: Construct a text generation loss based on the correct prediction probability of each word in the surgical description using a multimodal large model.

[0075] Specifically, text generation loss It is obtained using autoregressive cross-entropy loss, and its representation is as follows: in, For the first The surgical description of the th surgical segment is obtained by minimizing the negative log-likelihood, given the th ... surgical fragments and historical generation Under the condition of word-by-word maximization of predicting the next word The probability of obtaining the text generation loss can effectively guide the multimodal large model to learn semantic mappings that conform to clinical practice from surgical descriptions, thereby generating accurate and coherent predictive operation descriptions.

[0076] Step S333: Construct a score loss based on the numerical difference between the surgical score and the predicted score.

[0077] Specifically, based on surgical scores With predicted score The numerical differences between them are used to construct the rating loss. The scoring loss is expressed in the form of mean squared error (MSE) to constrain the predicted score to approximate the surgical score, and is defined as follows: Step S335: Construct a composite loss function by weighted summation of text generation loss and rating loss.

[0078] Specifically, a composite loss function is constructed by combining text generation loss and rating loss: Where L represents the composite loss function; and These are weighting coefficients used to balance the text generation loss and the scoring loss. For example, a deep learning optimizer (AdamW optimizer) with a learning rate of 5e-5 and a batch size of 16 is used, trained in parallel on four A100 GPUs. The weighting coefficients are set to λ1=1.0 and λ2=0.5, prioritizing the accuracy of the text description. An early stopping mechanism is triggered when the composite loss function on the validation set fails to decrease further within five consecutive training epochs.

[0079] This embodiment constructs a composite loss function to jointly optimize the multimodal large model. It constructs a text generation loss based on the correct prediction probability of each word in the surgical description, and then constructs a scoring loss based on the numerical difference between the predicted score and the surgical score. The text generation loss and the scoring loss are weighted and summed to form a composite loss function, which enables the multimodal large model to improve simultaneously in the two tasks of generating predicted operation descriptions and predicting scores. The generated predicted operation descriptions are more consistent with the surgical descriptions, and the predicted scores are closer to the surgical scores, thus possessing greater professionalism and practicality.

[0080] In one optional embodiment, a surgical video to be evaluated is acquired and input into the surgical evaluation model obtained by the training method in steps S1 to S5, resulting in a target prediction operation description and target prediction score for each surgical segment in the surgical video. Use cases are as follows: In one optional implementation scenario: the surgical evaluation model monitors the surgical video to be evaluated in real time and outputs a target prediction operation description and a target prediction score for the current surgical segment. When the target prediction score falls below a preset quality threshold... An alarm will be automatically triggered. Surgical segments with a predicted score below this threshold can be represented as: The system displays a description of its target prediction operation, along with playable low-scoring clips, and shows a curve of the target prediction score change and textual improvement suggestions.

[0081] Accordingly, please refer to Figure 7 A block diagram of a surgical evaluation model training device provided in this application embodiment, the device comprising: The surgical evaluation unit 101 is used to acquire surgical videos, crop surgical segments based on the surgical videos, and score and describe the surgical segments to obtain surgical scores and surgical descriptions, respectively.

[0082] The model acquisition unit 103 is used to construct a multimodal large model. It uses surgical fragments, surgical scores, and surgical descriptions as training samples and trains the multimodal large model using a composite loss function constructed by combining text generation loss and scoring loss to obtain the trained multimodal large model. Among them, the text generation loss is used to characterize the difference between the surgical description and the predicted operation description output by the multimodal large model, and the scoring loss is used to characterize the difference between the surgical score and the predicted score output by the multimodal large model.

[0083] The validation unit 105 is used to validate the trained multimodal large model and update the trained multimodal large model based on the validation results to obtain the surgical evaluation model.

[0084] In some optional implementations, the surgical evaluation unit 101 includes: The surgical video is decoded to obtain pixel information for each frame.

[0085] The pixel differences between adjacent frames are obtained based on the pixel information of each frame.

[0086] Based on the pixel information of each frame and the pixel differences between adjacent frames, the surgical video is cropped to obtain surgical segments.

[0087] In some optional implementations, the surgical evaluation unit 101 includes: A sliding time window is used to smooth the pixel differences between adjacent frames to obtain the cumulative motion intensity.

[0088] The starting frame of each surgical segment is determined based on the cumulative motion intensity and a preset threshold.

[0089] The end frame of each surgical segment is determined based on a preset continuous low-motion state.

[0090] The surgical video is cropped based on the start and end frames to obtain multiple surgical segments.

[0091] In some optional implementations, the model acquisition unit 103 includes: Construct a large multimodal model that includes visual encoding units, text encoding units, and multimodal fusion units.

[0092] The surgical segments are input into the visual encoding unit to obtain video segment features.

[0093] The surgical description is encoded using a text encoding unit to obtain the operation description text.

[0094] The video segment features and operation description text are input into the multimodal fusion unit to obtain a multimodal joint feature representation.

[0095] Based on multimodal joint feature representation, prediction operation descriptions and prediction scores are generated respectively.

[0096] Based on the text generation loss constructed from the prediction operation description and the rating loss constructed from the prediction rating, a composite loss function is obtained, and a multimodal large model is trained using the composite loss function to obtain the trained multimodal large model.

[0097] In some optional implementations, the model acquisition unit 103 includes: Visual feature vectors are extracted from surgical fragments, and frame-level feature sequences are constructed based on these visual feature vectors.

[0098] A new input sequence is obtained by adding a preset learning label before the frame-level feature sequence.

[0099] Temporal modeling is performed on the new input sequence, and the feature vectors corresponding to the preset learning labels are determined as video segment features of the surgical segment.

[0100] In some optional implementations, the model acquisition unit 103 includes: The text generation loss is constructed based on the correct prediction probability of each word in the surgical description using a multimodal large model.

[0101] A score loss is constructed based on the numerical difference between the surgical score and the predicted score.

[0102] A composite loss function is constructed by weighting and summing the text generation loss and the rating loss.

[0103] In this embodiment, a surgical evaluation model training device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0104] Please see Figure 8 , Figure 8 This application provides a schematic diagram of the structure of a computer device, as shown in the embodiment of the present application. Figure 8 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0105] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0106] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0107] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0108] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0109] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0110] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0111] The apparatus and units described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0112] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0113] Those skilled in the art will understand that the embodiments of this application can be provided as methods or apparatus. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, and devices according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0118] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0119] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0120] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for training a surgical evaluation model, characterized in that, The method includes: The surgical video is acquired, and the video is cropped to obtain surgical segments. The surgical segments are then scored and described to obtain a surgical score and a surgical description, respectively. A multimodal large model is constructed using the surgical segment, surgical score, and surgical description as training samples. The multimodal large model is trained using a composite loss function combining text generation loss and rating loss to obtain the trained multimodal large model. The text generation loss characterizes the difference between the surgical description and the predicted operational description output by the multimodal large model, and the rating loss characterizes the difference between the surgical score and the predicted score output by the multimodal large model. The trained multimodal large model is obtained by constructing a multimodal large model containing visual encoding units, text encoding units, and multimodal fusion units. The process involves: inputting the surgical segment into the visual encoding unit to obtain video segment features; encoding the surgical description through the text encoding unit to obtain operation description text; inputting the video segment features and the operation description text into the multimodal fusion unit to obtain multimodal joint feature representation; generating predicted operation descriptions and predicted scores based on the multimodal joint feature representations; obtaining a composite loss function based on the text generation loss constructed from the predicted operation descriptions and the score loss constructed from the predicted scores; and training the multimodal large model using the composite loss function to obtain the trained multimodal large model. The trained multimodal large model is validated, and the trained multimodal large model is updated based on the validation results to obtain a surgical evaluation model.

2. The method according to claim 1, characterized in that, The process of cropping the surgical video to obtain surgical segments includes: The surgical video is decoded to obtain pixel information for each frame; The pixel differences between adjacent frames are obtained based on the pixel information of each frame; Based on the pixel information of each frame and the pixel differences between adjacent frames, the surgical video is cropped to obtain surgical segments.

3. The method according to claim 2, characterized in that, The step of cropping the surgical video to obtain surgical segments based on the pixel information of each frame and the pixel differences between adjacent frames includes: The cumulative motion intensity is obtained by smoothing the pixel differences between adjacent frames using a sliding time window: Based on the cumulative motion intensity and the preset threshold, the starting frame of each surgical segment is determined; The end frame of each surgical segment is determined based on a preset continuous low motion state. The surgical video is cropped based on the start frame and the end frame to obtain multiple surgical segments.

4. The method according to claim 1, characterized in that, Inputting the surgical segment into the visual encoding unit to obtain video segment features includes: Visual feature vectors are extracted from the surgical segment, and a frame-level feature sequence is constructed based on the visual feature vectors; A new input sequence is obtained by adding a preset learning label before the frame-level feature sequence; Temporal modeling is performed on the new input sequence, and the feature vector corresponding to the preset learning label is determined as the video segment feature of the surgical segment.

5. The method according to claim 1, characterized in that, The construction method of the composite loss function includes: The text generation loss is constructed based on the correct prediction probability of each word in the surgical description by the multimodal large model; The score loss is constructed based on the numerical difference between the surgical score and the predicted score; The composite loss function is constructed by weighted summing of the text generation loss and the rating loss.

6. A surgical evaluation method, characterized in that, The method includes: Obtain the surgical video to be evaluated, input the surgical video to be evaluated into the surgical evaluation model obtained by the training method according to any one of claims 1-5, and obtain the target prediction operation description and target prediction score for each surgical segment in the surgical video to be evaluated.

7. A surgical evaluation model training device, characterized in that, The device includes: The surgical evaluation unit is used to acquire surgical videos, crop the surgical videos to obtain surgical segments, and score and describe the surgical segments to obtain surgical scores and surgical descriptions, respectively. The model acquisition unit is used to construct a multimodal large model. Using the surgical segment, the surgical score, and the surgical description as training samples, it trains the multimodal large model using a composite loss function constructed by combining text generation loss and rating loss, thereby obtaining the trained multimodal large model. The text generation loss characterizes the difference between the surgical description and the predicted operational description output by the multimodal large model, and the rating loss characterizes the difference between the surgical score and the predicted score output by the multimodal large model. The trained multimodal large model is obtained by constructing a multimodal large model including a visual encoding unit, a text encoding unit, and a multimodal fusion unit. A multimodal large model is constructed. The surgical segment is input into the visual encoding unit to obtain video segment features. The surgical description is encoded by the text encoding unit to obtain operation description text. The video segment features and the operation description text are input into the multimodal fusion unit to obtain a multimodal joint feature representation. Based on the multimodal joint feature representation, a predicted operation description and a predicted score are generated respectively. A composite loss function is obtained based on the text generation loss constructed from the predicted operation description and the score loss constructed from the predicted score. The multimodal large model is then trained using the composite loss function to obtain the trained multimodal large model. The verification unit is used to verify the trained multimodal large model and update the trained multimodal large model according to the verification results to obtain the surgical evaluation model.

8. A computer device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the surgical evaluation model training method of any one of claims 1 to 5 or the surgical evaluation method of claim 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the surgical evaluation model training method of any one of claims 1 to 5 or the surgical evaluation method of claim 6.

Images