A postoperative video analysis method and system for laparoscopic cholecystectomy

Abstract

The present application belongs to the technical field of image recognition, and is a laparoscopic cholecystectomy postoperative video analysis method and system. The system comprises: a video preprocessing module for preprocessing and quality screening; a multi-task joint network comprising: a shared space-time encoder for feature extraction of video segments and output of a multi-scale feature set; a semantic segmentation branch for output of a segmentation mask of a target class based on the multi-scale feature set and a stage context vector; a stage analysis branch for output of a surgical stage label and a stage transition time point based on the multi-scale feature set and a structure context vector; a cross-task coupling submodule for extracting structure information from the segmentation mask to generate a structure context vector; converting the stage probability vector output by the stage analysis branch into a stage context vector; and a result generation module for generating a structured postoperative analysis report based on the multi-task joint network output. The present application solves the problems of segmentation jitter and stage drift, parameter redundancy and high computational delay in existing postoperative video analysis.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and in particular relates to a method and system for postoperative video analysis of laparoscopic cholecystectomy. Background Technology

[0002] Laparoscopic cholecystectomy is one of the most common minimally invasive surgeries in general surgery, characterized by relatively standardized procedures, large amounts of video data, strong clinical teaching needs, and high requirements for complication prevention. Postoperative analysis of laparoscopic cholecystectomy videos using AI for structured analysis can provide doctors with replayable, quantifiable, and searchable postoperative evaluation results, and can also accumulate case-level evidence chains, providing fundamental data support for training, quality control, research, and product iteration.

[0003] Currently, video AI research surrounding this technique mainly focuses on semantic segmentation, stage recognition, key security structure recognition, key field of view evaluation, and model deployment optimization.

[0004] In semantic segmentation, objects such as the liver, gallbladder, Calot's triangle, cystic duct, cystic artery, common bile duct, and instruments are typically identified frame-by-frame. Common approaches include U-Net, DeepLab, FPN, PSPNet, hybrid CNN-Transformer, and video cue segmentation methods based on base models. These methods improve the visualization of surgical field structures, but most still rely heavily on single-frame or short-window segmentation, making them susceptible to the effects of smoke, blood, lens smudges, traction deformation, instrument occlusion, and cross-center imaging differences. This results in insufficient temporal continuity, poor generalization ability, and significant structural boundary jitter.

[0005] In terms of surgical stage identification, videos are typically classified time-by-time based on stages defined by datasets such as Cholec80, including preparation, Calot's triangle dissection, clamping and cutting, gallbladder removal, removal, and hemostasis and irrigation. Although long-term Transformer, multi-scale temporal modeling, transfer point perception, and online stage identification are commonly used to continuously improve the accuracy of stage identification, in the current AI video analysis architecture, surgical stage identification and semantic segmentation are usually processed as independent tasks. There is a lack of information interaction and collaborative verification channels between the two. This task separation makes it impossible to use stage context for artifact suppression in the segmentation results, and stage identification cannot be verified based on the real-time anatomical exposure status. As a result, the system is prone to segmentation jitter and stage drift when faced with interference such as smoke and occlusion, which reduces the structured nature and clinical reliability of postoperative quality control analysis. At the same time, pure temporal models often rely on high computing power and long sequence caching, making it difficult to balance the stability and cost of local hospital deployment.

[0006] In terms of model lightweighting, to adapt to localized deployment scenarios such as hospitals, methods such as network pruning, knowledge distillation, mixed-precision inference, quantized perceptual training, or low-rank decomposition are commonly used to reduce model size and computational cost. However, most lightweighting solutions only compress for a single task, leading to decreased segmentation accuracy and loss of boundary details after compression of segmentation models, and delayed stage transition recognition after compression of stage models. Furthermore, even after compressing two independent models, they still need to be deployed together, and the overall system's parameter redundancy and computational overhead issues are not fundamentally resolved. Summary of the Invention

[0007] This invention provides a method and system for postoperative video analysis of laparoscopic cholecystectomy, which solves the problems of segmentation jitter and stage drift caused by the independent semantic segmentation and stage recognition tasks in the prior art, as well as parameter redundancy and high computational latency caused by the serial deployment of independent models.

[0008] This invention provides a basic solution: a postoperative video analysis system for laparoscopic cholecystectomy, comprising: The video preprocessing module is used to preprocess and screen the quality of the input surgical videos to form video segments for analysis. A multi-task joint network is used to perform feature extraction, semantic segmentation and stage analysis on video clips, and output segmentation masks, surgical stage labels and stage transition time points; The multi-task collaborative network includes: A shared spatiotemporal encoder is used to extract multi-scale spatial texture features and temporal context features from video clips, and output a multi-scale feature set. The semantic segmentation branch is used to output a segmentation mask for the target category based on a multi-scale feature set and a stage context vector from the stage analysis branch. The stage analysis branch is used to output surgical stage labels and stage transition time points based on multi-scale feature sets and structural context vectors from the semantic segmentation branch. The cross-task coupling submodule is connected to the semantic segmentation branch and the stage analysis branch respectively. It is used to extract structural information based on the segmentation mask and generate a structural context vector to be input to the stage analysis branch; it is also used to convert the stage probability vector output by the stage analysis branch into a stage context vector to be input to the semantic segmentation branch. The results generation module is used to generate a structured postoperative analysis report based on the segmentation mask, surgical stage labels, and stage transition time points.

[0009] Preferably, in the cross-task coupling submodule, the structural information includes a structural statistical vector and a stability vector; The structural statistical vector is used to describe the spatial distribution and anatomical exposure status of each target category in the current frame; it includes the area proportion information of the target category, boundary length information, exposure score of key anatomical objects, and the relative positional relationship between multiple key targets; The stability vector is used to characterize the stability of the recognition of the preset key anatomical object in consecutive video frames.

[0010] More preferably, the stability vector calculation strategy is as follows: Calculate the continuous stability score of each preset key anatomical object based on the segmentation mask of the n consecutive frames before the current time, and combine the continuous stability scores of each key anatomical object into a stability vector. The formula for calculating the continuous stability score is as follows:

[0011] In the formula, Indicates the first Key anatomical objects calculated using frames as the center and based on continuous time windows Local continuity stability score, Indicates the first Key anatomical objects in the frame The segmentation mask; This represents the inter-frame alignment operator, used to transform adjacent frame masks to the same coordinate space; This represents the function for calculating the intersection-union ratio; Indicates the key anatomical object c in the first... Frame prediction confidence; This represents the confidence threshold for the corresponding key anatomical object c; Indicates the length of the stability evaluation window; This indicates an indicator function that takes the value 1 when the condition within the parentheses is true, and 0 otherwise.

[0012] More preferably, the preset key anatomical objects include the cystic duct, cystic artery, common bile duct, and Calot's triangle.

[0013] More preferably, the stage analysis branch includes: A short-time branch is used to capture local temporal patterns of the multi-scale feature set; Long-term branches are used to model long-term dependencies in the multi-scale feature set. A stage decoder, connected to the short-time branch and the long-time branch, is used to fuse short-time features and long-time features, and output a stage probability vector in combination with the structural context vector, and obtain the surgical stage label based on the stage probability vector; A boundary detection head, connected to the stage decoder, is used to output stage boundary probabilities based on stage features, and to determine the stage transition time point based on the stage boundary probabilities. The stage boundary probabilities are used to characterize whether the current moment is near a stage transition. The stage transition constraint unit is configured with a stage transition prior matrix. The stage decoder is constrained by the stage transition prior matrix during the decoding process. The stage transition prior matrix is ​​used to constrain the stage prediction at the current time based on the stage label at the previous time step.

[0014] More preferably, the training and deployment strategy for the multi-task joint network includes: Construct a teacher model with a parameter size larger than that of the multi-task joint network; Using a multi-task joint network as the student model, a knowledge distillation method is adopted, and one or more of the segmentation mask, stage probability vector and stage boundary probability output by the teacher model are used as supervision signals to distill and train the student model. Quantization-perception training was performed on the student model after distillation training. After training, the student model is used as the multi-task joint network for the deployment phase; Export the student model as an inference engine format and deploy it to the target hardware environment.

[0015] More preferably, a joint loss function is used to optimize the multi-task joint network, the joint loss function being expressed as:

[0016]

[0017]

[0018]

[0019] in, Represents semantic segmentation loss. Indicates the time-series consistency loss. Indicates the stage classification loss, Indicates the stage transition constraint loss. This represents the knowledge distillation loss. This indicates the quantization of the perceived training loss. These represent the weighting coefficients corresponding to each loss term; Dice loss is used to optimize the region overlap between the predicted segmentation mask and the ground truth segmentation mask. Pixel-level cross-entropy loss is used to optimize pixel-level classification accuracy. This is the boundary loss, used to enhance the positioning accuracy of the predicted boundary. , respectively, are the weighting coefficients for the corresponding loss terms; This represents the segmentation mask for frame t. This represents the segmentation mask for frame t-1. This represents the inter-frame alignment operator. express The norm, the temporal consistency loss, is used to constrain the segmentation results of adjacent frames to maintain temporal smoothness and consistency in the context of continuous target motion. This represents the stage prediction probability vector for frame t. Indicates the first The frame corresponds to the real stage label or supervision signal, and A is the stage transition prior matrix. Indicates based on the previous time stage label In the transition prior matrix The prior transition distribution selected in the middle, This represents the distribution distance function.

[0020] Preferably, the shared spatiotemporal encoder includes: Depthwise separable convolutional layers are used to extract local texture and edge information from video clips; A window attention layer group, connected to a depthwise separable convolutional layer group, is used to model mid-range spatial relationships with controlled computational cost. The hierarchical feature aggregation layer group, connected to the window attention layer group, is used to fuse semantic and detailed information from different downsampling levels and output a multi-scale feature set. A compact bottleneck quantization layer group, connected to a hierarchical feature aggregation layer group, is used to control computational complexity.

[0021] More preferably, when the stage analysis branch identifies that the current stage has entered the stage of duct or blood vessel clamping and cutting, or when the stage boundary probability output by the boundary detection head is greater than the preset frame number threshold, the continuous stability evaluation process of the key anatomical object is triggered. The evaluation process is as follows: Obtain the continuous stability score of each key anatomical object, and calculate the number of stable recognition frames for each key anatomical object based on the continuous stability score. The number of stable identification frames for each key anatomical object is compared with a preset frame threshold. If the number of stable identification frames for a key anatomical object is less than the preset frame threshold, the corresponding time segment is marked as a review segment and written into the structured postoperative analysis report. If the number of stable identification frames for each key anatomical object is greater than or equal to the preset frame number threshold, a prompt message indicating that the key structure has been sufficiently identified before clamping is generated.

[0022] It also includes a postoperative video analysis method for laparoscopic cholecystectomy, which uses the above-mentioned postoperative video analysis system for laparoscopic cholecystectomy to perform postoperative video analysis.

[0023] The principles and advantages of this invention are as follows: 1. By sharing a spatiotemporal encoder, feature extraction is performed simultaneously on two tasks: semantic segmentation and stage analysis. This allows the two tasks to share underlying visual features, avoiding redundant calculations when deploying independent models. This effectively reduces the overall number of parameters and memory usage, decreases the complexity of the engineering process, and facilitates deployment and operation on local hospital servers or workstations.

[0024] 2. By establishing bidirectional information interaction between the semantic segmentation branch and the stage analysis branch through cross-task coupling submodules, the segmentation results and stage identification enhance each other: the segmentation branch uses the stage context to suppress false positive regions that are inconsistent with the current stage, and the stage branch uses the structural context to perceive the current anatomical exposure status for process verification, which effectively reduces segmentation jitter and stage drift, and improves the recognition accuracy of key anatomical objects and the accuracy of stage discrimination.

[0025] Preferably, by introducing temporal consistency loss constraints, the segmentation results of adjacent frames are kept smooth in continuous motion scenarios, thus eliminating inter-frame jitter; by injecting clinical process prior knowledge through a stage transition prior matrix, misjudgments caused by non-physiological, non-standard, or abnormal stage switching are prevented, thereby enhancing the continuity, stability, and clinical rationality of long video analysis results.

[0026] Preferably, high-precision knowledge is transferred to a lightweight student model through teacher-student knowledge distillation, redundant channels are removed through structured pruning, and weights and activations are mapped to low-bit integers through quantized perceptual training, which effectively reduces the model size and inference latency, enabling the model to run efficiently on local hospital servers, workstations, or edge GPU environments.

[0027] Preferably, the postoperative analysis report output by the result generation module includes multi-dimensional data such as frame-by-frame segmentation mask, stage labels, stage transfer timestamps, duration of each stage, key structure exposure curves, continuous stability scores of key structures before clamping, review markers, and key frame indexes. It also supports the export of playback videos with overlaid annotations and structured JSON / database records, providing quantifiable and traceable data support for postoperative quality control, teaching evaluation, scientific research statistics, and case archiving. Attached Figure Description

[0028] Figure 1 This is a system logic block diagram of the present invention; Figure 2 This is a logic block diagram of an embodiment of the present invention; Figure 3 This is a schematic diagram of the training and deployment process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the multi-task collaborative network of the present invention; Figure 5 This is a schematic diagram of the structure of the stage analysis branch of the present invention; Figure 6 This is a schematic diagram of the lightweight training and local deployment process of the present invention. Detailed Implementation

[0029] The following detailed description illustrates the specific implementation method: The specific implementation process is as follows: (See details) Figures 1 to 2 A video analysis system for postoperative laparoscopic cholecystectomy includes a video preprocessing module, a multi-task network, and a result generation module. The video preprocessing module is used to preprocess and screen the quality of the input surgical videos to form video segments for analysis.

[0030] Preprocessing includes: de-identification, shot stabilization, frame extraction, segmentation, and color normalization. Specifically, the original video is uniformly converted to a preset frame rate. and preset resolution The input video stream, and using a length of Step size is Sliding window construction of video clips .

[0031] Quality screening includes blurry frame detection, severely occluded segment recognition, abnormal interruption detection, black screen detection, and duplicate frame detection.

[0032] In this embodiment, let the input laparoscopic cholecystectomy video be denoted as...

[0033] in, Indicates the first Frame image, This represents the total number of video frames. To utilize video timing information, a length of [length value missing] is used. The sliding window constructs a video clip, resulting in

[0034] in, Indicates the first The frame is the timing input segment of the current target frame.

[0035] See Figure 4 A multi-task joint network is used to perform feature extraction, semantic segmentation and stage analysis on video clips, and output segmentation mask, surgical stage labels and stage transition time points; Specifically, the multi-task joint network includes a shared spatiotemporal encoder, a semantic segmentation branch, a stage analysis branch, and a cross-task coupling submodule; A shared spatiotemporal encoder is used to process video clips. Multi-scale spatial texture features and temporal context features are extracted, and a multi-scale feature set is output. The shared spatiotemporal encoder employs a lightweight hybrid encoder; specifically, the shared spatiotemporal encoder includes: Depthwise separable convolutional layers are used to extract local texture and edge information from video clips; A window attention layer group, connected to a depthwise separable convolutional layer group, is used to model mid-range spatial relationships with controlled computational cost. The hierarchical feature aggregation layer group, connected to the window attention layer group, is used to fuse semantic and detailed information from different downsampling levels and output a multi-scale feature set. A compact bottleneck quantization layer group, connected to a hierarchical feature aggregation layer group, is used to control computational complexity.

[0036] The structure of the shared spatiotemporal encoder can take into account the requirements of local texture modeling capability, long-range dependency modeling capability and computational complexity control, and is suitable for complex scenarios in laparoscopic videos with smoke obscuration, tissue deformation, instrument interference and large scale changes.

[0037] In this embodiment, a shared spatiotemporal encoder is used. video clips Perform multi-scale feature extraction to obtain the output multi-scale feature set. :

[0038] in, Indicates the first Feature representation at different scales Indicates the number of feature scales. Indicates shared encoder parameters; Specifically, in this embodiment, the shared spatiotemporal encoder adopts a lightweight shared encoder, which can be CNN+Lite-Tramsformer.

[0039] The semantic segmentation branch is used to output a segmentation mask for the target category based on a multi-scale feature set and a stage context vector from the stage analysis branch.

[0040] The target categories are all object types (complete list) that can be identified by the semantic segmentation branch, including gallbladder, cystic duct, cystic artery, liver, adipose tissue, instruments, smoke area, background area and other predefined anatomical structures or surgical objects. The target categories can be configured according to clinical needs. Preferably, the target categories in this embodiment include liver, gallbladder, cystic duct, cystic artery, common bile duct, Calot's triangle, blood, smoke, gripper, electrocautery hook, scissors and background.

[0041] Semantic segmentation branches Receive multi-scale feature sets and the stage context vector from the stage analysis branch Output the segmentation mask for the target category, specifically the first... The pixel-level label image corresponding to the frame is denoted as:

[0042] In the formula, Used to characterize the pixel-level distribution of target categories in the current frame. This represents the semantic segmentation branch parameters.

[0043] The stage analysis branch is used to output surgical stage labels and stage transition time points based on multi-scale feature sets and structural context vectors from the semantic segmentation branch. The categories of surgical stages are configured according to hospital quality control standards, research needs, or teaching needs. Preferably, in this embodiment, the categories of surgical stages include the surgical field establishment or preparation stage, the Calot's triangle exposure stage, the tube and / or blood vessel clamping and cutting stage, the gallbladder dissection stage, the gallbladder removal stage, and the hemostasis, irrigation, and termination stage.

[0044] Furthermore, for different hospitals or different surgical procedures, secondary labels such as "critical safety field of view met" and "critical safety field of view not met" can be configured as supplementary outputs of the phase analysis results.

[0045] See Figure 5 The stage analysis branch includes: A short-time branch is used to capture local temporal patterns of the multi-scale feature set; Long-term branches are used to model long-term dependencies in the multi-scale feature set. A stage decoder, connected to the short-time branch and the long-time branch, is used to fuse short-time features and long-time features, and output a stage probability vector in combination with the structural context vector, and obtain the surgical stage label based on the stage probability vector; A boundary detection head, connected to the stage decoder, is used to output stage boundary probabilities based on stage features, and to determine the stage transition time point based on the stage boundary probabilities. The stage boundary probabilities are used to characterize whether the current moment is near a stage transition. The stage transition constraint unit is configured with a stage transition prior moment. The stage decoder is constrained by the stage transition prior matrix during the decoding process. The stage transition prior matrix is ​​used to constrain the stage prediction at the current moment based on the stage label at the previous moment.

[0046] Phase Analysis Branch Receive multi-scale feature sets and the structural context vector from the semantic segmentation branch Output stage probability vector in, , Indicates the number of surgical stage categories. This represents the branch parameters for stage analysis; based on the stage probability vector, the first... Frame corresponding stage label .

[0047] Specifically, in this embodiment, the short-time branch is used to capture local temporal patterns such as abrupt changes in instrument movement and rapid structural exposure; the long-time branch is used to capture long-term dependencies corresponding to the overall progression of the surgical procedure. The output features of the short-time and long-time branches are concatenated and input into the stage decoder, ultimately outputting a stage probability vector.

[0048] The final stage of decoding can be represented as:

[0049] in, This represents a vector concatenation operation. This indicates the short-time branch output characteristics. This indicates the output characteristics of the long-term branch. This indicates the structural context information from the split branch. This represents the learnable parameters. Based on the stage prediction results... With boundary detection results Joint optimization can reduce stage drift and improve the accuracy of stage switching location recognition.

[0050] The stage transition prior matrix is:

[0051] in, Represents the stage transition prior matrix The Middle line, number The elements of the column are used to characterize the surgical stage. Transfer to surgical stage Prior probability, allowance, or transition weight; This indicates the number of surgical stage categories. When the predicted stage is... At that time, it is required that the current time phase Compared to the previous moment phase The preset transfer reachability relationship is satisfied. Preferably, the constraint relationship can be written as:

[0052] in and These represent the stage labels for the previous and current moments, respectively. This indicates the threshold for allowed stage transitions.

[0053] By incorporating the stage transition prior matrix into the stage prediction sequence, jumps that do not conform to clinical operational logic can be suppressed, thereby reducing misjudgments caused by non-physiological, non-standard, or abnormal stage switching.

[0054] The cross-task coupling submodule is connected to the semantic segmentation branch and the stage analysis branch respectively. It is used to extract structural information based on the segmentation mask and generate a structural context vector to be input to the stage analysis branch; it is also used to convert the stage probability vector output by the stage analysis branch into a stage context vector to be input to the semantic segmentation branch.

[0055] Specifically, the cross-task coupling submodule includes a segmentation guidance unit and a stage feedback unit.

[0056] The segmentation guidance unit is used to extract structural information based on the segmentation mask and generate a structural context vector, which is then input to the stage analysis branch; it is also used to generate a gating vector based on the segmentation mask to modulate the feature response of the stage analysis branch.

[0057] In the cross-task coupling submodule, the structural information includes structural statistical vectors and stability vectors; The structural statistical vector is used to describe the spatial distribution and anatomical exposure status of each target category in the current frame; it includes the area proportion information of the target category, boundary length information, exposure score of key anatomical objects, and the relative positional relationship between multiple key targets; specifically, the structural statistical vector ,Right now

[0058] in, This indicates the area percentage information for one or more target categories. This indicates information about the target boundary length, boundary sharpness, or shape stability. The score indicates the degree of exposure of key anatomical objects. It indicates the relative positional relationship, distance relationship, angular relationship or topological relationship between multiple key targets.

[0059] The predetermined key anatomical objects are determined in advance based on the laparoscopic cholecystectomy procedure, key safety field assessment requirements, and risk control needs before clamping and cutting ducts and / or vessels. In this embodiment, the predetermined key anatomical objects include the cystic duct, cystic artery, common bile duct, and Calot's triangle.

[0060] The stability vector is used to characterize the stability of the recognition of the preset key anatomical object in consecutive video frames.

[0061] The stability vector calculation strategy is as follows: Calculate the continuous stability score of each preset key anatomical object based on the segmentation mask of the n consecutive frames before the current time, and combine the continuous stability scores of each key anatomical object into a stability vector. The formula for calculating the continuous stability score is as follows:

[0062] In the formula, Indicates the first Key anatomical objects calculated using frames as the center and based on continuous time windows The local continuity stability score is used to evaluate the stability of the structure in the vicinity of the current time. Indicates the first Key anatomical objects in the frame The segmentation mask; This represents the inter-frame alignment operator, used to transform adjacent frame masks to the same coordinate space; This represents the function for calculating the intersection-union ratio; Indicates the key anatomical object c in the first... Frame prediction confidence; This represents the confidence threshold for the corresponding key anatomical object c; Indicates the length of the stability evaluation window; This indicates an indicator function that takes the value 1 when the condition within the parentheses is true, and 0 otherwise.

[0063] Based on this, the continuous stability of each key structure is used to form a stability vector. and structural statistical vectors The structural context information that constitutes the analysis branches of the common phase can be represented as:

[0064] in, Represents the learnable mapping matrix. This represents the concatenation result of the structural statistics vector and the continuous stability vector. This represents the structural context vector input to the stage analysis branch.

[0065] By introducing the continuous stability vector of the key structure The phase analysis branch can not only utilize the anatomical exposure status of the current frame, but also the stable identification information of multiple consecutive frames before key operations such as clamping and cutting occur. This reduces the phase misjudgment and quality control evaluation deviation caused by momentary occlusion, smoke, blood, lens contamination or instrument interference, and improves the continuity, reliability and interpretability of postoperative video analysis results.

[0066] The stage feedback unit is used to convert the stage probability vector output by the stage analysis branch into a stage context vector and input it into the semantic segmentation branch; it is also used to generate a gating vector based on the stage probability vector and modulate the channel attention weight or spatial attention weight of the semantic segmentation branch.

[0067] Specifically, in this embodiment, the first gating vector in the cross-task coupling module, which guides the stage analysis based on the segmentation result, and the second gating vector, which modulates the segmentation branch in reverse based on the stage result, can be represented as follows:

[0068] in, This indicates that global pooling, area statistics, boundary statistics, region relation encoding, or a combination thereof are performed on the segmentation results. This represents the Sigmoid function. The gating vector represents the learnable parameters. The channel weights, attention weights, or feature responses of the stage analysis branch and the semantic segmentation branch can be modulated respectively to achieve bidirectional prior injection, thereby improving the structural sensitivity of stage recognition and the stage consistency of segmentation results.

[0069] See Figure 6 The training and deployment strategies for multi-task joint networks include: Construct a teacher model with a parameter size larger than that of the multi-task joint network; Using a multi-task joint network as the student model, a knowledge distillation method is adopted, and one or more of the segmentation mask, stage probability vector and stage boundary probability output by the teacher model are used as supervision signals to distill and train the student model. Quantization-perception training was performed on the student model after distillation training. After training, the student model is used as the multi-task joint network for the deployment phase; Export the student model as an inference engine format and deploy it to the target hardware environment.

[0070] like Figure 3As shown, specifically in this embodiment, during the training phase, it is preferable to construct the model using a combination of publicly available data and in-hospital data. Public data can provide relatively standardized stage labels and partial segmentation labels, while in-hospital data can provide equipment differences, surgeon differences, and business data distribution that are closer to real clinical applications. For videos with insufficient or incomplete annotations, pseudo-labels can be generated first using the teacher model, and then combined with manual sampling verification and screening to form a high-confidence training sample set.

[0071] In some embodiments, the teacher model may employ a spatiotemporal model with a large parameter scale, a video segmentation model assisted by a basic visual model, or other high-performance models suitable for surgical video analysis, to output pseudo-segmentation results and pseudo-stage labels. Furthermore, a segmentation confidence threshold may be set for the pseudo-labels. and stage confidence threshold Only if the segmentation prediction confidence satisfies Or the stage prediction confidence level meets Only when the time is right will the corresponding samples be included in the subsequent distillation training or joint training process, in order to reduce the adverse effects of low-quality pseudo-labels on model training.

[0072] Furthermore, before formal supervised training, self-supervised pre-training can be performed to improve the shared encoder's ability to represent texture patterns, instrument motion patterns, and tissue dynamic changes in laparoscopic surgery videos. The self-supervised pre-training tasks include, but are not limited to, one or more of the following: mask reconstruction, temporal sequence discrimination, adjacent video segment comparison learning, and keyframe consistency learning. After self-supervised pre-training, multi-task joint fine-tuning is then performed based on the labeled videos.

[0073] The above-mentioned self-supervised pre-training and multi-task joint fine-tuning process based on labeled videos can be represented as: in, This indicates a collection of videos that are not labeled. This indicates a collection of videos that have been labeled. This indicates the segmentation of supervisory information. Indicates phased monitoring information, This represents the initial parameters of the shared encoder obtained through pre-training.

[0074] like Figure 3 As shown, the shared spatiotemporal encoder in this embodiment Preferably, it is composed of multiple functional modules. Specifically, the shared spatiotemporal encoder includes: a low-level depthwise separable convolutional module for extracting local texture and edge information; a lightweight window attention module for modeling mid-range spatial relationships with controlled computational cost; and a multi-scale feature aggregation module for fusing semantic and detail information from different downsampling levels. By combining the above modules, the requirements for local structure modeling capability, context modeling capability, and deployment complexity control can be met.

[0075] In this embodiment, during the deployment phase, the optimized student model can be converted into the ONNX model format, and TensorRT or other inference engines can be used to generate an optimized computation graph suitable for execution on the target hardware platform.

[0076] The multi-task joint network is optimized using a joint loss function, which is expressed as follows:

[0077]

[0078]

[0079]

[0080] in, Represents semantic segmentation loss. Indicates the time-series consistency loss. Indicates the stage classification loss, Indicates the stage transition constraint loss. This represents the knowledge distillation loss. This indicates the quantization of the perceived training loss. These represent the weighting coefficients corresponding to each loss term; Dice loss is used to optimize the region overlap between the predicted segmentation mask and the ground truth segmentation mask. Pixel-level cross-entropy loss is used to optimize pixel-level classification accuracy. This is the boundary loss, used to enhance the positioning accuracy of the predicted boundary. , respectively, are the weighting coefficients for the corresponding loss terms; This represents the segmentation mask for frame t. This represents the segmentation mask for frame t-1. This represents the inter-frame alignment operator. express The norm, the temporal consistency loss, is used to constrain the segmentation results of adjacent frames to maintain temporal smoothness and consistency in the context of continuous target motion. This represents the stage prediction probability vector for frame t. Indicates the first The frame corresponds to the real stage label or supervision signal, and A is the stage transition prior matrix. Indicates based on the previous time stage label In the transition prior matrix The prior transition distribution selected in the middle, This represents the distribution distance function.

[0081] Distillation loss can be expressed as:

[0082] in, and These represent the intermediate features of the teacher model and the student model, respectively. This represents the feature dimension alignment operator. Indicates the distillation temperature coefficient. and These represent the boundary response diagrams output by the teacher model and the student model, respectively. This represents the weighting coefficient of the corresponding loss term.

[0083] To adapt to local hospital deployment environments, edge computing environments, or resource-constrained workstation environments, a teacher-student distillation strategy is adopted to optimize the model for lightweight use. The teacher model... With high accuracy as the optimization goal, the student model The optimization goals are low parameter count, low computational complexity, and high throughput. During training, knowledge distillation loss is introduced. and quantized perception training loss Among them, knowledge distillation loss This is used to constrain the student model to approximate one or more of the teacher model's intermediate feature representation, category response distribution, boundary attention map, and stage transition cues; quantifying the perceptual training loss. This is used to reduce the deviation in feature space, output space, or prediction accuracy before and after model quantization, so as to improve the robustness and accuracy preservation capability when deploying in subsequent low-bit scenarios.

[0084] After distillation training is complete, structured channel pruning can be performed. Preferably, scoring can be based on the importance of convolutional channels. Alternatively, the scaling factor of the batch normalization layer can be adjusted to prune low-importance channels, and model performance can be restored through fine-tuning in a few subsequent rounds.

[0085] Preferably, when the stage analysis branch identifies that the current stage has entered the stage of clamping and cutting the duct or blood vessel, or when the stage boundary probability output by the boundary detection head is greater than the preset frame number threshold, the continuous stability evaluation process of the key anatomical object is triggered. The evaluation process is as follows: Obtain the continuous stability score of each key anatomical object, and calculate the number of stable recognition frames for each key anatomical object based on the continuous stability score. The number of stable identification frames for each key anatomical object is compared with a preset frame threshold. If the number of stable identification frames for a key anatomical object is less than the preset frame threshold, the corresponding time segment is marked as a review segment and written into the structured postoperative analysis report. The preset frame count threshold is determined based on a validation dataset. Specifically, clinicians label the exposure status of key anatomical objects in the video segment before clipping as stable or unstable, and iterate through candidate thresholds. The selected threshold is the one where the F1 score (a comprehensive evaluation index of stable exposure recognition results, calculated from precision and recall, used to evaluate the accuracy and completeness of stable exposure judgment), Youden index, or precision reaches its maximum. The preset frame count threshold ranges from 5 to 30 frames; in this embodiment, 10 frames is preferred. By comparing the number of stable identification frames of key anatomical objects with the preset frame count threshold, it is confirmed whether the key anatomical objects are in a stable state during the current time period, ensuring that key anatomical structures have been continuously and reliably identified before the clipping operation, providing a quantitative basis for postoperative quality control.

[0086] The formula for calculating the stable identification frame count of key anatomical objects is as follows:

[0087] in, Indicates the first The frame represents the continuous stability score of the key anatomical object c calculated by the evaluation center; Indicates the length of the backtracking time window; This indicates that the continuous stability score threshold condition is met within the backtracking time window. The number of valid stable identification frames, used to represent the number of valid frames that meet the stable identification conditions, is a discrete statistical indicator. That is to say, Used to describe the continuous stability of key anatomical objects near a single moment, while Further statistical analysis of the stable states at multiple time points is used to evaluate whether the continuous stability identification requirements were met before the occurrence of critical operations. and These are used to describe stability information at different levels, enabling the system to capture the local continuous stability of key anatomical objects near each frame and to accumulate and statistically analyze the number of stable identification frames in the time window dimension. This improves the reliability and interpretability of the determination of the adequacy of identification of key structures before clamping, and provides a quantitative basis for postoperative quality control that is both fine-grained and holistic.

[0088] In this way, the present invention can correlate frame-by-frame segmentation results, stage identification results and continuous stability evidence before clamping in the output stage, so that the postoperative quality control results include not only single-frame identification conclusions, but also continuous temporal evidence before the occurrence of key operations, thereby improving the interpretability and reliability of postoperative review, teaching evaluation and case screening processes.

[0089] The results generation module is used to generate a structured postoperative analysis report based on the segmentation mask, surgical stage labels, and stage transition time points.

[0090] In this embodiment, based on the above method, the system can output one or more of the following analysis results for the entire laparoscopic cholecystectomy video: frame-by-frame segmentation mask, frame-by-frame stage labels, stage transition timestamps, duration statistics of each stage, key structure exposure curves, danger zone proximity scores, exposure adequacy scores, keyframe indexes, playback videos with overlay annotations, and case-level structured JSON records, XML records, or database records.

[0091] For postoperative quality control scenarios, quality control evaluation indicators can be further calculated based on the above output results. These quality control evaluation indicators include, but are not limited to, one or more of the following: Calot triangle exposure adequacy score, number of frames for stable identification of key channels before clamping, continuous stability score of key structures before clamping, verification mark for insufficient stable identification of key structures, frequency of instrument-tissue conflict during gallbladder dissection, duration of critical safety field of view, and number of abnormal retreats.

[0092] The output reports and playback videos obtained from the analysis can be further written into the hospital's case management system, quality control management system, teaching platform, or scientific research database.

[0093] It also includes a postoperative video analysis method for laparoscopic cholecystectomy, which uses the aforementioned postoperative video analysis system for laparoscopic cholecystectomy to perform postoperative video analysis.

[0094] In this embodiment, the input image resolution can be set to 512×512 or 640×640; the video segment length L can be 8 to 32 frames; the time window of the stage analysis head can cover an equivalent frame length range of 30 to 120 seconds; the number of output categories of the semantic segmentation branch can be set to 8 to 15; and the number of student model parameters can be controlled within the range of 5M to 25M. The model can be run offline in batch processing on a single consumer-grade GPU or a hospital local workstation GPU, or it can be executed at a lower speed in a high-performance CPU environment.

[0095] To further illustrate the technical effects of this invention, an exemplary experimental verification was conducted, comparing this invention with an independent segmentation model, an independent stage model serial analysis scheme, and a dual-task model that only shares a backbone. The test set can use retrospective laparoscopic cholecystectomy postoperative videos. Evaluation metrics include the segmentation mean Dice, stage identification accuracy, mean absolute error at stage transition time points, number of model parameters, and offline inference speed. The comparison results are shown in Table 1.

[0096] Table 1 Comparison of different postoperative video analysis protocols

[0097] As shown in Table 1, adding bidirectional coupling and stage transition priors to the shared backbone improves segmentation accuracy, stage identification accuracy, and transition point localization accuracy. The student model, after distillation, pruning, and quantization, significantly reduces the number of parameters and improves inference speed while maintaining key accuracy. Furthermore, under the above experimental conditions, adding a continuous stability evaluation of key structures before clamping increases the recall rate of re-triggered segments with insufficient stability identification of key structures from 82.8% to 90.6%, which is beneficial for postoperative quality control review and teaching playback. The above data are used to illustrate the technical effects of this embodiment under specific sample and hardware conditions and do not constitute a limitation on the scope of protection of this invention.

[0098] Through the above technical solutions, this invention can simultaneously achieve imaging of key anatomical objects, analysis of surgical procedures, and lightweight deployment optimization in postoperative application scenarios, and improve the analytical performance of each sub-task through a multi-task coupling mechanism. For hospital business applications, this invention is particularly suitable for scenarios such as postoperative quality control, surgical teaching, scientific research statistics, case screening, key segment playback, AI-assisted evaluation, and continuous model iteration. Compared with existing single-task video analysis solutions, this invention can better meet the comprehensive needs of systematic, structured, and deployable analysis in real clinical practice.

[0099] The above are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A video analysis system for postoperative laparoscopic cholecystectomy, characterized in that, include: The video preprocessing module is used to preprocess and screen the quality of the input surgical videos to form video segments for analysis. A multi-task joint network is used to perform feature extraction, semantic segmentation and stage analysis on video clips, and output segmentation masks, surgical stage labels and stage transition time points; The multi-task collaborative network includes: A shared spatiotemporal encoder is used to extract multi-scale spatial texture features and temporal context features from video clips, and output a multi-scale feature set. The semantic segmentation branch is used to output a segmentation mask for the target category based on a multi-scale feature set and a stage context vector from the stage analysis branch. The stage analysis branch is used to output surgical stage labels and stage transition time points based on multi-scale feature sets and structural context vectors from the semantic segmentation branch. The cross-task coupling submodule is connected to the semantic segmentation branch and the stage analysis branch respectively. It is used to extract structural information based on the segmentation mask and generate a structural context vector to be input to the stage analysis branch; it is also used to convert the stage probability vector output by the stage analysis branch into a stage context vector to be input to the semantic segmentation branch. The results generation module is used to generate a structured postoperative analysis report based on the segmentation mask, surgical stage labels, and stage transition time points.

2. The postoperative video analysis system for laparoscopic cholecystectomy according to claim 1, characterized in that: In the cross-task coupling submodule, the structural information includes structural statistical vectors and stability vectors; The structural statistical vector is used to describe the spatial distribution and anatomical exposure status of each target category in the current frame; This includes information on the area proportion of target categories, boundary length, exposure scores of key anatomical objects, and the relative positional relationships between multiple key targets; The stability vector is used to characterize the stability of the recognition of the preset key anatomical object in consecutive video frames.

3. The postoperative video analysis system for laparoscopic cholecystectomy according to claim 2, characterized in that: The stability vector calculation strategy is as follows: Calculate the continuous stability score of each preset key anatomical object based on the segmentation mask of the n consecutive frames before the current time, and combine the continuous stability scores of each key anatomical object into a stability vector. The formula for calculating the continuous stability score is as follows: In the formula, Indicates the first Key anatomical objects calculated using frames as the center and based on continuous time windows Local continuity stability score, Indicates the first Key anatomical objects in the frame The segmentation mask; This represents the inter-frame alignment operator, used to transform adjacent frame masks to the same coordinate space; This represents the function for calculating the intersection-union ratio; Indicates the key anatomical object c in the first... Frame prediction confidence; This represents the confidence threshold for the corresponding key anatomical object c; Indicates the length of the stability evaluation window; This indicates an indicator function that takes the value 1 when the condition within the parentheses is true, and 0 otherwise.

4. The postoperative video analysis system for laparoscopic cholecystectomy according to claim 2 or 3, characterized in that: The preset key anatomical objects include the cystic duct, cystic artery, common bile duct, and Calot's triangle.

5. The postoperative video analysis system for laparoscopic cholecystectomy according to claim 3, characterized in that: The phase analysis branches include: A short-time branch is used to capture local temporal patterns of the multi-scale feature set; Long-term branches are used to model long-term dependencies in the multi-scale feature set. A stage decoder, connected to the short-time branch and the long-time branch, is used to fuse short-time features and long-time features, and output a stage probability vector in combination with the structural context vector, and obtain the surgical stage label based on the stage probability vector; A boundary detection head, connected to the stage decoder, is used to output stage boundary probabilities based on stage features, and to determine the stage transition time point based on the stage boundary probabilities. The stage boundary probabilities are used to characterize whether the current moment is near a stage transition. The stage transition constraint unit is configured with a stage transition prior matrix. The stage decoder is constrained by the stage transition prior matrix during the decoding process. The stage transition prior matrix is ​​used to constrain the stage prediction at the current time based on the stage label at the previous time step.

6. The postoperative video analysis system for laparoscopic cholecystectomy according to claim 5, characterized in that: Training and deployment strategies for multi-task joint networks include: Construct a teacher model with a parameter size larger than that of the multi-task joint network; Using a multi-task joint network as the student model, a knowledge distillation method is adopted, and one or more of the segmentation mask, stage probability vector and stage boundary probability output by the teacher model are used as supervision signals to distill and train the student model. Quantization-perception training was performed on the student model after distillation training. After training, the student model is used as the multi-task joint network for the deployment phase; Export the student model as an inference engine format and deploy it to the target hardware environment.

7. The postoperative video analysis system for laparoscopic cholecystectomy according to claim 6, characterized in that: The multi-task joint network is optimized using a joint loss function, which is expressed as follows: in, Represents semantic segmentation loss. Indicates the time-series consistency loss. Indicates the stage classification loss, Indicates the stage transition constraint loss. This represents the loss from knowledge distillation. This represents the quantized perceptual training loss. These represent the weighting coefficients corresponding to each loss term; Dice loss is used to optimize the region overlap between the predicted segmentation mask and the ground truth segmentation mask. Pixel-level cross-entropy loss is used to optimize pixel-level classification accuracy. This is the boundary loss, used to enhance the positioning accuracy of the predicted boundary. , respectively, are the weighting coefficients for the corresponding loss terms; This represents the segmentation mask for frame t. This represents the segmentation mask for frame t-1. This represents the inter-frame alignment operator. express The norm, the temporal consistency loss, is used to constrain the segmentation results of adjacent frames to maintain temporal smoothness and consistency in the context of continuous target motion. This represents the stage prediction probability vector for frame t. Indicates the first The frame corresponds to the real stage label or supervision signal, and A is the stage transition prior matrix. Indicates based on the previous time stage label In the transition prior matrix The prior transition distribution selected in the middle, This represents the distribution distance function.

8. The postoperative video analysis system for laparoscopic cholecystectomy according to claim 1, characterized in that: The shared spatiotemporal encoder includes: Depthwise separable convolutional layers are used to extract local texture and edge information from video clips; A window attention layer group, connected to a depthwise separable convolutional layer group, is used to model mid-range spatial relationships with controlled computational cost. The hierarchical feature aggregation layer group, connected to the window attention layer group, is used to fuse semantic and detailed information from different downsampling levels and output a multi-scale feature set. A compact bottleneck quantization layer group, connected to a hierarchical feature aggregation layer group, is used to control computational complexity.

9. The postoperative video analysis system for laparoscopic cholecystectomy according to claim 5, characterized in that: When the stage analysis branch identifies that the current stage has entered the stage of clamping and cutting the tube or blood vessel, or when the stage boundary probability output by the boundary detection head is greater than the preset frame number threshold, the continuous stability evaluation process of the key anatomical object is triggered. The evaluation process is as follows: Obtain the continuous stability score of each key anatomical object, and calculate the number of stable recognition frames for each key anatomical object based on the continuous stability score. The number of stable identification frames for each key anatomical object is compared with a preset frame threshold. If the number of stable identification frames for a key anatomical object is less than the preset frame threshold, the corresponding time segment is marked as a review segment and written into the structured postoperative analysis report. If the number of stable identification frames for each key anatomical object is greater than or equal to the preset frame number threshold, a prompt message indicating that the key structure has been sufficiently identified before clamping is generated.

10. A method for postoperative video analysis of laparoscopic cholecystectomy, characterized in that, include: Postoperative video analysis was performed using the laparoscopic cholecystectomy postoperative video analysis system as described in any one of claims 1 to 9.

Images