A method for classifying actions of an endoluminal surgical instrument based on kinematic appearance fusion and related apparatus
By fusing kinematic and appearance features into intracavitary surgical videos and combining them with temporal smoothing using a hidden Markov model, accurate classification and stable identification of instrument movements are achieved. This supports graded risk warnings and surgical operation log recording, solving the problems of unstable identification and insufficient evaluation in existing technologies.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHUHAI HENGQIN ALL-STAR MEDICAL TECHNOLOGY CO LTD
- Filing Date
- 2026-05-27
- Publication Date
- 2026-06-23
AI Technical Summary
Existing methods for instrument motion recognition in intracavitary surgical videos suffer from problems such as insufficient single-modal information, easy confusion due to similar motion patterns, instability between frames, unstable classification output, and limited time-series smoothing capabilities, which cannot support downstream risk assessment.
A kinematic appearance fusion-based approach is adopted. By acquiring intracavitary video frame sequences and instrument tip trajectory data, and combining instance segmentation, kinematic feature extraction, appearance feature fusion, multi-class confidence prediction, and hidden Markov model temporal smoothing, the probability distribution of instrument action categories is output, triggering graded risk warnings and surgical operation log recording.
It achieves accurate classification of instrument movement types, solves the confusion caused by similar movement patterns, improves inter-frame stability, and outputs category probability distributions to support downstream risk assessment, thereby enhancing the safety and recording accuracy of surgical procedures.
Smart Images
Figure CN122265343A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical image processing, and in particular to a method and related device for classifying the motions of intracavitary surgical instruments based on kinematic appearance fusion. Background Technology
[0002] Accurate identification of instrument movement types (including cutting, electrocoagulation, dissection, clamping, and traction) during endovascular surgery is crucial for triggering graded risk warnings and automatically recording surgical operation logs. Identifying instrument movements in endovascular videos faces challenges such as insufficient single-modal information, easy confusion of similar movements, and inter-frame instability.
[0003] Existing methods for instrument motion recognition in intracavitary videos have the following problems: First, existing methods mainly perform instrument motion recognition based on a single modality (image or kinematic data), without introducing dual-source fusion of kinematic and appearance features. Therefore, existing methods are prone to misclassification when the motion patterns are similar. Second, existing methods perform motion classification based on features of a single frame image, without introducing a temporal dimension to handle inter-frame instability. Therefore, existing methods have unstable recognition results in scenarios with instrument shaking or motion transitions. Third, the classification output of existing methods is discrete categories, without outputting category probability distributions, which cannot be used for downstream confidence-based risk assessment mechanisms. Fourth, the temporal smoothing of existing methods is based on simple sliding window averaging, without introducing temporal modeling based on state transition priors. Therefore, the temporal smoothing capability of existing methods is limited. Summary of the Invention
[0004] To address the above technical problems, this application provides a method and related device for classifying the motions of intracavitary surgical instruments based on kinematic appearance fusion.
[0005] Firstly, a method for classifying the movements of intracavitary surgical instruments based on kinematic appearance fusion is provided, comprising the following steps: S1. Acquire an intracavitary video frame sequence and instrument tip trajectory data, wherein the intracavitary video frame sequence includes at least 5 consecutive images arranged in chronological order, and the instrument tip trajectory data includes the three-dimensional coordinates of the instrument tip at the corresponding time. S2. Perform instrument instance segmentation on each frame in the intracavitary video frame sequence to obtain the instrument mask and instrument position; S3. Extract kinematic features based on the trajectory data of the instrument tip; S4. Fusion of appearance features based on the image region corresponding to the instrument mask; S5. Perform multi-class confidence prediction based on the kinematic features and the appearance features to obtain the probability distribution of the device action categories; S6. Perform temporal smoothing on the probability distribution of the device action categories based on the Hidden Markov Model, and output the device action category corresponding to the maximum probability.
[0006] In any embodiment of this application, the kinematic features include velocity, acceleration, and attitude angular velocity. The appearance features include the instrument's RGB texture features and the response features of the tissue surrounding the instrument.
[0007] In any embodiment of this application, the instance segmentation network includes an instance segmentation network based on a convolutional neural network; the instrument mask is output in the form of a binary image, and the instrument position is calculated based on the centroid coordinates of the instrument mask.
[0008] In any embodiment of this application, the multi-class confidence prediction is performed based on the following formula: P(a|F) = softmax(W·F + b), Where P(a|F) is the posterior probability distribution of the action category based on the fusion feature F, which is obtained by concatenating the kinematic feature and the appearance feature.
[0009] In any embodiment of this application, the temporal smoothing is performed based on the Viterbi decoding algorithm of the Hidden Markov Model; the state set of the Hidden Markov Model includes cutting state, electrocoagulation state, peeling state, clamping state and traction state; the state transition probability matrix is obtained based on the state transition prior statistics of clinical surgical operations.
[0010] In a second aspect, an intracavitary surgical instrument motion classification device based on kinematic appearance fusion is provided, comprising: a video and trajectory acquisition unit, an instrument instance segmentation unit, a kinematic feature extraction unit, an appearance feature fusion unit, a multi-class confidence prediction unit, and a temporal smoothing output unit, which respectively execute steps S1 to S6 of the method described in the first aspect.
[0011] Thirdly, an electronic device is provided, including a processor and a storage unit, the storage unit being used to store computer program code, the code containing computer instructions, wherein when the processor executes these instructions, the electronic device performs the method described in the first aspect.
[0012] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in the first aspect.
[0013] Fifthly, a computer program product is provided, the computer program product comprising a computer program or instructions that, when the computer program or instructions are run on a computer, cause the computer to perform the method described in the first aspect.
[0014] It should be understood that the above general descriptions and subsequent specific descriptions are for illustrative and explanatory purposes only and do not impose any limitations on this application.
[0015] In this application, compared with the prior art, the technical problem to be solved by this application is: to accurately classify the types of instrument actions (including cutting, electrocoagulation, peeling, clamping and traction) in intracavitary video images and instrument tip trajectory data, to handle the confusion caused by similar action patterns, to handle the instability between frames, and to output the category probability distribution to support the downstream risk assessment mechanism. The technical means employed in this application and their non-obvious aspects compared to existing technologies include: First, this application addresses the problem of insufficient information in a single modality based on a dual-source fusion mechanism of kinematic and appearance features, a design not seen in existing methods based solely on a single modality (image or kinematics); Second, this application introduces state transition prior execution temporal smoothing based on a Hidden Markov Model, a design not seen in existing methods based solely on sliding window averaging; Third, this application outputs a five-class probability distribution of action categories, a design not seen in existing methods that only output discrete categories; Fourth, this application jointly inputs kinematic features (velocity + acceleration + attitude angular velocity) and appearance features (RGB texture + tissue response) into the classifier, a design not seen in existing methods based solely on a single feature; Fifth, this application uses instrument action categories to trigger graded risk warnings and automatic recording of surgical operation logs, a design not seen in existing methods that only output action categories but have no downstream applications. The related devices described in this application include devices, electronic devices, computer-readable storage media, and computer program products. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be explained below.
[0017] Figure 1 This is a flowchart illustrating a method for classifying the motions of intracavitary surgical instruments based on kinematic appearance fusion, as provided in an embodiment of this application.
[0018] Figure 2 This is a schematic diagram of the structure of an intracavitary surgical instrument motion classification device based on kinematic appearance fusion, provided in an embodiment of this application.
[0019] Figure 3 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of this application. Detailed Implementation
[0020] To enable those skilled in the art to more fully understand the technical solution of this application, the technical solution of this application will be explained in detail and clearly with reference to the accompanying drawings.
[0021] In the specification, claims, and related drawings of this application, the terms "first," "second," etc., are used only to distinguish different elements and do not imply any particular order. Meanwhile, the use of "comprising" and "having," and variations thereof, indicates non-exclusive inclusion.
[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for classifying the motions of intracavitary surgical instruments based on kinematic appearance fusion, as provided in an embodiment of this application. The method includes steps S1 to S6.
[0023] S1. Video and Instrument Trajectory Acquisition: Intracavitary video frame sequences are acquired through an intracavitary imaging device, and instrument tip trajectory data at corresponding moments are acquired through a surgical instrument tracking system.
[0024] In this embodiment, the total number of frames in the intracavitary video frame sequence ranges from 5 to 30 frames. The inter-frame time interval ranges from 33 milliseconds to 100 milliseconds. The instrument tip trajectory data is aligned to the corresponding frame in the video frame sequence at each frame.
[0025] S2. Instrument instance segmentation: Perform instrument instance segmentation on each frame in the intracavitary video frame sequence to obtain the instrument mask and instrument position.
[0026] The device instance segmentation is performed based on an instance segmentation network. This instance segmentation network includes, but is not limited to, Mask R-CNN, YOLACT, SOLO, or other networks with instance segmentation capabilities. This application does not rely on a specific network architecture.
[0027] In this embodiment, the instrument mask is output as a binary image, and the instrument position is calculated based on the centroid coordinates of the instrument mask. This instrument instance segmentation can distinguish multiple surgical instruments existing simultaneously in a single frame image.
[0028] S3. Kinematic feature extraction: Extract kinematic features based on the trajectory data of the instrument tip.
[0029] In this embodiment, the kinematic features include velocity, acceleration, and attitude angular velocity. The velocity is calculated by dividing the difference in instrument tip coordinates between adjacent frames by the inter-frame time interval. The acceleration is calculated by dividing the difference in velocity between adjacent frames by the inter-frame time interval. The attitude angular velocity is calculated based on the inter-frame rate of change of the instrument's attitude angle.
[0030] The introduction of kinematic features addresses the clinical prior that different instruments in endoscopic surgery have different kinematic patterns. The action recognition based on kinematic features is not a routine step in general video analysis that is based solely on image information.
[0031] S4. Appearance feature fusion: Fuse appearance features based on the image region corresponding to the instrument mask.
[0032] The appearance features include the RGB texture features of the instrument and the response features of the tissue surrounding the instrument.
[0033] In this embodiment, the RGB texture features of the instrument are calculated based on the RGB channel statistics of the instrument mask region, and the tissue response features around the instrument are calculated based on the color change and deformation features of the outer region of the instrument mask. The appearance features and the kinematic features are concatenated to form a fused feature vector.
[0034] S5. Multi-class confidence prediction: Perform multi-class confidence prediction based on the kinematic features and the appearance features to obtain the probability distribution of the device action category.
[0035] In this embodiment, the multi-class confidence prediction is performed based on the following formula: P(a|F) = softmax(W·F + b), Where P(a|F) is the posterior probability distribution of the action category based on the fusion feature F, W is the classification weight matrix, and b is the bias term. The fusion feature F is obtained by concatenating the kinematic feature and the appearance feature. The probability distribution of the instrument action category includes cutting probability, electrocoagulation probability, peeling probability, clamping probability, and traction probability, and the sum of the five probabilities is equal to 1.
[0036] S6. Temporal Smoothing and Output: Based on the Hidden Markov Model, perform temporal smoothing on the probability distribution of the device action categories and output the device action category corresponding to the maximum probability.
[0037] In this embodiment, the temporal smoothing is performed using a Viterbi decoding algorithm based on a Hidden Markov Model (HMM). The state set of the HMM includes cutting, electrocoagulation, dissection, clamping, and traction states. The state transition probability matrix is obtained based on prior statistics of state transitions during clinical surgical operations. The emission probability is obtained based on the probability distribution of instrument action categories output in step S5.
[0038] The temporal smoothing mechanism based on Hidden Markov Models differs from the conventional steps based on sliding window averaging. The temporal modeling based on state transition priors ensures the stability of the action classification in action transition scenarios, and the temporal modeling based on Hidden Markov Models originates from the state transition patterns of clinical surgical operations.
[0039] Clinical Application Example 1: In laparoscopic total mesorectal resection for rectal cancer, the method detects the movement category of surgical instruments in real time. When the time-smoothed instrument movement category is cutting and the distance between the instrument tip coordinates and key anatomical structures is less than 5 mm, a level 3 risk warning is triggered; simultaneously, the instrument movement category and timestamp are automatically written into the surgical operation log.
[0040] Clinical Application Example 2: During pancreaticoduodenectomy, the method detects the instrument movement category in real time during the bile duct dissection phase. When the time-smoothed instrument movement category is dissection and the duration is greater than 5 seconds, the method automatically records the dissection movement and its duration into the surgical operation log for postoperative surgical quality assessment.
[0041] Please see Figure 2 , Figure 2 This is a schematic diagram of a kinematic appearance fusion-based intracavitary surgical instrument motion classification device provided in an embodiment of this application. The kinematic appearance fusion-based intracavitary surgical instrument motion classification device 1 includes: The video and trajectory acquisition unit 11 is used to acquire intracavitary video frame sequences and instrument tip trajectory data.
[0042] The instrument instance segmentation unit 12 is used to perform instrument instance segmentation on each frame in the intracavitary video frame sequence.
[0043] The kinematic feature extraction unit 13 is used to extract kinematic features based on the trajectory data of the instrument tip.
[0044] The appearance feature fusion unit 14 is used to fuse appearance features based on the image region corresponding to the instrument mask.
[0045] The multi-class confidence prediction unit 15 is used to perform multi-class confidence prediction based on the kinematic features and the appearance features.
[0046] The temporal smoothing output unit 16 is used to perform temporal smoothing on the probability distribution of the device action category based on the hidden Markov model and output it.
[0047] Please see Figure 3 , Figure 3This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of this application. The electronic device 2 includes a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21, the memory 22, the input device 23, and the output device 24 are communicatively connected via a bus. The memory 22 is used to store computer program code, which contains computer instructions. When the processor 21 executes the instructions, the electronic device 2 executes the kinematic appearance fusion-based intracavitary surgical instrument motion classification method.
[0048] The various embodiments in the application have their own focus. For ease of description and simplification, the same or similar content may not be described in detail in different embodiments. For details, please refer to the relevant descriptions in the other embodiments mentioned above.
[0049] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The following describes in detail another electronic device provided by the embodiments of this application, taking electronic device 2 as an example.
[0050] The method described in this application can be implemented based on hardware, software, or a combination of hardware and software. The hardware implementation can be based on application-specific integrated circuits, field-programmable gate arrays, or other programmable logic devices. The software implementation can be based on a general-purpose processor executing a computer program stored in a storage medium.
Claims
1. A method for classifying the motions of intracavitary surgical instruments based on kinematic appearance fusion, characterized in that... It includes the following steps: S1. Acquire an intracavitary video frame sequence and instrument tip trajectory data, wherein the intracavitary video frame sequence includes at least 5 consecutive images arranged in chronological order, and the instrument tip trajectory data includes the three-dimensional coordinates of the instrument tip corresponding to the time of each frame; S2. Perform device instance segmentation on each frame of the intracavitary video frame sequence to obtain the device mask and device position. The device instance segmentation is performed based on an instance segmentation network. S3. Extract kinematic features based on the trajectory data of the instrument tip, the kinematic features including velocity, acceleration, and attitude angular velocity; S4. Based on the image region corresponding to the instrument mask, the appearance features are fused, including the instrument's RGB texture features and the response features of the tissue surrounding the instrument; S5. Perform multi-class confidence prediction based on the kinematic features and the appearance features to obtain the probability distribution of instrument action categories, which includes cutting probability, electrocoagulation probability, peeling probability, clamping probability, and traction probability; S6. Perform temporal smoothing on the probability distribution of the instrument action categories based on a Hidden Markov Model (HMM). The HMM includes a set of states and a state transition probability matrix. The set of states includes cutting state, electrocoagulation state, peeling state, clamping state, and traction state. The temporal smoothing is performed based on the Viterbi decoding algorithm to obtain the smoothed probability distribution of the instrument action categories and output the instrument action category corresponding to the maximum probability.
2. The method according to claim 1, characterized in that... The instance segmentation network is an instance segmentation network based on a convolutional neural network; the instrument mask is output in the form of a binary image; the instrument position is calculated based on the centroid coordinates of the instrument mask.
3. The method according to claim 1, characterized in that... The extraction of kinematic features in step S3 specifically includes: the velocity being equal to the difference between the three-dimensional coordinates of the instrument tip in adjacent frames divided by the inter-frame time interval; the acceleration being equal to the difference between the velocities in adjacent frames divided by the inter-frame time interval; and the attitude angular velocity being equal to the change in the instrument attitude angle between adjacent frames divided by the inter-frame time interval.
4. The method according to claim 1, characterized in that... The multi-class confidence prediction in step S5 is performed according to the following formula: P(a|F) = softmax(W·F + b), Where P(a|F) is the posterior probability distribution of the action category based on the fusion feature F, W is the classification weight matrix, and b is the bias term; the fusion feature F is obtained by concatenating the kinematic feature and the appearance feature.
5. The method according to claim 1, characterized in that... The state transition probability matrix of the hidden Markov model is obtained based on the prior statistics of state transitions in clinical surgical operations.
6. The method according to claim 1, characterized in that... The instrument action category is used to trigger graded risk warnings or for automatic recording of surgical operation logs; when the instrument action category is cutting, the graded risk warning level is the highest; when the instrument action category is traction, the graded risk warning level is the lowest.
7. A device for classifying the motions of intracavitary surgical instruments based on kinematic appearance fusion, characterized in that... ,include: The video and trajectory acquisition unit is configured to perform step S1 as described in claim 1. The device instance segmentation unit is configured to perform step S2 as described in claim 1. A kinematic feature extraction unit is used to perform step S3 as described in claim 1; An appearance feature fusion unit is used to perform step S4 as described in claim 1; A multi-class confidence prediction unit is configured to perform step S5 as described in claim 1. A timing smoothing output unit is used to perform step S6 as described in claim 1.
8. An electronic device, characterized in that... The electronic device includes a processor and a memory, the memory storing computer program instructions, wherein when the processor executes the computer program instructions, the electronic device performs the method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that... It stores computer program instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that... It includes computer program instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 6.