A method for classifying actions of an endoluminal surgical instrument based on kinematic appearance fusion and related apparatus

Abstract

The application provides a lumen surgery instrument action classification method based on kinematic appearance fusion and related devices. The method comprises the following steps: S1. acquiring a lumen surgery video frame sequence and an instrument tip motion trajectory; S2. performing appearance feature extraction on the lumen surgery video frame sequence to obtain an appearance feature vector; S3. performing kinematic feature extraction on the instrument tip motion trajectory to obtain a kinematic feature vector; S4. performing feature-level fusion on the appearance feature vector and the kinematic feature vector to obtain a fusion feature vector; S5. performing classification prediction based on the fusion feature vector to obtain an action category probability distribution; and S6. performing time sequence smoothing on the action category probability distribution based on a hidden Markov model to output a final instrument action category and a confidence degree. The application also provides related devices corresponding to the method, and the related devices comprise a device, an electronic device, a computer readable storage medium and a computer program product.

Description

Technical Field

[0001] This application relates to the field of medical image processing, and in particular to a method and related device for classifying the motions of intracavitary surgical instruments based on kinematic appearance fusion. Background Technology

[0002] Accurate identification of instrument movement types (including cutting, electrocoagulation, dissection, clamping, and traction) during endovascular surgery is crucial for triggering graded risk warnings and automatically recording surgical operation logs. Identifying instrument movements in endovascular videos faces challenges such as insufficient single-modal information, easy confusion of similar movements, and inter-frame instability.

[0003] Existing methods for instrument motion recognition in intracavitary videos suffer from the following problems: First, existing methods primarily rely on a single modality (image or kinematic data) for instrument motion recognition, failing to incorporate dual-source fusion of kinematic and appearance features. Consequently, these methods are prone to misclassification when motion patterns are similar. Second, existing methods classify motions based on features from a single frame, neglecting to address inter-frame instability through temporal dimensions. Consequently, their recognition results are unstable in scenarios involving instrument jitter or transitional motions. Third, existing methods output discrete categories without providing a probability distribution, making them unsuitable for downstream confidence-based risk assessment mechanisms. Fourth, the temporal smoothing of existing methods relies on simple sliding window averaging, lacking temporal modeling based on state transition priors, thus limiting their temporal smoothing capabilities. Summary of the Invention

[0004] To address the above technical problems, this application provides a method and related device for classifying the motions of intracavitary surgical instruments based on kinematic appearance fusion.

[0005] Firstly, a method for classifying the movements of intracavitary surgical instruments based on kinematic appearance fusion is provided, comprising the following steps: S1. Acquire an intracavitary video frame sequence and instrument tip trajectory data, wherein the intracavitary video frame sequence includes at least 5 consecutive images arranged in chronological order, and the instrument tip trajectory data includes the three-dimensional coordinates of the instrument tip at the corresponding time. S2. Perform instrument instance segmentation on each frame in the intracavitary video frame sequence to obtain an instrument mask and instrument position. The instrument instance segmentation is performed based on an instance segmentation network, which includes an instance segmentation network based on a convolutional neural network. The instrument mask is output in the form of a binary image, and the instrument position is calculated based on the centroid coordinates of the instrument mask. S3. Extract kinematic features based on the trajectory data of the instrument tip. The kinematic features include velocity, acceleration, and attitude angular velocity. The velocity is calculated by dividing the difference between the three-dimensional coordinates of the instrument tip in adjacent frames by the inter-frame time interval. The acceleration is calculated by dividing the difference between the velocities in adjacent frames by the inter-frame time interval. The attitude angular velocity is calculated by the rate of change of the instrument attitude angle between adjacent frames. S4. Based on the image region corresponding to the instrument mask, fuse appearance features, including instrument RGB texture features and tissue response features around the instrument; S5. Perform multi-class confidence prediction based on the kinematic features and the appearance features to obtain the probability distribution of the device action category; S6. Perform temporal smoothing on the probability distribution of the device action categories based on the Hidden Markov Model, and output the device action category corresponding to the maximum probability.

[0006] In any embodiment of this application, the multi-class confidence prediction is performed based on the following formula: P(a|F) = softmax(W·F + b), Where P(a|F) is the posterior probability distribution of the action category based on the fusion feature F, which is obtained by splicing the kinematic feature and the appearance feature. The probability distribution of the instrument action category includes cutting probability, electrocoagulation probability, peeling probability, clamping probability, and traction probability, and the sum of the five probabilities is equal to 1.

[0007] In any embodiment of this application, the time-series smoothing is performed based on the Viterbi decoding algorithm of the Hidden Markov Model; the state set of the Hidden Markov Model includes cutting state, electrocoagulation state, peeling state, clamping state, and traction state; the state transition probability matrix is ​​obtained based on the state transition prior statistics of clinical surgical operations.

[0008] In a second aspect, an intracavitary surgical instrument motion classification device based on kinematic appearance fusion is provided, which is used to execute the method described in the first aspect, including: a video and trajectory acquisition unit, an instrument instance segmentation unit, a kinematic feature extraction unit, an appearance feature fusion unit, a multi-class confidence prediction unit, and a temporal smoothing output unit, which respectively execute steps S1 to S6 of the method described in the first aspect.

[0009] Thirdly, an electronic device is provided, including a processor and a storage unit, the storage unit being used to store computer program code containing computer instructions, wherein when the processor executes these instructions, the electronic device performs the method described in the first aspect.

[0010] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in the first aspect.

[0011] Fifthly, a computer program product is provided, the computer program product comprising a computer program or instructions that, when the computer program or instructions are run on a computer, cause the computer to perform the method described in the first aspect.

[0012] It should be understood that the above general descriptions and subsequent specific descriptions are for illustrative and explanatory purposes only and do not impose any limitations on this application.

[0013] In this application, compared with the prior art, the technical problem to be solved by this application is: to accurately classify the types of instrument actions (including cutting, electrocoagulation, peeling, clamping and traction) in intracavitary video images and instrument tip trajectory data, to handle the confusion caused by similar action patterns, to handle the instability between frames, and to output the category probability distribution to support the downstream risk assessment mechanism. The technical means employed in this application and their non-obvious aspects compared to existing technologies include: First, this application addresses the problem of insufficient information in a single modality based on a dual-source fusion mechanism of kinematic and appearance features, a design not seen in existing methods based solely on a single modality (image or kinematics); Second, this application introduces state transition prior execution temporal smoothing based on a Hidden Markov Model, a design not seen in existing methods based solely on sliding window averaging; Third, this application outputs a five-class probability distribution of action categories, a design not seen in existing methods that only output discrete categories; Fourth, this application jointly inputs kinematic features (velocity + acceleration + attitude angular velocity) and appearance features (RGB texture + tissue response) into the classifier, a design not seen in existing methods based solely on a single feature; Fifth, this application uses instrument action categories to trigger graded risk warnings and automatic recording of surgical operation logs, a design not seen in existing methods that only output action categories but have no downstream applications. The related devices described in this application include devices, electronic devices, computer-readable storage media, and computer program products. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be explained below.

[0015] Figure 1 This is a flowchart illustrating a method for classifying the motions of intracavitary surgical instruments based on kinematic appearance fusion, as provided in an embodiment of this application.

[0016] Figure 2This is a schematic diagram of the structure of an intracavitary surgical instrument motion classification device based on kinematic appearance fusion, provided in an embodiment of this application.

[0017] Figure 3 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To enable those skilled in the art to more fully understand the technical solution of this application, the technical solution of this application will be explained in detail and clearly with reference to the accompanying drawings.

[0019] In the specification, claims, and related drawings of this application, the terms "first," "second," etc., are used only to distinguish different elements and do not imply any particular order. Meanwhile, the use of "comprising" and "having," and variations thereof, indicates non-exclusive inclusion.

[0020] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for classifying the motions of intracavitary surgical instruments based on kinematic appearance fusion, as provided in an embodiment of this application. The method includes steps S1 to S6.

[0021] S1. Video and Instrument Trajectory Acquisition: Intracavitary video frame sequences are acquired through an intracavitary imaging device, and instrument tip trajectory data at corresponding moments are acquired through a surgical instrument tracking system.

[0022] In this embodiment, the total number of frames in the intracavitary video frame sequence ranges from 5 to 30 frames. The inter-frame time interval ranges from 33 milliseconds to 100 milliseconds. The instrument tip trajectory data is aligned to the corresponding frame in the video frame sequence at each frame.

[0023] S2. Instrument instance segmentation: Perform instrument instance segmentation on each frame in the intracavitary video frame sequence to obtain the instrument mask and instrument position.

[0024] The device instance segmentation is performed based on an instance segmentation network. This instance segmentation network includes, but is not limited to, Mask R-CNN, YOLACT, SOLO, or other networks with instance segmentation capabilities. This application does not rely on a specific network architecture.

[0025] In this embodiment, the instrument mask is output as a binary image, and the instrument position is calculated based on the centroid coordinates of the instrument mask. This instrument instance segmentation can distinguish multiple surgical instruments existing simultaneously in a single frame image.

[0026] S3. Kinematic feature extraction: Extract kinematic features based on the trajectory data of the instrument tip.

[0027] In this embodiment, the kinematic features include velocity, acceleration, and attitude angular velocity. The velocity is calculated by dividing the difference in the coordinates of the instrument tip in adjacent frames by the inter-frame time interval. The acceleration is calculated by dividing the difference in velocity in adjacent frames by the inter-frame time interval. The attitude angular velocity is calculated based on the inter-frame rate of change of the instrument's attitude angle.

[0028] The introduction of kinematic features addresses the clinical prior that different instruments in endoscopic surgery have different kinematic patterns. The action recognition based on kinematic features is not a routine step in general video analysis that is based solely on image information.

[0029] S4. Appearance feature fusion: Fuse appearance features based on the image region corresponding to the instrument mask.

[0030] The appearance features include the RGB texture features of the instrument and the response features of the tissue surrounding the instrument.

[0031] In this embodiment, the RGB texture features of the instrument are calculated based on the RGB channel statistics of the instrument mask region, and the tissue response features surrounding the instrument are calculated based on the color change and deformation features of the outer region of the instrument mask. The appearance features and the kinematic features are concatenated to form a fused feature vector.

[0032] S5. Multi-class confidence prediction: Perform multi-class confidence prediction based on the kinematic features and the appearance features to obtain the probability distribution of the device action category.

[0033] In this embodiment, the multi-class confidence prediction is performed based on the following formula: P(a|F) = softmax(W·F + b) Where P(a|F) is the posterior probability distribution of the action category based on the fusion feature F, W is the classification weight matrix, and b is the bias term. The fusion feature F is obtained by concatenating the kinematic feature and the appearance feature. The probability distribution of the instrument action category includes cutting probability, electrocoagulation probability, peeling probability, clamping probability, and traction probability, and the sum of the five probabilities is equal to 1.

[0034] S6. Temporal Smoothing and Output: Based on the Hidden Markov Model, perform temporal smoothing on the probability distribution of the device action categories and output the device action category corresponding to the maximum probability.

[0035] In this embodiment, the temporal smoothing is performed using a Viterbi decoding algorithm based on a Hidden Markov Model (HMM). The state set of the HMM includes cutting, electrocoagulation, dissection, clamping, and traction states. The state transition probability matrix is ​​obtained based on prior statistics of state transitions during clinical surgical operations. The emission probability is obtained based on the probability distribution of instrument action categories output in step S5.

[0036] The temporal smoothing mechanism based on Hidden Markov Models differs from the conventional steps based on sliding window averaging. The temporal modeling based on state transition priors ensures the stability of the action classification in action transition scenarios, and the temporal modeling based on Hidden Markov Models originates from the state transition patterns of clinical surgical operations.

[0037] Clinical Application Example 1: In laparoscopic total mesorectal resection for rectal cancer, the method detects the movement category of surgical instruments in real time. When the time-smoothed instrument movement category is cutting and the distance between the instrument tip coordinates and key anatomical structures is less than 5 mm, a level 3 risk warning is triggered; simultaneously, the instrument movement category and timestamp are automatically written into the surgical operation log.

[0038] Clinical Application Example 2: During pancreaticoduodenectomy, the method detects the instrument movement category in real time during the bile duct dissection phase. When the time-smoothed instrument movement category is dissection and the duration is greater than 5 seconds, the method automatically records the dissection movement and its duration into the surgical operation log for postoperative surgical quality assessment.

[0039] Please see Figure 2 , Figure 2 This is a schematic diagram of a endoscopic surgical instrument motion classification device based on kinematic appearance fusion, provided in an embodiment of this application. The instrument motion classification device 1 includes: The video and trajectory acquisition unit 11 is used to acquire intracavitary video frame sequences and instrument tip trajectory data.

[0040] The instrument instance segmentation unit 12 is used to perform instrument instance segmentation on each frame in the intracavitary video frame sequence.

[0041] The kinematic feature extraction unit 13 is used to extract kinematic features based on the trajectory data of the instrument tip.

[0042] The appearance feature fusion unit 14 is used to fuse appearance features based on the image region corresponding to the instrument mask.

[0043] The multi-class confidence prediction unit 15 is used to perform multi-class confidence prediction based on the kinematic features and the appearance features.

[0044] The temporal smoothing output unit 16 is used to perform temporal smoothing on the probability distribution of the device action category based on the hidden Markov model and output it.

[0045] Please see Figure 3 , Figure 3This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of this application. The electronic device 2 includes a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21, the memory 22, the input device 23, and the output device 24 are communicatively connected via a bus. The memory 22 stores computer program code, which contains computer instructions. When the processor 21 executes the instructions, the electronic device 2 executes the kinematic appearance fusion-based intracavitary surgical instrument motion classification method.

[0046] The various embodiments in the application have their own focus. For ease of description and simplification, the same or similar content may not be described in detail in different embodiments. For details, please refer to the relevant descriptions in the other embodiments mentioned above.

[0047] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The following describes in detail another electronic device provided by the embodiments of this application, taking electronic device 2 as an example.

[0048] The method described in this application can be implemented based on hardware, software, or a combination of hardware and software. The hardware implementation can be based on application-specific integrated circuits, field-programmable gate arrays, or other programmable logic devices. The software implementation can be based on a general-purpose processor executing a computer program stored in a storage medium.

Claims

1. A method for classifying an action of an endoluminal surgical instrument based on kinematic appearance fusion, characterized by, Includes the following steps: S1. Acquire intracavitary video frame sequence and instrument tip trajectory data, wherein the intracavitary video frame sequence includes at least 5 consecutive images arranged in chronological order, and the instrument tip trajectory data includes the three-dimensional coordinates of the instrument tip corresponding to the time of each frame image. S2. Perform instrument instance segmentation on each frame of the intracavitary video frame sequence to obtain an instrument mask and instrument position. The instrument instance segmentation is performed based on an instance segmentation network, which includes an instance segmentation network based on a convolutional neural network. The instrument mask is output in the form of a binary image, and the instrument position is calculated based on the centroid coordinates of the instrument mask. S3. Extract kinematic features based on the trajectory data of the instrument tip. The kinematic features include velocity, acceleration, and attitude angular velocity. The velocity is calculated by dividing the difference between the three-dimensional coordinates of the instrument tip in adjacent frames by the inter-frame time interval. The acceleration is calculated by dividing the difference between the velocities in adjacent frames by the inter-frame time interval. The attitude angular velocity is calculated by the rate of change of the instrument attitude angle between adjacent frames. S4. Based on the image region corresponding to the instrument mask, fuse appearance features, including instrument RGB texture features and tissue response features around the instrument; S5. Perform multi-class confidence prediction based on the kinematic features and the appearance features to obtain the probability distribution of the instrument action category. The probability distribution of the instrument action category includes cutting probability, electrocoagulation probability, peeling probability, clamping probability and traction probability, and the sum of the cutting probability, electrocoagulation probability, peeling probability, clamping probability and traction probability is 1. S6. Perform temporal smoothing on the probability distribution of the instrument action categories based on a Hidden Markov Model (HMM). The HMM includes a set of states and a state transition probability matrix. The set of states includes cutting state, electrocoagulation state, peeling state, clamping state, and traction state. The temporal smoothing is performed based on the Viterbi decoding algorithm to obtain the smoothed probability distribution of the instrument action categories. The instrument action category corresponding to the maximum probability value in the smoothed probability distribution is then output.

2. The method of claim 1, wherein, The multi-class confidence prediction is performed based on the following formula: P(a|F) = softmax(W·F + b), Where P(a|F) is the posterior probability distribution of the action category based on the fusion feature F, W is the preset classification weight matrix, and b is the preset bias term; the fusion feature F is obtained by concatenating the kinematic feature and the appearance feature.

3. The method of claim 1, wherein, The state transition probability matrix of the hidden Markov model is obtained based on the prior statistics of state transitions in clinical surgical procedures.

4. The method according to claim 1, characterized in that, The instrument action category is used to trigger graded risk warnings or for automatic recording of surgical operation logs; when the instrument action category is cutting, the graded risk warning level is the highest level; when the instrument action category is traction, the graded risk warning level is the lowest level.

5. A kinematic appearance fusion-based intracavitary surgical instrument motion classification device, used to perform the intracavitary surgical instrument motion classification method as described in claim 1, characterized in that, include: The video and trajectory acquisition unit is used to perform step S1 as described above; The instrument instance segmentation unit is used to perform step S2 as described above; A kinematic feature extraction unit is used to perform step S3 as described above; An appearance feature fusion unit is used to perform step S4 as described above; A multi-class confidence prediction unit is used to perform step S5 as described above; The timing smoothing output unit is used to perform step S6 as described above.

6. An electronic device, characterized in that, include: A processor and a storage unit, the storage unit being used to store computer program code, the computer program code containing computer instructions, wherein when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program comprising program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 4.

8. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 4.

Images