Multi-instrument visual tracking method for laparoscope-holding robot in field of view of laparoscope

Abstract

The invention provides a multi-task learning strategy-based multi-instrument visual tracking method in a laparoscope visual field, and the method comprises the steps: constructing a light-weight image segmentation network and a target regression network based on a light-weight LinkNet-18 neural network architecture, synchronously and rapidly extracting regions of all parts (a rod shaft and an end executor) of surgical instruments in a laparoscope image, and carrying out the recognition of all parts (a rod shaft and an end executor) of the surgical instruments in the laparoscope image; accurately positioning the joint position of the surgical instrument in the laparoscopic image in real time; on the basis of a multi-task learning strategy, an image segmentation network and a target regression network are coupled in an iterative interaction mode, a lightweight image segmentation and target regression task joint model is constructed, regions of all parts (a rod shaft and an end effector) of a surgical instrument in a laparoscopic image are synchronously and rapidly extracted, and meanwhile the joint position of the surgical instrument is accurately positioned. And calculating a visual tracking target point in the laparoscope image, and finally realizing real-time accurate tracking of multiple instruments in the field of view of the laparoscope.

Description

technical field [0001] The technical problem to be solved by the present invention is to rapidly and synchronously track multiple surgical instruments in the laparoscopic field of view based on the laparoscopic surgical video. The technology belongs to the field of laparoscopic surgery navigation and robotics. Background technique [0002] In laparoscopic surgery, the hand holding the mirror needs to hold and adjust the position of the laparoscope in time according to the requirements of the chief surgeon, so as to provide the surgeon with a stable and appropriate operative field image in time. However, holding the laparoscope for a long time may cause the hand holding the mirror. Fatigue, hand tremors, and distraction often lead to abnormally jittery images in the surgical field. In addition, it is difficult to cooperate between the mirror holder and the chief surgeon, and communication is inefficient, which ultimately greatly reduces the efficiency of the operation. Aimin...

AI Technical Summary

Problems solved by technology

Obviously, the single-instrument visual tracking method is limited to the application scenario where there is only one surgical instrument in the field of view of the laparoscope. However, surgical tasks such as organ tissue cutting and suturing in laparoscopic surgery usually require the surgeon to control two surgical instruments to complete, which is more complicated. The surgery requires a surgical assistant to control two surgical instruments

Images