An apparatus detection system and method applied to digestive endoscope surgery video analysis

Abstract

One of the technical solutions of the present application is to provide an instrument detection system applied to digestive endoscope surgery video analysis, characterized in that it comprises: an endoscope video acquisition module; an instrument detection module; an instrument state discrimination module; a surgery process analysis module; and an analysis result display module. Another technical solution of the present application is to provide an instrument detection method applied to digestive endoscope surgery video analysis and realized based on the aforementioned instrument detection system. The present application utilizes deep learning and computer vision technology to dynamically identify the type and open / close state of an endoscope instrument. The system and method can be used to detect and identify instruments in real time during endoscopic surgery, and then to mark the surgery process in real time, and can also be used to retrospectively analyze historical videos, so that viewers can locate the operation process of different instruments through the system.

Description

TECHNICAL FIELD

[0001] The present application relates to an instrument detection system applied to digestive endoscopy surgery video analysis and an instrument detection method realized by the system. BACKGROUND

[0002] In the endoscopy surgery process, there are nearly ten kinds of commonly used endoscopic instruments, including biopsy forceps, hook knives, Dual knives, IT knives, snare, metal clips, nylon ropes, HIBO knives, APC, injection needles, etc. And these instruments often have multiple different states, including closing, opening and releasing, etc. In order to complete the analysis of the digestive endoscopy surgery video, different endoscopic instruments need to be identified and the state distinguished.

[0003] The existing digestive endoscopic instrument intraoperative identification method is mainly artificial identification, which relies completely on the physical strength, attention, memory, checking time and operation experience of the operator to identify the instrument type and judge the instrument state. And since the instruments are frequently replaced in a digestive endoscopy surgery, the recording process needs to consume a lot of manpower, and the judgment of most operators is not accurate enough, and it is even more impossible to complete accurate recording or marking work in the endoscopy report or surgery video. SUMMARY

[0004] The technical problem to be solved by the present application is that the existing digestive endoscopic instrument intraoperative identification method is artificial identification.

[0005] In order to solve the above technical problems, one technical scheme of the present application provides an instrument detection system applied to digestive endoscopy surgery video analysis, characterized in that it comprises:

[0006] An endoscopic video acquisition module: used to acquire a video stream from an intraoperative endoscopic host, disassemble the video stream into image frames, and input the image frames into an instrument detection module at a fixed frequency;

[0007] An instrument detection module: after reading the image frames sent from the endoscopic video acquisition module, the instrument detection module predicts whether the image frames contain endoscopic instruments: if the current image frame contains endoscopic instruments, the instrument detection module outputs the coordinates of the edge frame center of the instrument head of the endoscopic instruments on the current image frame and the edge frame; if the current image frame does not contain endoscopic instruments, the instrument detection module outputs an empty result;

[0008] An instrument state discrimination module: if the instrument detection module outputs a non-empty result, the instrument state discrimination module analyzes the instrument state of the endoscopic instruments according to the output result of the instrument detection module and outputs the recognized instrument state;

[0009] The surgical procedure analysis module is used for storing and analyzing the output results of the instrument detection module and the instrument state discrimination module, calculating the time of each endoscope instrument in the entire surgical video based on the recognition results of the instrument detection module and the instrument state discrimination module, and drawing a visualization result;

[0010] The analysis result display module is used for displaying the original image frames obtained by the endoscope video acquisition module, the edge frame obtained by the instrument detection module, the instrument state obtained by the instrument state discrimination module, and the visualization result obtained by the surgical procedure analysis module.

[0011] Preferably, the video stream includes a real-time video stream obtained from an intraoperative endoscope host or a retrospective video stream obtained based on a historical storage surgical video file obtained from the intraoperative endoscope host.

[0012] Preferably, the instrument detection module predicts whether an endoscope instrument exists in the image frame through a pre-trained neural convolutional neural network Yolo v5.

[0013] Preferably, when the instrument state discrimination module judges the instrument state, the edge frame and the coordinates of the edge frame center on the current image frame are sent from the instrument detection module, the image block of a fixed pixel size near the edge frame center is intercepted, the pre-trained neural convolutional network ResNet18 is inputted, the instrument state of the current endoscope instrument is discriminated, and the recognized instrument state is outputted.

[0014] Preferably, the instrument state discrimination module discriminates the instrument state of the current endoscope instrument through the instrument state to represent whether the recognized endoscope instrument is in contact with the human tissue.

[0015] Another technical scheme of the present application provides an instrument detection method applied to the analysis of a digestive endoscope surgical video based on the foregoing instrument detection system, characterized by comprising the following steps:

[0016] Step S1: obtaining a video stream through an endoscope video acquisition module, and decomposing the video stream into single image frames I raw , and inputting the single image frames I raw into the instrument detection module at a fixed frequency;

[0017] Step S2: the instrument detection module detects whether the input single image frame I raw contains an endoscope instrument: if the endoscope instrument is recognized, the image cut I ROI containing the entire edge frame is cropped with the edge frame center of the instrument head of the endoscope instrument as the center; if the endoscope instrument is not recognized, an empty result is outputted;

[0018] If there are p different types of endoscopic instruments, the instrument detection module uses x e {1, 2, …, p} to represent the serial number value of the type corresponding to the recognized endoscopic instrument; if the instrument detection module does not recognize the endoscopic instrument, x takes 0;

[0019] Step S3: If the output of step S2 is a non-empty result, the image patch I ROI The input instrument state discrimination module judges the instrument state of the endoscopic instrument, and the obtained instrument state is represented by y, y e {0, 1, 2, …, m}, wherein {1, 2, …, m} represents m kinds of recognized instrument states, and 0 represents that no endoscopic instrument is found;

[0020] Step S4: Based on the detection results in steps S2 and S3, the time series X 1:N of instrument recognition is calculated N , wherein x n e {0, 1, 2, …, p} is the instrument detection module recognition result of the nth single image frame I raw , n e {1, 2, …, N}; at the same time, the time series Y 1:N of instrument state is calculated N , wherein y n e {0, 1, 2, …, m} is the recognition result of the instrument state discrimination module of the nth single image frame I raw , n e {1, 2, …, N};

[0021] The surgical process analysis module calculates the appearance time and time occupancy of each instrument in the surgical video based on the calculated time series, and draws a visualization result;

[0022] Step S5: The analysis result display module displays: the original image frame obtained through the endoscopic video acquisition module, the edge box obtained through the instrument detection module, the instrument state obtained through the instrument state discrimination module, and the visualization result obtained through the surgical process analysis module.

[0023] Preferably, in step S2, the instrument detection module detects whether there is an endoscopic instrument in the input single image frame I raw through a deep neural network Yolov5.

[0024] Preferably, in step S2, when training the deep neural network Yolov5, the training set is composed of multiple video screenshots containing p different types of endoscopic instruments and multiple instrument-free video background screenshots, and the instrument head edge box in the training set picture is labeled by a doctor.

[0025] Preferably, in step S3, the instrument state discrimination module determines the instrument state of the endoscope instrument through the deep neural network ResNet.

[0026] Preferably, in step S4, the surgery process analysis module calculates the appearance time and time occupancy of each instrument in the surgery video based on the time sequence X 1:N ={x1,x2,…,x N} and draws a visualization result in the form of a heat map.

[0027] The present application utilizes deep learning and computer vision technology to dynamically identify the type and open / close state of the endoscope instrument. The system and method can be used to real-time detect and identify the instrument during endoscopic surgery, and then real-time label the surgery process, or retrospectively analyze historical videos, and the viewer can locate the operation process of different instruments through the system.

[0028] Compared with the prior art, the present application has the following advantages:

[0029] (1) The type and state of the instrument appearing in the endoscopic surgery can be detected and discriminated in real time;

[0030] (2) Whether an instrument appears in the operation field of view can be identified;

[0031] (3) The type of commonly used endoscopic instruments can be identified;

[0032] (4) Different states of the same endoscopic instrument can be identified. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a system function module diagram;

[0034] Figure 2 is a method flowchart;

[0035] Figure 3 is an example of an endoscopic surgery video image frame;

[0036] Figure 4A and Figure 4B is an example of image frame instrument identification result and slicing;

[0037] Figure 5 illustrates the Yolo v5 network structure;

[0038] Figure 6 illustrates the ResNet18 network structure;

[0039] Figure 7 is a schematic diagram of an analysis result display module. DETAILED DESCRIPTION

[0040] The application will be further described below in connection with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not used to limit the scope of the application. Furthermore, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content of the application, and these equivalent forms also fall within the scope defined by the appended claims.

[0041] As shown in the accompanying drawings, the apparatus detection system for video analysis of digestive endoscopy surgery disclosed by the application comprises the following modules: Figure 1

[0042] Endoscope video acquisition module: an image acquisition card is used to obtain real-time video stream from the HDMI interface of the endoscope host computer during surgery, or to read the historical storage of the surgical video file from the HDMI interface of the endoscope host computer during surgery, to obtain a retrospective video stream. For the convenience of discussion, the real-time video stream and the retrospective video stream are uniformly defined as video stream. The endoscope video acquisition module is also used to input the video stream into the apparatus detection module at a fixed frequency after the video stream is disassembled into image frames.

[0043] Apparatus detection module: after reading the image frames sent from the endoscope video acquisition module, the pre-trained neural convolutional neural network Yolo v5 is used to predict whether there is an endoscope apparatus in the image frames. If there is an endoscope apparatus in the current image frame, the coordinates of the edge frame center of the apparatus head of the endoscope apparatus on the current image frame and the edge frame are output; if there is no endoscope apparatus in the current image frame, an empty result is output.

[0044] Apparatus state discrimination module: if the apparatus detection module outputs a non-empty result, the apparatus state discrimination module further analyzes the apparatus state of the endoscope apparatus. When the apparatus state discrimination module judges the apparatus state, based on the edge frame and the coordinates of the edge frame center on the current image frame sent from the apparatus detection module, the image block of 224x224 pixels near the edge frame center is intercepted, and the image block is input into the pre-trained neural convolutional network ResNet18, and the apparatus state of the current endoscope apparatus is discriminated and the recognized apparatus state is output. In this embodiment, the apparatus state includes the on-off state of the current endoscope apparatus.

[0045] Surgical process analysis module: used to store and analyze the output results of the apparatus detection module and the apparatus state discrimination module, based on the recognition results of the apparatus detection module and the apparatus state discrimination module, the time of each endoscope apparatus in the entire surgical video is calculated, and the visualized results are drawn in the form of heat map (Heatmap).

[0046] ​Analysis Results Display Module: Used to display: the original image frames obtained by the endoscopic video acquisition module, the bounding boxes obtained by the instrument detection module, the instrument status obtained by the instrument status discrimination module, and the heat map obtained by the surgical process analysis module.

[0047] This invention also provides a device detection method based on the above system for video analysis of digestive endoscopy surgery, such as... Figure 2 As shown, the specific steps include:

[0048] Step S1: Obtain the video stream through the endoscope video acquisition module and decompose the video stream into single image frames I. raw Single image frame I raw The pixel size is H raw ×W raw Single image frame I raw Input the instrument detection module at a fixed frequency.

[0049] In this embodiment, a single image frame I raw The pixel size is 512×512, such as Figure 3 As shown, the data is input to the instrument detection module at a frequency of 25fps.

[0050] Step S2: The instrument detection module detects the input single image frame I using a pre-trained deep neural network Yolov5. raw If an endoscopic instrument is detected, output the coordinates (X,Y) of the center of the instrument tip's bounding box on the image, as well as the width dW and height dH of the bounding box. Then, crop an image block I of size 224×224 pixels centered at the bounding box center (X,Y). ROI If no endoscopic instrument is detected, an empty result will be output.

[0051] In this invention, there are p different types of endoscopic instruments. If the deep neural network YOLOv5 identifies an endoscopic instrument, then x∈{1,2,…,p} represents the index value of the type corresponding to the identified endoscopic instrument. In this embodiment, the endoscopic instruments include biopsy forceps, hook scalpel, Dual scalpel, IT scalpel, snare, metal clamp, nylon rope, Hyposurgical knife, APC, and injection needle, corresponding to x∈{1,2,…,10}, where x represents the instrument identification result of the current single image frame. If the deep neural network YOLOv5 does not identify an endoscopic instrument, then x is 0.

[0052] The parameters of the YOLOv5 deep neural network are obtained through pre-training on a training set. In this embodiment, the training set consists of 2000 video screenshots of each of the aforementioned 10 endoscopic instruments and 5000 video background screenshots without instruments. The physician annotates the bounding boxes of the instrument heads in the training set images. After training, for each real-time input single image frame I... raw The Yolov5 deep neural network can output the recognition result x and the corresponding bounding box prediction result, such as... Figure 4A and Figure 4B As shown. The network structure of the Yolov5 deep neural network is as follows. Figure 5 As shown. If the deep neural network Yolov5 identifies the endoscopic instrument, it crops an image block I of size 224×224 pixels centered on the edge box. ROI If the Yolov5 deep neural network fails to identify the endoscopic instrument, it outputs an empty result.

[0053] Step S3: If the output of step S2 is non-empty, then cut the image into blocks I. ROI The input instrument status discrimination module uses a deep neural network, ResNet, to determine the instrument status of endoscopic instruments. The instrument status obtained by the deep neural network ResNet is represented by y, where y∈{0,1,2,…,m}, where {1,2,…,m} represents the m types of instrument statuses identified, and 0 indicates that no endoscopic instruments were detected.

[0054] In this embodiment, the instrument status is an on / off state to indicate whether the endoscopic instrument is in contact with the tissue. Then y∈{0,1,2}, where 0 represents no endoscopic instrument detected, 1 represents no contact between the endoscopic instrument and the tissue, and 2 represents contact between the endoscopic instrument and the tissue.

[0055] The parameters of the deep neural network ResNet18 are obtained through pre-training on the training set samples. The network structure of ResNet18 is as follows: Figure 5 As shown.

[0056] Step S4: Based on the detection results in Steps S2 and S3, calculate the time series X for device identification. 1:N ={x1,x2,…,x N}, where x n ∈{0,1,2,…,p} represents the nth single image frame I. raw The instrument detection module identification result, n∈{1,2,…,N}: if no endoscopic instrument is found, x n Set the value to 0; if an endoscopic instrument is present, take the instrument's serial number, where {1,2,…,p} represent the serial numbers of p different types of endoscopic instruments. In this embodiment, p = 10. Simultaneously, calculate the time series Y of the instrument status. 1:N ={y1,y2,…,yN}, where y n ∈{0,1,2,…,m} represents the nth single image frame I. raw The identification result of the instrument status discrimination module, n∈{1,2,…,N}: if there is no endoscopic instrument, then y n Set y to 0; if endoscopic instruments are present, then y n The status sequence value of the corresponding endoscopic instrument is retrieved. In this embodiment, m = 2. Based on this time series, the surgical process analysis module calculates the appearance time and duration of each instrument in the surgical video and visualizes the results using a heatmap.

[0057] Step S5: Analysis Results Display Module displays: the original image frames obtained by the endoscopic video acquisition module, the bounding boxes obtained by the instrument detection module, the instrument status obtained by the instrument status discrimination module, and the heatmap obtained by the surgical process analysis module. For example... Figure 7 As shown, this specifically displays the current single image frame I. raw The identified device bounding box, the detected device type and on / off state, and the device appearance time X are all included. 1:N ={x1,x2,…,x N The heatmap visualization results.

[0058] In implementing this invention, real-time endoscopic surgical video streams or locally stored historical video streams are connected to the analysis system provided by this invention for automatic analysis of the surgical videos. Subsequently, based on the analysis results from the display module, the video quality is evaluated and analyzed, and the segment of interest in the surgical video is located using a heatmap.

Claims

1. An instrument detection system for video analysis of digestive endoscopic surgery, characterized in that, include: Endoscopic video acquisition module: After acquiring the video stream from the intraoperative endoscope host, the video stream is broken down into image frames and the image frames are input into the instrument detection module at a fixed frequency; Instrument detection module: After reading the image frame sent from the endoscope video acquisition module, it predicts whether there is an endoscopic instrument in the image frame: if there is an endoscopic instrument in the current image frame, it outputs the coordinates of the center of the edge box of the instrument head on the current image frame and the edge box; if there is no endoscopic instrument in the current image frame, it outputs an empty result. Instrument Status Determination Module: If the output of the instrument detection module is not empty, the instrument status determination module analyzes the instrument status of the endoscopic instrument based on the output of the instrument detection module and outputs the identified instrument status. When the instrument status discrimination module determines the instrument status, it uses the edge box sent from the instrument detection module and the coordinates of the edge box center on the current image frame. It uses the edge box center as the cropping center to extract an image block of a fixed pixel size nearby. After inputting the image block into the pre-trained neural convolutional network ResNet18, it determines the instrument status of the current endoscopic instrument and outputs the identified instrument status. The instrument status discrimination module uses the instrument status to indicate whether the identified endoscopic instrument is in contact with human tissue. Surgical process analysis module: Used to store and analyze the output results of the instrument detection module and the instrument status discrimination module. Based on the recognition results of the instrument detection module and the instrument status discrimination module, it calculates the time occupied by each endoscopic instrument in the entire surgical video and draws the visualization results. Analysis Results Display Module: This module displays the original image frames obtained through the endoscopic video acquisition module, the bounding boxes obtained through the instrument detection module, the instrument status obtained through the instrument status discrimination module, and the visualization results obtained through the surgical process analysis module.

2. The instrument detection system for video analysis of digestive endoscopic surgery as described in claim 1, characterized in that, The video stream includes a real-time video stream acquired from the intraoperative endoscope host, or a retrospective video stream acquired based on historically stored surgical video files acquired from the intraoperative endoscope host.

3. The instrument detection system for video analysis of digestive endoscopic surgery as described in claim 1, characterized in that, The instrument detection module uses a pre-trained YOLOv5 neural convolutional neural network to predict whether endoscopic instruments are present in an image frame.

4. A method for instrument detection applied to the video analysis of digestive endoscopic surgery based on the instrument detection system of claim 1, characterized in that, Includes the following steps: Step S1: Obtain the video stream through the endoscope video acquisition module and decompose the video stream into single image frames I. raw Single image frame I raw Input the instrument detection module at a fixed frequency; Step S2: The instrument detection module detects the input single image frame I. raw Does the image contain endoscopic instruments? If an endoscopic instrument is identified, then crop an image block I containing the entire bounding box of the instrument's head, centered on the edge of the bounding box. ROI If no endoscopic instrument is identified, an empty result will be output. Suppose there are p different types of endoscopic instruments. Then, the instrument detection module uses x∈{1,2,…,p} to represent the index value of the type corresponding to the detected endoscopic instrument. If the instrument detection module does not detect an endoscopic instrument, then x is 0. Step S3: If the output of step S2 is non-empty, then cut the image into blocks I. ROI The input instrument status discrimination module determines the instrument status of the endoscopic instruments. The obtained instrument status is represented by y, where y∈{0,1,2,…,m}, where {1,2,…,m} represents the m types of instrument statuses identified, and 0 indicates that no endoscopic instruments were found. Step S4: Based on the detection results in Steps S2 and S3, calculate the time series X for device identification. 1:N ={x1,x2,…,x N }, where x n ∈{0,1,2,…,p} represents the nth single image frame I. raw The instrument detection module identifies the results, n∈{1,2,…,N}; simultaneously, the time series Y of the instrument status is calculated. 1:N ={y1,y2,…,y N }, where y n ∈{0,1,2,…,m} represents the nth single image frame I. raw The recognition results of the instrument state discrimination module, n∈{1,2,…,N}; The surgical procedure analysis module calculates the appearance time and duration of each instrument in the surgical video based on the calculated time series, and then generates a visualization of the results. Step S5: Analysis Results Display Module Displays: Original image frames obtained through the endoscopic video acquisition module, bounding boxes obtained through the instrument detection module, instrument status obtained through the instrument status discrimination module, and visualization results obtained through the surgical process analysis module.

5. The instrument detection method for video analysis of digestive endoscopic surgery as described in claim 4, characterized in that, In step S2, the instrument detection module detects the input single image frame I using a deep neural network YOLOv5. raw Does it contain endoscopic instruments? 6. The instrument detection method for video analysis of digestive endoscopy surgery as described in claim 5, characterized in that, In step S2, when training the deep neural network YOLO v5, the training set consists of multiple video screenshots containing p different types of endoscopic instruments and multiple video background screenshots without instruments. The doctor annotates the edge boxes of the instrument heads in the training set images.

7. The instrument detection method for video analysis of digestive endoscopic surgery as described in claim 4, characterized in that, In step S3, the instrument status determination module determines the instrument status of the endoscopic instruments through a deep neural network ResNet.

8. The instrument detection method for video analysis of digestive endoscopy surgery as described in claim 4, characterized in that, In step S4, the surgical procedure analysis module is based on time series X. 1:N ={x1,x2,…,x N The appearance time and duration of each instrument in the surgical video were calculated, and the results were visualized as a heatmap.

Images