A method and system for detecting leakage from an incision in an eye surgery based on video data

Abstract

The application discloses an eye surgery incision leakage detection method and system based on video data, wherein the method steps comprise: extracting continuous video frame images from a fluorescein sodium staining video and segmenting a corneal region; screening video frames based on the set features of the corneal region; forming continuous video blocks by using the screened effective video frames; and judging whether incision leakage occurs and detecting the leakage position based on the video blocks. Through the accurate segmentation of the corneal region, the screening of the video frames, the selection and analysis of the continuous video blocks, and the application of the optical flow algorithm, the application can automatically judge whether there is a leakage liquid flow phenomenon in the video, and further deduce the source of the leakage liquid flow, i.e. the position of the leakage point. The technical scheme can effectively improve the accuracy and efficiency of the incision leakage detection after the eye surgery, reduce the error of manual inspection, and provide timely and accurate feedback for doctors.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, in particular to an eye surgery incision leakage detection method and system based on video data. BACKGROUND

[0002] In ophthalmic surgery, especially high-risk surgeries such as cataract surgery and vitrectomy, the healing of the surgical incision has an important influence on the postoperative recovery of the patient. Leakage of the surgical incision often leads to instability of intraocular pressure, which in turn affects visual recovery and even causes complications such as infection. Timely and accurate detection of whether there is leakage in the surgical incision is a crucial link in postoperative care of ophthalmic surgery. Therefore, carrying out effective leakage detection not only can improve the postoperative rehabilitation quality of patients, but also can reduce unnecessary medical risks.

[0003] Traditional leakage detection methods rely on the clinical experience of doctors, and usually judge whether there is leakage by observing the liquid outflow of the eye. For example, doctors will observe whether the liquid outflow through the incision by dropping physiological saline, pigment liquid, etc. in the surgical area. However, this manual observation method is limited by the experience of doctors and the limitations of real-time monitoring, and there is a certain risk of misdiagnosis and missed diagnosis, and it is impossible to accurately locate the specific position of the leakage.

[0004] With the development of medical imaging technology and computer vision technology, fluorescein sodium staining as a commonly used detection method after ophthalmic surgery has been gradually widely applied. Fluorescein sodium is a dye that can emit fluorescence under the excitation of a specific light source, and is usually used for eye examination. The dye can flow through the leakage of the eye surface and the incision area, enhancing the ophthalmologist's perception of surgical incision leakage. However, the existing fluorescein sodium staining detection method still relies on the naked eye observation of doctors, and cannot carry out large-scale, real-time and accurate automatic detection, and due to environmental lighting conditions, doctor subjective factors and other problems, there are certain challenges in accuracy and repeatability.

[0005] In order to improve the efficiency and accuracy of surgical incision leakage detection, some automatic detection methods based on video analysis have been gradually proposed. Through processing the video of fluorescein sodium staining, computer vision technology can analyze the dynamic changes in the image in real time, accurately identify the light-emitting area of fluorescence, and match it with the area of the surgical incision, and then judge whether there is leakage phenomenon. Although some researches try to use computer vision technology to analyze the fluorescence image, most of the existing technologies have some limitations, such as fluctuations in image quality, interference of background noise, and difficulties in tracking dynamic areas, etc., which limit the accuracy and reliability of leakage detection.

[0006] In addition, since the staining effect of sodium fluorescein depends on the flow state of the liquid, how to accurately extract the characteristics of the liquid flow, especially in the surgical incision area, becomes a major difficulty in automated detection. The dynamic changes in the shape, speed, and expansion range of the liquid flow require the use of advanced image processing techniques and pattern recognition algorithms for effective extraction and analysis.

[0007] Therefore, how to automatically detect whether there is leakage in the surgical incision based on the sodium fluorescein staining video through computer vision and artificial intelligence technology is still a problem to be solved. Achieving this goal not only improves the efficiency and accuracy of postoperative leakage detection in eye surgery, reduces human interference, but also provides timely and accurate feedback to doctors, helping them make better decisions, thereby improving the quality of patient recovery after surgery and reducing the occurrence of complications. SUMMARY

[0008] To solve the technical problems in the above background, the present application aims to use image processing, deep learning, and other technologies to automatically analyze and identify whether there is incision leakage in the video and locate the position of the incision leakage. This method can detect surgical incision leakage in real time, efficiently and accurately, and has important clinical application value.

[0009] To achieve the above-mentioned purpose, the present application provides an eye surgery incision leakage detection method based on video data, the steps of which include:

[0010] Extracting continuous video frame images from the sodium fluorescein staining video and segmenting the corneal region;

[0011] Screening video frames based on the collective characteristics of the corneal region;

[0012] Assembling the screened effective video frames into continuous video blocks;

[0013] Based on the video blocks, determining whether incision leakage occurs and detecting the leakage position.

[0014] Preferably, continuous video frames are extracted from the sodium fluorescein staining video, each frame representing an image at a time; then a pre-trained semantic segmentation neural network model Mask R-CNN is used to segment the corneal region of each frame image.

[0015] Preferably, the method of screening video frames includes calculating the radius and circularity coefficient of the corneal region in each frame image; removing video frame images whose corneal region radius or circularity coefficient is not within the set threshold range; wherein the calculation method of the circularity coefficient includes:

[0016] A = 4πS / C 2

[0017] Wherein A represents the circularity coefficient; S represents the area of the circle; C represents the circumference of the circle.

[0018] Preferably, for the qualified video frames, the video frames are selected in sequence according to time sequence frame by frame; each of the video blocks contains 10 continuous video frames; and the pre-processing of each selected video block includes: firstly, denoising and enhancing contrast of each frame image in the video block; extracting the green channel of each frame image, and then converting the green channel image into a gray image.

[0019] Preferably, the method for judging whether the incision leakage occurs includes: calculating the optical flow velocity field between adjacent two frames in the video block by using an optical flow algorithm, and converting the optical flow velocity and direction of each pixel point in the optical flow field into polar coordinate form; then, calculating the average value of the optical flow velocities of all adjacent frames in the video block, and when the average value exceeds a preset threshold value, it is determined that the video block has a liquid flow phenomenon and the incision leakage occurs.

[0020] Preferably, for the video block in which the liquid flow is detected, the source of the liquid flow is deduced by analyzing the optical flow velocity field, so as to deduce the position of the leakage point.

[0021] The application also provides an eye surgery incision leakage detection system based on video data, which is used to realize the above method and includes a segmentation module, a screening module, an assembly module and a detection module.

[0022] The segmentation module is used to extract continuous video frame images from a fluorescein sodium staining video and segment the corneal region.

[0023] The screening module is used to screen the video frames based on the collective features of the corneal region.

[0024] The assembly module is used to assemble the effective video frames screened out into continuous video blocks.

[0025] The detection module is used to judge whether the incision leakage occurs and detect the leakage position based on the video blocks.

[0026] Preferably, the working process of the segmentation module includes: extracting continuous video frames from a fluorescein sodium staining video, each frame representing an image at a time; and then segmenting the corneal region of each frame image by using a pre-trained semantic segmentation neural network model Mask R-CNN.

[0027] Compared with the prior art, the application has the following beneficial effects:

[0028] Through precise corneal segmentation, video frame screening, selection and analysis of consecutive video blocks, and the application of an optical flow algorithm, this technology can automatically determine whether leakage is present in a video and further infer the source of the leakage, i.e., the location of the leakage point. This technical solution can effectively improve the accuracy and efficiency of post-operative incision leakage detection, reduce errors caused by manual inspection, and provide timely and accurate feedback to doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention;

[0031] Figure 2 Schematic diagram of a semantic segmentation neural network model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Example 1

[0035] like Figure 1 FIG. 1 is a flow chart of the method of this embodiment, and the steps include:

[0036] S1. Extract continuous video frames from fluorescein sodium stained video and segment the corneal area.

[0037] Accurate segmentation of the corneal region is a prerequisite for leakage fluid flow detection. In this embodiment, a deep learning-based video frame two-dimensional image segmentation technique is used to automatically extract the corneal region in each frame. Continuous video frames are extracted from the fluorescein sodium staining video, with each frame representing an image at a certain time. Then a pre-trained semantic segmentation neural network model MaskR-CNN is used to segment the corneal region of each frame image. This pre-trained deep learning model can identify the edge of the cornea and accurately extract its region. Through this step, the corneal region can be accurately extracted under different shooting angles and image quality, providing high-quality regional information for subsequent leakage fluid flow detection. All video frames in the subsequent modules are analyzed based on the segmented corneal region images.

[0038] The structure of the above-mentioned semantic segmentation neural network model Mask R-CNN is shown in Figure 2 Mask R-CNN is a semantic segmentation neural network model improved based on Faster R-CNN, and its core structure includes a feature extraction network (such as ResNet-FPN), a region proposal network (RPN), a RoIAlign layer, and three parallel output branches (classification, regression, and mask prediction). The model eliminates the alignment error of the feature map and the original image through the RoIAlign layer, and uses a multi-task loss function for end-to-end training. Among them, the mask branch uses a full convolutional network structure to output independent pixel-level segmentation results for each candidate region, significantly improving the accuracy of instance segmentation.

[0039] S2. Screen the video frames based on the collective features of the corneal region.

[0040] After segmenting the corneal region, video frames that meet certain standards need to be selected. The main task of this module is to judge the effectiveness of the video frames based on the geometric features of the corneal region. The specific process is as follows: for the corneal region in each frame image, calculate its radius and circularity coefficient.

[0041] These indicators reflect the geometric shape of the cornea. If the radius or circularity coefficient of the corneal region in a frame is not within the set threshold range, it is considered that the quality of the frame is insufficient, which may be due to the deformation, blur or other factors of the corneal region in the video, resulting in ineffective subsequent analysis. If the corneal region of a frame does not meet the threshold condition, the frame is excluded and not further processed. The remaining valid video frames will be used as input for the subsequent modules. The formula for calculating the circularity coefficient is as follows:

[0042] A=4πS / C 2

[0043] Where A represents the circularity coefficient; S represents the area of the circle; C represents the circumference of the circle.

[0044] S3. The effective video frames screened out are grouped into continuous video blocks.

[0045] Among the remaining effective video frames, continuous video blocks are selected according to the time sequence. The selection of the video blocks is for further analysis of the seepage liquid flow. The specific process is as follows: from the remaining video frames, frames are selected in order according to the time sequence. Each video block contains 10 continuous video frames, which can ensure that the video block has sufficient time information for liquid flow analysis. If it is found during the selection process that a frame has a skip frame or a discontinuous timestamp, the video block is ignored. The purpose of this is to ensure the integrity of the video block and avoid false analysis due to frame loss or time inconsistency. The qualified video frames are organized into video blocks, each of which contains 10 continuous video images for subsequent seepage liquid flow analysis.

[0046] For each selected video block, pre-processing is performed, including denoising and contrast enhancement of each frame in the video block to improve the accuracy of subsequent image analysis. Since the liquid is usually more obvious in the green channel after sodium fluorescein staining, the green channel of each frame image is extracted, and then the green channel image is further converted into a grayscale image.

[0047] S4. Based on the video block, determine whether a cut seepage occurs and detect the seepage position.

[0048] Optical flow algorithm (Optical Flow) is used to analyze the motion between each frame of the video block. Optical flow can capture pixel-level motion between video frames. By analyzing the pixel changes between consecutive frames, the speed and direction of motion are calculated. In the case of liquid flow, the flow of seepage liquid is characterized by a large optical flow speed of some pixel points or regions in the image. Therefore, the optical flow velocity field can effectively reflect the position and speed of the liquid flow. Specifically, first, the OpenCV library function cv2.calcOpticalFlowFarneback is used to calculate the optical flow velocity field between two adjacent frames in the video block, and the general OpenCV library function cv2.cartToPolar is used to convert the optical flow speed and direction of each pixel point in the optical flow field into polar coordinate form. Then, the average value of the optical flow speed of all adjacent frames in the video block is calculated. If the average value exceeds a predetermined threshold, it is considered that the video block has liquid flow phenomenon and cut seepage occurs.

[0049] For the video block detected to have a cut leakage such as liquid flow, by analyzing the optical flow velocity field, the source of the liquid flow can be deduced, and thus the position of the leakage point can be inferred. The pixel point with the maximum velocity in the optical flow velocity field usually corresponds to the starting position of the liquid flow, i.e., the potential leakage point. The leakage point is usually located in the region with the most significant change in the optical flow velocity field, and thus the leakage point can be accurately located by calculating the local change of the optical flow. Specifically, for each frame image in the video block, the first 100 pixel points with the maximum velocity in the optical flow velocity field are counted, these pixel points form a contour, the convex hull of the contour is calculated by using the OpenCV library function cv2.convexHull, and then the minimum enclosing circle of the convex hull is calculated by using the OpenCV library function cv2.minEnclosingCircle. The center of the inscribed circle is the inferred cut leakage position.

[0050] Embodiment Two

[0051] The embodiment also provides an ocular surgery cut leakage detection system based on video data, which comprises a segmentation module, a screening module, an assembly module and a detection module. The segmentation module is used to extract continuous video frame images from a fluorescein sodium staining video and segment a corneal region. The screening module is used to screen video frames based on the collective features of the corneal region. The assembly module is used to assemble the effective video frames screened to form continuous video blocks. The detection module is used to judge whether a cut leakage occurs and detect the leakage position based on the video blocks.

[0052] The following will be combined with the embodiment to explain in detail how the present application solves the technical problems in actual work.

[0053] First, the segmentation module is used to extract continuous video frame images from a fluorescein sodium staining video and segment a corneal region.

[0054] The accurate segmentation of the corneal region is the premise of the leakage liquid flow detection. The embodiment adopts a video frame two-dimensional image segmentation technology based on deep learning to automatically extract the corneal region in each frame. Continuous video frames are extracted from a fluorescein sodium staining video, and each frame represents an image at a time. Then a pre-trained semantic segmentation neural network model MaskR-CNN is used to segment the corneal region of each frame image. The pre-trained deep learning model can identify the edge of the cornea and accurately extract the region thereof. Through this step, the corneal region can be accurately extracted under different shooting angles and image qualities, and high-quality regional information is provided for the subsequent leakage liquid flow detection. All video frames of the subsequent modules are analyzed based on the segmented images of the corneal region.

[0055] The structure of the above-mentioned semantic segmentation neural network model Mask R-CNN is as follows: Figure 2As shown, Mask R-CNN is a semantic segmentation neural network model improved based on Faster R-CNN, and its core structure includes a feature extraction network (such as ResNet-FPN), a region proposal network (RPN), a RoIAlign layer, and three parallel output branches (classification, regression, and mask prediction). The model eliminates the alignment error of the feature map and the original image through the RoIAlign layer, and uses a multi-task loss function for end-to-end training. Among them, the mask branch uses a full convolutional network structure to output independent pixel-level segmentation results for each candidate region, which significantly improves the accuracy of instance segmentation.

[0056] After that, the screening module screens the video frames based on the collective features of the corneal region.

[0057] After the corneal region is segmented, video frames that meet certain standards need to be selected. The main task of this module is to judge the effectiveness of the video frames according to the geometric features of the corneal region. The specific process is as follows: for the corneal region in each frame of image, calculate its radius and circularity coefficient.

[0058] These indicators reflect the geometric shape of the cornea. If the radius or circularity coefficient of the corneal region in a frame is not within the set threshold range, it is considered that the quality of the frame is insufficient, which may be due to the deformation, blur or other factors of the corneal region in the video, resulting in ineffective subsequent analysis. If the corneal region of a frame does not meet the threshold condition, the frame is excluded and not further processed. The remaining valid video frames will be used as input for the subsequent modules. The formula for calculating the circularity coefficient is as follows:

[0059] A=4πS / C 2

[0060] Where A represents the circularity coefficient; S represents the area of the circle; C represents the circumference of the circle.

[0061] The composition module will screen out valid video frames and form continuous video blocks.

[0062] In the remaining valid video frames, continuous video blocks are selected according to the time sequence. The selection of video blocks is to further analyze the flow of the leakage liquid. The specific process is as follows: from the remaining video frames, select them in order according to the time sequence. Each video block contains 10 consecutive video frames, which can ensure that the video block has enough time information for liquid flow analysis. If a frame is found to have a skip frame or a discontinuous timestamp during the selection process, the video block is ignored. The purpose of this is to ensure the integrity of the video block and avoid incorrect analysis due to frame loss or time inconsistency. The video frames that meet the conditions are organized into video blocks, each containing 10 consecutive video images, which are used for subsequent analysis of the flow of the leakage liquid.

[0063] For each selected video block, first, each frame image in the video block is denoised, enhanced contrast, etc. to improve the accuracy of subsequent image analysis. Since the liquid is usually more obvious in the green channel after sodium fluorescein staining, the green channel of each frame image is extracted, and then the green channel image is further converted into a grayscale image.

[0064] Finally, the detection module determines whether a cut leakage occurs and detects the leakage position based on the video block.

[0065] The optical flow algorithm (Optical Flow) is used to analyze the motion between each frame image in the video block. The optical flow method can capture the pixel-level motion between video frames, and by analyzing the pixel changes between consecutive frames, the speed and direction of motion are calculated. In the case of liquid flow, the flow of the leaking liquid is characterized by a larger optical flow speed in some pixel points or regions in the image, and therefore, the optical flow speed field can effectively reflect the position and speed of the liquid flow. Specifically, first, the OpenCV library function cv2.calcOpticalFlowFarneback is used to calculate the optical flow speed field between two adjacent frames in the video block, and the OpenCV library function cv2.cartToPolar is used to convert the optical flow speed and direction of each pixel point in the optical flow field into polar coordinate form. Then, the average value of the optical flow speed of all adjacent frames in the entire video block is calculated, and if the average value exceeds a predetermined threshold, it is considered that the video block has a liquid flow phenomenon.

[0066] For the video block where the cut leakage is detected, the source of the liquid flow can be deduced by analyzing the optical flow speed field, so as to infer the position of the leakage point. The pixel point with the largest speed in the optical flow speed field usually corresponds to the starting position of the liquid flow, i.e. the potential leakage point, and the leakage point is usually located in the region with the most significant change in the optical flow speed field, so the leakage point can be accurately located by calculating the local change of the optical flow. Specifically, for each frame image in the video block, the first 100 pixel points with the largest speed in the optical flow speed field are counted, these pixel points form a contour, and the convex hull of the contour is calculated using the OpenCV library function cv2.convexHull, and then the minimum enclosing circle of the convex hull is calculated using the OpenCV library function cv2.minEnclosingCircle. The center of the inscribed circle is the inferred cut leakage position.

[0067] The above-described embodiments are only descriptions of the preferred modes of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A method for detecting leakage of an ocular surgical incision based on video data, the method comprising the steps of: The application relates to a method for detecting corneal incision leakage, comprising the following steps: ​ extracting continuous video frame images from a fluorescein sodium staining video and segmenting corneal regions; extracting continuous video frames from the fluorescein sodium staining video, each frame representing an image at a time; then segmenting the corneal regions of each frame image by using a pre-trained semantic segmentation neural network model Mask R-CNN; screening video frames based on the collective features of the corneal regions; the method for screening video frames comprises the following steps: calculating the radius and circularity coefficient of the corneal region in each frame image; and eliminating video frame images whose corneal region radius or circularity coefficient is not within a set threshold range; wherein the method for calculating the circularity coefficient comprises the following steps: A = 4πS / C 2 wherein A represents the circularity coefficient; S represents the area of a circle; and C represents the circumference of a circle; composing continuous video blocks from the screened effective video frames; for video frames meeting the requirements, selecting the frames in sequence according to the time sequence; each video block comprises 10 continuous video frames; and pre-processing each selected video block, which comprises the following steps: firstly, denoising and enhancing the contrast of each frame image in the video block; extracting the green channel of each frame image, and then converting the green channel image into a gray-scale image; judging whether incision leakage occurs and detecting the leakage position based on the video block.

2. The method of claim 1, wherein, the method for judging whether incision leakage occurs comprises the following steps: calculating the optical flow velocity field between adjacent two frames in the video block by using an optical flow algorithm, and converting the optical flow velocity and direction of each pixel point in the optical flow field into polar coordinate form; then, calculating the average value of the optical flow velocities of all adjacent frames in the video block, and when the average value exceeds a preset threshold value, it is determined that the video block has a liquid flow phenomenon and incision leakage occurs.

3. The method of claim 2, wherein the video data is captured by a camera positioned to view the eye of the patient during the eye surgery. for the video block in which liquid flow is detected, the source of the liquid flow is analyzed by analyzing the optical flow velocity field, so as to deduce the position of the leakage point.

4. A system for ocular surgery incision leakage detection based on video data, the system being configured to implement the method of any one of claims 1 to 3, characterized in that, The application relates to a method for detecting corneal incision leakage, comprising the following steps: a segmentation module, a screening module, a composing module and a detection module; the segmentation module is used for extracting continuous video frame images from a fluorescein sodium staining video and segmenting corneal regions; the screening module is used for screening video frames based on the collective features of the corneal regions; the composing module is used for composing continuous video blocks from the screened effective video frames; the detection module is used for judging whether incision leakage occurs and detecting the leakage position based on the video block.

5. The video data-based ocular surgery incision leakage detection system of claim 4, wherein, the working process of the segmentation module comprises the following steps: extracting continuous video frames from a fluorescein sodium staining video, each frame representing an image at a time; and then segmenting the corneal regions of each frame image by using a pre-trained semantic segmentation neural network model Mask R-CNN.

Images