A method, apparatus and device for quantitative assessment of colonic perfusion in colorectal surgery

Abstract

Embodiments of the present application disclose a method, device and equipment for quantitative evaluation of colon perfusion in colorectal surgery. The method comprises: acquiring an initial video frame sequence of a target colorectal region; performing video de-jittering on the initial video frame sequence, and aligning the remaining frames of the initial video frame sequence with the first frame to obtain a target video frame sequence; performing ICG fluorescent color extraction on a preset region of interest in each frame image of the target video frame sequence through HSV color space conversion in frame order to obtain observation data; fitting a target ICG curve according to the observation data, and calculating a fluorescent intensity factor and a perfusion time factor according to the target ICG curve; and inputting the fluorescent intensity factor, the perfusion time factor and curve parameters of the target ICG curve into a trained XGBoost machine learning model to predict a score result of the target colorectal region. Thus, the burden on doctors is reduced, and the efficiency and safety of surgery are improved.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the medical technical field, and in particular to a quantitative evaluation method, device and equipment for colon perfusion in colorectal surgery. BACKGROUND

[0002] Colorectal cancer is one of the most common gastrointestinal cancers in the world, and the incidence rate is increasing year by year. With the development of surgical techniques and chemotherapy regimens, the treatment effect of cancer has been improved. However, despite the improvement in surgical techniques and postoperative care, the incidence of anastomotic complications is still as high as 10-20%. It is known that about 10% of the population has poor development of collateral circulation branches around the splenic flexure of the colon, and these differences in vascular anatomy can lead to colon ischemia in some patients after colorectal surgery. Acute ischemia at the anastomotic site can lead to anastomotic leakage or colon necrosis, and chronic ischemia can cause anastomotic stenosis, which requires reoperation. The most commonly used method to evaluate colon perfusion is for surgeons to observe the color change or pulsation of small blood vessels in the colon wall with their own eyes. Sometimes, it is not easy to accurately detect the small changes in the microcirculation of the colon wall through visual observation, especially when the blood vessels are buried in adipose tissue due to visceral obesity, and even experienced surgeons will have a significantly reduced accuracy in evaluating colon perfusion.

[0003] The recently developed fluorescence camera system using indocyanine green (ICG) can be used to easily observe the perfusion state of the colon during robotic or laparoscopic surgery. However, there are some problems in predicting the existence of colon ischemia by only qualitatively evaluating the expression of ICG fluorescence. Even if ICG fluorescence appears in the rectum, colon ischemia can occur if the blood flow rate is lower than the physiological requirement, so a reliable quantitative analysis is needed to predict anastomotic complications. The current quantitative analysis method for ICG fluorescence mainly includes: first, fix the camera at a certain distance from the sample according to the focal length of the camera, then intravenously inject ICG, and completely turn off the lights in the operating room to exclude external light. Then the laparoscope camera emits near-infrared light, causing the ICG in the blood vessels to emit fluorescence, and the fluorescence image is output to the screen, then record the fluorescence video of the colorectum, and use video analysis and modeling tools to manually measure the changes in fluorescence intensity one by one to generate a colon perfusion map. In order to quantitatively evaluate the colon perfusion, manually calculate the fluorescence intensity factor and the perfusion time factor from each ICG fluorescence map, and then compare these factors with the normal colorectal factors through medical experience and artificially, so as to judge the perfusion state of the colorectum.

[0004] It can be seen that the existing method for quantitatively evaluating ICG fluorescence expression to judge the colon perfusion condition is complicated and harsh in implementation conditions, which increases the burden of doctors. Manual measurement and calculation are time-consuming and complex, which delays the progress of surgery and causes certain risks to the safety of patients. In addition, the final evaluation result is difficult to guarantee in accuracy, and doctors need to obtain enough experience through years of practice. The diversity of collateral circulation blood vessels and blood flow paths also affects the factors such as fluorescence intensity, camera exposure time and shooting distance. Different ICG curve modes of each patient will prompt different perfusion states. SUMMARY

[0005] The embodiments of the present application provide a quantitative evaluation method, device and equipment for colon perfusion in colorectal surgery, so as to realize intelligent quantitative evaluation of colon perfusion, reduce the burden of doctors and improve the efficiency and safety of surgery.

[0006] In the first aspect, the embodiments of the present application provide a quantitative evaluation method for colon perfusion in colorectal surgery, which comprises:

[0007] obtaining an initial video frame sequence of a target colorectal region;

[0008] performing video deblurring on the initial video frame sequence, and aligning the remaining frames of the initial video frame sequence except the first frame with the first frame to obtain a target video frame sequence;

[0009] extracting ICG fluorescence color in a preset region of interest in each frame image of the target video frame sequence by HSV color space conversion in frame order to obtain observation data of ICG fluorescence intensity changing with time;

[0010] fitting a target ICG curve according to the observation data, and calculating a fluorescence intensity factor and a perfusion time factor according to the target ICG curve;

[0011] inputting the fluorescence intensity factor, the perfusion time factor and the curve parameters of the target ICG curve into a trained XGBoost machine learning model to predict a score result of the target colorectal region.

[0012] Optionally, the video deblurring on the initial video frame sequence and the alignment of the remaining frames of the initial video frame sequence except the first frame with the first frame to obtain the target video frame sequence comprises:

[0013] perform video deblurring on the initial video frame sequence using a Lucas-Kanade optical flow algorithm, and align the remaining frames of the initial video frame sequence other than the first frame to the first frame using an enhanced correlation coefficient maximization algorithm.

[0014] Optionally, the performing video deblurring on the initial video frame sequence using a Lucas-Kanade optical flow algorithm comprises:

[0015] traversing the initial video frame sequence, collecting feature points in each frame of image using a Lucas-Kanade optical flow algorithm, and calculating a rigid transformation between each two consecutive frames of image according to the feature points;

[0016] calculating original trajectories of each frame of image according to the rigid transformation, and smoothing the original trajectories using a moving average filter to obtain smoothed trajectories of each frame of image;

[0017] calculating a trajectory difference between the smoothed trajectories and the original trajectories, and calculating a smoothed transformation between each two consecutive frames of image according to the trajectory difference and the rigid transformation;

[0018] applying the smoothed transformation to the corresponding each frame of image.

[0019] Optionally, the aligning the remaining frames of the initial video frame sequence other than the first frame to the first frame using an enhanced correlation coefficient maximization algorithm comprises:

[0020] converting each frame of image of the initial video frame sequence into a gray-scale image;

[0021] for each of the remaining frames, estimating a warping matrix of a preset motion model between the remaining frame and the first frame using an enhanced correlation coefficient maximization algorithm, and aligning using the warping matrix.

[0022] Optionally, the extracting ICG fluorescence color in a preset region of interest in each frame of image of the target video frame sequence according to frame sequence through HSV color space conversion to obtain observation data of ICG fluorescence intensity changing with time comprises:

[0023] taking average ICG brightness in the preset region of interest as the observation data.

[0024] Optionally, the fitting a target ICG curve according to the observation data comprises:

[0025] fitting the target ICG curve using a non-linear least squares method improved by a RANSAC algorithm according to a preset ICG kinetic model and the observation data.

[0026] Optionally, the fitting the target ICG curve according to the preset ICG kinetic model and the observation data uses a nonlinear least squares method improved by a RANSAC algorithm, and the fitting includes:

[0027] Randomly selecting a preset number of points in the observation data as inliers each time;

[0028] For the inliers obtained each time, fitting a candidate ICG curve according to the inliers using the nonlinear least squares method, then inputting the observation data into the candidate ICG curve, and calculating the number of inliers in the observation data;

[0029] Selecting the candidate ICG curve with the largest number of inliers as the target ICG curve.

[0030] In a second aspect, the embodiments of the present application further provide a device for quantitative evaluation of colon perfusion in colorectal surgery, and the device includes:

[0031] An initial video frame sequence acquisition module is configured to acquire an initial video frame sequence of a target colorectal region;

[0032] A target video frame sequence acquisition module is configured to perform video de-jittering on the initial video frame sequence, align the remaining frames of the initial video frame sequence except for a first frame with the first frame, and obtain a target video frame sequence;

[0033] An observation data extraction module is configured to perform ICG fluorescence color extraction on a preset region of interest in each frame image of the target video frame sequence in a frame order through HSV color space conversion, and obtain observation data of ICG fluorescence intensity changing over time;

[0034] An ICG curve fitting module is configured to fit a target ICG curve according to the observation data, and calculate a fluorescence intensity factor and a perfusion time factor according to the target ICG curve;

[0035] A score result prediction module is configured to input the fluorescence intensity factor, the perfusion time factor, and curve parameters of the target ICG curve into a trained XGBoost machine learning model, so as to predict a score result of the target colorectal region.

[0036] In a third aspect, the embodiments of the present application further provide a computer device, and the computer device includes:

[0037] One or more processors;

[0038] A memory configured to store one or more programs;

[0039] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for quantitative evaluation of colon perfusion in colorectal surgery provided by any of the embodiments of the present application.

[0040] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the method for quantitative evaluation of colon perfusion in colorectal surgery provided by any of the embodiments of the present application.

[0041] The embodiments of the present application provide a method for quantitative evaluation of colon perfusion in colorectal surgery, which first acquires an initial video frame sequence of a target colorectal region, then performs video de-jittering on the obtained initial video frame sequence, and aligns the remaining frames of the initial video frame sequence except the first frame with the first frame to obtain a target video frame sequence, then extracts ICG fluorescent color in a preset region of interest in each frame image of the target video frame sequence in frame order through HSV color space transformation to obtain observation data of ICG fluorescent intensity changing with time, then fits a target ICG curve according to the observation data, and calculates a fluorescent intensity factor and a perfusion time factor according to the target ICG curve, and finally inputs the fluorescent intensity factor, the perfusion time factor and the curve parameters of the target ICG curve into a trained XGBoost machine learning model to predict a score result of the target colorectal region. The method for quantitative evaluation of colon perfusion in colorectal surgery provided by the embodiments of the present application reduces the influence of camera jitter by using video de-jittering and picture alignment, and removes the influence of factors such as background and light source reflection by using HSV color space transformation, thereby reducing the shooting requirements of the fluorescent video, avoiding complex operations and environmental conditions, and at the same time, realizing intelligent quantitative evaluation of colon perfusion through automatic curve fitting and automatic calculation of factors and scores, thereby reducing the burden of doctors and improving the efficiency and safety of surgery. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 A flowchart of the method for quantitative evaluation of colon perfusion in colorectal surgery provided by the first embodiment of the present application;

[0043] Figure 2 A structural schematic diagram of the device for quantitative evaluation of colon perfusion in colorectal surgery provided by the second embodiment of the present application;

[0044] Figure 3 A structural schematic diagram of the computer device provided by the third embodiment of the present application. DETAILED DESCRIPTION

[0045] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0046] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0047] Example 1

[0048] Figure 1 This is a flowchart illustrating a method for quantitatively assessing colonic perfusion during colorectal surgery according to Embodiment 1 of the present invention. This embodiment is applicable to situations where colonic perfusion is quantitatively assessed based on ICG fluorescence imaging during colorectal surgery. This method can be executed by the quantitative assessment device for colonic perfusion during colorectal surgery provided in this embodiment of the invention. This device can be implemented in hardware and / or software, and is generally integrated into a computer device. Figure 1 As shown, the specific steps include the following:

[0049] S11. Obtain the initial video frame sequence of the target colorectal region.

[0050] Specifically, in colorectal surgery, surgeons no longer need to fix the laparoscopic camera at a certain distance from the colorectal region, nor do they need to turn off the operating room lights. They simply need to hold the laparoscopic camera and aim it at the target colorectal area (i.e., the affected area) to capture the required initial video frame sequence. Of course, surgeons can also fix the laparoscopic camera in place for easier surgical manipulation.

[0051] S12. Perform video decluttering on the initial video frame sequence, and register and align the remaining frames of the initial video frame sequence (excluding the first frame) with the first frame to obtain the target video frame sequence.

[0052] Specifically, in extracting the ICG curve, since camera shake can cause distortion, and ICG curve distortion is the main cause of misjudgment of ICG curve microcirculation perfusion analysis, therefore, the initial video frame can be first subjected to video shake reduction and picture alignment to reduce the influence of camera shake. Video shake reduction refers to a series of methods used to reduce the influence of camera motion on the final video. The motion of the camera is translation (i.e. motion in the x, y, z directions) or rotation (yaw, pitch, roll), and the video shake reduction method includes mechanical, optical and digital stabilization methods, etc. After completing the video shake reduction, the remaining frames are aligned with the first frame, i.e. the target video frame sequence required for subsequent ICG curve extraction is obtained.

[0053] Optionally, the video shake reduction of the initial video frame sequence and the alignment of the remaining frames of the initial video frame sequence except the first frame with the first frame to obtain the target video frame sequence comprises: using the Lucas-Kanade optical flow algorithm to perform video shake reduction on the initial video frame sequence, and using the enhanced correlation coefficient maximization algorithm to align the remaining frames of the initial video frame sequence except the first frame with the first frame.

[0054] Further optionally, the video shake reduction of the initial video frame sequence using the Lucas-Kanade optical flow algorithm comprises: traversing the initial video frame sequence, using the Lucas-Kanade optical flow algorithm to collect feature points in each frame of image, and calculating the rigid transformation between each two consecutive frames of image according to the feature points; calculating the original trajectory of each frame of image according to the rigid transformation, and smoothing the original trajectory using a moving average line filter to obtain the smoothed trajectory of each frame of image; calculating the trajectory difference between the smoothed trajectory and the original trajectory, and calculating the smoothed transformation between each two consecutive frames of image according to the trajectory difference and the rigid transformation; applying the smoothed transformation to the corresponding each frame of image.

[0055] Specifically, the embodiment can use a digital stabilization method, which does not need special sensors to estimate the motion of the camera. Specifically, the Lucas-Kanade optical flow algorithm can be used to realize video stabilization. The method mainly has three steps: motion estimation, motion smoothing and image synthesis. The first stage derives the transformation parameters between two consecutive frames. The second stage filters out unwanted motion. Finally, the stable video is reconstructed. After obtaining the initial video frame sequence, the feature points in each frame image can be tracked first, and the rigid (Euclidean) transformation from the previous frame to the current frame in each two consecutive frames can be calculated. Then, according to the obtained rigid transformation, the original trajectory of each frame image can be calculated by adding the motion between the frames, and the moving average filter can be used to smooth the obtained original trajectory to obtain the smoothed trajectory (such as x, y and angle) of each frame image. Then, the trajectory difference between the smoothed trajectory and the original trajectory can be calculated, and the trajectory difference is added back to the original corresponding rigid transformation to obtain the smoothed transformation between each two consecutive frame images. By applying the smoothed transformation to each corresponding frame image, video stabilization can be realized. After that, the border artifact problem can be alleviated by scaling the video around the center by a small amount (such as 5%).

[0056] Further, the use of the enhanced correlation coefficient maximization algorithm to align the remaining frames of the initial video frame sequence except the first frame with the first frame comprises: converting each frame image of the initial video frame sequence into a grayscale image; for each of the remaining frames, using the enhanced correlation coefficient maximization algorithm to estimate the distortion matrix of the preset motion model between the remaining frame and the first frame, and using the distortion matrix to align.

[0057] Specifically, in a typical image alignment problem, we have two images of a scene that can be related by a motion model. Different image alignment algorithms use different techniques and assumptions to estimate the parameters of these motion models. Once these parameters are known, one image can be warped directly to align with the other. The embodiment can use an enhanced correlation coefficient (ECC) maximization image alignment method to align the remaining frames of the initial video frame sequence with the first frame. After video stabilization is completed, each frame image of the initial video frame sequence can be converted into a grayscale image first. Then, for each of the remaining frames, a homography transformation motion model can be selected as the preset motion model, and a corresponding spatial storage model can be assigned. The similarity measure of the enhanced correlation coefficient maximization algorithm can be used to estimate the distortion matrix of the preset motion model between the remaining frame and the first frame. Thus, the distortion matrix can be applied to one of the remaining frame and the first frame to align with the other.

[0058] S13, performing ICG fluorescence color extraction on the preset region of interest in each frame image of the target video frame sequence in frame order through HSV color space conversion to obtain observation data of ICG fluorescence intensity changing over time.

[0059] Specifically, during the extraction of the ICG curve, distortion may also occur due to the background and light source reflection, etc. Therefore, the influence of the background and light source reflection, etc. can be removed by using HSV (hue, saturation and value) color space conversion. After the target video frame sequence is obtained, ICG fluorescence color extraction can be performed on the preset region of interest in each frame image of the target video frame sequence in frame order through HSV color space conversion, so as to obtain the observation data of ICG fluorescence intensity changing over time. According to different ICG camera manufacturers, ICG fluorescence emission is synthesized into different colors on the image, generally green or blue. Therefore, different thresholds can be applied to each fluorescence emission color to extract the ICG fluorescence color. In this process, due to the difference in color, the background and reflection of light can be removed.

[0060] Optionally, the ICG fluorescence color extraction on the preset region of interest in each frame image of the target video frame sequence in frame order through HSV color space conversion to obtain the observation data of ICG fluorescence intensity changing over time comprises: taking the average ICG brightness in the preset region of interest as the observation data to reduce the influence of camera shaking and low light noise.

[0061] Before the ICG fluorescence color extraction on the preset region of interest in each frame image of the target video frame sequence in frame order through HSV color space conversion, the preset region of interest can be set first. There can be multiple preset regions of interest, and each can be subjected to ICG fluorescence color extraction, so that the ICG curves of multiple points can be obtained to evaluate the colon perfusion conditions at multiple positions. The quality of the target video frame sequence determines the size of the preset region of interest. If the target video frame sequence is shaking and dark, the preset region of interest should be increased, and vice versa. The preset region of interest should be reduced to improve the spatial resolution. In the embodiment, a preset region of interest of 5x5 pixels can be used by default. Further, due to the background, light source reflection, camera shaking, etc., there can be abnormal points that differ greatly from the average value. Therefore, after the ICG fluorescence color extraction is completed, the median filter can be used to remove the abnormal values and interference points.

[0062] S14, fitting a target ICG curve according to the observation data, and calculating a fluorescence intensity factor and a perfusion time factor according to the target ICG curve.

[0063] Specifically, the target ICG curve is the ICG fluorescence intensity curve fitted from the observation data. Since the ICG fluorescence intensity is positively correlated with the ICG fluorescence content concentration, the ICG fluorescence content concentration curve is similar to the ICG fluorescence intensity curve. The fitting method can be selected as a nonlinear least squares method. After the curve fitting is completed, one or more target ICG curves can be obtained corresponding to one or more preset regions of interest, and then the fluorescence intensity factor and the perfusion time factor can be automatically calculated from each target ICG curve. The fluorescence intensity factor can include the fluorescence intensity at the baseline, the fluorescence difference between the maximum intensity and the baseline intensity, and the fluorescence slope, etc., and the perfusion time factor can include the time from the first fluorescence increase to the maximum value, the time from the first fluorescence increase to half of the maximum value, and the time ratio, etc.

[0064] Optionally, the target ICG curve is fitted from the observation data, including: using a nonlinear least squares method improved by a RANSAC algorithm to fit the target ICG curve according to a preset ICG kinetic model and the observation data.

[0065] Specifically, first, the spatiotemporal dynamics of ICG can be mathematically modeled using a differential equation to obtain a preset ICG kinetic model. The model considers three main determinants of ICG pharmacokinetics, including elimination, perfusion and permeability. The arterial input function (C A ) is determined by the clearance of ICG in the liver and is expressed as an exponential decay function with a time constant τ. Although some studies have assumed a double exponential model to consider the long-term and short-term decay of ICG, the long-term decay is in the order of hours, and the ICG decay can be approximated as a single exponential function in 10 minutes. The half-life of ICG decay is 3-5 minutes, and its relationship with time is expressed as:

[0066] C A = C0e -t / τ

[0067] Where t represents time, and C0represents the initial concentration of ICG. The transport of ICG through the vascular lumen, especially the arterial interstitial space to the capillary and venous interstitial space (C CV ), represents perfusion, which is expressed by Fick's law, with a flow rate of F, and the formula is:

[0068] C‘ CV = F(C A -C CV )-K e C CV +K i C T

[0069] Where K e represents the extravasation parameter, and Ki represents the permeability parameter. Permeability indicates the bidirectional transport of ICG between the intravascular and extravascular compartments (C T ), which is given by:

[0070] C’ T = K e C CV -K i C T

[0071] The sum of the above equations describes the ICG kinetics of the tissue and the vascular compartments with their volume fractions, which mathematically describes the regional ICG kinetics, which is given by:

[0072] I = (V A C A + V cv C cv + V T C T ) / (V A + V cv + V T )

[0073] where I represents the ICG fluorescence content concentration, V A represents the volume fraction of the arterial compartment, V CV represents the volume fraction of the capillary and venous compartments, and V T represents the volume fraction of the extravascular compartment.

[0074] According to the above formula, the curve is fitted by a non-linear least squares method. When we use a model to describe a series of data in reality, there will always be a certain deviation between the predicted results of the model and the actual measured results, which is called residual error. The purpose of the non-linear least squares method is to adjust the parameters of the model so that the total residual error is minimized, so it is suitable for the case where the actual error is small. Due to background, light source reflection, camera shaking and other reasons, there will be many outliers, which will affect the accuracy of curve fitting, and further cause misjudgment of ICG curve microcirculation perfusion analysis. Therefore, the RANSAC algorithm can be used to improve the non-linear least squares method in this embodiment. The basic assumption of RANSAC is that the "in-group" data can describe its data distribution through several groups of model parameters, while the "out-group" data is not suitable for modeling data. Data will be affected by noise, and noise refers to outliers, such as extreme noise or incorrect interpretation of measurements or incorrect assumptions about data.

[0075] Further optionally, the fitting the target ICG curve according to the preset ICG kinetic model and the observation data using a nonlinear least squares method improved by a RANSAC algorithm comprises: randomly selecting a preset number (e.g., 80%) of points in the observation data as in-group points (i.e., as valid points other than noise points and invalid points) multiple times (e.g., a maximum number of iterations can be set); for the in-group points obtained each time, fitting a candidate ICG curve using the nonlinear least squares method according to the in-group points, then inputting the observation data into the candidate ICG curve, and calculating the number of in-group points in the observation data; and selecting the candidate ICG curve with the largest number of in-group points as the target ICG curve.

[0076] S15, inputting the fluorescence intensity factor, the perfusion time factor, and the curve parameters of the target ICG curve into the trained XGBoost machine learning model to predict a score result of the target colorectal region.

[0077] Specifically, an experienced surgeon can compare these factors with normal colorectal factors through medical experience to determine the perfusion state of the colon, and can assess the risk of complications through experience by considering the color of the intestine and the peristalsis of the colon wall. The present embodiment can collect the scores of the experienced surgeon on the calculated ICG curve and parameters as a training set, and train the XGBoost machine learning model. Then, the trained XGBoost machine learning model can be used to calculate the score result of the target colorectal region according to the fluorescence intensity factor, the perfusion time factor, and the curve parameters, thereby realizing intelligent quantitative evaluation.

[0078] The technical scheme provided by the embodiment of the present application firstly acquires an initial video frame sequence of a target colorectal region, then performs video de-jittering on the obtained initial video frame sequence, aligns the remaining frames of the initial video frame sequence except the first frame with the first frame, obtains a target video frame sequence, extracts ICG fluorescent color from a preset region of interest in each frame image of the target video frame sequence in frame order through HSV color space transformation, obtains observation data of ICG fluorescent intensity changing with time, then fits a target ICG curve according to the observation data, calculates a fluorescent intensity factor and a perfusion time factor according to the target ICG curve, and finally inputs the fluorescent intensity factor, the perfusion time factor and the curve parameters of the target ICG curve into a trained XGBoost machine learning model, so as to predict a score result of the target colorectal region. By using video de-jittering and picture alignment, the influence of camera jitter is reduced, and by using HSV color space transformation, the influence of factors such as background and light source reflection is removed, so that the shooting requirements of the fluorescent video are reduced, complex operations and environmental conditions are avoided, and through automatic fitting of the curve and automatic calculation of the factor and the score, intelligent quantitative evaluation of colon perfusion is realized, so that the burden of doctors is reduced and the efficiency and safety of the operation are improved.

[0079] Embodiment two

[0080] Figure 2 The structure diagram of the quantitative evaluation device for colon perfusion in colorectal surgery provided by the second embodiment of the present application is shown in the figure. The device can be realized by hardware and / or software, and can be integrated in a computer device for executing the quantitative evaluation method for colon perfusion in colorectal surgery provided by any embodiment of the present application. As shown in the figure, the device comprises: Figure 2

[0081] An initial video frame sequence acquisition module 21 is configured to acquire an initial video frame sequence of a target colorectal region;

[0082] A target video frame sequence acquisition module 22 is configured to perform video de-jittering on the initial video frame sequence, align the remaining frames of the initial video frame sequence except the first frame with the first frame, and obtain a target video frame sequence;

[0083] An observation data extraction module 23 is configured to extract ICG fluorescent color from a preset region of interest in each frame image of the target video frame sequence in frame order through HSV color space transformation, and obtain observation data of ICG fluorescent intensity changing with time;

[0084] An ICG curve fitting module 24 is configured to fit a target ICG curve according to the observation data, and calculate a fluorescent intensity factor and a perfusion time factor according to the target ICG curve; ​

[0085] The scoring result prediction module 25 is configured to input the fluorescence intensity factor, the perfusion time factor and the curve parameters of the target ICG curve into the trained XGBoost machine learning model to predict the scoring result of the target colorectal region.

[0086] The technical scheme provided by the embodiment of the present application firstly acquires the initial video frame sequence of the target colorectal region, then performs video deblurring on the obtained initial video frame sequence, and aligns the remaining frames of the initial video frame sequence except the first frame with the first frame to obtain a target video frame sequence, then extracts ICG fluorescence color in the preset region of interest in each frame image of the target video frame sequence in sequence according to the frame order through HSV color space transformation to obtain observation data of ICG fluorescence intensity changing with time, then fits the target ICG curve according to the observation data, and calculates the fluorescence intensity factor and the perfusion time factor according to the target ICG curve, and finally inputs the fluorescence intensity factor, the perfusion time factor and the curve parameters of the target ICG curve into the trained XGBoost machine learning model to predict the scoring result of the target colorectal region. By using video deblurring and picture alignment, the influence of camera shaking is reduced, and by using HSV color space transformation, the influence of background and light source reflection and other factors is removed, thereby reducing the shooting requirements of the fluorescence video and avoiding complex operations and environmental conditions. At the same time, through automatic fitting of the curve and automatic calculation of the factor and the score, intelligent and quantitative evaluation of colon perfusion is realized, thereby reducing the burden of doctors and improving the efficiency and safety of surgery.

[0087] On the basis of the above technical scheme, optionally, the target video frame sequence acquisition module 22 is specifically configured to:

[0088] The initial video frame sequence is deblurred using the Lucas-Kanade optical flow algorithm, and the remaining frames of the initial video frame sequence except the first frame are aligned with the first frame using the enhanced correlation coefficient maximization algorithm.

[0089] On the basis of the above technical scheme, optionally, the target video frame sequence acquisition module 22 comprises:

[0090] A rigid transformation calculation unit is configured to traverse the initial video frame sequence, collect feature points in each frame image using the Lucas-Kanade optical flow algorithm, and calculate the rigid transformation between each two consecutive frame images according to the feature points.

[0091] A smooth trajectory obtaining unit is configured to calculate the original trajectory of each frame image according to the rigid transformation, and smooth the original trajectory using a moving average line filter to obtain the smooth trajectory of each frame image.

[0092] a smooth transformation calculation unit configured to calculate a trajectory difference between the smooth trajectory and the original trajectory, and calculate a smooth transformation between each two consecutive image frames according to the trajectory difference and the rigid transformation;

[0093] a smooth transformation application unit configured to apply the smooth transformation to each corresponding image frame.

[0094] On the basis of the above technical solutions, optionally, the target video frame sequence acquisition module 22 comprises:

[0095] a grayscale image conversion unit configured to convert each image frame of the initial video frame sequence into a grayscale image;

[0096] a registration alignment unit configured to, for each of the remaining frames, estimate a warping matrix of a preset motion model between the remaining frame and the first frame using an enhanced correlation coefficient maximization algorithm, and perform registration alignment using the warping matrix.

[0097] On the basis of the above technical solutions, optionally, the observation data extraction module 23 is specifically configured to:

[0098] take the average ICG brightness in the preset region of interest as the observation data.

[0099] On the basis of the above technical solutions, optionally, the ICG curve fitting module 24 is specifically configured to:

[0100] fit the target ICG curve using a nonlinear least squares method improved by a RANSAC algorithm according to a preset ICG kinetic model and the observation data.

[0101] On the basis of the above technical solutions, optionally, the ICG curve fitting module 24 comprises:

[0102] an inlier point selection unit configured to randomly select a preset number of points as inlier points in the observation data multiple times;

[0103] an inlier point number calculation unit configured to, for each of the inlier points obtained by selection, fit a candidate ICG curve using the nonlinear least squares method according to the inlier points, then input the observation data into the candidate ICG curve, and calculate an inlier point number in the observation data;

[0104] a target curve determination unit configured to select the candidate ICG curve with the largest inlier point number as the target ICG curve.

[0105] The device for quantitatively evaluating colon perfusion in colorectal surgery provided by the embodiment of the present application can execute the method for quantitatively evaluating colon perfusion in colorectal surgery provided by any embodiment of the present application, has the function modules and beneficial effects corresponding to the execution method.

[0106] It is worth noting that in the above embodiment of the device for quantitatively evaluating colon perfusion in colorectal surgery, each unit and module included is only divided according to functional logic, but is not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for the convenience of mutual differentiation, and are not used to limit the protection scope of the present application.

[0107] Embodiment three

[0108] Figure 3 The structural schematic diagram of the computer device provided by the embodiment three of the present application shows the block diagram of an exemplary computer device suitable for implementing the embodiments of the present application. Figure 3 The displayed computer device is only an example, and should not bring any limitation to the functions and use range of the embodiment of the present application. As Figure 3 shown, the computer device includes a processor 31, a memory 32, an input device 33 and an output device 34; the number of processors 31 in the computer device can be one or more, Figure 3 for example, the processor 31 in the computer device, the memory 32, the input device 33 and the output device 34 can be connected through a bus or other means, Figure 3 for example, through bus connection.

[0109] The memory 32 as a kind of computer readable storage medium can be used to store software programs, computer executable programs and modules, such as the program instructions / modules (for example, the initial video frame sequence acquisition module 21, the target video frame sequence acquisition module 22, the observation data extraction module 23, the ICG curve fitting module 24 and the score result prediction module 25 in the device for quantitatively evaluating colon perfusion in colorectal surgery) of the method for quantitatively evaluating colon perfusion in colorectal surgery in the embodiment of the present application. The processor 31 executes the software programs, instructions and modules stored in the memory 32, thereby executing various function applications and data processing of the computer device, i.e. realizing the above-mentioned method for quantitatively evaluating colon perfusion in colorectal surgery.

[0110] The memory 32 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; and the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 32 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some examples, the memory 32 can further include a memory disposed remotely with respect to the processor 31, which can be connected to the computer device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0111] The input device 33 can be used to obtain an initial video frame sequence of a target colorectal region, and to generate key signal inputs related to user settings and function controls of the computer device, etc. The output device 34 can include a display screen, and can be used to show the final scoring result to the user, etc.

[0112] Embodiment Four

[0113] The embodiment four of the present application also provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to perform a quantitative evaluation method of colon perfusion in colorectal surgery, the method comprising:

[0114] obtaining an initial video frame sequence of a target colorectal region;

[0115] performing video de-jittering on the initial video frame sequence, and aligning the remaining frames of the initial video frame sequence with the first frame to obtain a target video frame sequence;

[0116] extracting ICG fluorescent color in a preset region of interest in each frame image of the target video frame sequence according to frame order through HSV color space conversion, to obtain observation data of ICG fluorescent intensity changing with time;

[0117] fitting a target ICG curve according to the observation data, and calculating a fluorescent intensity factor and a perfusion time factor according to the target ICG curve;

[0118] inputting the fluorescent intensity factor, the perfusion time factor, and the curve parameters of the target ICG curve into a trained XGBoost machine learning model, to predict a scoring result of the target colorectal region.

[0119] The storage media can be any available memory device or storage device. The term "storage media" as used herein includes one or both of storage media and memory media. The term "storage media" includes, but is not limited to, magnetic storage media such as a hard disk, floppy disk, magnetic strip, and the like; optical storage media such as optical disk or the like; solid state storage media such as flash memory, battery backup memory, and the like; and the like. The term "computer readable media" includes, but is not limited to, one or both of storage media and memory media.

[0120] Of course, a storage medium provided by an embodiment of the present application includes computer executable instructions that are not limited to the method operations described above, but can also perform related operations in the method for quantitative evaluation of colon perfusion in colorectal surgery provided by any embodiment of the present application.

[0121] A computer readable signal medium can include a propagated data signal with computer executable program code embodied therein. The propagated data signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that is not a storage medium or is not a memory medium. A computer readable signal medium can include, but is not limited to, a computer readable medium that transmits, propagates, or otherwise transfers program code to a computer system.

[0122] Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0123] Those skilled in the art can clearly understand the present application by the above description of the embodiments, and the present application can be realized by software and necessary general hardware, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a floppy disk, a read-only memory (ROM), a random access memory (RAM), a FLASH, a hard disk, or an optical disc, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0124] It is noted that the above are only the preferred embodiments of the present application and the technical principles applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and various obvious changes, re-adjustments and substitutions can be made by those skilled in the art without departing from the scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for quantitative assessment of colonic perfusion in colorectal surgery, characterized in that, The method comprises the following steps: acquiring an initial video frame sequence of a target colorectal region; performing video deblurring on the initial video frame sequence, and aligning the remaining frames of the initial video frame sequence to the first frame to obtain a target video frame sequence; extracting ICG fluorescence color from a preset region of interest in each frame image of the target video frame sequence in frame order through HSV color space conversion to obtain observation data of ICG fluorescence intensity changing over time; fitting a target ICG curve according to the observation data, and calculating a fluorescence intensity factor and a perfusion time factor according to the target ICG curve; inputting the fluorescence intensity factor, the perfusion time factor, and curve parameters of the target ICG curve into a trained XGBoost machine learning model to predict a score result of the target colorectal region; the step of performing video deblurring on the initial video frame sequence and aligning the remaining frames of the initial video frame sequence to the first frame to obtain a target video frame sequence comprises: performing video deblurring on the initial video frame sequence using a Lucas-Kanade optical flow algorithm, and aligning the remaining frames of the initial video frame sequence to the first frame using an enhanced correlation coefficient maximization algorithm; the step of fitting a target ICG curve according to the observation data comprises: fitting the target ICG curve using a non-linear least squares method improved by a RANSAC algorithm according to a preset ICG dynamic model and the observation data, wherein the preset ICG dynamic model considers three main determinants of ICG pharmacokinetics, including elimination, perfusion and permeability.

2. The method of quantitative assessment of colonic perfusion in colorectal surgery according to claim 1, characterized in that, the step of performing video deblurring on the initial video frame sequence using a Lucas-Kanade optical flow algorithm comprises: traversing the initial video frame sequence, collecting feature points in each frame image using a Lucas-Kanade optical flow algorithm, and calculating a rigid transformation between each two consecutive frame images according to the feature points; calculating original trajectories of the frame images according to the rigid transformation, and smoothing the original trajectories using a moving average line filter to obtain smoothed trajectories of the frame images; calculating trajectory differences between the smoothed trajectories and the original trajectories, and calculating smoothed transformations between each two consecutive frame images according to the trajectory differences and the rigid transformation; applying the smoothed transformations to the corresponding frame images.

3. The method of quantitative assessment of colonic perfusion in colorectal surgery according to claim 1, characterized in that, the step of aligning the remaining frames of the initial video frame sequence to the first frame using an enhanced correlation coefficient maximization algorithm comprises: converting each frame image of the initial video frame sequence into a grayscale image; for each of the remaining frames, estimating a distortion matrix of a preset motion model between the remaining frame and the first frame using an enhanced correlation coefficient maximization algorithm, and aligning the remaining frame to the first frame using the distortion matrix.

4. The method of quantitative assessment of colonic perfusion in colorectal surgery according to claim 1, characterized in that, the step of extracting ICG fluorescence color from a preset region of interest in each frame image of the target video frame sequence in frame order through HSV color space conversion to obtain observation data of ICG fluorescence intensity changing over time comprises: average ICG brightness in the preset region of interest as the observation data.

5. The method of quantitative assessment of colonic perfusion in colorectal surgery according to claim 1, characterized in that, The target ICG curve is fitted using a non-linear least squares method improved by a RANSAC algorithm according to a preset ICG kinetic model and the observation data. A preset number of points are randomly selected as inliers in the observation data multiple times. For the inliers obtained each time, a candidate ICG curve is fitted using the non-linear least squares method according to the inliers, then the observation data is input into the candidate ICG curve, and the number of inliers in the observation data is calculated. The candidate ICG curve with the largest number of inliers is selected as the target ICG curve.

6. A device for quantitative assessment of colonic perfusion in colorectal surgery, characterized in that, The method comprises the following steps: An initial video frame sequence acquisition module is configured to acquire an initial video frame sequence of a target colorectal region. A target video frame sequence acquisition module is configured to perform video deblurring on the initial video frame sequence, and align the remaining frames of the initial video frame sequence except for a first frame with the first frame to obtain a target video frame sequence. An observation data extraction module is configured to extract ICG fluorescence color in a preset region of interest in each frame image of the target video frame sequence in frame order through HSV color space conversion to obtain observation data of ICG fluorescence intensity changing over time. An ICG curve fitting module is configured to fit a target ICG curve according to the observation data, and calculate a fluorescence intensity factor and a perfusion time factor according to the target ICG curve. A score result prediction module is configured to input the fluorescence intensity factor, the perfusion time factor, and curve parameters of the target ICG curve into a trained XGBoost machine learning model to predict a score result of the target colorectal region. The target video frame sequence acquisition module is specifically configured to: perform video deblurring on the initial video frame sequence using a Lucas-Kanade optical flow algorithm, and align the remaining frames of the initial video frame sequence except for the first frame with the first frame using an enhanced correlation coefficient maximization algorithm. The ICG curve fitting module is specifically configured to: fit the target ICG curve using a non-linear least squares method improved by a RANSAC algorithm according to a preset ICG kinetic model and the observation data, wherein the preset ICG kinetic model considers three main determinants of ICG pharmacokinetics, including elimination, perfusion, and permeability.

7. A computer device, comprising: One or more processors; Memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for quantitative evaluation of colon perfusion in colorectal surgery according to any one of claims 1-5. The program is executed by the processor to implement the method for quantitative evaluation of colon perfusion in colorectal surgery according to any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images