A continuous grading scoring method based on safety-critical surgical fields

Abstract

A continuous hierarchical scoring method based on a safety-critical surgical field relates to the technical field of image processing, including obtaining a surgical video stream and preprocessing the video stream; setting a continuous scoring interval; constructing and training a discrete scoring prediction model based on a deep neural network; constructing and training a deep semantic segmentation model based on a deep neural network; using an activated region feature ratio calculation method, based on the continuous scoring interval, calculating the number of activated features of the discrete scoring prediction model, calculating the number of pixels in the scoring region of the deep semantic segmentation model, and calculating the ratio of the number of activated features to the number of pixels in the scoring region; calculating the weighted sum of the ratios to obtain a final continuous score; smoothing the final continuous score and outputting it; and being able to provide more detailed and real-time safety feedback for laparoscopic surgery.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a continuous grading and scoring method based on a safety-critical surgical field. Background Art

[0002] In laparoscopic cholecystectomy, the clarity of vision and structural visualization of the gallbladder triangle directly affect the safety of the operation. Currently, the more commonly used methods for evaluating the clarity of vision include: the critical view of safety (CVS) evaluation method, which generally adopts a discrete scoring system of 0-1-2, but this scoring system is relatively rough and difficult to dynamically reflect the changes in the visual field during the operation. In addition, the interval of the discrete scoring will cause a delay in the surgical safety assessment to a certain extent.

[0003] In addition, the safety-critical surgical field assessment method has the following defects:

[0004] a) Only consider the training and output of discrete scoring values: Currently, most safety vision scoring technologies based on neural networks and other technologies are designed for discrete scores, that is, the goal of the network is to directly learn and output limited scoring levels (such as 0, 1, 2), without considering the continuity requirements of the scoring. This architectural design limits the model's ability to perceive subtle changes during the surgical process, making the scoring lack sufficient accuracy and sophistication, thereby affecting the effect of real-time safety monitoring.

[0005] b) Lack of ability to capture features of surgical scenes: Existing evaluation models usually only perform routine feature extraction and scoring calculations, which makes it difficult for the model to perceive characteristic information about the surgical content when the surgical field of view, the degree of anatomical structure exposure, and the operation process undergo gradual changes. As a result, the scoring output lacks an accurate reflection of the real-time progress of the operation, thereby reducing the accuracy of the evaluation of intraoperative safety.

[0006] c) Limited clinical decision support capabilities: Discrete scoring can only provide simplified risk information in complex surgical scenarios and cannot support doctors in making accurate decisions during surgery. There is a high demand for continuous risk assessment during surgery, and the limitations of discrete scoring make it difficult to provide sufficient decision support in complex operation scenarios.

[0007] Therefore, we propose a method that can perform real-time scoring based on the real-time progress of the operation and the current content of the surgical video, and convert discrete scores into continuous scoring values, aiming to provide more detailed and real-time safety feedback for laparoscopic surgery. Summary of the invention

[0008] The purpose of the present invention is to provide a continuous grading and scoring method based on safety-critical surgical fields, which can provide more detailed and real-time safety feedback for laparoscopic surgery.

[0009] The present invention is achieved through the following technical solutions:

[0010] A continuous grading and scoring method based on safety-critical surgical fields, including:

[0011] Obtain surgical video stream and pre-process the video stream;

[0012] Set continuous scoring intervals;

[0013] Based on deep neural networks, a discrete rating prediction model is constructed and trained;

[0014] Based on deep neural networks, build and train deep semantic segmentation models;

[0015] The activated region feature ratio calculation method is used to calculate the number of activated features of the discrete score prediction model, the number of pixels in the score region of the deep semantic segmentation model, and the ratio of the number of activated features to the number of pixels in the score region based on the continuous score interval.

[0016] The weighted sum of the ratios was calculated to obtain the final continuous score;

[0017] The final continuous score is smoothed and output.

[0018] Furthermore, the process of constructing and training a discrete score prediction model includes:

[0019] Input the preprocessed video stream into the trained discrete score prediction model to obtain the discrete score of the model;

[0020] The calculation formula of the discrete score includes:

[0021]

[0022] in, Represents the discrete score output value of the model, which can be 0, 1, or 2; is the current input surgical video frame; Represents the scoring inference function of the neural network model, which generates a safety score value for the field of view through feature extraction and inference.

[0023] Furthermore, the specific processing steps of the deep semantic segmentation model include:

[0024] Based on deep neural network, a deep semantic segmentation model of gallbladder triangle is constructed;

[0025] The preprocessed video stream is received by the trained deep semantic segmentation model to obtain a segmentation map.

[0026] Furthermore, the segmentation map The calculation formula includes:

[0027]

[0028] in: is the input video frame; It is the prediction function of the segmentation model, which is used to output the segmentation map.

[0029] Furthermore, the process of setting the continuous scoring interval includes:

[0030] Determine a range;

[0031] The two end points and the middle point of the interval range are used as the starting points of the three discrete scores.

[0032] Furthermore, the process of calculating the number of activated features of the discrete score prediction model based on the continuous score interval includes:

[0033] Based on the continuous scoring interval, the starting score of each discrete scoring is mapped out;

[0034] Extract the scoring types with high starting scores from the feature map of the previous layer of the output layer of the discrete scoring model, and calculate the gradient map of the higher scoring types to the feature map;

[0035] Set a threshold and use the feature pixels in the gradient map that exceed the threshold as activated features;

[0036] Calculate the number of activated features.

[0037] Furthermore, the calculation of the number of activated features The formula includes:

[0038] ;

[0039] in: and are the width and height of the gradient image respectively; Indicates that the gradient map is at position Gray value of is the preset grayscale threshold; is an indicator function.

[0040] Furthermore, the number of pixels in the scoring area of ​​the deep semantic segmentation model is calculated The formula includes:

[0041] ;

[0042] in: and are the width and height of the segmentation map respectively; Represents the predicted value of the segmentation map at position (x, y). If the segmentation map is a triangular area, it is 1, otherwise it is 0.

[0043] Furthermore, the method for calculating the ratio of the number of activated features to the number of pixels in the scoring area includes:

[0044]

[0045] The weighted sum of the ratios The calculation method is:

[0046]

[0047] in is the number of higher-class categories than the current predicted category, is the starting score of the current video frame, is the final continuous score, Indicates the score of a scoring type that is one level higher than the current starting type.

[0048] Furthermore, the method for smoothing the final continuous score includes:

[0049] Initialize a smoothing window of size K;

[0050] A smoothing window is added to the final continuous score value calculated for each frame of the video stream;

[0051] When the number of rating values ​​in the window exceeds K, the window replaces the earliest rating value, so that the window always contains only the ratings closest to K frames;

[0052] Calculate the average score of K frames in the smoothing window, which is the smoothing score output of the current frame.

[0053] The technical solution of the present invention has at least the following advantages and beneficial effects:

[0054] The present invention discloses a continuous hierarchical scoring method based on a safety-critical surgical field. The method combines the activation features with the number of pixels in the gallbladder triangular segmentation area through an activation region feature ratio calculation method, and dynamically adjusts the score to capture significant features in the surgical field of view, making the surgical safety field of view score more delicate and able to adapt to the field of view status during surgery.

[0055] It also achieves real-time continuous scoring of the visual field safety during laparoscopic cholecystectomy, overcoming the shortcoming that traditional discrete scoring is difficult to capture subtle changes in visual field clarity during surgery.

[0056] In addition, a smooth transition from discrete values ​​to continuous scores was achieved by combining the discrete value prediction model, the gallbladder triangle score region segmentation model, and the activation region feature ratio calculation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A method flow chart of the present invention;

[0058] Figure 2 It is a flow chart of the activation area feature ratio calculation method of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0060] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0062] In the description of the present invention, it should be noted that if the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the product of the application is usually placed when used. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0063] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0064] Example 1

[0065] As attached Figure 1-2 As shown in FIG, a continuous grading scoring method based on safety-critical surgical fields includes:

[0066] Obtain surgical video stream and pre-process the video stream;

[0067] The video stream is real-time video data, and the source of the video stream may be a laparoscope or other equipment;

[0068] In addition, the preprocessing steps include:

[0069] a) Input normalization: adjust the input video to a uniform resolution, and convert the original image into a 224x224 pixel input frame using center cropping and scaling operations;

[0070] b) Sampling: 10 frames per second.

[0071] Set continuous scoring intervals;

[0072] The continuous scoring interval is mainly used to indicate that the three major scoring points of the critical view of safety (CVS) of laparoscopic cholecystectomy are converted from discrete 0, 1, and 2 scores to continuous scores, providing physicians with refined surgical references. The three major scoring points mainly include: 1) Completely remove the fat and fibrous tissue in the gallbladder triangle; 2) Free the lowest (1 / 3) part of the gallbladder bed to expose the gallbladder plane. The gallbladder plane refers to the white fibrous tissue gap where the gallbladder is attached to the liver; 3) Visually, there are "only" two anatomical structures entering the gallbladder (i.e., the cystic duct and the cystic artery);

[0073] In this article, the continuous scoring interval takes 0-100 as an example to cover the surgical field of view status under different safety levels; the purpose of the implementation method of this step is to map the discrete scores 0, 1 and 2 to the starting score value (benchmark score) of the continuous scoring interval respectively, so as to ensure the refinement of the scoring and clinical applicability.

[0074] Based on deep neural networks, a discrete scoring prediction model was constructed and trained; this model is used to automatically identify and score the safety status of the current surgical field of view in the surgical video stream, and output discrete scores for the three major scoring points of the critical safety field of view (CVS) in laparoscopic cholecystectomy;

[0075] Based on deep neural networks, build and train deep semantic segmentation models;

[0076] The model has the ability to segment the gallbladder triangle and output a gallbladder triangle annotation mask, which contains detailed annotations of the gallbladder triangle, including key anatomical structures in the gallbladder triangle, such as the cystic duct and cystic artery; the model is used to ensure a clear view of the anatomical structure during the scoring process;

[0077] The activated region feature ratio calculation method is used to calculate the number of activated features of the discrete score prediction model, the number of pixels in the score region of the deep semantic segmentation model, and the ratio of the number of activated features to the number of pixels in the score region based on the continuous score interval. This ratio reflects the distribution density and significance of features associated with higher safety score types in the field of view, and can provide an appropriate improvement for the current starting score.

[0078] The weighted sum of the ratios was calculated to obtain the final continuous score to more finely reflect the real-time changes during the surgery;

[0079] The final continuous score is smoothed and output.

[0080] Example 2

[0081] The process of constructing and training a discrete score prediction model includes:

[0082] Input the preprocessed video stream into the trained discrete score prediction model to obtain the discrete score of the model;

[0083] The model is specifically designed to identify the critical safety vision (CVS) status during surgery, covering the following three scoring points:

[0084] Remove fat and fibrous tissue from the gallbladder triangle;

[0085] Free the lower 1 / 3 of the gallbladder bed and expose the gallbladder plane;

[0086] Ensure that the cystic duct and cystic artery are clearly visible in the gallbladder triangle;

[0087] The model is used to accurately predict the input frame as a score of 0 (unscoreable), 1 (low safety), or 2 (high safety), and the model can be obtained from the safety-critical surgical field assessment method;

[0088] During real-time surgery, the model receives the current video frame As input, its features are automatically extracted, and the safety field of view is scored according to the feature value. The score value output by the discrete scoring prediction model It will be used as the basic value of the continuous scoring conversion process and used for subsequent score continuity processing;

[0089] In addition, the calculation formula of the discrete score includes:

[0090]

[0091] in, Represents the discrete score output value of the model, which can be 0, 1, or 2; is the current input surgical video frame; Represents the scoring inference function of the neural network model, which generates a safety score value for the field of view through feature extraction and inference.

[0092] Example 3

[0093] The specific processing steps of the deep semantic segmentation model include:

[0094] Based on a deep neural network, a deep semantic segmentation model of the gallbladder triangle area is constructed. This model is used to accurately identify and segment the gallbladder triangle in the surgical field of view. The model can be obtained from the gallbladder key area segmentation method; for details, see the journal "Murali A, Alapatt D, Mascagni P, et al. The endoscapes dataset for surgical scene segmentation, object detection, and critical view of safety assessment: official splits and benchmark[J]. arXiv preprint arXiv:2312.12429, 2023.";

[0095] The preprocessed video stream is received by the trained deep semantic segmentation model to obtain a segmentation map;

[0096] That is, the segmentation map The calculation formula includes:

[0097]

[0098] in: is the input video frame; It is the prediction function of the segmentation model, which is used to output the segmentation map;

[0099] The segmentation result image of the gallbladder triangle area, i.e., the segmentation map, includes the pixel-level segmentation of the area.

[0100] Example 4

[0101] The process of setting the continuous scoring interval includes:

[0102] Determine a range;

[0103] The two end points and the middle point of the interval range are used as the starting points of the three discrete scores;

[0104] The process of determining the starting point, that is, the mapping process of discrete scores, includes:

[0105] a) Assume that the discrete score value is X, which takes the value of 0, 1 or 2. The discrete score value X will correspond to a continuous score starting point Y between 0 and 100, and the mapping will be performed according to the following rules:

[0106] When X=0, the mapping starting score Y=0, indicating the lowest surgical field safety level;

[0107] When X=1, the mapping starting score Y=50, indicating a moderate surgical field safety level;

[0108] When X=2, the mapping starting score Y=100, indicating the highest surgical field safety level.

[0109] b) The mapping formula is defined as:

[0110]

[0111] This function ensures that each discrete scoring value has a corresponding continuous starting score as the basis for the scoring process.

[0112] In addition, before the activation region feature ratio calculation method begins, the current starting score is recorded as If the highest score (i.e. 100 points) is not reached, further calculations with higher score types ( , ,……) The relevant activation area feature ratio is used as the weight, and then the current score is weighted and accumulated to calculate a more reasonable continuous score value;

[0113] In the specific steps, the process of calculating the number of activated features of the discrete score prediction model based on the continuous score interval includes:

[0114] Based on the continuous rating interval, the starting score of each discrete rating is mapped out; that is, the current video frame is calculated by the discrete rating prediction model to have a discrete rating value of 0, 1 or 2, and then the corresponding evaluation score is mapped to 0, 50 or 100 according to the continuous rating interval;

[0115] Extract the scoring type with a high starting score from the feature map of the previous layer of the output layer of the discrete scoring model, and calculate the gradient map of the higher scoring type to the feature map. For example, if the evaluation score corresponding to the current video frame is 0, then in the neural network, extract the scoring type with an evaluation score of 50 from the feature layer, and then calculate the gradient map of the scoring type with an evaluation score of 50;

[0116] The calculation formula of the gradient map is:

[0117]

[0118] in: Indicates a rating type that is one level higher than the current rating; Discrete rating prediction model The feature map of the previous layer of the output layer; is the gradient map of the class to the feature map;

[0119] Set a threshold and use the feature pixels in the gradient map that exceed the threshold as activated features;

[0120] Calculate the number of activated features.

[0121] The calculation of the number of activated features The formula includes:

[0122] ;

[0123] in: and are the width and height of the gradient image respectively; Indicates that the gradient map is at position Gray value of is the preset grayscale threshold; is an indicator function.

[0124] In addition, the number of pixels in the scoring area of ​​the deep semantic segmentation model is calculated The formula includes:

[0125] ;

[0126] in: and are the width and height of the segmentation map respectively; Represents the predicted value of the segmentation map at position (x, y). If the segmentation map is a triangular area, it is 1, otherwise it is 0.

[0127] In addition, the ratio of the number of activated features to the number of pixels in the scoring area is calculated as the weight, and the calculation process is:

[0128] ;

[0129] If there is a higher score type such as If it exists, repeat the above steps to calculate The R values ​​of all high-scoring types are calculated and finally the weighted sum is calculated to obtain the final continuous score Score;

[0130]

[0131] in is the number of higher-class categories than the current predicted category, is the starting score of the current video frame, is the final continuous score, Indicates the score of a scoring type that is one level higher than the current starting type.

[0132] Example 5

[0133] The method for smoothing the final continuous score, in order to improve the stability of the surgical field safety score and avoid the sharp jump of the score value due to single frame fluctuation, includes:

[0134] Initialize a smoothing window of size K to store the continuous score values ​​of the most recent K frames;

[0135] A smoothing window is added to the final continuous score value calculated for each frame of the video stream;

[0136] When the number of rating values ​​in the window exceeds K, the window replaces the earliest rating value, so that the window always contains only the ratings closest to K frames;

[0137] Assume that the current frame is the t-th frame, calculate the average score of the K frames in the smoothing window, which is the smoothing score output of the current frame;

[0138] The specific calculation formula is:

[0139]

[0140] Finally, the smoothed score As the final score output of the current frame. Through smoothing, the system effectively reduces the score jump caused by the sudden change of the single-frame score, making the safety assessment of the surgical field more stable and smooth, providing doctors with continuous and stable real-time score feedback during the operation. In addition, the mechanism averages the continuous scores of the most recent K frames to smooth the score output and ensure the continuity and stability of the score.

[0141] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A continuous grading and scoring method based on safety-critical surgical fields, characterized by: include: Obtain surgical video stream and pre-process the video stream; Set continuous scoring intervals; Based on deep neural networks, a discrete rating prediction model is constructed and trained; Based on deep neural networks, build and train deep semantic segmentation models; The activated region feature ratio calculation method is used to calculate the number of activated features of the discrete score prediction model, the number of pixels in the score region of the deep semantic segmentation model, and the ratio of the number of activated features to the number of pixels in the score region based on the continuous score interval. The process of calculating the number of activated features of the discrete score prediction model based on the continuous score interval includes: Based on the continuous scoring interval, the starting score of each discrete scoring is mapped out; Extract the scoring types with high starting scores from the feature map of the previous layer of the output layer of the discrete scoring model, and calculate the gradient map of the higher scoring types to the feature map; Set a threshold and use the feature pixels in the gradient map that exceed the threshold as activated features; Calculate the number of activated features; The calculation of the number of activated features The formula includes: ； in: and are the width and height of the gradient image respectively; Indicates that the gradient map is at position Gray value of is the preset grayscale threshold; is an indicator function; The number of pixels in the scoring area of ​​the calculated deep semantic segmentation model The formula includes: ； in: and are the width and height of the segmentation map respectively; Indicates the predicted value of the segmentation map at position (x, y). If the segmentation map is a triangular area, it is 1, otherwise it is 0; The method for calculating the ratio of the number of activated features to the number of pixels in the scoring area includes: ； The weighted sum of the ratios The calculation method is: in is the number of higher-class categories than the current predicted category, is the starting score of the current video frame, is the final continuous score, Indicates the score of a higher-level scoring type than the current starting type; The weighted sum of the ratios was calculated to obtain the final continuous score; Smooth the final continuous score and output it; The method for smoothing the final continuous score includes: Initialize a smoothing window of size K; A smoothing window is added to the final continuous score value calculated for each frame of the video stream; When the number of rating values ​​in the window exceeds K, the window replaces the earliest rating value, so that the window always contains only the ratings closest to K frames; Calculate the average score of K frames in the smoothing window, which is the smoothing score output of the current frame.

2. The continuous grading and scoring method based on the safety-critical surgical field according to claim 1 is characterized by: The process of constructing and training a discrete score prediction model includes: Input the preprocessed video stream into the trained discrete score prediction model to obtain the discrete score of the model; The calculation formula of the discrete score includes: in, Represents the discrete score output value of the model, which can be 0, 1, or 2; is the current input surgical video frame; Represents the scoring inference function of the neural network model, which generates a safety score value for the field of view through feature extraction and inference.

3. The continuous grading and scoring method based on the safety-critical surgical field according to claim 1 is characterized by: The specific processing steps of the deep semantic segmentation model include: Based on deep neural network, a deep semantic segmentation model of gallbladder triangle is constructed; The preprocessed video stream is received by the trained deep semantic segmentation model to obtain a segmentation map.

4. The continuous grading and scoring method based on the safety-critical surgical field according to claim 3 is characterized by: The segmentation map The calculation formula is: include: in: is the input video frame; It is the prediction function of the segmentation model, which is used to output the segmentation map.

5. The continuous grading and scoring method based on safety-critical surgical fields according to claim 1 is characterized by: The process of setting the continuous scoring interval includes: Determine a range; The two end points and the middle point of the interval range are used as the starting points of the three discrete scores.

Images