Real-time detection and diagnosis system and method for intraoperative malignant biliary stricture based on DSOC

Abstract

The present application relates to the detection and diagnosis of malignant biliary strictures, in order to solve the problem that the prior art cannot accurately locate and sample in real time during DSOC operation, the present application provides a real-time detection and diagnosis system and method based on malignant biliary strictures during DSOC operation, including data acquisition and quality control module for collecting video data during DSOC operation, output quality control frame sequence; The first stage morphological feature detection module is used for detecting and positioning a plurality of malignant related morphological features, and outputting the boundary box coordinates and the confidence; The second stage benign and malignant diagnosis classification module is used for carrying out benign and malignant two classification on the quality control frame sequence, and outputs the malignant probability and the two classification results; The explainability module is used for generating a heat map and spatial consistency analysis, and obtaining a visual video frame sequence; The deployment demonstration module is used for real-time receiving, displaying and storing the intraoperative DSOC video data, the boundary box coordinates and the confidence, the video level diagnosis result and the visual video frame sequence.

Description

Technical Field

[0001] This invention relates to the field of malignant biliary stricture detection, and more specifically to a real-time detection and diagnosis system and method for malignant biliary stricture based on DSOC surgery. Background Technology

[0002] Early identification and accurate diagnosis of malignant biliary stricture (MBS) are crucial for improving patient prognosis, but there is still a certain risk of misdiagnosis in clinical practice. Currently, brushing cytology and forceps biopsy guided by conventional endoscopic retrograde cholangiopancreatography (ERCP) are important means of assessing biliary stricture, but their sensitivity is limited and they are highly dependent on the operator's experience, making false negatives common.

[0003] Digital single-operator cholangioscopy (DSOC) can obtain high-resolution direct images within the bile duct and perform targeted sampling, allowing direct observation of the morphology of mucosa, blood vessels, and protrusions for targeted biopsies. However, image interpretation is influenced by experience, resulting in insufficient consistency, although theoretically it can improve diagnostic capabilities. Nevertheless, the morphology of bile duct lesions exhibits significant heterogeneity, and image interpretation is subjective and subject to inter-observer variability. Even with DSOC-guided sampling, there is still a risk of missed diagnoses, poor consistency, and a high sampling error rate. Furthermore, it cannot provide real-time, intraoperative accurate localization and sampling guidance.

[0004] In recent years, deep learning (DL), especially convolutional neural networks (CNNs), has shown transformative potential in the field of endoscopic image analysis. It can automatically extract discriminative features, reducing reliance on subjective human interpretation. Currently, computer-aided detection (CADe) and computer-aided diagnosis (CADx) systems related to cholangioscopy mostly focus on retrospective binary classification of static biliary stricture images, still struggling to provide accurate and interpretable clinical decision support. Furthermore, the establishment of reliable multi-center datasets and integrated platforms for clinical validation remain to be improved.

[0005] Traditional machine learning methods rely on manual feature engineering, making it difficult to characterize the spatiotemporal patterns inherent in dynamic cholangioscopy videos. On the other hand, deep learning models are "black boxes" in their decision-making process and lack transparency, which may raise concerns about diagnostic reliability and liability. Therefore, there is an urgent need for interpretable AI-assisted clinical decision-making solutions. Summary of the Invention

[0006] To address the limitations of existing technologies in assessing biliary stricture, such as limited sensitivity, susceptibility to false negatives, reliance on operator experience leading to high error rates, and inability to provide accurate localization and sampling guidance during DSOC procedures, this invention provides a real-time detection and diagnosis system and method for malignant biliary strictures during DSOC procedures.

[0007] To achieve the above objectives and complete the above inventive concept, the present invention adopts the following technical solution:

[0008] The real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery is unique in that it includes a data acquisition and quality control module, a first-stage morphological feature detection module, a second-stage benign and malignant diagnosis and classification module, an interpretability module, and a deployment demonstration module.

[0009] The data acquisition and quality control module is used to collect data from DSOC intraoperative video streams or video files, filter quality control static images and remove blurry frames, and output a quality control frame sequence.

[0010] The first-stage morphological feature detection module includes an optimized YOLOv11 target detection network, which is used to detect and locate multiple malignant related morphological features in the quality control frame sequence, and output bounding box coordinates and confidence scores.

[0011] The second-stage benign and malignant diagnosis classification module includes a ResNet18 convolutional neural network, which is used to perform benign and malignant binary classification on the quality control frame sequence based on the bounding box coordinates and confidence level, output the malignant probability and binary classification result at the frame level, and aggregate the binary classification results of multiple consecutive quality control frame sequences to obtain video-level diagnostic results.

[0012] The interpretability module is used to generate heatmaps of corresponding quality control frames from the obtained video-level diagnostic results using the Grad-CAM and Grad-CAM++ algorithms, overlay the heatmaps onto the quality control frame sequence, and perform spatial consistency analysis with the bounding box coordinates and confidence region to obtain a visualized video frame sequence.

[0013] The deployment demonstration module is used to receive, display, and store intraoperative DSOC video data, bounding box coordinates and confidence levels, video-level diagnostic results, and visualized video frame sequences in real time.

[0014] Furthermore, the data acquisition and quality control module includes a receiving unit, a decoding unit, a data preprocessing unit, and a quality control screening unit arranged sequentially.

[0015] The receiving unit is used to receive data information from DSOC intraoperative video streams or video files;

[0016] The decoding unit is used to perform container parsing and decoding on the data information of the DSOC intraoperative video stream or video file to extract a continuous video frame sequence.

[0017] The data preprocessing unit is used to perform resolution unification, color space conversion and normalization preprocessing on the extracted continuous video frame sequence to obtain a quality frame sequence.

[0018] The quality control screening unit is used to filter the quality frame sequence according to preset quality control rules to obtain the quality control frame sequence.

[0019] Furthermore, the preset quality control rules are to filter out quality frame sequences that are blurry, overexposed, obscured, have rinsing bubbles, and have bile interference.

[0020] Furthermore, the optimized YOLOv11 target detection network includes an anchor box adaptation unit, a loss function and class imbalance unit, a small target detection enhancement unit, and a data augmentation and robustness improvement unit, which are used to detect the scale differences and fine-grained features of multiple malignant related morphological features in the quality control frame sequence in real time and optimize them to output accurate bounding box coordinates and confidence scores.

[0021] The anchor frame adaptive unit uses the k-means clustering algorithm to generate a set of anchor frames that match the feature scale of cholangioscopy, which is used to adaptively adjust the relevant parameters in the optimized YOLOv11 target detection network according to the scale difference of malignant-related morphological features.

[0022] The loss function and the class imbalance unit are used to jointly optimize the target localization error and the class prediction error;

[0023] The small target detection enhancement unit adopts a denser multi-scale training strategy for the fusion of multi-scale features and the enhancement of detailed features.

[0024] The data augmentation and robustness enhancement unit employs brightness and contrast perturbation and fuzzy simulation enhancement strategies to improve the model's adaptability to DSOC images under different lighting, viewing angles, noise, and artifact conditions through image enhancement and perturbation simulation.

[0025] Furthermore, the aforementioned malignant morphological features include mucosal surface irregularities, abnormal blood vessels, and nodular protrusions.

[0026] Furthermore, the second-stage benign / malignant diagnosis and classification module trims the quality control frame sequence according to the bounding box coordinates to obtain candidate regions of interest;

[0027] The cropped candidate regions of interest are input into a ResNet18 convolutional neural network for benign and malignant binary classification, and the malignancy probability and binary classification result at the frame level are output. At the same time, the binary classification results of multiple consecutive quality control frame sequences are aggregated and calculated to output stable video-level diagnostic results.

[0028] Furthermore, the interpretability module includes a heatmap generation unit, an overlay and fusion unit, and a spatial consistency analysis unit;

[0029] The heatmap generation unit uses Grad-CAM and Grad-CAM++ algorithms to generate heatmaps for the corresponding quality control frames.

[0030] The overlay and fusion unit is used to overlay the heat map onto the quality control frame sequence;

[0031] The spatial consistency analysis unit is used to perform spatial consistency analysis on the quality control frame sequence with the superimposed heatmap and the bounding box coordinates and confidence region to obtain a visual video frame sequence containing the heatmap and bounding box, which is used to perform visual attribution of misjudged quality control frame sequences.

[0032] Furthermore, the deployment demonstration module includes an offline processing unit and an online processing unit;

[0033] The offline processing unit is used to receive and integrate DSOC intraoperative video data, bounding box coordinates and confidence scores, malignancy probability, binary classification results, heatmaps and video-level diagnostic results in real time.

[0034] The online processing unit is used to display, store, and output DSOC intraoperative video data in real time.

[0035] Furthermore, this invention also proposes a real-time detection and diagnosis method for malignant biliary strictures during DSOC surgery, employing a real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery. Its unique feature lies in the inclusion of the following steps:

[0036] S1: Data Acquisition and Quality Control

[0037] The data acquisition and quality control module collects data from the DSOC intraoperative video stream or video file, filters quality control static images from the DSOC intraoperative video stream or video file data and removes blurry frames, and outputs a quality control frame sequence.

[0038] S2: Morphological Feature Detection

[0039] The optimized YOLOv11 target detection network in the first-stage morphological feature detection module is used to detect and locate multiple malignant related morphological features in the quality control frame sequence, and output the results.

[0040] S3: Classification of Benign and Malignant Diagnostic Methods

[0041] Candidate regions of interest (ROIs) in the quality control frame sequence are cropped based on the bounding box coordinates. The cropped ROIs are then input into a ResNet18 convolutional neural network for binary classification of benign and malignant cases, and the frame-level malignancy probability and binary classification result are output. The binary classification results of multiple consecutive quality control frame sequences are aggregated to obtain video-level diagnostic results.

[0042] S4: Interpretability Analysis

[0043] Heatmaps of the corresponding quality control frames are generated using the Grad-CAM and Grad-CAM++ algorithms. The heatmaps are then overlaid onto the quality control frame sequence, and spatial consistency analysis is performed with the bounding box coordinates and confidence regions detected in step S2 to obtain a visual video frame sequence containing heatmaps and bounding boxes.

[0044] S5: Deploy the demo application

[0045] The deployment demonstration module receives and integrates DSOC intraoperative video data, bounding box coordinates and confidence levels, video-level diagnostic results and visualization video frame sequences in real time from steps S1-S4, and completes the real-time display, storage and demonstration output of the data.

[0046] The present invention also proposes a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the method for real-time detection and diagnosis of malignant biliary stricture based on DSOC surgery.

[0047] The beneficial effects of this invention are:

[0048] 1. The present invention is based on a real-time detection and diagnosis system for malignant biliary stricture during DSOC surgery. Through a lightweight deep learning architecture, it can perform low-latency processing on the video stream during DSOC surgery and output benign and malignant diagnostic results and lesion location information in real time during the DSOC surgery process, which significantly improves the accuracy and timeliness of biliary stricture surgery.

[0049] 2. This invention collects and processes data from DSOC intraoperative video streams or video files through a data acquisition and quality control module, performs standardized quality control, and uniformly receives, decodes, preprocesses, and screens the DSOC intraoperative video streams or video files. This reduces the impact of blurring, overexposure, occlusion, irrigation bubbles, and bile interference on subsequent analysis, which helps to reduce subjective differences among different surgeons in the process of identifying and judging stenotic lesions, improves the consistency and objectivity of intraoperative assessment results, and can effectively ensure the accuracy and effectiveness of subsequent model analysis.

[0050] 3. In response to the characteristics of small lesions and indistinct boundaries in biliary strictures, this invention utilizes an optimized YOLOv11 target detection network to effectively target and locate various malignant morphological features. By optimizing small target detection, anchor frame size, and loss function, the feature capture capability of small lesions is significantly improved, effectively increasing the detection rate and effectiveness, greatly reducing the risk of misdiagnosis and missed diagnosis, and providing technical support for the early intervention of malignant biliary strictures.

[0051] 4. The second-stage benign and malignant diagnostic classification module of the present invention performs benign and malignant binary classification using a ResNet18 convolutional neural network. At the same time, it aggregates and calculates the binary classification results of multiple consecutive quality control frame sequences. By using a sliding window for multi-frame prediction fusion, the randomness of single-frame judgment can be effectively avoided, and the consistency and accuracy of clinical diagnosis can be improved.

[0052] 5. This invention breaks through the "black box" problem of deep learning by using an interpretability module, providing clinically understandable morphological evidence for the model's diagnostic results. The spatial consistency analysis of heatmaps and detection results can effectively enhance the credibility of the diagnosis. At the same time, it can also perform visual attribution of misjudged samples, providing a basis for data cleaning and model iteration.

[0053] 6. The deployment demonstration module of this invention is divided into an offline processing unit and an online processing unit. The offline processing unit can render the analysis results with superimposed bounding boxes, diagnostic labels and heat maps onto the DSOC intraoperative display screen in real time, directly assisting the endoscopic operation. The online processing unit uploads and automatically analyzes the DSOC intraoperative video data in real time, and securely stores the data uploaded by the system and the analysis results. It supports subsequent manual review, re-annotation and data entry into the database, forming a data closed-loop mechanism for continuous iterative optimization, which effectively improves the reliability and accuracy of detection and diagnosis results.

[0054] 7. The present invention provides a simple and easy-to-implement method for real-time detection and diagnosis of malignant biliary strictures during DSOC surgery. It can effectively locate and sample various malignant morphological features of biliary stricture lesions in real time and accurately during DSOC surgery, and automatically classify and output the diagnosis of benign and malignant biliary strictures, greatly improving the consistency and accuracy of detection and diagnosis. At the same time, the interpretability module provides interpretable and visualized detection results, making the output of detection and diagnosis aligned with clinical morphological evidence, effectively improving the credibility of detection data. Attached Figure Description

[0055] Figure 1 This is a system block diagram of the real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery based on the present invention;

[0056] Figure 2This is a flowchart of the real-time detection and diagnosis of malignant biliary stricture during DSOC surgery based on the present invention;

[0057] Figure 3 This is a schematic diagram of data detection in Embodiment 2 of the present invention, comprising the data acquisition and quality control module, the first-stage morphological feature detection module, the second-stage benign / malignant diagnosis and classification module, and the interpretability module.

[0058] Figure 4 This is a schematic diagram showing the results of deploying the demonstration module in Embodiment 2 of the present invention. Detailed Implementation

[0059] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Example 1

[0061] The real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery provided in this embodiment of the invention, such as... Figure 1 As shown, the real-time detection and diagnosis system includes a data acquisition and quality control module, a first-stage morphological feature detection module (CADe), a second-stage benign and malignant diagnosis and classification module (CADx), an interpretability module, and a deployment demonstration module;

[0062] The data acquisition and quality control module is used to collect data from DSOC intraoperative video streams or video files, filter quality control static images and remove blurry frames, and output quality control frame sequences. The DSOC intraoperative video streams or video files are retrospective, multi-center DSOC data. This module is responsible for the intake and cleaning of the front-end data of DSOC intraoperative video streams or video files, which is the foundation for ensuring the accuracy of subsequent system model analysis.

[0063] The data acquisition and quality control module includes a receiving unit, a decoding unit, a data preprocessing unit, and a quality control screening unit arranged sequentially.

[0064] The receiving unit receives data information from the DSOC intraoperative video stream or video file, and then the decoding unit performs container parsing and decoding on the data information of the DSOC intraoperative video stream or video file to extract a continuous video frame sequence. The data preprocessing unit performs resolution unification, color space conversion and normalization preprocessing on the extracted continuous video frame sequence to obtain a quality frame sequence. The quality control screening unit filters the quality frame sequence according to preset quality control rules to obtain a quality control frame sequence. The preset quality control rules are to filter out quality frame sequences with blur, overexposure, occlusion, rinsing bubbles and bile interference.

[0065] The first-stage morphological feature detection module includes an optimized YOLOv11 target detection network, which is used to detect and locate multiple malignant morphological features in the quality control frame sequence, and output bounding box coordinates and confidence scores. This module is mainly used for the precise localization of biliary stricture lesions and guidance for targeted sampling.

[0066] The optimized YOLOv11 target detection network includes an anchor box adaptation unit, a loss function and class imbalance unit, a small target detection enhancement unit, and a data augmentation and robustness improvement unit. It is used to detect the scale differences and fine-grained features of multiple malignant related morphological features in the quality control frame sequence in real time and optimize them to output accurate bounding box coordinates and confidence scores.

[0067] Among them, the anchor frame adaptive unit uses the k-means clustering algorithm to generate an anchor frame set that matches the feature scale of the cholangioscope. This set is used to adaptively adjust the relevant parameters in the optimized YOLOv11 target detection network according to the scale differences of malignant morphological features, so as to improve the localization accuracy of malignant morphological features at different scales.

[0068] The loss function and class imbalance unit are used to jointly optimize the target localization error and class prediction error, reduce the impact of class imbalance on the detection results, and improve the detection capability of malignant correlation morphological features of a few classes.

[0069] The small target detection enhancement unit strengthens the high-resolution branch at the feature pyramid level and adopts a denser multi-scale training strategy to improve the detection capability of fine blood vessel texture and superficial mucosal irregularities, and is used for the fusion of multi-scale features and the enhancement of detailed features.

[0070] The data augmentation and robustness enhancement unit incorporates data augmentation based on cholangioscopy-related interference factors, including irrigation reflections, bile obstruction, compression noise, and brightness variations. It employs brightness and contrast perturbation and fuzzy simulation enhancement strategies to improve the model's adaptability to DSOC images under different lighting, viewing angles, noise, and artifact conditions through image enhancement and perturbation simulation. This improves the stability, cross-center capability, and generalization ability of the detection results. Multiple malignancy-related morphological features include mucosal surface irregularities, abnormal vessels, and nodular protrusions. The optimized YOLOv11 target detection network is specifically optimized for the scale differences and fine-grained features of lesions under cholangioscopy, enabling real-time detection of these three malignancy-related morphological features: mucosal surface irregularities, abnormal vessels, and nodular protrusions, and outputting accurate bounding box coordinates and confidence levels.

[0071] The second-stage benign / malignant diagnostic classification module includes a ResNet18 convolutional neural network, which performs binary classification of benign and malignant lesions on the quality control frame sequences based on bounding box coordinates and confidence levels. It outputs the malignancy probability and binary classification results at the frame level, and aggregates the binary classification results of multiple consecutive quality control frame sequences to obtain video-level diagnostic results. This module performs qualitative assessment of lesions based on the detection results, which can effectively improve the consistency and accuracy of clinical diagnosis.

[0072] The second-stage benign / malignant diagnosis and classification module crops the quality control frame sequence based on the bounding box coordinates to obtain candidate regions of interest. The cropped candidate regions of interest are then input into a ResNet18 convolutional neural network for benign / malignant binary classification, outputting the frame-level malignancy probability and binary classification results. Simultaneously, the binary classification results of multiple consecutive quality control frame sequences are aggregated and calculated to output stable video-level diagnostic results.

[0073] The interpretability module is used to generate heatmaps of corresponding quality control frames from the obtained video-level diagnostic results using the Grad-CAM and Grad-CAM++ algorithms. The heatmaps are then overlaid onto the quality control frame sequence, and spatial consistency analysis is performed with the bounding box coordinates and confidence regions to obtain a visualized video frame sequence. This module breaks the "black box" problem of deep learning, providing clinically understandable morphological evidence for the model's diagnostic results, greatly enhancing the trust of doctors' diagnoses.

[0074] The interpretability module includes a heatmap generation unit, an overlay and fusion unit, and a spatial consistency analysis unit. The heatmap generation unit uses Grad-CAM and Grad-CAM++ algorithms to generate heatmaps for the corresponding quality control frames. The overlay and fusion unit is used to overlay the heatmaps onto the quality control frame sequence.

[0075] The spatial consistency analysis unit is used to perform spatial consistency analysis on the quality control frame sequence of the overlaid heatmap and the bounding box coordinates and confidence region to obtain a visual video frame sequence containing the heatmap and bounding box. This is used to perform visual attribution on misjudged quality control frame sequences. Through this visual attribution, the system can clearly show "why it was judged as malignant / benign" and help identify the reasons for misjudgment such as equipment imaging artifacts.

[0076] The deployment demonstration module is used to receive, display, and store intraoperative DSOC video data, bounding box coordinates and confidence levels, video-level diagnostic results, and visualized video frame sequences in real time. This module is responsible for the practical application of the system in actual clinical scenarios and the collaborative management of multi-center data.

[0077] The deployment demonstration module includes an offline processing unit and an online processing unit. The offline processing unit is used to receive and integrate DSOC intraoperative video data, bounding box coordinates and confidence scores, malignancy probability, binary classification results, heat maps, and video-level diagnostic results in real time. The online processing unit is used to display, store, and demonstrate DSOC intraoperative video data in real time. It can also support subsequent manual review, re-annotation, and data entry into the database, thus building a data closed-loop mechanism for sustainable iterative optimization.

[0078] Furthermore, this embodiment also proposes a real-time detection and diagnosis method for malignant biliary strictures during DSOC surgery. The method employs the aforementioned real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery, including the following steps: Figure 2 As shown:

[0079] S1: Data Acquisition and Quality Control

[0080] The data acquisition and quality control module collects data from the DSOC intraoperative video stream or video file, filters quality control static images from the DSOC intraoperative video stream or video file data and removes blurry frames, and outputs a quality control frame sequence.

[0081] S2: Morphological Feature Detection

[0082] The optimized YOLOv11 target detection network in the first-stage morphological feature detection module is used to detect and locate multiple malignant related morphological features in the quality control frame sequence, and output the bounding box coordinates and confidence scores.

[0083] S3: Classification of Benign and Malignant Diagnostic Methods

[0084] Candidate regions of interest (ROIs) in the quality control frame sequence are cropped based on the bounding box coordinates. The cropped ROIs are then input into a ResNet18 convolutional neural network for binary classification of benign and malignant cases, and the frame-level malignancy probability and binary classification result are output. The binary classification results of multiple consecutive quality control frame sequences are aggregated to obtain video-level diagnostic results.

[0085] S4: Interpretability Analysis

[0086] Heatmaps of the corresponding quality control frames are generated using the Grad-CAM and Grad-CAM++ algorithms. The heatmaps are then overlaid onto the quality control frame sequence, and spatial consistency analysis is performed with the bounding box coordinates and confidence regions detected in step S2 to obtain a visual video frame sequence containing heatmaps and bounding boxes.

[0087] S5: Deploy the demo application

[0088] The deployment demonstration module receives and integrates DSOC intraoperative video data, bounding box coordinates and confidence levels, video-level diagnostic results and visualization video frame sequences in real time from steps S1-S4, and completes the real-time display, storage and demonstration output of the data.

[0089] This embodiment also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a real-time detection and diagnosis method for malignant biliary strictures during DSOC surgery.

[0090] Example 2

[0091] The difference between Example 2 and Example 1 is as follows:

[0092] like Figure 3 As shown, the data acquired by the data acquisition and quality control module includes 5,266 quality control static images of 149 patients in the DSOC intraoperative video stream or video file. The training / validation set is divided at the patient level. After the real-time detection and diagnosis system locks the data parameters, 25 patients from two independent centers are included. After processing 45 DSOC video data and removing blurry frames, approximately 27,000 frames are obtained for system analysis, of which approximately 5,177 are quality control images.

[0093] Based on relevant standards, relevant personnel annotated the bounding boxes of three types of malignant morphological features: mucosal surface irregularities, abnormal blood vessels, and nodular protrusions. The optimized YOLOv11 object detection network was used to process the scale differences and fine-grained features of the three types of malignant morphological features in the quality control frame sequence and optimized them to output accurate bounding box coordinates and confidence scores. Supervised learning training was performed on the labeled dataset. The training / validation was divided into patient units, with 90% for training using mAP@0.5 (mAP50), Precision, Recall, and F1-score; and 10% for internal validation, with an internal validation mAP50 of approximately 91.2%.

[0094] The second-stage benign / malignant diagnostic classification module uses a ResNet18 convolutional neural network to perform benign / malignant binary classification on the DSOC frame images of the quality control frame sequence, outputting the frame-level malignancy probability and binary classification results. Data partitioning adopts stratified sampling and training / validation partitioning at the patient level. Evaluation uses AUC, sensitivity, specificity, PPV, and NPV indicators. The internal validation AUC is approximately 0.960, and the external validation AUC is approximately 0.843.

[0095] The interpretability module uses Grad-CAM and Grad-CAM++ algorithms to generate heatmaps of corresponding quality control frames from the obtained video-level diagnostic results. The heatmaps are then overlaid onto the quality control frame sequence, and spatial consistency analysis is performed with the bounding box coordinates and confidence region to obtain a visualized video frame sequence. This sequence can be used to explain "why it was judged as malignant / benign" and help identify misjudgments such as equipment imaging artifacts.

[0096] The deployment demonstration module provides an online demonstration platform that supports uploading clinical videos, automated analysis, and result display, facilitating multi-center validation and collaboration. The system extracts / reads video frames and executes the first-stage morphological feature detection module and the second-stage benign / malignant diagnostic classification module for inference. It overlays bounding box coordinates and displays the prediction results, generating and displaying statistical outputs, including confidence distribution, detection and classification summary indicators, etc. Uploaded videos and analysis results are securely stored, supporting subsequent manual review, annotation, and data import for continuous model improvement.

[0097] like Figure 4 As shown, the acquired DSOC intraoperative video data is 30 seconds long and contains 5 nodular protrusions. The optimized YOLOv11 object detection network detected 1 irregular surface with a confidence score of 0.66, 0 irregular blood vessels, and 1 nodular protrusion with a confidence score of 0.25. The ResNet18 convolutional neural network classified it as malignant with a probability of 1.00, indicating high suspicion. The interpretability module focuses on the edge region of the nodules, and the deployment demonstration module shows the result as cholangiocarcinoma.

[0098] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery, characterized in that: It includes a data acquisition and quality control module, a first-stage morphological feature detection module, a second-stage benign and malignant diagnosis and classification module, an interpretability module, and a deployment demonstration module; The data acquisition and quality control module is used to collect data from DSOC intraoperative video streams or video files, filter quality control static images and remove blurry frames, and output a quality control frame sequence. The first-stage morphological feature detection module includes an optimized YOLOv11 target detection network, which is used to detect and locate multiple malignant related morphological features in the quality control frame sequence, and output bounding box coordinates and confidence scores. The second-stage benign and malignant diagnosis classification module includes a ResNet18 convolutional neural network, which is used to perform benign and malignant binary classification on the quality control frame sequence based on the bounding box coordinates and confidence level, output the malignant probability and binary classification result at the frame level, and aggregate the binary classification results of multiple consecutive quality control frame sequences to obtain video-level diagnostic results. The interpretability module is used to generate heatmaps of corresponding quality control frames from the obtained video-level diagnostic results using the Grad-CAM and Grad-CAM++ algorithms, overlay the heatmaps onto the quality control frame sequence, and perform spatial consistency analysis with the bounding box coordinates and confidence region to obtain a visualized video frame sequence. The deployment demonstration module is used to receive, display, and store intraoperative DSOC video data, bounding box coordinates and confidence levels, video-level diagnostic results, and visualized video frame sequences in real time.

2. The real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery according to claim 1, characterized in that: The data acquisition and quality control module includes a receiving unit, a decoding unit, a data preprocessing unit, and a quality control screening unit arranged sequentially. The receiving unit is used to receive data information from DSOC intraoperative video streams or video files; The decoding unit is used to perform container parsing and decoding on the data information of the DSOC intraoperative video stream or video file to extract a continuous video frame sequence. The data preprocessing unit is used to perform resolution unification, color space conversion and normalization preprocessing on the extracted continuous video frame sequence to obtain a quality frame sequence. The quality control screening unit is used to filter the quality frame sequence according to preset quality control rules to obtain the quality control frame sequence.

3. The real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery according to claim 2, characterized in that: The preset quality control rules are to filter out quality frame sequences that are blurry, overexposed, obscured, have rinsing bubbles, and have bile interference.

4. The real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery according to claim 2 or 3, characterized in that: The optimized YOLOv11 target detection network includes an anchor box adaptation unit, a loss function and class imbalance unit, a small target detection enhancement unit, and a data augmentation and robustness improvement unit. It is used to detect the scale differences and fine-grained features of multiple malignant related morphological features in the quality control frame sequence in real time and optimize them to output accurate bounding box coordinates and confidence scores. The anchor frame adaptive unit uses the k-means clustering algorithm to generate a set of anchor frames that match the feature scale of cholangioscopy, which is used to adaptively adjust the relevant parameters in the optimized YOLOv11 target detection network according to the scale difference of malignant-related morphological features. The loss function and the class imbalance unit are used to jointly optimize the target localization error and the class prediction error; The small target detection enhancement unit adopts a denser multi-scale training strategy for the fusion of multi-scale features and the enhancement of detailed features. The data augmentation and robustness enhancement unit employs brightness and contrast perturbation and fuzzy simulation enhancement strategies to improve the model's adaptability to DSOC images under different lighting, viewing angles, noise, and artifact conditions through image enhancement and perturbation simulation.

5. The real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery according to claim 4, characterized in that: The various malignant morphological features include irregular mucosal surfaces, abnormal blood vessels, and nodular protrusions.

6. The real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery according to claim 5, characterized in that: The second-stage benign / malignant diagnosis and classification module trims the quality control frame sequence according to the bounding box coordinates to obtain candidate regions of interest; The cropped candidate regions of interest are input into a ResNet18 convolutional neural network for benign and malignant binary classification, outputting the malignancy probability and binary classification results at the frame level. At the same time, the binary classification results of multiple consecutive quality control frame sequences are aggregated and calculated to output stable video-level diagnostic results.

7. The real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery according to claim 6, characterized in that: The interpretability module includes a heatmap generation unit, an overlay and fusion unit, and a spatial consistency analysis unit; The heatmap generation unit uses Grad-CAM and Grad-CAM++ algorithms to generate heatmaps for the corresponding quality control frames. The overlay and fusion unit is used to overlay the heat map onto the quality control frame sequence; The spatial consistency analysis unit is used to perform spatial consistency analysis on the quality control frame sequence with the superimposed heatmap and the bounding box coordinates and confidence region to obtain a visual video frame sequence containing the heatmap and bounding box, which is used to perform visual attribution of misjudged quality control frame sequences.

8. The real-time detection and diagnosis system for malignant biliary strictures during DSOC surgery according to claim 7, characterized in that: The deployment demonstration module includes an offline processing unit and an online processing unit; The offline processing unit is used to receive and integrate DSOC intraoperative video data, bounding box coordinates and confidence scores, malignancy probability, binary classification results, heatmaps and video-level diagnostic results in real time. The online processing unit is used to display, store, and output DSOC intraoperative video data in real time.

9. A method for real-time detection and diagnosis of malignant biliary stricture during DSOC surgery, employing the real-time detection and diagnosis system for malignant biliary stricture during DSOC surgery as described in any one of claims 1-8, characterized in that, Includes the following steps: S1: Data Acquisition and Quality Control The data acquisition and quality control module collects data from the DSOC intraoperative video stream or video file, filters quality control static images from the DSOC intraoperative video stream or video file data and removes blurry frames, and outputs a quality control frame sequence. S2: Morphological Feature Detection The optimized YOLOv11 target detection network in the first-stage morphological feature detection module is used to detect and locate multiple malignant related morphological features in the quality control frame sequence, and output the bounding box coordinates and confidence scores. S3: Classification of Benign and Malignant Diagnostic Methods Candidate regions of interest (ROIs) in the quality control frame sequence are cropped based on the bounding box coordinates. The cropped ROIs are then input into a ResNet18 convolutional neural network for binary classification of benign and malignant cases, and the frame-level malignancy probability and binary classification result are output. The binary classification results of multiple consecutive quality control frame sequences are aggregated to obtain video-level diagnostic results. S4: Interpretability Analysis Heatmaps of the corresponding quality control frames are generated using the Grad-CAM and Grad-CAM++ algorithms. The heatmaps are then overlaid onto the quality control frame sequence, and spatial consistency analysis is performed with the bounding box coordinates and confidence regions detected in step S2 to obtain a visual video frame sequence containing heatmaps and bounding boxes. S5: Deploy the demo application The deployment demonstration module receives and integrates DSOC intraoperative video data, bounding box coordinates and confidence levels, video-level diagnostic results and visualization video frame sequences in real time from steps S1-S4, and completes the real-time display, storage and demonstration output of the data.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the real-time detection and diagnosis method for malignant biliary strictures based on DSOC surgery as described in claim 9.

Images