A method and system for detecting an abnormal digestive endoscopic video

Abstract

The present application relates to a kind of detection methods and systems of abnormal digestive endoscope video, belong to medical image intelligent analysis field. Image quality pre-inspection such as brightness, definition, field of view range and artifact determination is carried out by real-time receiving endoscope video stream and frame through 5G network;The image frame after pre-inspection is input into multi-task deep learning model, and fine quality control index and lesion classification probability are output synchronously by shared feature extraction network and quality detection branch, lesion identification branch;Time sequence fusion determination is carried out to continuous frame sequence, and three-level logic is generated by grade prompt instruction according to operation quality anomaly, highly suspicious lesion and quality control not up to standard temporary hanging judgment, and corresponding visual and auditory feedback is executed on endoscope display. Operation quality control and lesion identification are included in unified inference framework, solve the problem of false positive diagnosis caused by quality control loss, and through automatic triggering structured remote consultation request and training examination interface docking, realize the technical closed loop from real-time decision support to skill continuous improvement.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical image analysis technology, specifically to a method and system for detecting abnormal digestive endoscopy videos. Background Technology

[0002] Digestive endoscopy has become a key tool for the early screening and minimally invasive treatment of gastrointestinal malignancies such as esophageal, gastric, and colorectal cancers. Its clinical application demand continues to grow rapidly with the aging population and increased health awareness. However, the training of digestive endoscopy physicians is lengthy and involves a steep learning curve. High-quality endoscopic resources are highly concentrated in regional medical centers, leaving primary hospitals facing a shortage of endoscopists and a lack of operational experience. This results in high costs for implementing tiered healthcare policies and makes it difficult for patients in primary care to access high-quality digestive endoscopy services locally. Meanwhile, although quality control management standards for digestive endoscopy have been gradually established, their actual application in primary hospitals lags behind. Problems such as inconsistent image acquisition quality, incomplete coverage of standard sections, and lesions being missed due to mucus adhesion or improper air volume persist, severely hindering the substantial improvement of diagnostic and treatment capabilities in primary hospitals.

[0003] To address the aforementioned issues, existing technical solutions primarily utilize telemedicine systems to provide video guidance from senior experts to primary care physicians, or employ independently operating computer-aided detection models to identify lesions in single-frame endoscopic images. However, these solutions generally treat operational quality control and lesion detection as two independent tasks. Quality control feedback relies on manual post-operative sampling, while the lesion detection model lacks the ability to predict whether the input image quality meets diagnostic requirements. It continues to output classification results even when images exhibit quality defects such as artifacts, reflections, or field-of-view deviations, leading to increased false positive rates and a lack of effective calibration of diagnostic confidence. Furthermore, existing systems lack joint temporal analysis of the fluctuation trends of quality control indicators and lesion probability in continuous video frame sequences, failing to distinguish between occasional operational errors and persistent quality degradation. They also lack a closed-loop mechanism from real-time detection to remote expert resource scheduling and continuous improvement of operator capabilities, making it difficult to meet the practical needs of primary care gastrointestinal endoscopy scenarios for highly robust, low-latency feedback and a complete technical closed-loop system. Summary of the Invention

[0004] To address the problems of the prior art, this invention provides a method and system for detecting abnormal digestive endoscopy videos.

[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: Firstly, a method for detecting abnormal digestive endoscopy videos, comprising the following steps: Step S1: Receive the ultra-high-definition endoscopic video stream and corresponding patient basic data collected by the endoscope host in real time via the 5G communication network; Step S2: Deframe the video stream into multiple endoscopic images, perform image quality assessment on each endoscopic image, the assessment including at least brightness uniformity detection, focus sharpness scoring, and determination of whether the effective diagnostic field of view is met, and generate corresponding quality labels. Step S3: Call the multi-task deep learning model built into the cloud server to perform two-dimensional parallel analysis on the quality-judged endoscopic images. The multi-task deep learning model includes a shared feature extraction network, a quality detection output branch, and a lesion recognition output branch. Step S4, the dual-dimensional analysis includes: the quality detection output branch further generates refined quality control indicators including artifact index, lesion deviation from center distance, and mucosal extensibility score based on the quality label and extracted image features; the lesion identification output branch identifies whether there are gastrointestinal mucosal lesion features in the image, and if there are lesions, the lesion area is segmented and a classification probability of benign or malignant tendency is given based on the extracted deep features; Step S5: Perform time-series fusion judgment on the quality control indicators and lesion classification results of multiple consecutive frames of images. When a preset abnormal situation is judged, a graded prompt instruction is generated. Step S6: Through the edge computing node, execute the corresponding level of real-time reminder operation on the display interface of the endoscope host according to the graded prompt instruction. The real-time reminder operation includes flashing warning box, superimposed label, or voice prompt.

[0006] In one specific implementation of the first aspect, the multi-task deep learning model is trained in the following manner: We collected and labeled sample images from physicians with experience of more than 500 endoscopic procedures, covering various endoscopic modes and models. Senior physicians performed overall annotation, single lesion annotation, and feature annotation on the sample images according to the preset annotation principles to establish a training set; Quality control-related annotations are used as supervisory information for the quality detection output branch, and lesion nature and location annotations are used as supervisory information for the lesion identification output branch. A joint loss function is designed to train the shared feature extraction network and the two branches end-to-end. The joint loss function L_total=α*L_quality+β*L_lesion, where L_quality includes the regression loss of the quality control index, L_lesion includes the loss of lesion classification and segmentation, α and β are dynamically adjusted weights, and when the quality control index is worse than the threshold, α is increased and the classification confidence of the lesion output branch for this frame is reduced.

[0007] In one specific implementation of the first aspect, the timing fusion determination in step S5 includes: When the artifact index of several consecutive frames is higher than the threshold or the lesion deviates from the center by more than the preset distance, it is judged as an abnormal operation quality and a first-level reminder instruction is generated. When the lesion classification probability of a single frame exceeds the high confidence threshold, and all the fine-grained quality control indicators corresponding to this frame meet the standards, it is judged as a highly suspicious lesion and a second-level reminder instruction is generated. When the lesion classification probability of multiple consecutive frames exceeds the high confidence threshold but any quality control indicator fails to meet the standard, the judgment is temporarily suspended, and a third-level reminder instruction is generated to only prompt the operator to adjust the field of view or rinse. The lesion classification judgment is re-performed after the quality control indicator returns to the standard.

[0008] In one specific implementation of the first aspect, step S6 is followed by: When the lesion classification result indicates malignancy and the confidence level remains above the threshold, the system automatically generates an emergency consultation request package with keyframe screenshots, magnified images of the lesion area, and snapshots of quality control indicators, and pushes this request package to the remote center of the digestive endoscopy intelligent platform through the 5G communication network.

[0009] In one specific embodiment of the first aspect, the image quality determination process in step S2 further includes: The system detects and marks surface deposits and reflective areas in the image, and uses the detection results as input parameters for the artifact index. When the artifact index exceeds the standard, the system inserts a suggestion message "rinse / vacuum" into the quality label and triggers a corresponding audio-visual alert.

[0010] Secondly, a detection system for abnormal digestive endoscopy videos, employing a method for detecting abnormal digestive endoscopy videos, includes: The 5G high-speed communication module is used to enable real-time bidirectional transmission of ultra-high-definition images and medical data between hospitals of different levels across regions; The video frame decoding and quality pre-inspection unit, installed in an edge computing server or local workstation, is used to perform the image quality judgment processing and generate preliminary quality labels. The cloud-based multi-task inference and quality control unit is deployed in the remote center of the "5G+Endoscopic Clinical Intelligent Information Platform". It carries the multi-task deep learning model and performs synchronous fine-grained quality control analysis and lesion abnormality detection on the uploaded image stream. The real-time feedback terminal, embedded in the display interface of the endoscope host, is used to receive graded prompts from the cloud or edge device and render the corresponding warning graphics and text. The remote consultation request generation unit is used to automatically encapsulate an emergency consultation request package and push it to the remote center when the triggering conditions are met.

[0011] In one specific implementation of the second aspect, the instant feedback terminal includes a tiered alert module that renders alerts according to the following rules: In response to the first-level alert command, a yellow pulse indicator and distance adjustment suggestions will be displayed at the edge of the screen; In response to the second-level alert command, a red bounding box is drawn in the center of the field of vision for the suspicious lesion area and a flashing effect is superimposed, while a continuous alert sound is emitted; In response to the Level 3 alert, the text "Please rinse / aspirate and then evaluate" scrolls across the top of the screen, and the display of lesion classification results is disabled.

[0012] In one specific implementation of the second aspect, the system further includes an assessment interface that interfaces with the "five-step seven-evaluation" training system. The assessment interface is used to compare the operator's endoscopic operation trajectory, quality control index scores, and lesion identification results detected by the system with the standard path, and output an operation skill assessment report and complication risk prediction value.

[0013] The beneficial effects of this invention are as follows: 1. This invention constructs a multi-task deep learning model based on a shared feature extraction architecture, integrating endoscopy operation quality control and lesion benign / malignant identification into a unified feedforward inference framework. This solves the information silo problem caused by the independent operation of the quality control module and the diagnostic module in existing technologies. During forward propagation, the model simultaneously outputs refined quality control indicators and lesion classification probabilities. Through parallel feature extraction and multi-dimensional indicator joint inference, the operation quality judgment results can directly participate in the dynamic calibration of diagnostic confidence, eliminating the risk of lesion misjudgment caused by quality defects such as image artifacts, field of view deviation, or insufficient mucosal extension. This significantly improves the robustness and diagnostic specificity of the real-time endoscopy detection system in complex clinical conditions.

[0014] 2. This invention introduces a temporal fusion judgment mechanism to jointly model the fluctuation trends of quality control indicators and lesion probabilities in continuous video frame sequences, establishing a three-level progressive alert logic for abnormal operation quality, highly suspicious lesions, and temporary suspension due to substandard quality control. This mechanism avoids false positive alarms triggered by instantaneous noise or accidental quality degradation in a single frame. Simultaneously, when a highly suspicious lesion is detected, it automatically triggers a structured remote consultation request containing keyframe screenshots, magnified images of the lesion area, and snapshots of quality control indicators, creating a low-latency, highly reliable task scheduling closed loop between primary care endoscopic operations and higher-level expert diagnostic resources. Through interface integration with the training and assessment system, the system can also compare the operation trajectory data recorded during the detection process with standard paths, outputting multi-dimensional evaluation results such as coverage integrity, operational standardization, and complication risk, achieving a complete technical closed loop from real-time decision support to continuous skill improvement. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the main process of the endoscopic video anomaly detection method of the present invention.

[0016] Figure 2 This is a schematic diagram of the multi-task deep learning model structure of the present invention.

[0017] Figure 3 This is a schematic diagram of the timing fusion determination and three-level reminder logic of the present invention. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figures 1 to 3 This paper presents a method and system for detecting abnormal digestive endoscopy videos.

[0020] I. System Overall Architecture and Workflow The present invention proposes a method and system for detecting abnormal digestive endoscopy videos. Its overall architecture is shown in the attached figure of the specification. It adopts a "5G+Endoscopy Clinical Intelligent Information Platform" remote center platform to provide application services for four business scenarios: "mobile remote diagnosis, remote surgical guidance, AI cloud quality control management, and remote teaching and training".

[0021] System physical deployment architecture description: This system comprises three levels of nodes deployed in a hierarchical manner: Level 1: Endoscopy Terminal Layer. Located in the endoscopy centers of hospitals at all levels, this layer includes endoscopy mainframes (including mainstream models with a market share exceeding 5%, such as Olympus GIF-HQ290, GIF-XQ260, GIF-Q260, GIF-H260Z, CF-H290I, Pentax EG-2990i, EG29-i10, and Fujifilm SN1G384G310), image acquisition cards, endoscopy monitors, and locally deployed edge computing workstations. The edge computing workstations are configured with at least: 8 CPU cores, 16GB GPU memory, and 32GB RAM, for performing video frame decoding and image quality pre-checking.

[0022] Level Two: 5G Communication and Edge Computing Layer. Rapid network deployment is achieved through 5G communication modules, leveraging the high bandwidth (uplink peak rate no less than 100Mbps) and low latency (end-to-end latency less than 20ms) of the 5G network to enable synchronous transmission and real-time sharing of 4K ultra-high-definition endoscopic images, and patient basic data from HIS, LIS, and PACS systems across hospitals of different levels in different regions. Edge computing nodes are deployed locally in the hospital or in a nearby 5G edge cloud, responsible for initial quality assessment and low-latency feedback.

[0023] Level 3: Cloud-based Intelligent Platform Layer. This refers to the remote center of the "5G+Endoscopic Clinical Intelligent Information Platform," deployed in provincial or regional medical centers. The cloud server cluster hosts a multi-task deep learning inference engine, providing AI cloud quality control management standards services and remote teaching and training services such as online surgical demonstrations by experts. The cloud configuration must be at least: a GPU cluster (such as NVIDIA A100 or equivalent computing power) and a distributed storage system.

[0024] Specific steps of the video detection method for abnormal digestive endoscopy.

[0025] Step S1: Real-time data reception.

[0026] Before the formal examination begins, the person in charge of image acquisition at the endoscopy center (who must be an associate chief physician or above and have performed more than 3,000 endoscopic procedures) provides one-on-one training to the acquisition physicians to ensure that the procedures are performed in accordance with regulations.

[0027] At the start of the examination, ultra-high-definition endoscopic video streams acquired by the endoscopy host are received in real time via a 5G communication network. The video parameters are: resolution no less than 3840×2160 pixels (4K), frame rate no less than 30 frames / second, and color depth no less than 10 bits. Simultaneously, basic patient data is received through the hospital information system or image archiving and communication system interface, including but not limited to: patient age, gender, chief complaint, past medical history, HIS examination request form information, LIS test results, and PACS historical images.

[0028] Data anonymization: At the data link layer, privacy fields such as patient name, ID number, hospital number, and contact information are anonymized and encrypted. The anonymization uses the national standard SM4 symmetric encryption algorithm, and the key is held only by authorized terminals, ensuring that patient privacy is not leaked during transmission and storage.

[0029] Step S2: Video frame de-framing and image quality assessment.

[0030] After receiving the video stream, the edge computing workstation performs frame decomposition based on the frame rate, breaking the video stream down into a sequence of single-frame endoscopic images. For each frame of endoscopic image, the following quality determination sub-step is performed: S2.1 Brightness Uniformity Detection: The image is converted from the RGB color space to the HSV color space, and the V channel (brightness channel) is extracted. The image is evenly divided into 9 grid regions (3×3), and the average brightness value of each region is calculated. When the average brightness difference between any two regions exceeds 40 (0-255 range), it is judged as brightness non-uniformity, and a "brightness non-uniformity" quality label is generated.

[0031] S2.2 Focus Sharpness Score: The image is convolved using the Laplacian operator, and the variance of the convolution result is used as the sharpness score. If the sharpness score is below the threshold of 120, it is judged as out of focus, and a "Focus Blur" quality label is generated.

[0032] S2.3 Effective diagnostic field of view determination: Extract the grayscale histogram of the image edge region (a ring-shaped region 10% of the image boundary width). When the grayscale variance of the edge region pixels is less than 30, it is determined that there is obvious field of view occlusion or incompleteness, and a "incomplete field of view" quality label is generated.

[0033] S2.4 Surface Attachment and Reflective Area Detection: After Gaussian filtering to denoise the image, a threshold segmentation method is used to detect reflective areas (areas with a pixel grayscale value greater than 230). For non-reflective areas, texture analysis (based on the contrast characteristics of the grayscale co-occurrence matrix) is applied. When the texture contrast of a local area is significantly lower than its surroundings, it is marked as a possible slime attachment area. When the proportion of suspected attachments or reflective areas exceeds 15% of the total image area, a suggestion message "rinse / vacuum required" is inserted into the quality label.

[0034] S2.5 Comprehensive Quality Label Generation: Based on the results of the above sub-steps, a quality label vector Q=[q_brightness,q_focus,q_fov,q_artifact] is generated for this frame image, where the values ​​of each dimension range from 0 to 100, with higher scores indicating better quality. When any dimension scores below 60, the frame is marked as "unqualified in quality".

[0035] Step S3: Invoking the multi-task deep learning model.

[0036] The quality-assessed endoscopic images are uploaded to the cloud inference platform via a 5G network, where the built-in multi-task deep learning model is invoked for two-dimensional parallel analysis.

[0037] Model structure description: The multi-task deep learning model adopts an architecture design with a shared feature extraction network and two output branches.

[0038] Shared Feature Extraction Network: ResNet-50 is used as the backbone network. Its final fully connected classification layer is removed, and the remaining layers are retained up to the global average pooling layer, outputting a 2048-dimensional feature vector. This backbone network is pre-trained on the ImageNet dataset and fine-tuned on a digestive endoscopy image dataset.

[0039] Quality control output branch: Receives the 2048-dimensional feature vector from the shared feature extraction network, passes it through two fully connected layers (2048, then 512, then 4), and outputs a refined quality control index quadruple [q_pseudoArtifact,q_deviation,q_distention,q_overall]: The artifact index q_pseudoArtifact comprehensively reflects the severity of artifacts such as slime, reflections, and bubbles in an image. Its value ranges from 0 to 100, where 0 indicates no artifacts and 100 indicates artifacts severely obscuring the field of view. When the index is greater than or equal to 70, it is considered that the artifact count exceeds the limit.

[0040] Lesion deviation from center, q_deviation: Calculate the Euclidean distance between the center of the detected lesion region and the image center, using the image center as the origin. Normalize to 0-100, where 0 indicates the lesion is perfectly centered, and 100 indicates the lesion is located at the edge of the image. A distance greater than or equal to 50 is considered a severe deviation.

[0041] Mucosal stretchability score q_distention: Based on mucosal fold texture features extracted by deep learning, it determines whether the inflation volume adequately stretches the mucosa. The distribution of scores from experienced physicians in standard examination positions is used as a reference, with 0 indicating extremely inadequate stretching and 100 indicating adequate and appropriate stretching. A score of 40 or below is considered inadequate stretching.

[0042] Overall quality score q_overall: The weighted sum of the above three indicators, with weights of 0.3, 0.3, and 0.4 respectively.

[0043] Lesion recognition output branch: Receives the 2048-dimensional feature vector output from the shared feature extraction network, and then: Region Proposal Subnetwork: A variant of the region proposal network in the Faster R-CNN framework is used to generate bounding boxes for candidate lesion regions.

[0044] Classification and segmentation sub-network: Perform ROIAlign operation on each candidate region to extract 7×7×2048 region features, and output classification probabilities through two fully connected layers (100352, 4096, 1N+1, where N is the number of lesion categories); in parallel, output pixel-level segmentation mask of lesion region through a fully convolutional upsampling structure.

[0045] The lesion categories include: early esophageal cancer, early gastric cancer, colorectal polyps, atrophic gastritis, ulcerative colitis, esophageal varices, protruding lesions of the digestive tract (mucosal lesions), protruding lesions of the digestive tract (submucosal lesions), and ulcerative lesions. For each identified lesion area, the model provides a probability distribution of whether the lesion belongs to any of the above categories, and summarizes these into two comprehensive indicators: a benign tendency probability and a malignant tendency probability.

[0046] Training methods for multi-task deep learning models: The training data were collected from the endoscopy centers of general hospitals at or above the secondary level in this region, and all data were approved and filed by their respective hospital's medical ethics committee.

[0047] Physician qualification requirements: Physicians collecting digestive endoscopy images must be proficient in operating equipment such as white light endoscopes, electronic chromoendoscopy, and magnifying endoscopes, and have more than 500 endoscopic procedures under their experience. Before the formal collection of endoscopic data, the image collection manager of the endoscopy center (with the title of associate chief physician or above and more than 3,000 endoscopic procedures under their experience) will provide one-on-one training to the collecting physicians.

[0048] Data acquisition equipment requirements: The endoscope equipment must cover mainstream endoscope types with a market share exceeding 5% in China, with specific models listed above. The data acquisition equipment must undergo regular quality testing to ensure it is within its validity period.

[0049] Image acquisition standards: The images are clear, the lesions are clearly visible, and there is no mucus or attached material on the surface (achieved by the collecting physician through rinsing, suction, and other operations). Multi-angle imaging, with the lesion located in the center of the field of view, including: Distant view: Displays the location, shape, color, etc. of the lesion; the distance from the edge of the image to the center of the lesion is not less than 40% of the length of the shorter side of the image. Mid-range image: Shows the size, borders, elevation or depression of the lesion, presence of erosion and bleeding, etc. The lesion area occupies 20%-50% of the image area; Close-up view: shows the condition of the mucosa on the surface of the lesion, micro-dimples, blood vessels, etc., with the lesion area occupying more than 50% of the image area; Multiple images of the same lesion (from different angles, using different endoscopic modes, and employing different staining methods) should maintain basic consistency in terms of air volume, mucosal extensibility, and lesion size, with an error range not exceeding 15%. The image brightness is moderate, and the grayscale value in the histogram is between 80 and 180.

[0050] Image annotation principles: Overall annotation: Select and annotate the entire endoscopic image, such as images of atrophic gastritis, ulcerative colitis, esophageal varices, etc. Single lesion annotation: Local selection and annotation of lesions displayed in the image, such as early esophageal cancer, early gastric cancer, colorectal polyps, etc. Lesion nature selection: After the lesion is selected, two physicians with the title of associate chief physician or above shall respectively label it as benign or malignant. If there is a discrepancy, a third chief physician shall make the decision. Endoscopic image feature annotation: After selecting the lesion, annotate the endoscopic features, such as annotating the mucosal lesion or submucosal lesion for the protruding lesion of the digestive tract.

[0051] Training set construction: Based on the collected labeled data, a training set containing no less than 100,000 high-quality labeled images is constructed, and randomly divided into training set, validation set and test set in a ratio of 7:1.5:1.5.

[0052] Loss function and training strategy: The joint loss function is designed as follows: L_total=α×L_quality+β×L_lesion in: L_quality represents the loss of the quality inspection branch, using the mean squared error loss function to calculate the Euclidean distance between the predicted values ​​and the labeled values ​​of the four dimensions of the refined quality control indicators. The labeled values ​​are the median value obtained by taking the scores independently from three senior physicians according to the definitions of each dimension.

[0053] L_lesion is the loss of the lesion identification branch, which includes classification loss (cross-entropy loss) and segmentation loss (Dice loss), with a weighting ratio of 1:1.

[0054] α and β are dynamically adjusted weights, both with an initial value of 1.0.

[0055] Dynamic weight adjustment mechanism: During training, for each training batch, the overall quality score q_overall of the current training sample frame is calculated. When the q_overall of a training sample frame is <60 (i.e., the quality control index is worse than the threshold), the loss calculation weight of that training sample frame is adjusted according to the following rules: α is increased to 1.5, and the classification confidence of the lesion output branch for that training sample frame is multiplied by a reduction factor of 0.5 before participating in the loss calculation. This mechanism allows the model to learn the prior knowledge during the training phase that "for images with poor quality, more attention should be paid to optimizing the quality dimension, while reducing the confidence of the diagnostic conclusion." When q_overall >= 60, α and β are restored to 1.0, and the confidence is not reduced.

[0056] The Adam optimizer was used for training, with an initial learning rate of 0.0001, a batch size of 16, and a training iteration of no less than 100 epochs. Early stopping was triggered when the validation set loss did not decrease for 5 consecutive epochs.

[0057] Step S4: Two-dimensional parallel analysis.

[0058] During model deployment and inference, for each uploaded image frame, the forward propagation process of the multi-task deep learning model simultaneously performs analysis in the following two dimensions: Quality control dimension output: The quality inspection output branch outputs refined quality control indicators, obtaining the artifact index, lesion deviation from the center distance, mucosal extension adequacy score and comprehensive quality score of the current frame.

[0059] Lesion recognition dimension output: Lesion recognition output branch output: (1) Binary judgment of whether a lesion area is detected; (2) If a lesion is detected, output the bounding box coordinates of the lesion region (in the format [x_center,y_center,width,height], normalized to the [0,1] interval) and the pixel-level segmentation mask; (3) The probability of the lesion being benign or malignant, and the category with the highest probability is used as the reference classification result.

[0060] Step S5: Timing Fusion Determination and Hierarchical Instruction Generation To ensure the stability and accuracy of the judgment and avoid false triggering caused by single-frame noise, the system performs time-series fusion judgment on the quality control indicators and lesion classification results of multiple consecutive frames (the sliding window size is set to 90 frames, i.e., about 3 seconds in length).

[0061] The timing fusion determination rules are as follows: The first-level alert criteria are as follows: when the number of frames with substandard quality (q_overall<60) in the sliding window exceeds 30% (i.e., more than 27 out of 90 frames are substandard), and the main reason for the substandard quality is that the artifact index exceeds the standard or the lesion deviates from the center by more than the threshold, it is judged as "operation quality is continuously abnormal" and a first-level alert instruction is generated.

[0062] The second-level alert criteria are as follows: when a single frame within the sliding window has a malignancy probability exceeding the high confidence threshold (set to 85%), and all the corresponding refined quality control indicators for that frame meet the standards (q_overall>=60), it is judged as a "highly suspicious lesion," and a second-level alert instruction is generated. To further reduce false positives, at least 15 frames within the sliding window must meet the above conditions before an alert is triggered.

[0063] The third-level alert criteria are as follows: When multiple consecutive frames (no fewer than 30 frames) within the sliding window show a lesion classification with a malignancy probability exceeding the high confidence threshold of 85%, but the overall quality score is below 60 (i.e., the quality control indicator fails to meet the standard), the judgment is temporarily suspended, and a third-level alert instruction is generated. This instruction prompts the operator to first improve image quality (such as adjusting the field of view, performing washing or aspiration), rather than directly giving a lesion conclusion. After the overall quality score in subsequent frames recovers to the standard, the system automatically re-determines the lesion classification for that area.

[0064] Step S6: Real-time execution of tiered reminders Edge computing nodes receive tiered alert instructions from the cloud and execute corresponding real-time alert operations on the endoscope host's display interface: In response to the first-level alert instruction: a yellow circular pulse icon (approximately 15mm in diameter, flashing at a frequency of 2Hz) is displayed on the right edge of the endoscope monitor screen (5% of the screen width from the right boundary and 20% of the screen height from the top boundary), along with text prompts including distance adjustment suggestions (such as "Please move closer to the center of the lesion").

[0065] In response to the second-level alert command: A continuously displayed red rectangle (3 pixels wide) is drawn in the center of the field of view around the suspected lesion area, with a red halo flashing effect (1Hz flashing frequency, 5-pixel halo radius) superimposed on the outer edge of the rectangle. Simultaneously, an intermittent alert tone with a frequency of 1000Hz, a duration of 200ms, and an interval of 800ms is emitted through the audio output channel. A text prompt is displayed in the information bar at the top of the screen, such as "Suspicious lesion detection: Malignancy probability XX%".

[0066] In response to the Level 3 alert instruction: A scrolling text prompt bar (blue background with white text, scrolling speed of 5 characters per second) is displayed in the top center area of ​​the screen (10% of the screen height from the top boundary and 60% of the screen width). The content is "The image quality does not meet the diagnostic requirements. Please rinse / absorb and then evaluate again." At this time, any lesion classification results are prohibited from being displayed, and only the quality control index values ​​are displayed.

[0067] Step S7: Generation and push of remote consultation request.

[0068] When the second-level reminder command is triggered, the system will automatically perform the following operations: Extract keyframe screenshots from the sliding window (select the frame with the highest probability of being malicious as the representative frame) and save them as lossless PNG format; Capture an enlarged image of the lesion area (crop it after expanding it 20% outward from the bounding box, with a resolution of no less than 512×512 pixels). Summarize the quality control index snapshots of this frame and the adjacent frames before and after it to form a quality control data table; The above data is packaged into an emergency consultation request package in JSON format, which includes: the requesting hospital identifier, patient de-identification information, Base64 encoded keyframe images, Base64 encoded magnified images of lesion areas, quality control data, and the category probability distribution output by the detection model; The request packet is pushed to the remote center of the intelligent digestive endoscopy platform via the 5G communication network, where online digestive endoscopy experts are dispatched to respond.

[0069] Message push strategy: Emergency consultation request packets are pushed using message queue telemetry transmission protocol to ensure that messages are not lost; a timeout retransmission mechanism is set up so that if no response confirmation is received from the remote center within 5 minutes, the message will be automatically retransmitted, up to a maximum of 3 retransmissions.

[0070] Integration with the training system.

[0071] The system also includes an assessment interface that connects with the "Five Steps and Seven Assessments" training system.

[0072] The "Five Steps and Seven Assessments" training system includes five stages: theoretical learning, teaching observation, model practice, case operation, and simulation assessment, as well as two measures: basic assessment and follow-up assessment.

[0073] Assessment Interface Function: The system records the trainee's operational trajectory data throughout the entire endoscopic procedure, including: Endoscope movement trajectory (obtained via the endoscope host sensor interface): advance / retreat distance, rotation angle, number of inflation / suction cycles and duration; AI-based quality control index scores for each time period: artifact index, visual field deviation, and mucosal stretchability; Lesion detection status: whether the lesion areas indicated by AI were observed, and whether any areas were missed; Examination time: total examination time and time spent at each anatomical site.

[0074] Evaluation report generation: The above data is compared item by item with the preset standard examination operation path in the system. The standard path is obtained by senior experts through statistical analysis of 100 high-quality operations of the same examination item, including the reasonable observation time range for each anatomical site (with the 95% confidence interval as the boundary), the normal range of lens advance and retreat speed (not exceeding 5mm / s), and the inflation volume curve trend, etc.

[0075] After comparison, an operational skills assessment report is generated, including: Coverage integrity score (out of 100): Whether all areas to be inspected were observed without omission; Operational compliance score (out of 100): Whether the mirror motion parameters are within the standard range; Image quality preservation score (out of 100): The average of the overall quality score q_overall determined by AI throughout the inspection process; Lesion detection concordance rate: The concordance rate between the lesion areas independently detected by trainees and the AI ​​detection results and expert confirmation results.

[0076] Complication Risk Prediction: Based on the similarity between the trainee's operation trajectory and the operation trajectories of historical complication cases, the probability of complications occurring during the trainee's current operation is calculated. When the predicted risk probability exceeds 10%, the system issues an early warning to the supervising instructor, prompting the instructor to strengthen intraoperative monitoring and guidance for the trainee. This threshold is consistent with the "five-step seven-evaluation" system's standard that requires an independent operation complication rate of less than 10% to graduate.

[0077] Example Example Scenario: An endoscopy examination is being performed on a 55-year-old male patient at the endoscopy center of a municipal-level primary hospital (hereinafter referred to as Hospital A). Hospital A is equipped with an Olympus GIF-HQ290 endoscopy unit and a matching edge computing workstation (8-core CPU, NVIDIA RTX 3080 16GB GPU, 32GB RAM), and is connected to a 5G network. A cloud-based intelligent platform is deployed at a tertiary hospital (Hospital B) in the provincial capital.

[0078] Preoperative preparation stage: The endoscopist at Hospital A obtained patient information through the HIS system: male, 55 years old, chief complaint of "upper abdominal discomfort for 3 months, occasional black stools", no family history of gastric cancer. The data was anonymized and synchronized to the edge computing workstation.

[0079] Inspection process (timeline description): t=0min0s: The patient is anesthetized, the physician inserts the endoscope, and the examination begins. The video stream is transmitted to the cloud inference platform at 4K / 30fps via a 5G network. The endoscope first enters the upper esophagus; at this point, the field of vision is clear, the overall quality score q_overall=85, and the AI ​​detects no abnormal lesions. The endoscope monitor displays no special indications.

[0080] t=2min30s: The endoscope reaches the greater curvature of the stomach body, and the physician observes a reddened area of ​​mucosa and moves the lens closer. At this time, the AI ​​detects the lesion area, with bounding box coordinates of approximately [0.45, 0.55, 0.25, 0.3]. However, the surface of this area is covered with a small amount of mucus. The AI ​​determines the artifact index q_pseudoArtifact=72 (exceeding the threshold of 70), and the overall quality score q_overall=55 (below 60). The lesion identification branch outputs a malignancy probability of 78% (below the high confidence threshold of 85%, but belonging to moderate suspicion). Since more than 30 consecutive frames meet the condition of "high probability of malignancy but quality control failure", the system triggers a level 3 alert instruction.

[0081] Endoscopic monitor display: A scrolling blue bar with white text appears at the top of the screen: "Image quality does not meet diagnostic requirements. Please rinse / absorb before reassessment." At this time, the lesion classification results are not displayed on the screen; the physician can see the red warning arrow for the artifact index in the quality control panel.

[0082] t=3min0s: Following instructions, the physician flushed the gastric mucosa surface with warm water through the endoscope's working channel before suctioning. After flushing, the mucosa in this area was clearly exposed, and the surface structure was visible. AI reassessed: the artifact index q_pseudoArtifact decreased to 25, and the overall quality score q_overall increased to 82. 45 frames after the quality control indicators returned to normal, the AI ​​reclassified the lesion.

[0083] t=3min4s: Lesion identification branch output update: The malignancy probability increases from 78% to 91% (exceeding the high confidence threshold of 85%), corresponding to the category "early gastric cancer (type IIc)", with a benign probability of 9%. At this point, if the malignancy probability exceeds 85% for more than 15 consecutive frames and all quality control indicators meet the standards, the system triggers a second-level alert instruction.

[0084] Endoscopic monitor display: A red rectangular border appears outside the suspicious lesion area, with a red halo flashing at a frequency of 1Hz around the outer edge of the border. An intermittent beeping sound is also emitted. The information bar at the top of the screen displays: "Suspicious lesion detection: 91% probability of malignancy, recommendation: Possible early gastric cancer (type IIc)." Based on the prompts, the physician performed multi-angle, multi-mode observation of the area (white light, NBI narrow-band imaging, indigo carmine staining), and properly acquired and archived distant, mid-range, and close-up images.

[0085] t=3min30s: When the system first triggers the second-level reminder command, it automatically generates an emergency consultation request package, which includes: A 4K screenshot of the frame with the highest probability of being malicious (the frame with a 91% probability); Enlarged view of the lesion area (512×512); Quality control data snapshot: q_pseudoArtifact=25,q_deviation=18,q_distention=75,q_overall=82; Patient desensitization information: 55 years old, male, chief complaint, no family history.

[0086] The request packet was pushed to the remote center of Hospital B via the 5G network. Professor Zhang, the gastroenterologist on duty at Hospital B that day (Associate Chief Physician, with more than 5,000 endoscopic procedures performed), received the push notification via his mobile app within 3 minutes. After reviewing the images and quality control data in the consultation packet, he remotely confirmed the AI's judgment—the lesion was consistent with the endoscopic characteristics of early gastric cancer (irregular boundaries, disordered microvessels on the surface) and recommended immediate targeted biopsy.

[0087] t=4min0s: The physician at Hospital A received Professor Zhang's remote advice, performed targeted biopsies at 4 points in the AI-marked area, and completed the subsequent standard gastroscopy procedure. The total examination time was 8 minutes and 30 seconds, and all areas to be examined were observed.

[0088] Postoperative assessment: The system generated an operation report for this examination: coverage integrity score 95 (pyloric area observation time slightly shorter than the lower limit of the standard), operation standardization score 92, image quality retention score 88 (mainly lower scores during the mucus adhesion stage). Lesion detection anastomosis rate: Suspicious lesions detected and suggested by AI were completely consistent with the physician's independent judgment and remote confirmation by experts.

[0089] One week post-surgery, the pathology report revealed severe dysplasia of glandular tissue in the greater curvature of the gastric body, with focal carcinoma (moderately differentiated adenocarcinoma, pT1a stage), and negative surgical margins. Due to early detection, the patient only required endoscopic submucosal dissection (ESD), avoiding surgery. The patient was discharged 5 days post-surgery and has had good follow-up results.

[0090] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting abnormal digestive endoscopy videos, characterized in that, Includes the following steps: Step S1: Receive the ultra-high-definition endoscopic video stream and corresponding patient basic data collected by the endoscope host in real time via the 5G communication network; Step S2: Deframe the video stream into multiple endoscopic images, perform image quality assessment on each endoscopic image, the assessment including at least brightness uniformity detection, focus sharpness scoring, and determination of whether the effective diagnostic field of view is met, and generate corresponding quality labels. Step S3: Call the multi-task deep learning model built into the cloud server to perform two-dimensional parallel analysis on the quality-judged endoscopic images. The multi-task deep learning model includes a shared feature extraction network, a quality detection output branch, and a lesion recognition output branch. Step S4, the dual-dimensional analysis includes: the quality detection output branch further generates refined quality control indicators including artifact index, lesion deviation from center distance, and mucosal extensibility score based on the quality label and extracted image features; the lesion identification output branch identifies whether there are gastrointestinal mucosal lesion features in the image, and if there are lesions, the lesion area is segmented and a classification probability of benign or malignant tendency is given based on the extracted deep features; Step S5: Perform time-series fusion judgment on the quality control indicators and lesion classification results of multiple consecutive frames of images. When a preset abnormal situation is judged, a graded prompt instruction is generated. Step S6: Through the edge computing node, execute the corresponding level of real-time reminder operation on the display interface of the endoscope host according to the graded prompt instruction. The real-time reminder operation includes flashing warning box, superimposed label, or voice prompt.

2. The method according to claim 1, characterized in that, The multi-task deep learning model was trained using the following method: We collected and labeled sample images from physicians with experience of more than 500 endoscopic procedures, covering various endoscopic modes and models. Senior physicians performed overall annotation, single lesion annotation, and feature annotation on the sample images according to the preset annotation principles to establish a training set; Quality control-related annotations are used as supervisory information for the quality detection output branch, and lesion nature and location annotations are used as supervisory information for the lesion identification output branch. A joint loss function is designed to train the shared feature extraction network and the two branches end-to-end. The joint loss function L_total=α*L_quality+β*L_lesion, where L_quality includes the regression loss of the quality control index, L_lesion includes the loss of lesion classification and segmentation, α and β are dynamically adjusted weights, and when the quality control index is worse than the threshold, α is increased and the classification confidence of the lesion output branch for this frame is reduced.

3. The method according to claim 1, characterized in that, The timing fusion determination in step S5 includes: When the artifact index of several consecutive frames is higher than the threshold or the lesion deviates from the center by more than the preset distance, it is judged as an abnormal operation quality and a first-level reminder instruction is generated. When the lesion classification probability of a single frame exceeds the high confidence threshold, and all the fine-grained quality control indicators corresponding to this frame meet the standards, it is judged as a highly suspicious lesion and a second-level reminder instruction is generated. When the lesion classification probability of multiple consecutive frames exceeds the high confidence threshold but any quality control indicator fails to meet the standard, the judgment is temporarily suspended, and a third-level reminder instruction is generated to only prompt the operator to adjust the field of view or rinse. The lesion classification judgment is re-performed after the quality control indicator returns to the standard.

4. The method according to claim 1, characterized in that, Step S6 is followed by: When the lesion classification result indicates malignancy and the confidence level remains above the threshold, the system automatically generates an emergency consultation request package with keyframe screenshots, magnified images of the lesion area, and snapshots of quality control indicators, and pushes this request package to the remote center of the intelligent digestive endoscopy platform through the 5G communication network.

5. The method according to claim 1, characterized in that, The image quality determination process in step S2 further includes: The system detects and marks surface deposits and reflective areas in the image, and uses the detection results as input parameters for the artifact index. When the artifact index exceeds the standard, the system inserts a "wash / vacuum" suggestion into the quality label and triggers a corresponding audio-visual alert.

6. A detection system for abnormal digestive endoscopy videos, using the method described in any one of claims 1 to 5, characterized in that, include: The 5G high-speed communication module is used to enable real-time bidirectional transmission of ultra-high-definition images and medical data between hospitals of different levels across regions; The video frame decoding and quality pre-inspection unit, installed in an edge computing server or local workstation, is used to perform the image quality judgment processing and generate preliminary quality labels. The cloud-based multi-task inference and quality control unit is deployed in the remote center of the "5G+Endoscopic Clinical Intelligent Information Platform". It carries the multi-task deep learning model and performs synchronous fine-grained quality control analysis and lesion abnormality detection on the uploaded image stream. The real-time feedback terminal, embedded in the display interface of the endoscope host, is used to receive graded prompts from the cloud or edge device and render the corresponding warning graphics and text. The remote consultation request generation unit is used to automatically encapsulate an emergency consultation request package and push it to the remote center when the triggering conditions are met.

7. The system according to claim 6, characterized in that, The instant feedback terminal includes a tiered alert module, which renders alerts according to the following rules: In response to the first-level alert command, a yellow pulse indicator and distance adjustment suggestions will be displayed at the edge of the screen; In response to the second-level alert command, a red bounding box is drawn in the center of the field of vision for the suspicious lesion area and a flashing effect is superimposed, while a continuous alert sound is emitted; In response to the Level 3 alert, the text "Please rinse / suck and then evaluate" scrolls across the top of the screen, and the display of lesion classification results is disabled.

8. The system according to claim 6, characterized in that, The system also includes an assessment interface that interfaces with the "five steps and seven evaluations" training system. The assessment interface is used to compare the operator's endoscopic operation trajectory, quality control index scores and lesion identification results detected by the system with the standard path, and output an operation skill assessment report and complication risk prediction value.

Images