Endoscopic image optimization, lesion reconstruction labeling and quantification method based on U-Net / ResNet fusion algorithm

Abstract

The invention discloses an endoscope image processing system based on a U-Net / ResNet fusion algorithm, which relates to the field of medical instrument image processing and is characterized by comprising a feature fusion processing module, a lesion quantitative analysis module, a three-dimensional reconstruction rendering module, a full-scene integration module and a lesion progress tracking module to form an input-processing-output data closed loop. The feature fusion module adopts a U-Net / ResNet 50 customized architecture and is combined with a differential enhancement strategy, and the detection rate of minimal lesions smaller than 5 mm reaches 92%; the lesion quantification module is dynamically calibrated through an equipment parameter library and a metal marking ball, calculation is carried out according to an exclusive formula, the error is smaller than or equal to 3%, the three-dimensional reconstruction module is accelerated and optimized through blocks, and real-time rendering larger than or equal to 15 fps is achieved through a common workstation; pACS docking and lightweight off-line deployment are supported, and the model is compressed to be within 50 MB. The system can be migrated to various endoscopes such as gastroscopes, the lesion segmentation Dice coefficient is larger than or equal to 0.89, and the diagnosis precision is improved.

Description

Technical Field

[0001] This invention relates to the field of medical device image processing technology, specifically to the interdisciplinary application of artificial intelligence with digestive and respiratory endoscopy equipment. In particular, it relates to a deep learning-based system for precise processing and quantification of endoscopic images. The system is applicable to various endoscopic techniques, including colonoscopy, gastroscopy, bronchoscopy, and surgical endoscopy. It can enhance the identification of minute lesions under seven pathological states: normal, mucosal congestion, mucosal edema, intestinal tuberculosis, intestinal polyps, ulcers, and tumors; accurately measure lesion area / volume; perform three-dimensional visualization reconstruction; and flexibly deploy in multiple scenarios. It is widely used in disease screening, preoperative assessment, efficacy tracking, and doctor-patient communication in medical institutions at all levels. Background Technology

[0002] Endoscopic examination is the "gold standard" for diagnosing diseases of hollow organs such as the digestive and respiratory tracts, and its core relies on the doctor's subjective interpretation of dynamic video images. With the development of artificial intelligence technology in the field of medical imaging, various auxiliary diagnostic tools have gradually entered the clinical practice, but the existing technologies generally have four major bottlenecks: (1) Single function: Most products only focus on lesion detection or image enhancement, and lack a closed-loop design for the entire process from recognition, quantification, reconstruction to output. (2) Insufficient accuracy: The retrospective analysis of this invention shows that traditional methods rely on manual estimation or simple pixel statistics, and do not fully consider the differences in equipment parameters. The average error in measuring ulcer area and tumor volume exceeds 15%. (3) Poor practicality: Three-dimensional reconstruction often requires the support of high-end GPU clusters, which is difficult to popularize in primary hospitals. Moreover, the models are complex and cannot be rendered in real time on ordinary workstations. (4) Difficult to implement: The system modules are fragmented, do not support PACS docking, and do not have a lightweight offline mode, making it difficult to adapt to the workflow of different levels of hospitals.

[0003] Table 1 shows a comparison of the compatibility between existing technologies and clinical needs:

[0004] Table 1. Comparison of Clinical Needs and Adaptability to Existing Technologies

[0005] Clinical needs dimension Current state of technology Demand satisfaction core bottleneck Detection of minute lesions (<5mm) The general enhancement algorithm is insufficient in improving the contrast of congestion / edema. ≤65% Lack of disease-specific enhancement logic Pathological Quantification Accuracy Manual estimation or rough counting, without a calibration mechanism. Average error ≥15% Lack of standardized quantification system Practicality of 3D Reconstruction It requires a dedicated server, has a low frame rate, and lacks clear organizational hierarchy. Coverage rate at the grassroots level <10% High hardware requirements and poor structural adaptability Scene adaptation flexibility The module is independent, cannot be integrated with PACS, and has no offline deployment capability. Clinical efficiency improvement <20% Fragmented functionality and high deployment costs

[0006] Therefore, there is an urgent need to construct an endoscopic image processing system that integrates "intelligent enhancement, precise quantification, efficient 3D reconstruction, and full-scene integration" to achieve translation from scientific research to clinical application. Based on the above needs, this invention provides the following technical solution: Summary of the Invention

[0007] The purpose of this invention is to overcome the core defects of existing endoscopic image processing technologies, such as high missed detection rate of small lesions (detection rate of lesions <5mm ≤65%), unreliable lesion quantification results (average measurement error of traditional methods ≥15%), large delay and high hardware threshold of 3D reconstruction (requiring dedicated servers, with a grassroots adoption rate of <10%), and poor system deployment flexibility (fragmented modules, not supporting PACS integration and offline mode). This invention provides an endoscopic image optimization, lesion reconstruction annotation and quantification method based on a customized fusion architecture of U-Net and ResNet50.

[0008] This method constructs a closed-loop data system that links five modules: feature fusion processing, lesion quantification analysis, 3D reconstruction rendering, full-scene integration, and lesion progression tracking. It can adapt to seven pathological states, including normal, mucosal congestion, mucosal edema, intestinal tuberculosis, intestinal polyps, ulcers, and tumors. It achieves the goals of precise enhanced identification of small lesions, high-precision quantification of lesion area / volume (error ≤3%), real-time 3D reconstruction on ordinary workstations (frame rate ≥15fps), and flexible deployment in multiple scenarios. Ultimately, it improves the accuracy of clinical diagnosis, operational efficiency, and patients' understanding of their condition, aligns with the hierarchical medical system policy, and promotes the transformation and application of endoscopic imaging AI technology from scientific research to primary care.

[0009] The core of the technical solution of this invention lies in the synergistic linkage of five core modules. The detailed technical implementation of each module is as follows: The system as a whole flows in the order of "image input → feature processing → quantization calculation → 3D reconstruction → result output". The five core modules work together in a coordinated manner, and the overall architecture is as follows: Figure 1 As shown. The system modules are as follows: (1) Feature fusion processing module: U-Net / ResNet50 customized fusion network is adopted, combined with LAB space A channel stretching and Canny edge detection to achieve pixel-level accurate segmentation in 7 pathological states; (2) Lesion quantification analysis module: The equipment parameter library and metal marker ball dynamic calibration are introduced, combined with SIFT frame alignment and pixel / voxel counting method to achieve high-precision quantification of area and volume (error ≤3%); (3) Full scene integration module: Supports DICOM / PNG format input, provides two deployment modes: PACS docking and lightweight offline, outputs an integrated report of "original image + algorithm-enhanced endoscopic image + three-dimensional annotation map", and is compatible with Excel export; (4) Extended function module: A lesion progression tracking module is added, which uses cosine similarity to compare historical and current image feature vectors to quantify the disease evolution trend and assist in long-term follow-up.

[0010] The parameter-effect relationship is explained as follows: For the Canny threshold selection, the Canny threshold for mucosal edema lesions is set to low 50 / high 150 because the gray-level gradient of the lesion edge is between 40 and 160. This threshold can remove more than 85% of noise pixels (such as mucus interference) while retaining 92% of the lesion edge pixels. Testing showed that the edge detection accuracy at this threshold is 15% higher than the conventional threshold (40 / 140). Regarding the voxel block size selection: using 32×32×32 voxel blocks is the optimal choice to balance reconstruction speed and accuracy; while 24×24×24 voxel blocks increase speed by 10%, accuracy decreases by 3%; 40×40×40 voxel blocks increase accuracy by 2%, but decrease speed by 25%, which cannot meet real-time requirements.

[0011] Detailed technical implementation of each module:

[0012] (1) Feature Fusion Processing Module This module adopts a deep fusion architecture of U-Net and ResNet50 to solve the problem of poor adaptability of general algorithms to 7 pathological states. The network structure is designed as follows: The encoder uses ResNet50 as the backbone network. The conv2_x layer extracts high-frequency edge texture features (1 / 4 resolution original image), and the conv3_x to conv5_x layers extract low- and medium-frequency background structure features (1 / 8, 1 / 16, and 1 / 32 resolution original images) in sequence. All high- and low-frequency features are unified into 256 channels by 1×1 convolution and then stitched along the channel dimension. Decoding stage: U-Net uses transposed convolution (3×3, stride 2) to upsample step by step. Each layer is added and fused with the corresponding scale encoder feature map pixel by pixel. Output layer: 1×1 convolution + Sigmoid activation function to generate a single-channel lesion probability mask.

[0013] (2) Pathological state-specific enhancement strategies implement differentiated enhancement for 7 pathological states, including normal mucosa and mucosal congestion. The specific enhancement methods and parameter settings are shown in Table 2:

[0014] Table 2. Image Enhancement Strategies for 7 Pathological States

[0015] Pathological state Enhancement methods Parameter settings normal mucosa No additional enhancements, original texture features preserved. — Mucosal congestion Convert RGB to LAB color space, stretch grayscale of channel A. 50–200 (determined based on grayscale distribution of 150 images, with a 40% improvement in contrast). Mucosal edema Feature fusion followed by Canny edge detection enhances boundary definition. Canny threshold: low 50 / high 150, 3×3 morphological closing operation for noise reduction (removing 85%+ noise) intestinal tuberculosis LAB space L-channel stretching + edge detection L channel 40-210, Canny threshold 45 / 145 Intestinal polyps HSV spatial S-channel enhancement + contour extraction S-channel gain 1.3x, Canny threshold 55 / 155 ulcer Multi-channel fusion (RGB+LAB) + area threshold filtering <![CDATA[Fusion weight 1:1, minimum detection area 0.5mm 2 > tumor High-frequency feature weight enhancement + 3D edge enhancement High-frequency weight 0.6, superimposed with 3×3×3 voxel edge detection

[0016] To accurately evaluate the segmentation performance, this invention uses the Dice coefficient to measure the similarity between the model's output segmentation results and expert annotations. The calculation formula is as follows: Where A represents the lesion mask region output by the model, and B represents the standard lesion region annotated by two associate chief physicians in a double-blind manner (annotation consistency Kappa ≥ 0.85). The Dice coefficient typically ranges from [0,1]. A Dice coefficient ≥ 0.90 indicates that the segmentation result is highly consistent with the manual annotation and has clinical application value. Experimental results show that after adopting the above enhancement strategy, the model segmentation Dice coefficient was ≥ 0.89 in all seven pathological states, and the detection rate of small lesions < 5 mm increased to 92%, an improvement of 27 percentage points compared to traditional methods.

[0017] (2) Pathological Quantitative Analysis Module This module establishes a three-in-one quantitative closed loop of "calibration-calculation-early warning" to ensure the reliability of measurement results under 7 pathological conditions. Equipment Parameter Calibration Unit The equipment parameter calibration unit pre-stores a database of pixel spacing of mainstream endoscopic equipment. The specific preset parameters are shown in Table 3, which can directly match different types of endoscopic equipment:

[0018] Table 3. Preset Pixel Spacing Parameters for Mainstream Endoscopic Devices (Data based on official technical manuals and clinical calibration for each device)

[0019] Endoscopic equipment model Preset pixel spacing (mm / pixel) Applicable endoscope types Olympus CV-290 0.05 Colonoscopy / Gastroscopy Pentax EPK-i7000 0.048 colonoscopy Fujifilm EPX-4450 0.052 Bronchoscope Mindray H160 0.05 Gastroscopy / colonoscopy

[0020] Dynamic calibration: Place a standard metal marker ball with a diameter of 5mm, and the actual pixel pitch of the device is... Calculate using the following formula: (1), where, The actual diameter of the standard metal marker ball is 5mm. The pixel diameter of the marker sphere in the image.

[0021] The inter-frame alignment calculation unit uses the SIFT algorithm to extract mucosal texture points and marker sphere feature points (≥50 stable points per frame), RANSAC to remove mismatches (error > 2 pixels), and calculates the homography matrix to complete translation / rotation correction; if the axial sampling interval between adjacent image frames is unknown, the interval is estimated by using double markers with a spacing of 2 mm.

[0022] Quantitative calculation formula:

[0023] Ulcer area: Among them, the total number of lesion pixels The contours are detected and statistically analyzed using the cv2.findContours function.

[0024] Tumor / polyp volume: ,in, Obtained by stacking 3D masks using SimpleITK. This represents the axial sampling interval.

[0025] To further improve the accuracy of lesion quantification, the following two operational guidelines must be met when measuring tumors: First, control the distance between the fiberoptic endoscope lens and the surface of the tumor, which should be kept stable within the range of 10-12 mm. This distance range minimizes measurement errors. Distance calibration can be achieved through the scale on the endoscope itself or by connecting an external auxiliary ruler. Second, ensure the measurement angle. Ensure that the lens is directly facing the maximum diameter of the tumor to avoid "perspective shortening" errors caused by lens tilt. If the angle deviation is difficult to adjust, the measurement conditions can be optimized by changing the patient's position or fine-tuning the angle of the endoscope.

[0026] The abnormal warning mechanism will automatically trigger a prompt when the following conditions occur: ulcer area > 50mm 2 Tumor volume > 2000 mm 3 →A pop-up window prompts "Please check image integrity"; the deviation between the dynamic calibration pixel spacing and the preset value is >10% → the secondary calibration process is initiated. Test results: Compared with pathological slides, the measurement error is ≤3% under all 7 pathological conditions.

[0027] (3) 3D Reconstruction and Rendering Module

[0028] Specifically optimized for the tubular structure of the colorectal region, it balances accuracy and efficiency across seven pathological states. The tissue type differentiation strategy clearly distinguishes between diseased tissue and normal intestinal wall layers. The system is configured with a dual-threshold transfer function, with specific parameters shown in Table 4, enabling visual differentiation of different tissues.

[0029] Table 4. Parameter Table for Differentiating Tissue Types in 3D Reconstruction

[0030] Organization type CT value range (HU) transparency Display effect Intestinal wall tissue -200~-51HU 0.6 Semi-transparent gray, with visible stratification (determined based on density distribution of 200 normal tissue samples). Diseased tissue (congestion / edema / polyps / ulcers / mass) 20～100 1.0 Opaque white with clear edges normal mucosa -50～19 0.8 Translucent light gray, distinguishing it from lesions.

[0031] The visualization was scored by 10 experts with the title of associate chief physician or above based on clinical visualization effect (out of 10), with an average score of 9.5. This visualization effect clearly presents the boundary relationship between the tumor and surrounding tissues, helping to accurately determine the depth of tumor infiltration (T-staging). Rendering speed optimization adopted 32×32×32 voxel block processing, supporting parallel computing; early ray termination was implemented for three consecutive lesion-free voxel blocks to ensure real-time performance. Reconstruction time was reduced from 12 seconds to 5 seconds, improving efficiency by 58.3%.

[0032] Real-time performance verification under Intel i5-10400+GTX1650+8GB memory environment: reconstruction frame rate ≥15fps, video memory usage not exceeding 1.8GB; the maximum diameter deviation of the tumor measured by this module is ≤1.2%, and the infiltration depth deviation is ≤2.6%, which meets the preoperative planning requirements.

[0033] (4) The full-scenario integration module enables seamless connection across platforms, multiple terminals, and the entire link, adapting to medical institutions of different levels.

[0034] Input / output compatibility: Supports PNG, JPG, and DICOM 3.0 input formats; automatically identifies device model and loads corresponding RGB gain and resolution parameters; outputs "three..." Figure 1 The report includes the original image, the algorithm-enhanced endoscopic image (red border + pathology type label), and a three-dimensional parameter annotation map (blue annotation of maximum diameter, area / volume, and depth of invasion).

[0035] The parameter table for compatibility with multiple endoscopes is as follows:

[0036] Table 5. Multi-endoscopic fitting parameter table

[0037] Endoscopic type Adapt to pathological conditions Core parameter adjustment Verification effect Gastroscopy (such as Olympus CV-290) Gastric mucosal congestion, gastric ulcer, gastric polyp, normal mucosa <![CDATA[1. Enhancement strategy: For gastric ulcer, RGB and HSV channels are fused with a weight ratio of 1:1.2, and the minimum detection area is 0.4 mm 2 ; 2. Three-dimensional reconstruction: The CT value of the gastric wall tissue is -180 to 20 HU (transparency 0.7), and the CT value of the lesion tissue is 30 to 110 HU (transparency 1.0); 3. Pixel pitch calibration: The preset value is 0.049 mm / pixel]]> Quantitative error ≤2.8%, detection rate of minute lesions (<5mm) ≥91%. Bronchoscope (such as Fujifilm EPX-4450) Bronchial mucosal congestion, polyps, masses, normal mucosa 1. Enhancement Strategy: Polyps were enhanced with 1.5x gain in the HSV spatial S channel and a Canny threshold of 60 / 160; 2. 3D Reconstruction: Bronchial wall CT value -220 to -30 HU (transparency 0.65), lesion tissue CT value 25 to 95 HU (transparency 1.0); 3. Pixel Spacing Calibration: Preset value 0.053 mm / pixel. Quantization error ≤3.1%, reconstruction frame rate ≥13fps (ordinary workstation)

[0038] The system is designed with dual deployment modes to meet the needs of medical institutions at different levels. The differences in the technical implementation and applicable scenarios of the two modes are shown in Table 6.

[0039] Table 6 Comparison of Technical Parameters for Dual Deployment Modes of the System

[0040] model Technical Implementation Applicable Scenarios PACS integration mode Develop a C-MOVE / C-STORE interface based on DCMTK, and configure the AE title "AI_ENDOSCOPE", server IP (e.g., 192.168.1.100), and port 104. Tertiary hospital, networked environment Lightweight offline mode TensorRT INT8 quantization compression was used, and quantization calibration was performed using a calibration set (100 images of various pathological states). Key dimensions of the lesion feature vectors were preserved (256 core features were not lost), resulting in a model size of ≤50MB, an installation package of ≤200MB, and a startup time of <30 seconds. Primary hospitals, without internet access

[0041] The data export function supports exporting quantitative data to an Excel spreadsheet (fields: Patient ID, Examination Time, Pathology Type, Lesion Location, Area / Volume, Measurement Error), "three Figure 1 The results of the "body" test are exported as PNG format with a resolution of 1920×1080 for easy consultation.

[0042] (5) Extended Functional Modules: The lesion progression tracking module calls the lesion feature vector output by the feature fusion module and uses the cosine similarity algorithm to compare the differences between historical and current image features; Quantitative indicators: lesion size change rate (accuracy 0.1%), morphological evolution trend (cosine similarity ≥0.8 indicates stable lesion, <0.8 indicates significant change). Clinical Application: For chronic lesions such as intestinal tuberculosis and ulcers, it can track changes before and after treatment, provide data support for adjusting the treatment plan, and avoid overtreatment.

[0043] The core innovations of this invention are: (1) pioneering a customized AI full-process system for 7 pathological states: constructing a U-Net / ResNet50 customized fusion architecture, optimizing enhancement strategies for different pathological states, achieving a detection rate of 92% for small lesions <5mm (previous technology ≤65%); (2) establishing a closed-loop quantitative system of "hardware-algorithm-verification": integrating equipment parameter library and dynamic calibration of metal marker balls, with area / volume measurement error ≤3% for 7 pathological states, replacing subjective estimation; (3) breaking through the hardware barrier of 3D reconstruction: optimizing the dual threshold transfer function and block acceleration for colorectal structure, achieving real-time rendering of ≥15fps on ordinary workstations, and improving the accuracy of preoperative T staging to ≥84% (traditional 65%); (4) innovating a full-scene adaptation solution: supporting PACS docking and lightweight offline dual deployment mode, compressing the model to ≤50MB to adapt to primary hospitals; the core technology can be transferred to various endoscopic scenarios such as gastroscopy and bronchoscopy, with a lesion segmentation Dice coefficient ≥0.89.

[0044] This invention achieves breakthroughs in multiple dimensions, including the detection of minute lesions and the quantification of lesions, through innovative designs such as algorithm customization and quantification. The comparison results with existing technologies in each dimension are shown in Table 7.

[0045] Table 7 Comparison of the beneficial effects of the present invention and the prior art

[0046] Dimension Current technical performance This invention demonstrates Increase Minimal lesion detection rate ≤65% 92% ↑27 percentage points Quantization error ≥15% ≤3% ↓80% 3D Reconstruction Frame Rate <10fps (dedicated equipment) ≥15fps (standard workstation) Speed ​​increased by 50%, hardware requirements lowered Singleton processing time 30 minutes (manual labor + tools) 5 minutes (fully automatic) Efficiency increased by 83.3% Patient's understanding of their condition 35% 95% ↑60 percentage points Positive detection rate at the grassroots level — Increase by 25% Reduce missed diagnoses Postoperative complication rate 8% 3% A decrease of 62.5% (observation results after considering this system solution).

[0047] This system aligns with the national hierarchical medical system policy and the strategy of replacing outdated equipment with domestically produced ones, and has significant social benefits and promotional value.

[0048] The core technology protection boundary of this invention is defined as follows: (1) Algorithm layer: U-Net / ResNet50 exclusive fusion logic for endoscopic lesions (including the fusion method of ResNet50 high and low frequency feature layer extraction and U-Net), and customized enhancement strategies for 7 pathological states (such as stretching of LAB space A channel 50~200 range for congested lesions, edge detection of Canny threshold 50 / 150 for edematous lesions, etc.); (2) Quantization layer: dynamic calibration method based on endoscopic equipment parameter library and 5mm diameter metal marker ball (including calibration formula of "pixel spacing = 5mm / marker ball pixel diameter"), and ulcer area ( ), tumor / polyp volume ( (3) System layer: “Feature fusion - lesion quantification - three-dimensional reconstruction - full scene integration” four major module collaborative architecture and data flow logic, PACS docking DICOM C-MOVE / C-STORE protocol implementation scheme, lightweight model compressed to ≤50MB by TensorRT, and “original image + enhanced image + three-dimensional annotation map” three Figure 1"Body" output mode; (4) Extension layer: core technologies include adaptation methods for multiple endoscope types such as gastroscopy and bronchoscopy (achieved by adjusting parameters such as color space and edge threshold), and disease progression tracking algorithm based on cosine similarity. Attached Figure Description

[0049] Figure 1 System overall architecture diagram;

[0050] Figure 2 Technology hierarchy and scenario expansion diagram;

[0051] Figure 3 Functional flow and module association diagram. Detailed Implementation

[0052] Example 1:

[0053] 1. System hardware and software environment configuration

[0054] 1.1 Hardware Configuration: Core Processing Terminal: Intel i7-12700K (12 cores, 20 threads, base frequency 3.6GHz), 32GB DDR4 3200MHz memory, NVIDIA RTX3060 12GB graphics card, 1TB SSD (read speed 3500MB / s); Endoscopic Equipment: Olympus CV-290 electronic colonoscope (1920×1080, 30fps, CMOS sensor), Mindray H160 electronic gastroscope; Auxiliary Equipment: 5mm diameter standard metal marker ball (error ±0.01mm), PACS server (Intel Xeon E3-1230 v5, 16GB memory, 10TB SATA hard drive).

[0055] 1.2 Software Dependencies: Operating System: Windows 11 Professional 64-bit (22H2); Programming Language: Python 3.8.10 (64-bit); Core Libraries and Versions: TensorFlow 2.8.0 (GPU version, supports CUDA 11.2), OpenCV 4.5.5 (including contrib module), SimpleITK 2.1.1, DCMTK 3.6.6 (compiled with Visual Studio 2019), TensorRT 8.4.3.1, PyQt5 5.15.6.

[0056] Example 2:

[0057] 1. Feature fusion module training and validation

[0058] 1.1 Dataset Construction Data Source: Colonoscopy images from January 2023 to October 2025 at Baodi Hospital Affiliated to Tianjin Medical University (ethical approval obtained, informed consent obtained from patients, ethics approval number: BDYY-2024-LC-005); Data Composition: A total of 1000 cases (200 normal cases, 150 cases of mucosal congestion, 150 cases of mucosal edema, 50 cases of intestinal tuberculosis, 200 cases of intestinal polyps, 150 cases of ulcers, and 100 cases of tumors), labeled by two associate chief physicians using a double-blind method (labeling consistency Kappa≥0.87); Preprocessing: Divided into training set (700 cases), validation set (200 cases), and test set (100 cases) in a 7:2:1 ratio, with a uniform resolution of 1024×512, random flipping, and brightness enhancement of ±10%;

[0059] 1.2 Network training parameter optimizer: Adam (initial learning rate 0.001, decaying to 0.9 times the previous rate every 5 epochs); loss function: weighted sum of Dice loss function and cross-entropy loss function (weight ratio 1:1); batch size 16, training epochs 50, early stopping if the validation loss does not decrease for 5 consecutive epochs.

[0060] 1.3 Model performance test set Dice coefficient 0.92±0.03, detection rate of minute lesions ≥88% in 7 pathological states, single frame processing time 0.3 seconds, meeting real-time requirements.

[0061] Example 3:

[0062] 1. Quantization module calibration and calculation verification

[0063] The calibration process is as follows: ① Device docking: Connect the Olympus CV-290 endoscope device, and the system automatically loads the preset pixel spacing of 0.05mm / pixel; ② Marker shooting: Take a picture of a standard metal marker ball with a diameter of 5mm through the endoscope and record its pixel diameter in the image; ③ Calculation calibration: The pixel diameter of the marker ball is measured to be 100 pixels. Calculate the actual spacing according to formula (1) = 5mm / 100 pixels = 0.05mm / pixel. The deviation from the preset value is 0%, and the calibration is passed.

[0064] Quantization example ulcer: Total number of pixels 20000, area calculated according to formula (2) = 20000 × (0.05) 2 =50mm 2 (Pathological measurement: 51 mm) 2 (Error 1.96%); Tumor: Total number of voxels 64000, axial sampling interval 0.5mm, volume calculated according to formula (3) = 64000 × (0.05) 2 ×0.5=80mm 3 (Pathological measurement: 81 mm) 3 (Error 1.23%).

[0065] Example 4:

[0066] 3D Reconstruction Performance Test

[0067] In an environment with an Intel i5-10400+GTX1650+8GB memory, loading 20 frames of 1024×512 images: the reconstruction frame rate was stable at 15fps, and there was no stuttering during rotation and dissection; the maximum diameter of the tumor was measured to be 8.2mm (the actual pathological measurement was 8.3mm, with an error of 1.2%), and the infiltration depth was 3.8mm (the pathological measurement was 3.9mm, with an error of 2.6%).

[0068] Example 5:

[0069] 1. Full-scenario deployment and output verification

[0070] PACS integration settings: AE title "AI_ENDOSCOPE", IP [192.168.1.100](192.168.1.100), port 104; Input patient ID → Execute C-FIND query → Download images via C-MOVE → After processing, return the DICOM SR report via C-STORE. Lightweight deployment with TensorRT INT8 resulted in a quantized model size of 48MB. 50 images were processed continuously in a network-free environment, and the results were saved to "D:\Endoscopy Processing Results". Exporting Excel and PNG files showed no data loss.

[0071] Example 6:

[0072] To verify the applicability of the system in different clinical scenarios, a multi-center clinical trial was conducted. The data volume and key results for each scenario are shown in Table 8.

[0073] Table 8 Summary of Multicenter Clinical Validation Results

[0074] Scene Data volume Key Results Preoperative T-staging in tertiary hospitals The scenario data volume is 400 cases (200 cases each from Tianjin Medical University Affiliated Baodi Hospital and a certain tertiary-level Class A hospital). The accuracy rate was 84.7% (compared to 65% with traditional methods), and the postoperative complication rate decreased from 8% to 3% after combining these methods. Primary screening at the grassroots level 300 cases (150 cases each from a secondary hospital in Baodi District, Tianjin and a primary hospital in Baodi District, Tianjin) The positive detection rate increased by 26%, reducing missed diagnoses by 25 cases and identifying 15 early-stage tumors. 7 pathological states adapted 700 cases (100 cases each) The quantization error is ≤3% in each state, and the segmentation Dice is ≥0.89. Ulcer progression tracking One case (ulcerative colitis) <![CDATA[The area before treatment was 120 mm 2 , and 80 mm after 3 months 2 . The cosine similarity was 0.72, indicating improvement and guiding drug reduction]]>

[0075] All validation data came from retrospective studies approved by the hospital's ethics committee (ethics approval numbers: BDYY-2023-LC-001, BDZYY-2024-LC-003). All patients signed informed consent forms, and the data were de-identified. All images were double-blindly annotated by two experts with the title of associate chief physician or above, with an annotation consistency Kappa value ≥0.87, ensuring the reliability of the gold standard for annotation.

[0076] Example 7:

[0077] Gastroscopy Scene Adaptation Verification

[0078] Equipment: Mindray H160 electronic gastroscope (1920×1080 resolution), Intel i3-10100 processor + GTX1630 graphics card + 8GB memory;

[0079] Data: Images of 50 patients with gastric ulcers (approved by the Ethics Committee of Baodi Hospital Affiliated to Tianjin Medical University, ethics approval number: QLYY-2024-LC-003), labeled in a double-blind manner by two chief physicians of gastroenterology;

[0080] Procedure: Load endoscopy-specific parameters (HSV channel fusion weight 1:1.2, CT value range -180~20HU), and execute the "calibration-segmentation-quantization-reconstruction" process;

[0081] Results: The ulcer area measurement error was 2.7%, the 3D reconstruction frame rate was 14fps, and the accuracy of lesion edge annotation was 93%, which met the clinical diagnostic requirements.

[0082] Example 8:

[0083] Bronchoscopy scenario adaptation verification

[0084] Equipment: Fujifilm EPX-4450 bronchoscope, Intel i5-10400 + GTX1650 + 8GB RAM;

[0085] Data: Images of 30 patients with bronchial polyps (approved by the Ethics Committee of Baodi Hospital Affiliated to Tianjin Medical University);

[0086] Results: Polyp volume measurement error was 3.0%, micropolyps (<3mm) detection rate was 90%, and PACS docking transmission delay was <2 seconds.

[0087] Example 9: Multi-center Data Consistency Verification

[0088] Validation objective: To verify the measurement consistency of this system in different medical institutions, with different endoscopic equipment, and with different operators, so as to ensure the clinical universality of the technical solution.

[0089] Participating institutions: 2 tertiary hospitals (Tianjin Medical University Affiliated Baodi Hospital and a tertiary hospital in Tianjin) and 2 primary hospitals (a secondary hospital in Baodi District, Tianjin and a primary hospital in Baodi District, Tianjin), covering medical levels in different regions.

[0090] Experimental equipment:

[0091] Tertiary hospitals: Olympus CV-290 electronic colonoscope, Fujifilm EPX-4450 bronchoscope, Mindray H160 electronic gastroscope;

[0092] Primary care hospitals: Mindray H160 electronic gastroscope, Pentax EPK-i7000 colonoscope;

[0093] Processing terminals: Standardized workstations with uniform configuration of Intel i5-10400 processor + GTX1650 graphics card + 8GB memory to eliminate the impact of hardware differences.

[0094] Experimental data: 50 standardized imaging samples were selected (including 5-8 cases of each of 7 pathological states, of which 40% were small lesions <5mm). The samples were uniformly labeled by a third-party expert team (3 chief physicians) as the gold standard for consistency evaluation.

[0095] Each institution assigned two operators (one senior physician and one junior physician) to perform measurements according to the same operating procedures. Each operator repeated the measurement three times for each of the 50 standardized image samples, recording the area / volume measurement values ​​and the operation time; at the same time, the 3D reconstruction frame rate of each sample was recorded.

[0096] Verification metrics:

[0097] Intragroup correlation coefficient (ICC): assesses the consistency of repeated measurements by the same operator, the consistency of measurements by different operators, and the consistency of measurements by different institutions. ICC ≥ 0.95 is considered excellent. Coefficient of variation (CV): calculates the dispersion of measured values ​​in each scenario. CV ≤ 5% is considered acceptable. Operation time: calculates the time of the entire process from image import to report export to assess clinical efficiency.

[0098] Verification results:

[0099] Repeated measurements by the same operator yielded an ICC of 0.98 and a CV of 1.2%; measurements by different operators yielded an ICC of 0.97 and a CV of 1.8%.

[0100] The ICC was 0.96 and the CV was 2.3% as measured by different institutions. The average operation time was 4.8 minutes (±0.5 minutes). After training, operators in primary hospitals could complete the operation independently, with no obvious learning threshold.

[0101] Conclusion: This system maintains high consistency and stability across different institutions, personnel, and equipment scenarios, meeting the standardized application needs of medical institutions at all levels.

[0102] Example 10: Practical Verification of Lightweight Model in Primary Hospitals

[0103] Verification scenario: A community hospital (without PACS system, without high-end GPU equipment, and with an unstable network) mainly conducts routine gastrointestinal endoscopy screening services.

[0104] Equipment configuration:

[0105] Endoscopic equipment: Mindray H160 electronic gastroscope, Pentax EPK-i7000 colonoscope;

[0106] Processing terminal: Intel i3-10100 processor + GTX1630 graphics card + 8GB memory (common configuration for basic tasks), 1TB hard drive;

[0107] Software environment: Windows 10 Professional 64-bit, with the lightweight offline version of this system installed (installation package size 198MB, startup time 22 seconds).

[0108] Validation period: 3 months (a total of 300 cases of gastrointestinal endoscopy image processing were completed, including 180 cases of colonoscopy and 120 cases of gastroscopy).

[0109] Verification content:

[0110] Compatibility: Test the success rate of importing PNG, JPG, and DICOM 3.0 format images, and the completeness of exporting Excel and PNG format reports;

[0111] Performance metrics: Record 3D reconstruction frame rate and quantization error to assess device compatibility;

[0112] Clinical value: To statistically analyze the positive detection rate and the false negative rate, and to compare the differences with the traditional manual interpretation method;

[0113] Ease of use: A questionnaire survey was conducted to collect the system's ease of use ratings from operators (2 general practitioners and 1 nurse) (out of 10).

[0114] Verification results:

[0115] Compatibility: 100% success rate for image import; no data loss or formatting issues during report export; Excel spreadsheets can be directly used for electronic medical record archiving.

[0116] Performance: The average frame rate of 3D reconstruction is 14.8fps (minimum 13.2fps) (under the hardware configuration of primary hospitals), the average quantization error is 2.9% (≤3% standard), and there are no stutters or crashes;

[0117] Clinical value: The positive detection rate was increased by 26% compared with the traditional manual method, and the false negative rate was reduced from 8.3% to 2.7%. Twelve cases of early polyps and eight cases of microulcers were detected (which were not detected by the traditional method).

[0118] Ease of use: Operators gave it an average score of 8.9, believing that the "automatic parameter loading" and "one-click report generation" functions greatly reduced the difficulty of operation.

[0119] Conclusion: The lightweight offline mode of this system is fully adapted to the hardware conditions and working scenarios of primary hospitals, and can significantly improve the accuracy and efficiency of endoscopic screening at the primary level, meeting the needs of the implementation of the hierarchical medical system policy.

[0120] Example 11:

[0121] Clinical application validation of the disease progression tracking module

[0122] Case Information: The patient was a 45-year-old male diagnosed with ulcerative colitis (left colon involvement), requiring regular follow-up to monitor ulcer healing. Follow-up period: 6 months, with colonoscopies performed at month 0 (before treatment), month 3, and month 6 to collect imaging data.

[0123] Verification process:

[0124] Image import: Import DICOM format images from three examinations into this system and link them to historical data via patient ID;

[0125] Feature comparison: The system automatically calls the lesion progression tracking module to extract the feature vectors of the ulcer area in each examination and uses the cosine similarity algorithm to perform pairwise comparisons;

[0126] Quantitative analysis: Calculate the rate of change in ulcer area and the trend of morphological evolution, and generate a tracking report;

[0127] Clinical decision-making: Adjust treatment plans based on follow-up reports and evaluate the guiding value of the module for clinical intervention.

[0128] Verification results:

[0129] Month 0: Ulcer area 128mm 2 The cosine similarity of the eigenvectors (with the 3rd month) is 0.71;

[0130] Month 3: Ulcer area 82mm 2 The cosine similarity of the eigenvectors (with the 6th month) is 0.92;

[0131] Month 6: Ulcer area 45mm 2 ;

[0132] Treatment adjustments: If the cosine similarity is <0.8 at month 3, it indicates significant improvement in the disease, and the dosage of immunosuppressants should be reduced as prescribed by the doctor; if the cosine similarity is ≥0.8 at month 6, it indicates that the disease is stable, and the current treatment plan should be maintained.

[0133] Clinical feedback: The tracking report visually demonstrates the process of ulcer reduction and morphological repair, avoiding overtreatment. Patients tolerated the treatment well and experienced no serious adverse reactions.

[0134] Conclusion: The disease progression tracking module of this system can accurately quantify the treatment response of chronic diseases, providing objective data support for clinical protocol adjustments, and is especially suitable for diseases that require long-term follow-up, such as intestinal tuberculosis and ulcerative colitis.

[0135] In this invention, the appendix Figure 1The system architecture diagram presents the connections between the five core modules in a modular layout, annotating the core input and output data types of each module (e.g., the "feature fusion processing module" inputs "raw endoscopic images" and outputs "lesion probability masks"). Solid lines with arrows indicate the direction of data flow, clearly demonstrating the closed-loop logic of "image input → feature processing → quantization calculation → 3D reconstruction → result output"; (Attached) Figure 2 The diagram illustrates the technology hierarchy and application scenarios: The upper layer, the "Core Technology Innovation Layer," lists four original technologies, including the U-Net / ResNet fusion model and the precise quantization closed-loop mechanism; the lower layer, the "Basic Technology Support Layer," labels supporting technologies such as deep learning frameworks and the DICOM communication protocol; the right side indicates expansion directions such as "Gastroscopy Adaptation" and "Bronchoscopy Adaptation," demonstrating technology transferability; (See attached diagram) Figure 3 The functional flow and module association diagram is as follows: the full functional nodes are labeled in chronological order: "image acquisition → equipment calibration → feature enhancement → lesion segmentation → quantitative calculation → 3D reconstruction → report output → progress tracking". Each node corresponds to the associated core module, which intuitively presents the complete execution chain of the technical solution.

Claims

1. A method for endoscopic image optimization, lesion reconstruction annotation, and quantification based on a U-Net / ResNet fusion algorithm, characterized in that, A closed-loop data system is constructed, consisting of a feature fusion processing module, a lesion quantification analysis module, a 3D reconstruction and rendering module, a full-scene integration module, and extended function modules. The core function modules are linked in the order of "image input → feature processing → quantification calculation → 3D reconstruction → result output" to achieve enhanced recognition of small lesions under seven pathological states: normal, mucosal congestion, mucosal edema, intestinal tuberculosis, intestinal polyps, ulcers, and tumors; accurate measurement of lesion area / volume; 3D visualization reconstruction; and multi-scene deployment. The extended function modules, based on the output results of the core modules, achieve lesion progression tracking, accurate measurement of lesion area / volume, 3D visualization reconstruction, and multi-scene deployment. The system has a lesion segmentation Dice coefficient ≥0.89 and a lesion quantification error ≤3%. On a typical workstation equipped with an Intel i5-10400 processor, GTX1650 graphics card, and 8GB of memory, the 3D reconstruction frame rate is ≥15fps.

2. The method for endoscopic image optimization, lesion reconstruction annotation, and quantification based on the U-Net / ResNet fusion algorithm according to claim 1, characterized in that, The feature fusion processing module adopts a customized fusion architecture of U-Net and ResNet50: the encoder uses ResNet50 as the backbone network, where the conv2_x layer extracts high-frequency edge texture features at 1 / 4 of the original image's spatial resolution, and the conv3_x to conv5_x layers extract mid-to-low-frequency background structure features at 1 / 8, 1 / 16, and 1 / 32 of the original image's spatial resolution, respectively. To achieve effective feature fusion, all high- and low-frequency features are unified into 256 channels through a 1×1 convolution and then concatenated along the channel dimension. The decoder upsamples step by step through 3×3 transposed convolutions with a stride of 2. Each layer is added and fused pixel by pixel with the feature map of the encoder at the corresponding scale. The output layer generates a single-channel lesion probability segmentation mask through 1×1 convolution and a Sigmoid activation function.

3. The method for endoscopic image optimization, lesion reconstruction annotation, and quantification based on the U-Net / ResNet fusion algorithm according to claim 2, characterized in that, Differential enhancement strategies were employed for seven pathological conditions: For mucosal congestion lesions, after converting RGB to LAB space, grayscale stretching of the A channel within the range of 50–200 grayscale values ​​was performed (based on the grayscale range of 0–255 for 8-bit images); for mucosal edema lesions, feature fusion was followed by Canny edge detection (threshold 50 / 150) and 3×3 morphological closing operation; for intestinal tuberculosis lesions, stretching of the L channel in LAB space within the range of 40–210 was combined with Canny edge detection at a threshold of 45 / 145; for intestinal polyps, 1.3x gain enhancement of the S channel in HSV space was combined with contour extraction at a Canny threshold of 55 / 155; and for ulcer lesions, 1:1 weighted fusion of RGB and LAB channels was performed with a 0.5mm... 2 Minimum detection area screening; tumor lesions are detected using 0.6 weighted high-frequency feature enhancement combined with 3×3×3 voxel edge detection; normal mucosa retains original texture features.

4. The method for endoscopic image optimization, lesion reconstruction annotation, and quantification based on the U-Net / ResNet fusion algorithm according to claim 1, characterized in that, The lesion quantification analysis module includes a device parameter calibration unit, an inter-frame alignment calculation unit, a quantitative calculation formula, and an anomaly warning mechanism: the device parameter calibration unit pre-stores a database of pixel spacing of mainstream endoscopic devices, and calculates the lesion using a standard metal marker ball with a diameter of 5mm according to the formula. (in, =5mm, (For marking the diameter of the sphere pixels); dynamically calibrate the actual pixel spacing. ; Inter-frame The alignment calculation unit uses the SIFT algorithm to extract ≥50 feature points per frame. After RANSAC is used to remove mismatches with an error >2 pixels, the homography matrix is ​​calculated to complete the correction.

5. The method for endoscopic image optimization, lesion reconstruction annotation, and quantification based on the U-Net / ResNet fusion algorithm according to claim 4, characterized in that, The quantitative calculation formula is: ulcer area (N is the total number of pixels in the lesion); tumor / polyp volume (in, The total number of voxels in the lesions. (This refers to the axial sampling interval). When observing abnormal tissue, the distance between the fiberoptic lens and the tumor surface is kept stable within the range of 10–12 mm; the abnormality warning mechanism is activated when the ulcer area is >50 mm. 2 Tumor volume > 2000 mm 3 A prompt will be triggered if the pixel pitch calibration deviation is greater than 10%.

6. The method for endoscopic image optimization, lesion reconstruction annotation, and quantification based on the U-Net / ResNet fusion algorithm according to claim 1, characterized in that, The 3D reconstruction rendering module is configured with a dual threshold transfer function for the colorectal tubular structure: CT value of intestinal wall tissue -200 to 0 HU, transparency 0.6; CT value of lesion tissue 20 to 100 HU, transparency 1.0; CT value of normal mucosa -50 to 20 HU, transparency 0.8; and 32×32×32 voxel blocks are processed in parallel, with early ray termination implemented for three consecutive voxel blocks without lesions to ensure real-time performance.

7. The method for endoscopic image optimization, lesion reconstruction annotation, and quantification based on the U-Net / ResNet fusion algorithm according to claim 1, characterized in that, The full-scene integration module supports png, jpg, and DICOM 3.0 format input, automatically identifies the endoscope equipment model and loads the corresponding parameters; it provides dual deployment modes: PACS docking mode is based on DCMTK to develop C-MOVE / C-STORE interface, and the AE title is "AI_ENDOSCOPE"; the lightweight offline mode, after TensorRT INT8 quantization and compression, has a model size ≤50MB, an installation package ≤200MB, and a startup time <30 seconds.

8. The method for endoscopic image optimization, lesion reconstruction annotation, and quantification based on the U-Net / ResNet fusion algorithm according to claim 7, characterized in that, The following adjustments were made for the gastroscopy scenario: For gastric ulcer lesions, the enhancement strategy adopted RGB and HSV channels with a weighted fusion ratio of 1:1.2, with a CT value of -180 to 20 HU and a transparency of 0.7 for gastric wall tissue, and a CT value of 30 to 110 HU and a transparency of 1.0 for lesion tissue. For the bronchoscopy scenario, the enhancement strategy for polyp lesions adopted 1.5 times gain for the HSV spatial S channel and a Canny threshold of 60 / 160, with a CT value of -220 to -30 HU and a transparency of 0.65 for the bronchial wall.

9. The method for endoscopic image optimization, lesion reconstruction annotation, and quantification based on the U-Net / ResNet fusion algorithm according to claim 1, characterized in that, The extended functional module is a lesion progression tracking module. It calls the lesion feature vector output by the feature fusion processing module, uses the cosine similarity algorithm to compare the differences between historical and current images, and quantifies the lesion size change rate (accuracy 0.1%) and morphological trend. Based on clinical validation data, a cosine similarity ≥0.8 indicates that the lesion is stable, and <0.8 indicates significant change.

10. A method for endoscopic image optimization, lesion reconstruction annotation, and quantification based on a U-Net / ResNet fusion algorithm according to claim 1, characterized in that, The system outputs a "three-in-one" report, including the original endoscopic image, the enhanced image with a red border and pathology label, and the 3D image with blue parameter annotations; it supports exporting quantitative data containing fields such as patient ID and examination time to an Excel spreadsheet, and exporting the "three-in-one" results to PNG format (1920×1080 resolution).

Images