Intelligent auxiliary diagnosis system for early-stage cancer of digestive tract based on deep learning technology

Abstract

The present application relates to the technical field of intelligent auxiliary diagnosis, in particular to a gastrointestinal early cancer intelligent auxiliary diagnosis system based on deep learning technology. It comprises: an image acquisition and processing unit acquires images in the gastrointestinal tract and performs preprocessing; a lesion detection unit uses Faster R-CNN to locate the lesion area, and at the same time, combines endoscope pose data to calculate local curvature to generate a rotation angle adaptive anchor box to optimize the process of locating the lesion area; a lesion segmentation unit segments the lesion area and outputs the segmentation mask of the lesion; a lesion classification unit classifies the segmented lesion area based on the segmented lesion area using a convolutional neural network model to classify and diagnose different lesion types; a diagnosis fusion unit combines the lesion detection, lesion classification results and lesion segmentation mask to output an auxiliary diagnosis report. The present application effectively deals with motion interference and imaging artifacts in endoscopic imaging through a multi-stage collaborative optimization strategy.

Description

Technical Field

[0001] This invention relates to the field of intelligent assisted diagnostic technology, and more specifically, to an intelligent assisted diagnostic system for early gastrointestinal cancers based on deep learning technology. Background Technology

[0002] With the increasing incidence of gastrointestinal cancers, early detection and treatment are crucial for improving patient prognosis. However, gastrointestinal cancers, especially in their early stages, often lack obvious clinical symptoms, leading to many patients being diagnosed only when the disease has progressed to a later stage. At this point, treatment becomes more difficult and patient survival rates significantly decrease. Traditional endoscopy, as the primary screening method, heavily relies on the experience and judgment of physicians, resulting in significant subjective differences in practice and increasing the risk of misdiagnosis or missed diagnosis. Furthermore, given the massive amount of gastrointestinal endoscopic image data and its complex structural information, manual analysis is not only time-consuming and labor-intensive but also struggles to guarantee the consistency and accuracy of results. In addition, the uneven distribution of medical resources further limits many patients' access to timely and effective medical services, especially in remote areas where highly skilled medical professionals are scarce. Therefore, this paper proposes an intelligent auxiliary diagnostic system for early gastrointestinal cancers based on deep learning technology. Summary of the Invention

[0003] The purpose of this invention is to provide an intelligent auxiliary diagnostic system for early gastrointestinal cancer based on deep learning technology, in order to solve the problems mentioned in the background art, such as low detection rate and high risk of misdiagnosis and missed diagnosis of early cancer caused by large differences in doctors' subjective judgment, strong concealment of early lesions, many image artifacts and motion interference, and inaccurate positioning under complex curved structures.

[0004] To achieve the above objectives, the present invention provides an intelligent auxiliary diagnostic system for early gastrointestinal cancers based on deep learning technology, comprising:

[0005] An image acquisition and processing unit acquires and preprocesses images of the digestive tract.

[0006] The lesion detection unit uses Faster R-CNN to locate the lesion area based on the preprocessed image. At the same time, it calculates the local curvature by combining the endoscope pose data and generates an anchor frame with an adaptive rotation angle to optimize the process of locating the lesion area.

[0007] A lesion segmentation unit, which segments the lesion region and outputs a lesion segmentation mask;

[0008] The lesion classification unit, based on the segmented lesion region, uses a convolutional neural network model to classify and diagnose different lesion types in the segmented lesion region.

[0009] The diagnostic fusion unit combines lesion detection, lesion classification results, and lesion segmentation mask to output an auxiliary diagnostic report.

[0010] As a further improvement to this technical solution, the image acquisition and processing unit includes an image acquisition module and an image processing module;

[0011] The image acquisition module is used to acquire images of the inside of the digestive tract;

[0012] The image processing module is used to preprocess the acquired images.

[0013] As a further improvement to this technical solution, the lesion detection unit, based on the preprocessed image, uses Faster R-CNN to locate suspected lesion areas, including the following steps:

[0014] S1.1 Input the preprocessed image into the feature extraction subnetwork of the convolutional neural network;

[0015] S1.2 Extract multi-scale feature maps through multi-layer convolution and pooling operations;

[0016] S1.3 Generate multi-scale, multi-aspect-ratio anchor boxes at each location of the multi-scale feature map, and predict the overlap and positional offset of each anchor box with the lesion area through a lightweight convolutional network;

[0017] S1.4. Non-maximum suppression is used to remove overlapping anchor boxes and output candidate regions;

[0018] S1.5 Map the candidate regions to the corresponding positions of the multi-scale feature maps, and unify the features of candidate regions of different sizes into a fixed-size feature tensor through the ROI Align operation;

[0019] S1.6 Perform bounding box regression on the features of the candidate region to calculate the coordinates and size of the lesion location;

[0020] S1.7. Perform non-maximum suppression again to remove the remaining redundant boxes and obtain the final set of lesion regions;

[0021] S1.8 Output the bounding box coordinates and location confidence of each lesion region.

[0022] As a further improvement to this technical solution, in S1.3, multi-scale, multi-aspect-ratio anchor boxes are generated at each location of the multi-scale feature map, and the overlap and positional offset of each anchor box with the lesion region are predicted by a lightweight convolutional network, including the following steps:

[0023] S1.31. For the input multi-scale feature map, calculate the local curvature by combining the endoscope pose data, and generate an anchor box with adaptive rotation angle.

[0024] S1.32. Using the lightweight convolutional subnetwork set in the region proposal network, extract the corresponding local feature representation for each anchor box;

[0025] S1.33. Input the extracted features into two parallel fully connected layers;

[0026] The first layer is used for binary classification, outputting the probability that each anchor box is a lesion region; the second layer is used for bounding box regression, outputting the coordinate offset of the anchor box relative to the actual lesion region.

[0027] S1.34. Calculate the overlap score of each anchor frame based on the foreground probability output by the first layer.

[0028] S1.35. Use the foreground probabilities and position offsets of all anchor frames as the output of the region proposal network.

[0029] As a further improvement to this technical solution, in step S1.31, the calculation of local curvature based on endoscopic pose data to generate an anchor frame with adaptive rotation angle includes the following steps:

[0030] It reads data from the inertial measurement unit and lens depth sensor on the endoscope in real time to obtain the three-dimensional attitude information of the lens in the current frame;

[0031] Based on the aforementioned three-dimensional pose information, the angle between the lens optical axis and the cross-section of the digestive tract wall is calculated. And measure the distance from the lens to the mucosal surface. ;

[0032] According to the included angle and distance A localized digestive tract surface model is constructed to calculate the local curvature of that region. ;

[0033] When generating anchor boxes at each location in the feature map, the principal direction angle of the anchor boxes is adjusted according to the local curvature. Dynamic rotation adjustment is performed to align the anchor frame with the actual geometry of the digestive tract wall;

[0034] While maintaining the original multi-dimensional and aspect ratio design, the anchor frame, which has been rotated and adjusted, is translated and corrected to form an anchor frame with adaptive rotation angle.

[0035] As a further improvement to this technical solution, the lesion segmentation unit segments the lesion region and outputs a segmentation mask for the lesion, including the following steps:

[0036] S2.1 Receive the lesion area image block output by the lesion detection unit and perform preprocessing;

[0037] S2.2 Input the preprocessed lesion region image patch into a deep convolutional network based on the U-Net architecture to output a lesion probability map. ;

[0038] S2.3, Pathology probability chart Input a heart rate motion field estimation network, output the motion field And perform stabilization processing to generate a stable lesion probability map. It is used to compensate for the shaking of the boundaries of digestive tract lesions caused by heartbeat during endoscopic imaging;

[0039] S2.4, Probability chart of lesions Using probability threshold Binarization is performed to obtain a preliminary binary mask. ;

[0040] S2.5, Preliminary binary mask Perform an inverse motion-compensated transform to restore the original motion state and generate a motion-compensated binary mask. ;

[0041] S2.6, Apply the motion-compensated binary mask Restored to the original size of the lesion area;

[0042] S2.7 Output the segmentation mask for the lesion area .

[0043] As a further improvement to this technical solution, in step S2.3, the lesion probability map is... Input a heart rate motion field estimation network, output the motion field And perform stabilization processing to generate a stable lesion probability map. This includes the following steps:

[0044] S2.31, Pathology probability chart Input a heart rate motion field estimation network and output the corresponding motion field. ;

[0045] S2.32, Based on lesion probability map Calculate the probability gradient and its magnitude;

[0046] S2.33. Generate motion sensitivity heatmap based on gradient intensity. It is used to indicate the sensitivity of the boundary to the influence of heartbeat.

[0047] S2.34. Construct a time-domain band-stop filter for heart rate to suppress heart rate components;

[0048] S2.35, in motion sensitivity heatmap For regions where the value is below the lower threshold b, the probability values ​​of the corresponding pixels are processed by temporal band-stop filtering to generate a lesion probability map after motion-sensing boundary stabilization. To suppress periodic motion disturbances;

[0049] S2.36, in In the region, anisotropic diffusion equations are used to smooth the local boundaries, generating a lesion probability map after local boundary smoothing. Indicates the sensitivity threshold;

[0050] S2.37, the sports field Pathology probability map after motion-sensing boundary stabilization processing The lesion probability map is fused with the locally smoothed lesion probability map to output the final stable lesion probability map. .

[0051] As a further improvement to this technical solution, the method of using a convolutional neural network model to classify and diagnose different lesion types in the segmented lesion region includes the following steps:

[0052] S3.1 Receive the segmentation mask output by the lesion segmentation unit. ;

[0053] S3.2, Based on the segmentation mask The lesion region is cropped from the original image and normalized. Then, an artifact-aware focus self-correction network is used to enhance the normalized image patch, generating an enhanced lesion region image patch. and enhanced lesion area image patches Simultaneously, the lesion segmentation unit is input to replace the lesion region image block output by the lesion detection unit in step S2.1;

[0054] S3.3. Use deep convolutional neural networks as the lesion classification model;

[0055] S3.4, Enhance the lesion area image block Input the lesion classification model and extract deep semantic features step by step;

[0056] S3.5 Input the deep semantic features into the fully connected layer of the lesion classification model, and normalize them through the Softmax activation function to output the probability distribution of the lesion region belonging to each preset category;

[0057] S3.6 Based on the probability distribution, select the category corresponding to the highest probability as the final classification result of the lesion area, and record the highest probability value of the category as the classification confidence.

[0058] S3.7 Output the classification results for each lesion area.

[0059] As a further improvement to this technical solution, in step S3.2, image enhancement is performed on the normalized image patch using an artifact-aware focus self-correction network, including the following steps:

[0060] S3.21. Normalized lesion area image patch The lightweight artifact detection network ADNet is used for parallel inference to generate artifact heatmaps of the same size. ;

[0061] S3.22, Based on artifact heatmap Generate a focal self-correcting spatial weight mask This is used to dynamically suppress the influence of interfering regions on subsequent feature extraction;

[0062] S3.23, Using a focus-self-correcting spatial weight mask Original normalized lesion area image patch Perform local enhancement to generate enhanced image patches of the lesion area. .

[0063] As a further improvement to this technical solution, the diagnostic fusion unit receives the bounding box coordinates of the lesion region and its location confidence, classification results, and lesion segmentation mask. Based on the bounding box coordinates, it maps the lesion classification results and segmentation mask to the corresponding lesion region in the original endoscopic image and generates an auxiliary diagnostic report.

[0064] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0065] 1. In this invention, an adaptive anchor frame generation mechanism based on endoscopic pose data is introduced into the region proposal network of Faster R-CNN. By fusing IMU and depth sensor data in real time, the angle and distance between the lens pose and the digestive tract wall are calculated, local curvature is dynamically modeled, and the rotation angle of the anchor frame is adjusted accordingly to align its major axis with the direction of irregular lesions (such as flat or circumferential lesions) on the curved tract wall. This design effectively overcomes the problems of traditional axis-aligned rectangular anchor frames covering too much background and having poor fit on curved surfaces, making the candidate region more closely fit the actual lesion morphology. This significantly improves the lesion detection accuracy and localization precision in complex digestive tract curved environments, providing high-quality input for subsequent segmentation and classification tasks, thereby reducing the overall risk of false positives and false negatives.

[0066] 2. This invention effectively addresses motion interference and imaging artifacts in endoscopic imaging through a multi-stage collaborative optimization strategy. In the segmentation stage, a Heartbeat Motion Field Estimation Network (HME-Net) is introduced, combined with motion-aware boundary stabilization processing and anisotropic diffusion, to suppress boundary jitter caused by heartbeats and generate stable and clear segmentation masks. In the classification stage, an artifact-aware focus self-calibration network is employed, generating multi-class artifact heatmaps through a lightweight ADNet and constructing spatial weight masks to dynamically suppress the influence of interfering regions such as mucus, bubbles, and fluid edges. Simultaneously, CLAHE enhancement is fused to preserve details of potential lesions. This mechanism enhances image quality while retaining key diagnostic information, significantly improving the robustness of the classification model to noise and artifacts. Finally, a diagnostic fusion unit integrates the detection, segmentation, and classification results, outputting a structured report containing location, morphology, type, and a visualized segmentation map, greatly enhancing the system's clinical usability and diagnostic reliability. Attached Figure Description

[0067] Figure 1 This is an overall flowchart of the present invention;

[0068] The meanings of the labels in the diagram are as follows:

[0069] 1. Image acquisition and processing unit; 2. Lesion detection unit; 3. Lesion segmentation unit; 4. Lesion classification unit; 5. Diagnostic fusion unit. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0071] Example: Please refer to Figure 1 As shown, a deep learning-based intelligent auxiliary diagnostic system for early gastrointestinal cancers is provided, including:

[0072] Image acquisition and processing unit 1 acquires images of the digestive tract and performs preprocessing (acquired via endoscopic equipment).

[0073] In this embodiment, the image acquisition and processing unit 1 includes an image acquisition module and an image processing module;

[0074] The image acquisition module is used to acquire images of the inside of the digestive tract;

[0075] The image processing module is used to preprocess the acquired images, including noise removal, contrast adjustment, brightness correction, and cropping of irrelevant areas;

[0076] Based on the preprocessed image, the lesion detection unit 2 uses Faster R-CNN (a faster region-based convolutional neural network) to locate the lesion area. At the same time, it combines the endoscope pose data to calculate the local curvature and generate an anchor box with an adaptive rotation angle to optimize the process of locating the lesion area.

[0077] In this embodiment, the lesion detection unit 2 locates suspected lesion areas based on the preprocessed image using Faster R-CNN, including the following steps:

[0078] S1.1 Input the preprocessed image into the convolutional neural network feature extraction subnetwork (such as ResNet, VGG or SwinTransformer backbone network). The convolutional neural network feature extraction subnetwork takes the preprocessed endoscope image as input, and extracts the edge, texture and other features of the image through multiple convolutional layers and activation layers (such as ReLU). Then, it gradually downsamples through pooling layers to reduce the spatial size and enhance translation invariance. The superposition of multiple convolutions and pooling makes the network obtain multi-scale feature maps with different scales and semantic depths. Finally, it outputs a set of feature maps from shallow to deep layers, with the size gradually decreasing and the number of channels gradually increasing.

[0079] S1.2 Extracting multi-scale feature maps through multi-layer convolution and pooling operations (specifically: first, the preprocessed input image is passed through multiple convolutional layers, and features are extracted from the image using convolutional kernels of different sizes to capture details such as edges, textures, and colors of the lesion area; then, the spatial size of the feature map is gradually reduced through pooling layers (such as max pooling or average pooling) to achieve spatial dimensionality reduction of features and extraction of key information, while enhancing the translation invariance of the model; after repeating convolution and pooling operations multiple times, rich feature representations can be obtained at different scales, enabling the network to retain both local details and capture global contextual information), preserving significant features such as edge textures and color changes of the lesion area;

[0080] S1.3 Generate multi-scale, multi-aspect-ratio anchor boxes at each location of the multi-scale feature map. Predict the overlap (foreground probability) and position offset of each anchor box with the lesion region through a lightweight convolutional network. The lightweight convolutional network usually starts with a small input image and goes through a series of depthwise separable convolutions or a combination of pointwise convolutions and pointwise convolutions. Each layer is followed by batch normalization and non-linear activation (such as ReLU or Swish). Efficient feature extraction is achieved by reducing the number of parameters and computation. The network has a shallow number of layers and a moderate number of channels. The output of the intermediate layers maintains a small spatial size and channel depth, and finally outputs a compact feature representation.

[0081] The process involves generating multi-scale, multi-aspect-ratio anchor boxes at each location in the multi-scale feature map, and predicting the overlap (foreground probability) and positional offset of each anchor box with the lesion region using a lightweight convolutional network. This includes the following steps:

[0082] S1.31. For the input multi-scale feature map, calculate the local curvature by combining the endoscope pose data, and generate an anchor box with adaptive rotation angle.

[0083] Furthermore, traditional rectangular anchor frames struggle to conform to irregular lesions (including flat and circumferential lesions) on curved surfaces within the digestive tract, resulting in the localization frame containing a large amount of irrelevant mucosal background and reducing subsequent classification / segmentation accuracy. By combining endoscopic IMU pose data with depth sensor distance information to construct a local curvature model, and dynamically calculating the rotation angle based on curvature to generate an adaptive anchor frame, real-time alignment of the anchor frame shape with the local anatomical structure of the digestive tract is achieved. This effectively solves the problems of low conformity of traditional fixed rectangular anchor frames on curved digestive tract surfaces, easy inclusion of a large amount of irrelevant mucosal background, and mismatch between orientation and lesion direction. Its rotational adaptive mechanism allows the anchor frame's long axis to automatically conform to the lesion direction, and maintains spatial consistency with translation correction, reducing background interference and improving localization accuracy. The introduction of random angle perturbation enhances the model's robustness to changes in viewing angle and lens tilt, thereby providing a cleaner lesion region for the segmentation unit and a geometrically aligned high-quality image patch for the classification unit, significantly improving downstream task performance, reducing the missed detection rate of flat early cancers, and improving the accuracy of circumferential lesion boundary recognition.

[0084] Calculating local curvature based on endoscopic pose data to generate an anchor frame with adaptive rotation angle includes the following steps:

[0085] The system reads data from the inertial measurement unit (IMU) and the lens depth sensor on the endoscope in real time to obtain the three-dimensional attitude information of the lens in the current frame (the lens refers to the lens of the endoscope).

[0086] Based on the aforementioned three-dimensional pose information, the angle between the lens optical axis and the cross-section of the digestive tract wall is calculated. ( In the formula, Let be the unit vector of the lens optical axis in the world coordinate system. (This refers to the unit normal vector of a local digestive tract section), and the distance from the lens to the mucosal surface is measured. (Depth sensor output distance) The unit is millimeters, which has been converted to pixel coordinates using the endoscope lens focal length and image resolution (pixels / mm).

[0087] According to the included angle and distance A localized digestive tract surface model is constructed to calculate the local curvature of that region. (The local digestive tract surface model is as follows:) Local curvature The unit is (This refers to the local area of ​​the digestive tract wall that is directly opposite to or covered by the optical axis of the endoscope lens in the current frame.)

[0088] When generating anchor boxes at each location in the feature map, the principal direction angle of the anchor boxes is adjusted according to the local curvature. Perform dynamic rotation adjustment (dynamic rotation adjustment is based on the rotation angle) Rotate (rotation angle) (Unit: radians) The formula for calculating the rotation angle is: In the formula, The arctangent function is used to map the product of curvature and distance to a finite angle, avoiding excessively large rotation angles. The random perturbation angle (unit: radians) is used to introduce angle data augmentation during the training phase to improve the robustness of the model to rotation invariance. The value range is [-5°, 5°], which is set according to the angle fluctuation range under real shooting conditions. This aligns the anchor frame with the actual geometry of the digestive tract wall. The greater the curvature (the steeper the duct wall), the greater the required rotation angle, so that the long axis of the anchor frame fits the direction of the lesion.

[0089] While maintaining the original multi-scale and aspect ratio design, the anchor frame that has been rotated is translated and corrected (the translation correction step is based on the rotated anchor frame, aligning the center position of the anchor frame with the corresponding position in the feature map, calculating the offset of the anchor frame caused by rotation, and then translating and adjusting the center coordinates of the anchor frame to maintain the spatial correspondence with the grid points in the feature map), forming an anchor frame with adaptive rotation angle.

[0090] S1.32. Utilize the lightweight convolutional sub-network set in the Region Proposal Network (specifically: within the Region Proposal Network (RPN), a concise and computationally efficient convolutional neural network sub-module is used. This lightweight convolutional sub-network is responsible for extracting local features for each anchor box. The Region Proposal Network (RPN) takes a multi-scale feature map as input, first extracts the shared feature representation of each position through a 3×3 convolutional layer, and then branches into two parallel lightweight convolutional sub-networks: one for binary classification, outputting the probability that each anchor box is foreground (lesion) or background, and the other for bounding box regression, outputting the coordinate offset of the anchor box relative to the real target. The intermediate layers maintain the same spatial size as the input feature map, and finally generate the class scores and position adjustment parameters of multiple anchor boxes), to extract the corresponding local feature representation for each anchor box (here, the anchor box is a rotation angle adaptive anchor box).

[0091] S1.33. Input the extracted features into two parallel fully connected layers (or convolutional layers).

[0092] The first layer is used for binary classification, outputting the probability (i.e., overlap) of each anchor box as a lesion area (foreground) or background; the second layer is used for bounding box regression, outputting the coordinate offset of the anchor box relative to the actual lesion area (position adjustment parameters, including center coordinate offset and width and height scaling ratio).

[0093] S1.34. Based on the foreground probability output by the first layer, calculate the overlap score of each anchor box (specifically: based on the foreground probability output by the first layer, first normalize the probability value of each anchor box predicted as a lesion area, and this probability is used as the overlap score between the anchor box and the real lesion area), which is used for the subsequent screening and sorting of candidate areas.

[0094] S1.35, Use the foreground probability and position offset of all anchor frames as the output of the region proposal network;

[0095] S1.4. Use non-maximum suppression (NMS) to remove highly overlapping anchor boxes and output candidate regions (select candidate regions with confidence greater than the preset upper threshold c). The specific steps of using non-maximum suppression (NMS) to remove highly overlapping anchor boxes are as follows: First, sort all anchor boxes according to their foreground probabilities. Starting from the anchor box with the highest confidence, iterate through and retain the anchor box one by one. At the same time, remove other anchor boxes whose overlap (such as IoU) exceeds the upper threshold c. Repeat this process until all anchor boxes have been processed. Finally, output a set of candidate regions with high confidence and low overlap.

[0096] S1.5. Map candidate regions to corresponding positions on multi-scale feature maps. Use the ROI Align operation to unify the features of candidate regions of different sizes into a fixed-size feature tensor, which facilitates subsequent localization and refinement. The ROI Align operation is as follows: First, map each candidate region to the corresponding position on the multi-scale feature map and precisely divide the region into a fixed number of sub-grids. Then, use bilinear interpolation to sample the feature values ​​in each sub-grid to avoid quantization errors. Finally, integrate the sampling results of all sub-grids into a fixed-size feature tensor, so that candidate regions of different sizes and aspect ratios can be input into subsequent convolutional or fully connected layers for unified processing.

[0097] S1.6 Perform bounding box regression on the features of the candidate region to calculate the coordinates and size of the lesion location. The bounding box regression process involves inputting the features corresponding to the candidate region into the regression network. By predicting the coordinate offset relative to the initial anchor box (including the translation of the center point and the scaling ratio of the width and height), the position and size of the candidate box are finely adjusted, thereby calculating a more accurate bounding box coordinate of the lesion region.

[0098] S1.7. Perform non-maximum suppression (NMS) again to remove the remaining redundant boxes and obtain the final set of lesion regions;

[0099] S1.8 Output the bounding box coordinates and location confidence of each lesion region.

[0100] The lesion segmentation unit 3 segments the lesion region and outputs the lesion segmentation mask;

[0101] In this embodiment, the lesion segmentation unit 3 segments the lesion region and outputs a segmentation mask for the lesion, including the following steps:

[0102] S2.1 Receive the lesion area image block output by the lesion detection unit 2 and perform preprocessing, including adjusting to a fixed resolution (e.g., 224×224 pixels) and standardizing pixel values ​​(e.g., normalizing to the [0,1] range).

[0103] S2.2 Input the preprocessed lesion region image patch into a deep convolutional network based on the U-Net architecture to output a lesion probability map. , In the formula, The height of the lesion probability map (number of pixel rows). The width (number of pixel columns) of the lesion probability map is given by the deep convolutional network based on the U-Net architecture. The network takes the preprocessed lesion region image patch as input. First, the downsampling encoder part extracts progressively deeper semantic features through multi-layer convolution and pooling. The intermediate layer fuses the high-resolution features of the corresponding layer of the encoder with the corresponding layer of the decoder through skip connections. The decoder part gradually restores the spatial resolution through deconvolution or upsampling. Finally, the output is a lesion probability map of the same size as the input image, where each pixel value represents the probability that the location belongs to a lesion.

[0104] S2.3, Pathology probability chart Input a heart rate motion field estimation network, output the motion field (playground It is a two-dimensional motion vector field of size 224×224 (each pixel is described by two values ​​in the x and y directions), and is stabilized to generate a stable lesion probability map. It is used to compensate for the shaking of the boundaries of digestive tract lesions caused by heartbeat during endoscopic imaging;

[0105] The heartbeat, transmitted to the digestive tract (especially the esophagus and stomach fundus), causes continuous micro-vibrations on the mucosal surface, resulting in periodic blurring or jittering of the lesion probability map (U-Net output). The static segmentation model (U-Net) cannot model temporal motion, and the segmentation mask boundary vibrates with the heartbeat. Global motion compensation methods (such as video stabilization) ignore local anatomical characteristics and are prone to over-smoothing lesion edge details. This solution achieves high-precision geometric stabilization of the segmentation results under heartbeat interference. Compared to traditional methods that rely solely on global motion estimation or simple temporal smoothing, this solution uses probability gradient magnitude to generate a motion sensitivity heatmap, quantifying the sensitivity of lesion boundaries to motion. Based on this, a gradient-guided partitioning strategy is constructed: in low-sensitivity areas, temporal band-stop filtering is used to accurately suppress heartbeat frequency interference; in high-sensitivity areas, anisotropic diffusion is used to smooth noise along the edge tangent direction and protect the sharpness in the normal direction. Simultaneously, sub-pixel-level spatial deformation compensation is performed using the pixel displacement field predicted by HME-Net to achieve accurate alignment of the probability map. This method significantly reduces boundary jitter caused by heartbeats, avoids area misjudgment, and preserves the subtle topological features of lesion edges, thereby providing more stable lesion region input for classification units;

[0106] lesion probability map Input a heart rate motion field estimation network, output the motion field And perform stabilization processing to generate a stable lesion probability map. This includes the following steps:

[0107] S2.31, Pathology probability chart The input is a heartbeat motion field estimation network (which takes the lesion probability map of consecutive frames as input, passes through multiple layers of temporal convolutional networks to extract spatiotemporal features, captures the dynamic pixel displacement information of the lesion area caused by heartbeat, and the intermediate layers generate motion vector feature maps by fusing temporal and spatial information, finally outputting a two-dimensional motion field (containing two channels, horizontal and vertical), used to represent the motion direction and amplitude of each pixel), and outputs the corresponding motion field. (playground Unit: pixels / frame, calibrated parameters (pixels / mm) is related to physical displacement. This motion field represents the pixel displacement vector field of the lesion area caused by heartbeat, which is used to guide subsequent motion compensation and stabilization processing.

[0108] S2.32, Based on lesion probability map Calculate the probability gradient and its magnitude:

[0109] The probability gradient is:

[0110] ; ;

[0111] In the formula, This represents the gradient of the lesion probability map along the horizontal direction (x-axis), i.e., the rate of change of pixel values ​​in the horizontal direction. This represents the gradient of the lesion probability map along the vertical direction (y-axis), i.e., the rate of change of pixel values ​​in the vertical direction. Let x be an infinitesimal change in the x-direction (horizontal direction). Let be an infinitesimal change in the y-direction (vertical direction). Pathology probability map A small change in a certain direction;

[0112] Magnitude of the probability gradient for:

[0113] ;

[0114] S2.33. Generate motion sensitivity heatmap based on gradient intensity. This is used to indicate the sensitivity of the boundary to the influence of heartbeat:

[0115] ;

[0116] In the formula, The Gaussian kernel standard deviation is used to control motion sensitivity. Here, a hyperparameter with a value of 0.1 is used to adjust the range of gradient influence.

[0117] S2.34. Construct a time-domain band-stop filter for heart rate to suppress typical heart rate components:

[0118] ;

[0119] In the formula, This is the frequency response function of a time-domain band-stop filter, suppressing components near the heartbeat frequency. This is a frequency variable, representing the signal frequency (Hz). This represents the heart rate, set here to a typical value of 1.2Hz. This is the bandwidth parameter of the band-stop filter, which controls the suppression range; here it is 0.3Hz (based on clinical data statistics).

[0120] S2.35, in motion sensitivity heatmap For regions where the value is below the lower threshold b (the lower threshold b ranges from 0.1 to 0.5 and is dimensionless), the probability values ​​of the corresponding pixels are processed by temporal band-stop filtering to generate a lesion probability map after motion-sensing boundary stabilization. To suppress periodic motion disturbances;

[0121] ;

[0122] In the formula, pixel coordinates The filtered probability value after motion-sensing boundary stabilization processing. pixel coordinates The original lesion probability map value, To represent the operator for performing a Fourier transform on a function (image signal), the spatial domain signal is transformed into the frequency domain. This represents the inverse Fourier transform operator, which converts a frequency domain signal back to the spatial domain;

[0123] S2.36, in In the region, anisotropic diffusion equations are used to smooth the local boundaries, generating a lesion probability map after local boundary smoothing. In this embodiment, the sensitive threshold is represented. ;

[0124] Local boundary smoothing specifically involves:

[0125] ;

[0126] ;

[0127] In the formula, For the probability map of lesions at location and diffusion time pixel values, For gradient operators, For divergence operators, The diffusion coefficient depends on the gradient magnitude. This is a constant used to control the diffusion rate, suppress diffusion at the boundary, and prevent boundary blurring; its value ranges from 0.1 to 0.3.

[0128] S2.37, the sports field Pathology probability map after motion-sensing boundary stabilization processing The lesion probability map is fused with the locally smoothed lesion probability map to output the final stable lesion probability map. The integration process specifically involves: integrating the sports field Used for probabilistic maps after motion-sensing boundary stabilization processing Pixel-level motion compensation is performed, and the resulting lesion probability map (an optimized probability map obtained by smoothing and enhancing the lesion probability map of motion-sensitive areas using an anisotropic diffusion algorithm) is combined with the motion compensation and smoothing results through a weighted fusion method. This approach comprehensively considers the spatial correction guided by motion vectors and the continuity of smoothed boundaries, ultimately generating a stable and accurate lesion probability map. ;

[0129] S2.4, Probability chart of lesions Using probability threshold Binarization is performed to obtain a preliminary binary mask. :

[0130] ;

[0131] In the formula, For the pixel coordinates in the image, in the initial binary mask In the text, the value at this position is: This pixel was identified as a lesion. This pixel was determined to be non-lesion;

[0132] S2.5, Preliminary binary mask Perform an inverse motion-compensated transform to restore the original motion state (the result of the compensation segmentation network segmenting on the stable image is transformed back to the position of the original image patch), and generate a motion-compensated binary mask. Specifically, the inverse transformation of motion compensation involves: using the heartbeat motion field to estimate the motion vector field output by the network, calculating the displacement shift of each pixel during the stabilization process; then, based on the motion vector, moving the pixels in the binary mask in reverse to restore their corresponding positions in the original image, thus realizing the inverse spatial transformation operation of the mask.

[0133] S2.6, Apply the motion-compensated binary mask To restore the original lesion area size, if the image was scaled before being input into the segmentation network, the mask needs to be scaled proportionally. If the mask needs to be backmapped to the entire original image, the position is aligned according to the bounding box coordinates provided by the lesion detection unit 2.

[0134] S2.7 Output the segmentation mask for the lesion area (Segmentation mask of the lesion area) Binary mask based on motion compensation The size of the lesion area is obtained by aligning the size and position (i.e., step S3.5), and its size corresponds to the original lesion area.

[0135] Based on the segmented lesion regions, lesion classification unit 4 uses a convolutional neural network (CNN) model to classify and diagnose different lesion types in the segmented lesion regions;

[0136] In this embodiment, a convolutional neural network (CNN) model is used to classify and diagnose different lesion types in the segmented lesion regions, including the following steps:

[0137] S3.1 Receive the segmentation mask output by lesion segmentation unit 3 ;

[0138] S3.2, Based on the segmentation mask Lesion regions are cropped from the original image and normalized: they are uniformly scaled to a fixed size (e.g., 224×224 pixels) to fit the input requirements of the classification network. Then, an artifact-aware focus self-correction network is used to enhance the normalized image patches, generating enhanced lesion region image patches. and enhanced lesion area image patches Simultaneously, the lesion segmentation unit 3 is input to replace the lesion region image block output by the lesion detection unit 2 in step S2.1, so as to improve the segmentation accuracy of the lesion boundary while suppressing artifact interference.

[0139] In this study, mucus adhesion, bubble aggregation, or liquid reflection within the digestive tract can obscure lesion areas (the fine surface structure of early-stage cancer) or create false edges resembling lesions (bubble outlines are misjudged as raised boundaries). Global image enhancement (such as histogram equalization) simultaneously amplifies noise in artifact regions, and segmentation-based masking methods directly discard artifact regions, leading to the permanent loss of some lesion information. An anatomical semantic-guided local enhancement strategy achieves accurate restoration and morphological preservation of lesion features under artifact interference. Unlike traditional uniform enhancement or simple binary masking methods, this method utilizes a three-channel artifact heatmap (probabilities of mucus, bubbles, and liquid edges) output by a lightweight ADNet to accurately model the interference type and spatial distribution. Differentiated processing strategies are designed for different artifact types, such as morphological dilation of bubble regions to protect attached lesion tissue. Based on this, a focus-self-correcting spatial weight mask is generated, dynamically adjusting the suppression intensity at the pixel level: CLAHE is used to enhance local contrast in artifact areas to reveal masked fine textures, while clean areas retain the original signal to avoid over-enhancing noise. This method effectively relieves the suppression of lesion features by mucus, bubbles, and liquid edges, reduces the false positive rate of classification models, and maintains the topological continuity of lesion boundaries through a soft transition, while also preserving potential malignant features in liquid reflective areas.

[0140] Image enhancement is performed on normalized image patches using an artifact-aware focus self-correction network, including the following steps:

[0141] S3.21. Normalized lesion area image patch The lightweight artifact detection network ADNet is used for parallel inference (ADNet consists of 4 lightweight convolutional layers. The input is a normalized lesion region image patch (e.g., 224×224). Each convolutional layer is followed by batch normalization and activation functions to extract local semantic features. The network output is a three-channel artifact heatmap of the same size as the input, representing the pixel probabilities of mucus, bubbles, and liquid edges, which are used for subsequent focus self-correction spatial weight mask generation, thereby dynamically suppressing the influence of interference regions on feature extraction and lesion classification). This generates an artifact heatmap of the same size. :

[0142] ;

[0143] ADNet is a lightweight 4-layer convolutional neural network with an input resolution of 224×224 and an output heatmap of the same size as the input. The heatmap contains three channels of semantic information: channel 1: slime probability (0~1); channel 2: bubble probability (0~1); channel 3: liquid edge probability (0~1).

[0144] S3.22, Based on artifact heatmap Generate a focal self-correcting spatial weight mask This is used to dynamically suppress the influence of interfering regions on subsequent feature extraction.

[0145] ;

[0146] In the formula, The first channel of the artifact heatmap represents the probability value matrix (element values ​​range from 0 to 1) of the slime region. This is the second channel of the artifact heatmap, representing the probability value matrix (element values ​​range from 0 to 1) of the bubble region. The third channel of the artifact heatmap represents the probability value matrix (element values ​​range from 0 to 1) of the liquid edge region. This is a morphological expansion operation used to expand the boundaries of the bubble region and prevent important lesion information from being mistakenly suppressed. This is an element-wise weighted summation (the corresponding matrix elements are added together, including weight coefficients). This is the weighting coefficient for mucus artifacts, set according to the degree of interference caused by mucus artifacts in lesion identification. Mucus is usually present in a large area and is blurry, so it needs to be moderately suppressed. The value range is 0.5 to 0.8. The bubble artifact weighting coefficient is used because bubble artifacts produce obvious edges and cause significant interference, so they are generally given a large weight. Furthermore, dilation is used to avoid excessive suppression of edge information. The value range is 0.8 to 1.2. Set a small weight for the liquid edge artifact weighting coefficient to preserve detail and avoid over-smoothing. The value range is 0.3 to 0.6;

[0147] S3.23, Using a focus-self-correcting spatial weight mask Original normalized lesion area image patch Perform local enhancement to generate enhanced image patches of the lesion area. This approach balances preserving residual lesion characteristics within the interference area with enhancing contrast.

[0148] ;

[0149] In the formula, This is an image patch after adaptive contrast-limited histogram equalization, used to improve local contrast and detail in interference areas. For the element-wise product (Hadamard product), each pixel of the image is weighted.

[0150] S3.3. A deep convolutional neural network is used as the lesion classification model. The lesion classification model adopts an improved ResNet-50 as the backbone network and introduces attention mechanism modules, including channel attention (such as the SE module) and spatial attention (such as the CBAM module), to enhance the model's ability to pay attention to key lesion texture, color distribution and microstructure features.

[0151] S3.4, Enhance the lesion area image block The input lesion classification model is processed sequentially through multiple convolutional layers, batch normalization, activation functions (such as ReLU or Swish), and pooling layers to extract deep semantic features step by step.

[0152] S3.5 Input the deep semantic features into the fully connected layer of the lesion classification model, and normalize them through the Softmax activation function to output the probability distribution of the lesion region belonging to each preset category;

[0153] S3.6 Based on the probability distribution, select the category corresponding to the highest probability as the final classification result of the lesion area, and record the highest probability value of the category as the classification confidence.

[0154] S3.7 Output the classification results for each lesion region, including: lesion type label (identifying which type of lesion the lesion region belongs to (including adenoma, early-stage carcinoma, well-differentiated carcinoma, etc.)) and classification confidence (the predicted probability value of the corresponding lesion type, used to measure the reliability of the classification results).

[0155] The diagnostic fusion unit 5 combines lesion detection, lesion classification results, and lesion segmentation mask to output an auxiliary diagnostic report;

[0156] In this embodiment, the diagnostic fusion unit 5 receives the bounding box coordinates of the lesion region and its location confidence, classification results (lesion type label and classification confidence), and the segmentation mask of the lesion. Based on the bounding box coordinates, it maps the lesion classification results and segmentation mask to the corresponding lesion region in the original endoscopic image and generates an auxiliary diagnostic report. The report should include: the specific location, size, and morphological description of the lesion region; the classification results of the lesion type; the segmentation map of the lesion region (showing the lesion range); and recommended measures for each discovery (such as further examination, treatment plan suggestions, etc.).

[0157] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. An intelligent auxiliary diagnostic system for early-stage cancer of the digestive tract based on deep learning technology, characterized by, The application relates to a computer-aided diagnosis system for digestive tract diseases, which comprises: an image acquisition and processing unit (1) which acquires images in the digestive tract and performs preprocessing; a lesion detection unit (2) which, based on the preprocessed images, uses a Faster R-CNN to locate a lesion area, simultaneously calculates a local curvature in combination with endoscope pose data, and generates anchor boxes which are adaptive to rotation angles to optimize the process of locating the lesion area; wherein the lesion detection unit (2), based on the preprocessed images, uses a Faster R-CNN to locate a suspected lesion area, comprises the following steps: S1.1, inputting the preprocessed images into a convolutional neural network feature extraction subnet; S1.2, extracting multi-scale feature maps through multi-layer convolution and pooling operations; S1.3, generating multi-scale and multi-aspect-ratio anchor boxes at each position of the multi-scale feature maps, and predicting the coincidence degree and position offset of each anchor box with the lesion area through a lightweight convolutional network; S1.31, for the input multi-scale feature maps, calculate the local curvature in combination with the endoscope pose data to generate anchor boxes which are adaptive to rotation angles; real-time reading of inertial measurement unit data and lens depth sensor data carried by the endoscope to obtain three-dimensional pose information of the current frame lens; Based on the three-dimensional pose information, the included angle between the lens optical axis and the digestive tract wall section is calculated , and the distance from the lens to the mucosa surface is measured ; According to the included angle and the distance Constructing a local digestive tract surface model for calculating the local curvature of the region ; When generating the anchor box at each position of the feature map, the main direction angle of the anchor box is adjusted according to the local curvature Dynamic rotation adjustment is performed to align the anchor box with the actual geometric shape of the inner wall of the digestive tract; on the basis of maintaining the original multi-scale and aspect-ratio design, performing translation correction on the anchor boxes which have been adjusted in rotation to form anchor boxes which are adaptive to rotation angles; S1.32, using a lightweight convolutional subnet set in a region proposal network to extract corresponding local feature representations for each anchor box; S1.33, inputting the extracted features into two parallel fully connected layers; wherein the first layer is used for binary classification and outputs the probability of each anchor box being a lesion area; the second layer is used for boundary box regression and outputs the coordinate offset of the anchor box relative to the real lesion area; S1.34, calculating the coincidence score of each anchor box according to the foreground probability output by the first layer; S1.35, taking the foreground probability and position offset of all anchor boxes as the output of the region proposal network; S1.4, removing overlapping anchor boxes by using non-maximum suppression to output candidate regions; S1.5, mapping the candidate regions to the corresponding positions of the multi-scale feature maps, and uniformly converting the features of candidate regions of different sizes into fixed-size feature tensors through ROI Align operation; S1.6, performing boundary box regression on the features of the candidate regions to calculate the lesion position coordinates and size; S1.7, performing non-maximum suppression again to remove the remaining redundant boxes to obtain a final set of lesion regions; S1.8, outputting the boundary box coordinates of each lesion region and the confidence of the position; a lesion segmentation unit (3) which segments the lesion regions and outputs a segmentation mask of the lesion; a lesion classification unit (4) which, based on the segmented lesion regions, uses a convolutional neural network model to perform classification diagnosis of different lesion types on the segmented lesion regions; a diagnosis fusion unit (5) which combines the lesion detection, lesion classification results and the segmentation mask of the lesion to output an auxiliary diagnosis report. 2.The intelligent auxiliary diagnostic system for early-stage cancer in the digestive tract based on deep learning technology according to claim 1, characterized in that: The image acquisition and processing unit (1) comprises an image acquisition module and an image processing module; The image acquisition module is configured to acquire images inside the digestive tract; The image processing module is configured to pre-process the acquired images. 3.The intelligent auxiliary diagnostic system for early-stage cancer in the digestive tract based on deep learning technology according to claim 1, characterized in that: The lesion segmentation unit (3) segments the lesion region and outputs a segmentation mask of the lesion, including the following steps: S2.1, receiving the lesion region image block output by the lesion detection unit (2) and pre-processing; S2.2, input the pre-processed lesion region image block into a deep convolutional network based on U-Net architecture, and output a lesion probability map ; S2.3, generating a lesion probability map inputting the heartbeat motion field estimation network, outputting the motion field and performing a stabilization process to generate a stable lesion probability map for compensating for the boundary jitter of the gastrointestinal lesion caused by the heartbeat in endoscopic imaging; The heartbeat motion field estimation network takes the lesion probability map of consecutive frames as input, extracts spatio-temporal features through a multi-layer temporal convolution network, captures dynamic pixel displacement information of the lesion region caused by heartbeat, generates a motion vector feature map through fusion of time and space information in the middle layer, and finally outputs a two-dimensional motion field for representing the motion direction and amplitude of each pixel point; S2.4, lesion probability map adopting a probability threshold performing binarization to obtain a preliminary binary mask ; S2.5, to the preliminary binary mask performing a motion compensated inverse transform to revert to the original motion state, generating a motion compensated binary mask ; S2.6, the motion compensated binary mask restore to original lesion area size; S2.7, outputting a segmentation mask of the lesion region . 4.The intelligent auxiliary diagnostic system for early cancer in the digestive tract based on deep learning technology according to claim 3, characterized in that: In S2.3, the lesion probability map The input heartbeat motion field estimation network outputs a motion field and performs a stabilization process to generate a stabilized lesion probability map comprising the following steps: S2.31, generating a lesion probability map inputting a heartbeat motion field estimation network, outputting a corresponding motion field ; S2.32, based on the lesion probability map The probability gradient is calculated, and the magnitude of the probability gradient is calculated. S2.33, generating a motion sensitivity heat map from the gradient intensities to indicate the degree to which the border is affected by the heartbeat motion; S2.34, a time domain bandpass filter for the heartbeat frequency is constructed to suppress the heartbeat frequency component; S2.35, in the motion sensitivity heat map The region with the value lower than the lower threshold value b will be subjected to the time domain band filter processing to generate the lesion probability map after the motion-aware boundary stabilization processing to suppress periodic motion interference; S2.36、in In the region of S2.36, a local boundary smoothing is performed using an anisotropic diffusion equation to generate a local boundary smoothed lesion probability map, wherein, denotes a sensitivity threshold. S2.37, the sports field with the lesion probability map after the motion-aware boundary stabilization process and the lesion probability map after the local boundary smoothing, output the final stabilized lesion probability map . 5.The intelligent auxiliary diagnostic system for early cancer in the digestive tract based on deep learning technology according to claim 3, characterized in that: The lesion classification unit (4) classifies the segmented lesion region into different lesion types using a convolutional neural network model, including the following steps: S3.1, receiving the segmentation mask output by the lesion segmentation unit (3) ; S3.2、 according to the segmentation mask cropping the lesion region block from the original image and performing normalization processing, using the artifact perception focal self-correction network to perform image enhancement on the normalized image block, to generate an enhanced lesion region image block , and inputting the enhanced lesion region image block into the lesion segmentation unit (3) to replace the lesion region image block output by the lesion detection unit (2) in step S2.1; The focal point self-correction network with artifact perception is used for image enhancement on the normalized image block, including the following steps: S3.21、to the normalized lesion area image block , a lightweight artifact detection network ADNet is used for parallel inference to generate an artifact heat map of the same size ; S3.22, based on artifact heat map Generating focus self-correcting spatial weight mask for dynamically suppressing the impact of interference regions on subsequent feature extraction; S3.23, utilizing focal point self-correcting spatial weight mask performing local enhancement to generate an enhanced lesion region image block performing local enhancement to generate an enhanced lesion region image block ; S3.3, using a deep convolutional neural network as a lesion classification model; S3.4, the enhanced lesion region image block input the lesion classification model to extract deep semantic features step by step; S3.5, inputting deep semantic features into the full connection layer of the lesion classification model and performing normalization processing through a Softmax activation function to output the probability distribution of the lesion region belonging to each preset category; S3.6, based on the probability distribution, selecting the category corresponding to the maximum probability as the final classification result of the lesion region, and recording the maximum probability value of the category as the classification confidence; S3.7, outputting the classification result of each lesion region. 6.The intelligent auxiliary diagnostic system for early cancer in the digestive tract based on deep learning technology according to claim 1, characterized in that: The diagnosis fusion unit (5) receives the bounding box coordinates of the lesion region and its positioning confidence, the classification result, and the segmentation mask of the lesion, maps the lesion classification result and the segmentation mask to the corresponding lesion region in the original endoscope image according to the bounding box coordinates, and generates an auxiliary diagnosis report.

Images