Digestive tract early-stage cancer intelligent auxiliary diagnosis system based on deep learning technology

Abstract

The invention relates to the technical field of intelligent auxiliary diagnosis, in particular to an intelligent auxiliary diagnosis system for early-stage cancer of the digestive tract based on a deep learning technology. The method comprises the following steps: an image acquisition and processing unit acquires and preprocesses an image in a digestive tract; the lesion detection unit uses Faster R-CNN to position a lesion area, calculates a local curvature in combination with endoscope pose data, and generates an anchor frame with a self-adaptive rotation angle to optimize the process of positioning the lesion area; the lesion segmentation unit segments the lesion area and outputs a segmentation mask of the lesion; the lesion classification unit performs classification diagnosis of different lesion types on the segmented lesion areas by using a convolutional neural network model based on the segmented lesion areas; and the diagnosis fusion unit outputs an auxiliary diagnosis report in combination with the lesion detection, the lesion classification result and the lesion segmentation mask. According to the method, through a multi-stage collaborative optimization strategy, motion interference and imaging artifacts in endoscope imaging are effectively coped with.

Description

Technical Field

[0001] The present invention relates to the field of intelligent auxiliary diagnosis technology, and specifically to an intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology. Background Art

[0002] With the increasing incidence of gastrointestinal cancers, early detection and treatment are crucial for improving patient outcomes. However, gastrointestinal cancers, especially in their early stages, often lack obvious clinical symptoms, leading many patients to be diagnosed only at advanced stages, when treatment becomes more challenging and survival rates decline significantly. Traditional endoscopy, the primary screening method, relies heavily on the physician's experience and judgment. This leads to significant subjective variability in practice and increases the risk of misdiagnosis or missed diagnosis. Furthermore, given the massive volume of gastrointestinal endoscopic image data and its complex structural information, manual analysis is not only time-consuming and labor-intensive, but also difficult to ensure consistent and accurate results. Furthermore, the uneven distribution of medical resources further limits many patients' access to timely and effective medical services, especially in remote areas where highly skilled medical professionals are scarce. Therefore, an intelligent assisted diagnosis system for early gastrointestinal cancer based on deep learning technology was designed. Summary of the Invention

[0003] The purpose of the present invention is to provide an intelligent auxiliary diagnosis system for early gastrointestinal cancer based on deep learning technology, so as to solve the problems raised in the above-mentioned background technology, such as low early cancer detection rate and high risk of misdiagnosis and missed diagnosis in traditional endoscopic examinations due to large differences in doctors' subjective judgments, strong hidden nature of early lesions, many image artifacts and motion interference, and inaccurate positioning under complex curved surface structures.

[0004] To achieve the above objectives, the present invention provides an intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology, comprising:

[0005] An image acquisition and processing unit, which acquires images of the digestive tract and performs preprocessing;

[0006] A lesion detection unit, which locates the lesion area based on the preprocessed image using Faster R-CNN, and simultaneously calculates the local curvature in combination with the endoscope pose data to generate an anchor frame with adaptive rotation angle to optimize the process of locating the lesion area;

[0007] a lesion segmentation unit, which segments the lesion area and outputs a segmentation mask of the lesion;

[0008] A lesion classification unit, wherein the lesion classification unit uses a convolutional neural network model to classify and diagnose different lesion types based on the segmented lesion area;

[0009] A diagnosis fusion unit combines lesion detection, lesion classification results and lesion segmentation masks to output an auxiliary diagnosis report.

[0010] As a further improvement of the present technical solution, the image acquisition and processing unit includes an image acquisition module and an image processing module;

[0011] Wherein, the image acquisition module is used to acquire images of the interior of the digestive tract;

[0012] The image processing module is used to pre-process the collected images.

[0013] As a further improvement of the present technical solution, the lesion detection unit locates the suspected lesion area based on the preprocessed image using Faster R-CNN, including the following steps:

[0014] S1.1, input the preprocessed image into the convolutional neural network feature extraction subnet;

[0015] S1.2, extracting multi-scale feature maps through multi-layer convolution and pooling operations;

[0016] S1.3. Generate multi-scale, multi-aspect ratio anchor boxes at each position in the multi-scale feature map, and predict the overlap and position offset of each anchor box with the lesion area through a lightweight convolutional network.

[0017] S1.4. Use non-maximum suppression to remove overlapping anchor boxes and output candidate regions.

[0018] S1.5. Map the candidate regions to the corresponding positions of the multi-scale feature maps, and unify the features of the candidate regions of different sizes into a fixed-size feature tensor through the ROI Align operation;

[0019] S1.6. Perform bounding box regression on the features of the candidate region to calculate the lesion location coordinates and size;

[0020] S1.7. Perform non-maximum suppression again to remove the remaining redundant frames and obtain the final set of lesion regions.

[0021] S1.8. Output the bounding box coordinates of each lesion area and its positioning reliability.

[0022] As a further improvement of the present technical solution, in S1.3, a multi-scale, multi-aspect ratio anchor frame is generated at each position of the multi-scale feature map, and the overlap and position offset of each anchor frame with the lesion area are predicted by a lightweight convolutional network, including the following steps:

[0023] S1.31. Calculate the local curvature of the input multi-scale feature map in combination with the endoscope pose data to generate an anchor frame with adaptive rotation angle;

[0024] S1.32. Use the lightweight convolutional subnetwork set in the region proposal network to extract the corresponding local feature representation for each anchor box;

[0025] S1.33, input the extracted features into two parallel fully connected layers;

[0026] The first layer is used for binary classification and outputs the probability that each anchor box is the lesion area; the second layer is used for bounding box regression and outputs the coordinate offset of the anchor box relative to the actual lesion area;

[0027] S1.34. Calculate the overlap score for each anchor box based on the foreground probability output by the first layer.

[0028] S1.35. The foreground probability and position offset of all anchor boxes are used as the output of the region proposal network.

[0029] As a further improvement of the present technical solution, in S1.31, the local curvature is calculated in combination with the endoscope posture data to generate an anchor frame with adaptive rotation angle, including the following steps:

[0030] Read the inertial measurement unit data and lens depth sensor data of the endoscope in real time to obtain the three-dimensional posture information of the current frame lens;

[0031] Based on the three-dimensional posture information, calculate the angle between the lens optical axis and the digestive tract wall section , and measure the distance from the lens to the mucosal surface ;

[0032] According to the angle and distance Construct a local digestive tract surface model to calculate the local curvature of the area ;

[0033] When generating an anchor frame at each position of the feature map, the main direction angle of the anchor frame is adjusted according to the local curvature. Perform dynamic rotational adjustments to align the anchor frame with the actual geometry of the digestive tract wall;

[0034] On the basis of maintaining the original multi-scale and aspect ratio design, the anchor frame that has been rotated and adjusted is translated to form an anchor frame with adaptive rotation angle.

[0035] As a further improvement of the present technical solution, the lesion segmentation unit segments the lesion area and outputs a segmentation mask of the lesion, including the following steps:

[0036] S2.1. Receive the lesion area image block output by the lesion detection unit and perform preprocessing;

[0037] S2.2. Input the preprocessed lesion area image block into the deep convolutional network based on the U-Net architecture and output the lesion probability map ;

[0038] S2.3. Lesion Probability Map Input the heartbeat motion field estimation network and output the motion field , and perform stabilization processing to generate a stable lesion probability map , used to compensate for the jitter of digestive tract lesion boundaries caused by heartbeat in endoscopic imaging;

[0039] S2.4. Lesion Probability Map Using probability threshold Perform binarization to obtain a preliminary binary mask ;

[0040] S2.5. Preliminary binary mask Perform motion-compensated inverse transform to restore the original motion state and generate a motion-compensated binary mask ;

[0041] S2.6, the binary mask after motion compensation Restoration to the original lesion area size;

[0042] S2.7. Output segmentation mask of lesion area .

[0043] As a further improvement of this technical solution, in S2.3, the lesion probability map Input the heartbeat motion field estimation network and output the motion field , and perform stabilization processing to generate a stable lesion probability map , including the following steps:

[0044] S2.31. Change the lesion probability map Input the heartbeat motion field estimation network and output the corresponding motion field ;

[0045] S2.32, based on lesion probability map Calculate the probability gradient and calculate the magnitude of the probability gradient;

[0046] S2.33. Generating motion sensitivity heatmap based on gradient strength , which is used to indicate how sensitive the boundary is to the heartbeat motion;

[0047] S2.34. Construct a time-domain band-stop filter for the heartbeat frequency to suppress the heartbeat frequency component;

[0048] S2.35, Motion Sensitivity Heat Map For areas where the value is lower than the lower threshold b, the probability value of the corresponding pixel is processed by time domain band-stop filtering to generate a lesion probability map after motion-aware boundary stabilization processing. , to suppress periodic motion interference;

[0049] S2.36, in In the region of , the anisotropic diffusion equation is used to perform local boundary smoothing to generate the lesion probability map after local boundary smoothing, where represents the sensitivity threshold;

[0050] S2.37, the sports field and the lesion probability map after motion-aware boundary stabilization It is fused with the lesion probability map after local boundary smoothing to output the final stable lesion probability map. .

[0051] As a further improvement of the present technical solution, the use of a convolutional neural network model to classify and diagnose different lesion types in the segmented lesion area includes the following steps:

[0052] S3.1. Receive the segmentation mask output by the lesion segmentation unit ;

[0053] S3.2, according to the segmentation mask The lesion area block is cropped from the original image and normalized. The normalized image block is enhanced using the artifact-aware focus self-correction network to generate an enhanced lesion area image block. , and the enhanced lesion area image block At the same time, the lesion segmentation unit is input to replace the lesion area image block output by the lesion detection unit in step S2.1;

[0054] S3.3. Using deep convolutional neural networks as lesion classification models;

[0055] S3.4, the enhanced lesion area image block Input the lesion classification model and extract deep semantic features step by step;

[0056] S3.5. Input the deep semantic features into the fully connected layer of the lesion classification model and normalize them using the Softmax activation function to output the probability distribution of the lesion area belonging to each preset category;

[0057] S3.6. Based on the probability distribution, select the category corresponding to the maximum probability as the final classification result for the lesion area, and record the maximum probability value of the category as the classification confidence;

[0058] S3.7. Output the classification results of each lesion area.

[0059] As a further improvement of the present technical solution, in S3.2, performing image enhancement on the normalized image block using an artifact-aware focus self-correction network includes the following steps:

[0060] S3.21, for the normalized lesion area image block , using the lightweight artifact detection network ADNet parallel reasoning to generate artifact heat maps of the same size ;

[0061] S3.22, based on artifact heat map Generate focal self-correction spatial weight mask , used to dynamically suppress the influence of interference areas on subsequent feature extraction;

[0062] S3.23. Using focal self-correction spatial weight mask The original normalized lesion area image block Perform local enhancement to generate enhanced image blocks of the lesion area .

[0063] As a further improvement of the present technical solution, the diagnostic fusion unit receives the bounding box coordinates of the lesion area and its positioning reliability, the classification results, and the segmentation mask of the lesion, and maps the lesion classification results and the segmentation mask to the corresponding lesion area in the original endoscopic image according to the bounding box coordinates, and generates an auxiliary diagnosis report.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. In this invention, a rotation-angle adaptive anchor frame generation mechanism based on endoscopic pose data is introduced into the Faster R-CNN region proposal network. By fusing IMU and depth sensor data in real time, the angle and distance between the lens pose and the digestive tract wall are calculated, and the local curvature is dynamically modeled. The rotation angle of the anchor frame is adjusted accordingly to align its long axis with the direction of irregular lesions (such as flat and circumferential lesions) on the curved tube wall. This design effectively overcomes the problem of traditional axis-aligned rectangular anchor frames covering too much background and poor fit on curved surface structures, allowing candidate regions to more closely fit the actual lesion morphology. This significantly improves the accuracy of lesion detection and positioning precision in the complex curved environment of the digestive tract, provides high-quality input for subsequent segmentation and classification tasks, and thus reduces the overall risk of false detection and missed detection.

[0066] 2. In the present invention, a multi-stage collaborative optimization strategy is used to effectively deal with motion interference and imaging artifacts in endoscopic imaging. In the segmentation stage, the heartbeat motion field estimation network (HME-Net) is introduced in combination with motion-aware boundary stabilization processing and anisotropic diffusion to suppress boundary jitter caused by the heartbeat and generate a stable and clear segmentation mask. In the classification stage, an artifact-aware focal self-correction network is used to generate multi-category artifact heat maps through a lightweight ADNet, and a spatial weight mask is constructed to dynamically suppress the influence of interfering areas such as mucus, bubbles, and liquid edges. At the same time, CLAHE is integrated to enhance the retention of potential lesion details. This mechanism retains key diagnostic information while enhancing image quality, significantly improving the robustness of the classification model to noise and artifacts. Finally, the diagnostic fusion unit integrates the detection, segmentation, and classification results, and outputs a structured report containing location, morphology, type, and visual segmentation map, greatly enhancing the clinical practicality and diagnostic reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is the overall flow chart of the present invention;

[0068] The meaning of each number in the figure is:

[0069] 1. Image acquisition and processing unit; 2. Lesion detection unit; 3. Lesion segmentation unit; 4. Lesion classification unit; 5. Diagnostic fusion unit. DETAILED DESCRIPTION

[0070] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0071] Example: See Figure 1 As shown, an intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology is provided, including:

[0072] The image acquisition and processing unit 1 acquires images of the digestive tract and performs preprocessing (acquired by an endoscope);

[0073] In this embodiment, the image acquisition and processing unit 1 includes an image acquisition module and an image processing module;

[0074] Wherein, the image acquisition module is used to acquire images of the interior of the digestive tract;

[0075] The image processing module is used to pre-process the collected images, including noise removal, contrast adjustment, brightness correction, cropping of irrelevant areas, etc.;

[0076] Lesion detection unit 2 uses Faster R-CNN (Faster Region-based Convolutional Neural Network) to locate the lesion area based on the preprocessed image. At the same time, it calculates the local curvature based on the endoscope posture data and generates an anchor frame with adaptive rotation angle to optimize the process of locating the lesion area.

[0077] In this embodiment, the lesion detection unit 2 locates the suspected lesion area based on the preprocessed image using Faster R-CNN, including the following steps:

[0078] S1.1. The preprocessed image is input into a convolutional neural network feature extraction subnet (such as a backbone network such as ResNet, VGG, or SwinTransformer). The convolutional neural network feature extraction subnet takes the preprocessed endoscopic image as input, extracts image features such as edges and textures through multiple convolutional layers and activation layers (such as ReLU), and then gradually downsamples through pooling layers to reduce the spatial size and enhance translation invariance. The superposition of multiple layers of convolution and pooling enables the network to obtain multi-scale feature maps of different scales and semantic depths. Finally, it outputs a set of feature maps with gradually decreasing size and increasing number of channels from shallow to deep layers;

[0079] S1.2. Extract multi-scale feature maps through multi-layer convolution and pooling operations (specifically: first, pass the preprocessed input image through multiple convolution layers, using convolution kernels of different sizes to extract features from the image to capture detailed information such as edges, textures, and colors of the lesion area; then, gradually reduce the spatial size of the feature map through pooling layers (such as maximum pooling or average pooling), achieving spatial dimensionality reduction of features and extraction of key information, while enhancing the model's translation invariance; after repeated convolution and pooling operations, rich feature representations can be obtained at different scales, allowing the network to preserve local details while capturing global contextual information), retaining significant features such as edges, textures, and color changes in the lesion area;

[0080] S1.3. Generate multi-scale, multi-aspect ratio anchor boxes at each position in the multi-scale feature map. Use a lightweight convolutional network to predict the overlap (foreground probability) and position offset of each anchor box with the lesion area. The lightweight convolutional network usually starts with a smaller input image and passes through a series of convolutional layers that combine depthwise separable convolution or point-wise convolution with point-by-point convolution. Each layer is followed by batch normalization and nonlinear activation (such as ReLU or Swish). Efficient feature extraction is achieved by reducing the number of parameters and computation. The network has a shallow number of layers and a moderate number of channels. The output of the intermediate layers maintains a small spatial size and channel depth, and ultimately outputs a compact feature representation.

[0081] Among them, a multi-scale, multi-aspect ratio anchor frame is generated at each position of the multi-scale feature map, and the overlap (foreground probability) and position offset of each anchor frame with the lesion area are predicted through a lightweight convolutional network. The following steps are included:

[0082] S1.31. Calculate the local curvature of the input multi-scale feature map in combination with the endoscope pose data to generate an anchor frame with adaptive rotation angle;

[0083] Furthermore, traditional rectangular anchor frames are difficult to fit irregular lesions (including flat lesions and circumferential lesions) on the curved surface of the digestive tract, resulting in the positioning frame containing a large amount of irrelevant mucosal background, reducing the accuracy of subsequent classification / segmentation. By combining the endoscope IMU posture data with the depth sensor distance information to construct a local curvature model, and dynamically calculating the rotation angle based on the curvature to generate an adaptive anchor frame, real-time alignment of the anchor frame morphology with the local anatomical structure of the digestive tract is achieved, effectively solving the problems of traditional fixed rectangular anchor frames having low fit on the curved digestive tract surface, being prone to containing a large amount of irrelevant mucosal background, and having a mismatch between the direction and the direction of the lesion. Its rotational adaptive mechanism allows the long axis of the anchor frame to automatically fit the direction of the lesion, and cooperates with translation correction to maintain spatial consistency, reduce background interference, and improve positioning accuracy. The introduction of random angle perturbations enhances the model's robustness to changes in perspective and lens tilt, thereby providing the segmentation unit with a purer lesion area and the classification unit with geometrically aligned high-quality image blocks, significantly improving downstream task performance, reducing the missed detection rate of flat early cancers, and improving the accuracy of circumferential lesion boundary recognition.

[0084] The local curvature is calculated based on the endoscope pose data to generate an anchor frame with adaptive rotation angle, which includes the following steps:

[0085] Read the inertial measurement unit (IMU) data and lens depth sensor data of the endoscope in real time to obtain the three-dimensional posture information of the lens in the current frame (the lens refers to the endoscope lens);

[0086] Based on the three-dimensional posture information, calculate the angle between the lens optical axis and the digestive tract wall section ( , where is the unit vector of the lens optical axis in the world coordinate system, is the unit normal vector of the local digestive tract section), and measures the distance from the lens to the mucosal surface (Depth sensor output distance The unit is millimeters, which has been converted to pixel coordinates using the endoscope lens focal length and image resolution (pixels / mm);

[0087] According to the angle and distance Construct a local digestive tract surface model to calculate the local curvature of the area (The surface model of the local digestive tract is: , local curvature The unit is ), where the area refers to the local digestive tract inner wall area that the optical axis of the endoscope lens in the current frame faces or covers;

[0088] When generating an anchor frame at each position of the feature map, the main direction angle of the anchor frame is adjusted according to the local curvature. Perform dynamic rotation adjustment (dynamic rotation adjustment is based on the rotation angle Rotate (rotation angle Unit: radian), the calculation formula of the rotation angle is: , where is the inverse tangent function, which is used to map the product of curvature and distance to a finite angle to avoid excessive rotation angles. is a random perturbation angle (unit: radians), used to introduce angle data augmentation during training and improve the model's robustness to rotation invariance. Its value range is [-5°, 5°], set based on the angular fluctuation range under real-world imaging conditions. This aligns the anchor frame with the actual geometry of the digestive tract wall. The greater the curvature (the steeper the wall), the larger the required rotation angle, so that the anchor frame's long axis aligns with the lesion's direction.

[0089] While maintaining the original multi-scale and aspect ratio design, the rotated anchor frame is subjected to translation correction (the translation correction step is based on the rotated anchor frame, spatially aligning the center position of the anchor frame with the corresponding position in the feature map, calculating the offset of the anchor frame caused by the rotation, and then performing translation adjustment on the center coordinates of the anchor frame to keep it consistent with the spatial correspondence with the grid points in the feature map), forming a rotation-adaptive anchor frame;

[0090] S1.32. Utilize the lightweight convolutional subnetwork set in the region proposal network (specifically, a convolutional neural network submodule with a simple structure and efficient computation is used within the region proposal network (RPN). This lightweight convolutional subnetwork is responsible for extracting local features for each anchor box. The region proposal network (RPN) takes a multi-scale feature map as input and first extracts a shared feature representation for each position through a 3×3 convolutional layer. It then branches into two parallel lightweight convolutional subnetworks: one for binary classification, outputting the probability of each anchor box being foreground (lesion) or background, and the other for bounding box regression, outputting the coordinate offset of the anchor box relative to the true target. The intermediate layer maintains the same spatial size as the input feature map, and finally generates the category scores and position adjustment parameters of multiple anchor boxes). Extract the corresponding local feature representation for each anchor box (the anchor box here is a rotation-adaptive anchor box).

[0091] S1.33. Input the extracted features into two parallel fully connected layers (or convolutional layers);

[0092] The first layer is used for binary classification and outputs the probability (i.e., overlap) of each anchor box as a lesion area (foreground) or background. The second layer is used for bounding box regression and outputs the coordinate offset of the anchor box relative to the true lesion area (position adjustment parameters, including center coordinate offset and width and height scaling).

[0093] S1.34. Calculate the overlap score for each anchor box based on the foreground probability output by the first layer (specifically, based on the foreground probability output by the first layer, first normalize the probability value of each anchor box predicted to be a lesion area. This probability serves as the overlap score between the anchor box and the actual lesion area). This score is used for subsequent screening and ranking of candidate areas.

[0094] S1.35. Take the foreground probability and position offset of all anchor boxes as the output of the region proposal network;

[0095] S1.4. Use non-maximum suppression (NMS) to remove highly overlapping anchor boxes and output candidate regions (screen out candidate regions with confidence greater than a preset upper threshold c). Non-maximum suppression (NMS) is used to remove highly overlapping anchor boxes as follows: first, sort all anchor boxes according to their foreground probability, starting with the anchor box with the highest confidence, and traverse and retain each anchor box one by one. At the same time, remove other anchor boxes whose overlap (such as IoU) exceeds the upper threshold c. Repeat this process until all anchor boxes have been processed, and finally output a set of candidate regions with high confidence and small overlap.

[0096] S1.5. Map the candidate regions to the corresponding positions of the multi-scale feature maps. Use the ROI Align operation to unify the features of candidate regions of different sizes into a fixed-size feature tensor to facilitate subsequent positioning refinement. The ROI Align operation is as follows: first, each candidate region is mapped to the corresponding position of the multi-scale feature map and accurately divided into a fixed number of sub-grids by region; then, bilinear interpolation is used to sample the feature values ​​in each sub-grid to avoid quantization errors; finally, the sampling results of all sub-grids are integrated into a fixed-size feature tensor, so that candidate regions of different sizes and aspect ratios can be input into the subsequent convolutional or fully connected layer for unified processing;

[0097] S1.6. Perform bounding box regression on the features of the candidate region to calculate the lesion location coordinates and size. The bounding box regression process inputs the features corresponding to the candidate region into the regression network. By predicting the coordinate offset relative to the initial anchor box (including the translation of the center point and the width and height scaling), the position and size of the candidate box are finely adjusted to calculate more accurate bounding box coordinates of the lesion region.

[0098] S1.7. Perform non-maximum suppression (NMS) again to remove the remaining redundant boxes and obtain the final set of lesion regions.

[0099] S1.8. Output the bounding box coordinates of each lesion area and its positioning reliability.

[0100] The lesion segmentation unit 3 segments the lesion area and outputs a segmentation mask of the lesion;

[0101] In this embodiment, the lesion segmentation unit 3 segments the lesion area and outputs a segmentation mask of the lesion, including the following steps:

[0102] S2.1. Receive the lesion area image block output by lesion detection unit 2 and perform preprocessing, including adjusting it to a fixed resolution (e.g., 224×224 pixels) and normalizing the pixel values ​​(e.g., to the interval [0, 1]);

[0103] S2.2. Input the preprocessed lesion area image block into the deep convolutional network based on the U-Net architecture and output the lesion probability map , , where is the height of the lesion probability map (number of pixel rows), is the width of the lesion probability map (number of pixel columns). The deep convolutional network based on the U-Net architecture takes the preprocessed lesion area image block as input. First, the downsampling encoder part extracts gradually deeper semantic features through multi-layer convolution and pooling. The intermediate layer fuses the high-resolution features of the corresponding encoder layer with the corresponding decoder layer through skip connections. The decoder part gradually restores the spatial resolution through deconvolution or upsampling, and finally outputs a lesion probability map of the same size as the input image. Each pixel value represents the probability that the location belongs to a lesion.

[0104] S2.3. Lesion Probability Map Input the heartbeat motion field estimation network and output the motion field (playground It is a two-dimensional motion vector field with a size of 224×224. Each pixel uses two values ​​to describe its displacement in the x and y directions. It is stabilized to generate a stable lesion probability map. , used to compensate for the jitter of digestive tract lesion boundaries caused by heartbeat in endoscopic imaging;

[0105] The transmission of cardiac pulsation to the digestive tract (especially the esophagus and gastric fundus) causes continuous, subtle jittering of the mucosal surface, resulting in periodic blurring or jittering of the lesion probability map (U-Net output). Static segmentation models (U-Net) are unable to model temporal motion, causing segmentation mask boundaries to fluctuate with the heartbeat. Global motion compensation methods (such as video stabilization) ignore local anatomical characteristics and tend to oversmooth lesion edge details. This approach achieves high-precision geometric stabilization of segmentation results despite heartbeat interference. Compared to traditional methods that rely solely on global motion estimation or simple temporal smoothing, this approach uses the probability gradient magnitude to generate a motion sensitivity heatmap to quantify the sensitivity of lesion boundaries to motion. Based on this, a gradient-guided segmentation processing strategy is developed: temporal band-stop filtering is used to precisely suppress heartbeat frequency interference in low-sensitivity areas, while anisotropic diffusion is used to smooth noise along the edge tangent direction and preserve sharpness along the normal direction in high-sensitivity areas. Furthermore, the pixel displacement field predicted by HME-Net is combined to perform sub-pixel spatial deformation compensation, achieving precise alignment of the probability maps. This method significantly reduces boundary jitter caused by heartbeats, avoids area misjudgment, and preserves subtle topological features of lesion edges, thereby providing more stable lesion area input for the classification unit.

[0106] Disease probability map Input the heartbeat motion field estimation network and output the motion field , and perform stabilization processing to generate a stable lesion probability map , including the following steps:

[0107] S2.31. Change the lesion probability map Input the heartbeat motion field estimation network (the heartbeat motion field estimation network takes the lesion probability map of continuous frames as input, extracts spatiotemporal features through a multi-layer temporal convolutional network, captures the dynamic pixel displacement information of the lesion area caused by the heartbeat, and the middle layer generates a motion vector feature map by fusing time and space information. Finally, it outputs a two-dimensional motion field (including horizontal and vertical channels) to represent the motion direction and amplitude of each pixel point). Output the corresponding motion field (playground Unit: pixel / frame, calibrated parameters (pixels / mm) is associated with physical displacement), the motion field represents the pixel displacement vector field of the lesion area caused by heartbeat, which is used to guide subsequent motion compensation and stabilization processing;

[0108] S2.32, based on lesion probability map Calculate the probability gradient and calculate the magnitude of the probability gradient:

[0109] The probability gradient is:

[0110] ; ;

[0111] Where, is the gradient of the lesion probability map along the horizontal direction (x-axis), that is, the rate of change of the pixel value in the horizontal direction, is the gradient of the lesion probability map along the vertical direction (y-axis), that is, the rate of change of the pixel value in the vertical direction, is the infinitesimal change in the x direction (horizontal direction), is the infinitesimal change in the y direction (vertical direction), is the lesion probability map A small change in a certain direction;

[0112] Magnitude of the probability gradient for:

[0113] ;

[0114] S2.33. Generating motion sensitivity heatmap based on gradient strength , which indicates how sensitive the boundary is to heartbeat motion:

[0115] ;

[0116] Where, To control the motion sensitivity, the Gaussian kernel standard deviation is a hyperparameter that adjusts the gradient influence range and its value is 0.1.

[0117] S2.34. Construct a time-domain band-stop filter for the heartbeat frequency to suppress typical heartbeat frequency components:

[0118] ;

[0119] Where, is the frequency response function of the time domain band-stop filter, which suppresses the components near the heartbeat frequency. is the frequency variable, representing the signal frequency (Hz), is the heart rate, here it is set to a typical value of 1.2Hz, is the bandwidth parameter of the band-stop filter, which controls the suppression range. Here it is 0.3 Hz (based on clinical data statistics);

[0120] S2.35, Motion Sensitivity Heat Map For areas where the value is lower than the lower threshold b (the value range of the lower threshold b is between 0.1 and 0.5, dimensionless), the probability value of the corresponding pixel is processed by time domain band-stop filtering to generate a lesion probability map after motion-aware boundary stabilization processing. , to suppress periodic motion interference;

[0121] ;

[0122] Where, is the pixel coordinate The filtered probability value after motion-aware boundary stabilization is is the pixel coordinate The original lesion probability map value at To represent the operator for Fourier transform of the function (image signal), the spatial domain signal is converted to the frequency domain. represents the inverse Fourier transform operator, which converts the frequency domain signal back to the spatial domain;

[0123] S2.36, in In the region of , the anisotropic diffusion equation is used to perform local boundary smoothing to generate the lesion probability map after local boundary smoothing, where Indicates the sensitivity threshold. In this embodiment ;

[0124] The specific steps for local boundary smoothing are:

[0125] ;

[0126] ;

[0127] Where, is the lesion probability map at location and diffusion time The pixel value of is the gradient operator, is the divergence operator, is the diffusion coefficient, which depends on the gradient size, It is a constant that controls the diffusion rate. It is used to suppress the diffusion at the boundary and prevent the boundary from being blurred. Its value range is 0.1 to 0.3.

[0128] S2.37, the sports field and the lesion probability map after motion-aware boundary stabilization It is fused with the lesion probability map after local boundary smoothing to output the final stable lesion probability map. The fusion process is as follows: Used to stabilize the probability map after motion-aware boundary processing Perform pixel-level motion compensation and combine it with the lesion probability map after local boundary smoothing (the optimized probability map obtained by smoothing the lesion probability map of the motion-sensitive area and enhancing the boundary using the anisotropic diffusion algorithm). The motion compensation results and the smoothing results are integrated through the weighted fusion method. The spatial correction guided by the motion vector and the continuity of the smoothed boundary are comprehensively considered to finally generate a stable and accurate lesion probability map. ;

[0129] S2.4. Lesion Probability Map Using probability threshold Perform binarization to obtain a preliminary binary mask :

[0130] ;

[0131] Where, is the pixel coordinate position in the image, in the preliminary binary mask The value of this position is: : The pixel is judged to be a lesion, : The pixel is judged as non-lesional;

[0132] S2.5. Preliminary binary mask Perform motion-compensated inverse transform to restore the original motion state (compensate the segmentation network's segmentation result on the stable image and convert it back to the position of the original image block) and generate a motion-compensated binary mask The inverse motion compensation transform is specifically performed as follows: using the motion vector field output by the heartbeat motion field estimation network, the displacement offset of each pixel point during the stabilization process is calculated; then, according to the motion vector, the pixels in the binary mask are reversely moved to restore them to their corresponding positions in the original image, thus achieving the inverse operation of the spatial transformation of the mask;

[0133] S2.6, the binary mask after motion compensation Restore the original lesion area size. If it was scaled before being input to the segmentation network, the mask needs to be scaled accordingly. If the mask needs to be mapped back to the entire original image, it is aligned according to the bounding box coordinates provided by the lesion detection unit 2;

[0134] S2.7. Output segmentation mask of lesion area (Segmentation mask of the lesion area It is a binary mask based on motion compensation , obtained after size and position alignment (i.e., step S3.5), whose size corresponds to the original lesion area.

[0135] The lesion classification unit 4 uses a convolutional neural network (CNN) model to classify and diagnose different lesion types based on the segmented lesion area.

[0136] In this embodiment, a convolutional neural network (CNN) model is used to classify and diagnose different lesion types on the segmented lesion area, including the following steps:

[0137] S3.1. Receive the segmentation mask output by lesion segmentation unit 3 ;

[0138] S3.2, according to the segmentation mask The lesion area block is cropped from the original image and normalized: uniformly scaled to a fixed size (e.g., 224×224 pixels) to adapt to the classification network input requirements, and the normalized image block is enhanced using the artifact-aware focus self-correction network to generate the enhanced lesion area image block , and the enhanced lesion area image block At the same time, the image block of the lesion area is input into the lesion segmentation unit 3 to replace the lesion area image block output by the lesion detection unit 2 in step S2.1, so as to improve the segmentation accuracy of the lesion boundary while suppressing artifact interference;

[0139] Mucus adhesion, bubble accumulation, or liquid reflections within the digestive cavity can obscure lesions (microscopic surface structures of early-stage cancer) or create false edges that resemble lesions (bubble outlines are misinterpreted as ridge boundaries). Global image enhancement (such as histogram equalization) can also amplify noise in artifact regions, while segmentation-based masking methods directly discard artifact regions, resulting in permanent loss of some lesion information. An anatomically semantically guided local enhancement strategy achieves accurate restoration of lesion features and morphological preservation despite artifact interference. Unlike traditional unified enhancement or simple binary masking methods, this method utilizes a three-channel artifact heatmap (mucus, bubble, and liquid edge probabilities) output by a lightweight ADNet to accurately model the interference type and spatial distribution. Differentiated processing strategies are then designed for different artifact types, such as morphological dilation of bubble regions to protect attached lesion tissue. Based on this, a focus-self-corrected spatial weighted mask is generated, dynamically adjusting the suppression strength at the pixel level: CLAHE is used to enhance local contrast in artifact regions to reveal obscured subtle textures, while clean regions preserve the original signal to avoid over-enhancing noise. This method effectively relieves the suppression of lesion features by mucus, bubbles, and liquid edges, reduces the false positive rate of the classification model, and maintains the topological continuity of the lesion boundary through a soft transition. It can also preserve potential malignant features in liquid reflective areas.

[0140] Image enhancement is performed on the normalized image patches using an artifact-aware focal self-correction network, which includes the following steps:

[0141] S3.21, for the normalized lesion area image block , using the lightweight artifact detection network ADNet parallel inference (the lightweight artifact detection network ADNet consists of 4 layers of lightweight convolution, the input is the normalized lesion area image block (such as 224×224), each layer of convolution is followed by batch normalization and activation function to extract local semantic features; the network outputs a three-channel artifact heat map of the same size as the input, which represents the pixel probabilities of mucus, bubbles and liquid edges respectively, and is used for subsequent focus self-correction spatial weight mask generation, thereby dynamically suppressing the influence of interference areas on feature extraction and lesion classification), generating an artifact heat map of the same size :

[0142] ;

[0143] ADNet is a 4-layer lightweight convolutional neural network with an input resolution of 224×224. It outputs a heatmap of the same size as the input. The heatmap contains three channels of semantic information: Channel 1: mucus probability (0-1); Channel 2: bubble probability (0-1); Channel 3: liquid edge probability (0-1);

[0144] S3.22, based on artifact heat map Generate focal self-correction spatial weight mask , used to dynamically suppress the impact of interference areas on subsequent feature extraction:

[0145] ;

[0146] Where, is the first channel of the artifact heat map, representing the probability value matrix of the mucus area (element value range 0~1), The second channel of the artifact heat map represents the probability value matrix of the bubble area (element value range is 0~1). The third channel of the artifact heat map represents the probability value matrix of the liquid edge area (element value range 0~1). It is a morphological dilation operation used to expand the bubble area boundary to avoid the false suppression of important lesion information. is the element-by-element weighted summation (the corresponding matrix elements are added together, including weight coefficients), is the mucus artifact weight coefficient, which is set according to the degree of interference caused by mucus artifacts on lesion identification. Mucus usually appears in a large area and is blurred, so it needs to be moderately suppressed. The value range is 0.5~0.8, is the bubble artifact weight coefficient. Bubble artifacts produce obvious edges and significant interference, so the weight is generally large, and expansion is used to avoid excessive suppression of edge information. The value range is 0.8~1.2, For the liquid edge artifact weight coefficient, set a smaller weight to maintain details and avoid over-smoothing. The value range of is 0.3~0.6;

[0147] S3.23. Using focal self-correction spatial weight mask The original normalized lesion area image block Perform local enhancement to generate enhanced image blocks of the lesion area , taking into account both preserving the characteristics of residual lesions in the interference area and improving contrast:

[0148] ;

[0149] Where, The image block is processed by adaptive contrast limited histogram equalization, which is used to enhance the local contrast and details of the interference area. It is the element-by-element product (Hadamard product), which performs weighted processing on each pixel of the image;

[0150] S3.3. Use a deep convolutional neural network as the lesion classification model. The lesion classification model uses a modified ResNet-50 as the backbone network and introduces attention mechanism modules, including channel attention (such as the SE module) and spatial attention (such as the CBAM module), to enhance the model's ability to focus on key lesion texture, color distribution, and microstructural features;

[0151] S3.4, the enhanced lesion area image block The input lesion classification model passes through multiple convolutional layers, batch normalization, activation functions (such as ReLU or Swish), and pooling layers to extract deep semantic features step by step;

[0152] S3.5. Input the deep semantic features into the fully connected layer of the lesion classification model and normalize them using the Softmax activation function to output the probability distribution of the lesion area belonging to each preset category;

[0153] S3.6. Based on the probability distribution, select the category corresponding to the maximum probability as the final classification result for the lesion area, and record the maximum probability value of the category as the classification confidence;

[0154] S3.7. Output the classification results for each lesion area, including: lesion type label (identifying which type of lesion the lesion area belongs to (including adenoma, early cancer, well-differentiated cancer, etc.)), classification confidence (the predicted probability value of the corresponding lesion type, used to measure the credibility of the classification result).

[0155] The diagnosis fusion unit 5 combines the lesion detection, lesion classification results and the lesion segmentation mask to output an auxiliary diagnosis report;

[0156] In this embodiment, the diagnostic fusion unit 5 receives the bounding box coordinates of the lesion area and its positioning confidence, the classification result (lesion type label and classification confidence), and the segmentation mask of the lesion. According to the bounding box coordinates, the lesion classification result and the segmentation mask are mapped to the corresponding lesion area in the original endoscopic image, and an auxiliary diagnosis report is generated. The report content should include: the specific location, size, and morphological description of the lesion area; the classification result of the lesion type; the segmentation map of the lesion area (showing the range of the lesion); and recommended measures for each finding (such as further examination, treatment plan recommendations, etc.).

[0157] The basic principles, main features, and advantages of the present invention are shown and described above. It should be understood by those skilled in the art that the present invention is not limited to the above-described embodiments. The above-described embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention, and such changes and modifications fall within the scope of the invention claimed.

Claims

1. An intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology, characterized by: include: An image acquisition and processing unit (1), wherein the image acquisition and processing unit (1) acquires images of the digestive tract and performs preprocessing; A lesion detection unit (2) locates the lesion area based on the preprocessed image using Faster R-CNN, and simultaneously calculates the local curvature in combination with the endoscope posture data to generate an anchor frame with adaptive rotation angle to optimize the process of locating the lesion area; A lesion segmentation unit (3), wherein the lesion segmentation unit (3) segments the lesion area and outputs a segmentation mask of the lesion; A lesion classification unit (4), wherein the lesion classification unit (4) performs classification diagnosis of different lesion types on the segmented lesion area using a convolutional neural network model; A diagnosis fusion unit (5) is provided, wherein the diagnosis fusion unit (5) combines the lesion detection, the lesion classification result and the segmentation mask of the lesion to output an auxiliary diagnosis report.

2. The intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology according to claim 1 is characterized by: The image acquisition and processing unit (1) comprises an image acquisition module and an image processing module; Wherein, the image acquisition module is used to acquire images of the interior of the digestive tract; The image processing module is used to pre-process the collected images.

3. The intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology according to claim 1 is characterized by: The lesion detection unit (2) locates the suspected lesion area based on the preprocessed image using Faster R-CNN, including the following steps: S1.1, input the preprocessed image into the convolutional neural network feature extraction subnet; S1.2, extracting multi-scale feature maps through multi-layer convolution and pooling operations; S1.

3. Generate multi-scale, multi-aspect ratio anchor boxes at each position in the multi-scale feature map, and predict the overlap and position offset of each anchor box with the lesion area through a lightweight convolutional network. S1.

4. Use non-maximum suppression to remove overlapping anchor boxes and output candidate regions. S1.

5. Map the candidate regions to the corresponding positions of the multi-scale feature maps, and unify the features of the candidate regions of different sizes into a fixed-size feature tensor through the ROI Align operation; S1.

6. Perform bounding box regression on the features of the candidate region to calculate the lesion location coordinates and size; S1.

7. Perform non-maximum suppression again to remove the remaining redundant frames and obtain the final set of lesion regions. S1.

8. Output the bounding box coordinates of each lesion area and its positioning reliability.

4. The intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology according to claim 3 is characterized by: In S1.3, a multi-scale, multi-aspect ratio anchor frame is generated at each position of the multi-scale feature map, and the overlap and position offset of each anchor frame with the lesion area are predicted by a lightweight convolutional network, including the following steps: S1.

31. Calculate the local curvature of the input multi-scale feature map in combination with the endoscope pose data to generate an anchor frame with adaptive rotation angle; S1.

32. Use the lightweight convolutional subnetwork set in the region proposal network to extract the corresponding local feature representation for each anchor box; S1.33, input the extracted features into two parallel fully connected layers; The first layer is used for binary classification and outputs the probability that each anchor box is the lesion area; the second layer is used for bounding box regression and outputs the coordinate offset of the anchor box relative to the actual lesion area; S1.

34. Calculate the overlap score for each anchor box based on the foreground probability output by the first layer. S1.

35. The foreground probability and position offset of all anchor boxes are used as the output of the region proposal network.

5. The intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology according to claim 4 is characterized by: In S1.31, the local curvature is calculated in combination with the endoscope pose data to generate an anchor frame with adaptive rotation angle, including the following steps: Read the inertial measurement unit data and lens depth sensor data of the endoscope in real time to obtain the three-dimensional posture information of the current frame lens; Based on the three-dimensional posture information, calculate the angle between the lens optical axis and the digestive tract wall section , and measure the distance from the lens to the mucosal surface ; According to the angle and distance Construct a local digestive tract surface model to calculate the local curvature of the area ; When generating an anchor frame at each position of the feature map, the main direction angle of the anchor frame is adjusted according to the local curvature. Perform dynamic rotational adjustments to align the anchor frame with the actual geometry of the digestive tract wall; On the basis of maintaining the original multi-scale and aspect ratio design, the anchor frame that has been rotated and adjusted is translated to form an anchor frame with adaptive rotation angle.

6. The intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology according to claim 1 is characterized by: The lesion segmentation unit (3) segments the lesion area and outputs a segmentation mask of the lesion, including the following steps: S2.1, receiving the lesion area image block output by the lesion detection unit (2) and performing preprocessing; S2.

2. Input the preprocessed lesion area image block into the deep convolutional network based on the U-Net architecture and output the lesion probability map ; S2.

3. Lesion Probability Map Input the heartbeat motion field estimation network and output the motion field , and perform stabilization processing to generate a stable lesion probability map , used to compensate for the jitter of digestive tract lesion boundaries caused by heartbeat in endoscopic imaging; S2.

4. Lesion Probability Map Using probability threshold Perform binarization to obtain a preliminary binary mask ; S2.

5. Preliminary binary mask Perform motion-compensated inverse transform to restore the original motion state and generate a motion-compensated binary mask ; S2.6, the binary mask after motion compensation Restoration to the original lesion area size; S2.

7. Output segmentation mask of lesion area .

7. The intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology according to claim 6 is characterized by: In S2.3, the lesion probability map Input the heartbeat motion field estimation network and output the motion field , and perform stabilization processing to generate a stable lesion probability map , including the following steps: S2.

31. Change the lesion probability map Input the heartbeat motion field estimation network and output the corresponding motion field ; S2.32, based on lesion probability map Calculate the probability gradient and calculate the magnitude of the probability gradient; S2.

33. Generating motion sensitivity heatmap based on gradient strength , which is used to indicate how sensitive the boundary is to the heartbeat motion; S2.

34. Construct a time-domain band-stop filter for the heartbeat frequency to suppress the heartbeat frequency component; S2.35, Motion Sensitivity Heat Map For areas where the value is lower than the lower threshold b, the probability value of the corresponding pixel is processed by time domain band-stop filtering to generate a lesion probability map after motion-aware boundary stabilization processing. , to suppress periodic motion interference; S2.36, in In the region of , the anisotropic diffusion equation is used to perform local boundary smoothing to generate the lesion probability map after local boundary smoothing, where represents the sensitivity threshold; S2.37, the sports field and the lesion probability map after motion-aware boundary stabilization It is fused with the lesion probability map after local boundary smoothing to output the final stable lesion probability map. .

8. The intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology according to claim 6 is characterized by: The lesion classification unit (4) uses a convolutional neural network model to classify and diagnose different lesion types on the segmented lesion area, including the following steps: S3.

1. Receive the segmentation mask output by the lesion segmentation unit (3) ; S3.2, according to the segmentation mask The lesion area block is cropped from the original image and normalized. The normalized image block is enhanced using the artifact-aware focus self-correction network to generate an enhanced lesion area image block. , and the enhanced lesion area image block At the same time, the lesion segmentation unit (3) is input to replace the lesion area image block output by the lesion detection unit (2) in step S2.1; S3.

3. Using deep convolutional neural networks as lesion classification models; S3.4, the enhanced lesion area image block Input the lesion classification model and extract deep semantic features step by step; S3.

5. Input the deep semantic features into the fully connected layer of the lesion classification model and normalize them using the Softmax activation function to output the probability distribution of the lesion area belonging to each preset category; S3.

6. Based on the probability distribution, select the category corresponding to the maximum probability as the final classification result for the lesion area, and record the maximum probability value of the category as the classification confidence; S3.

7. Output the classification results of each lesion area.

9. The intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology according to claim 8, characterized in that: In S3.2, performing image enhancement on the normalized image block using an artifact-aware focus self-correction network includes the following steps: S3.21, for the normalized lesion area image block , using the lightweight artifact detection network ADNet parallel reasoning to generate artifact heat maps of the same size ; S3.22, based on artifact heat map Generate focal self-correction spatial weight mask , used to dynamically suppress the influence of interference areas on subsequent feature extraction; S3.

23. Using focal self-correction spatial weight mask The original normalized lesion area image block Perform local enhancement to generate enhanced image blocks of the lesion area .

10. The intelligent auxiliary diagnosis system for early digestive tract cancer based on deep learning technology according to claim 1 is characterized by: The diagnostic fusion unit (5) receives the bounding box coordinates of the lesion area and its positioning reliability, the classification result, and the segmentation mask of the lesion, maps the lesion classification result and the segmentation mask to the corresponding lesion area in the original endoscopic image according to the bounding box coordinates, and generates an auxiliary diagnosis report.

Images