Digestive tract auxiliary detection method and device based on deep learning

Abstract

The invention discloses a deep learning-based digestive tract auxiliary detection method and device, and relates to the technical field of auxiliary detection, and the method comprises the steps: carrying out the original image collection of the digestive tract of a patient through endoscope equipment based on an event-driven sampling strategy, carrying out the denoising, size adjustment and quality improvement processing of the original image, and obtaining a preprocessed image; using Canny edge detection to extract a suspected lesion area of the preprocessed image; in combination with a super-resolution generative adversarial network and a self-attention mechanism, performing super-resolution reconstruction and local enhancement on the preprocessed image to obtain a high-definition enhanced image; and based on the lesion segmentation mask and the high-definition enhanced image, lesion types are classified by using a pre-trained feature extraction network. According to the method, the super-resolution generative adversarial network and the self-attention mechanism are combined, super-resolution reconstruction and local enhancement are performed on the preprocessed image, image details are enhanced, especially the definition of a minimal lesion area is enhanced, and the omission ratio is effectively reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of auxiliary detection, and particularly to a method and device for gastrointestinal tract auxiliary detection based on deep learning. Background Art

[0002] In recent years, the detection technology of gastrointestinal diseases has developed rapidly. Especially in the field of endoscopic image analysis, the introduction of deep learning technology has significantly improved the accuracy and efficiency of lesion area detection. Traditional gastrointestinal tract auxiliary detection methods mainly rely on doctors' experience and manual annotation tools. This method is not only time-consuming and laborious, but also easily affected by subjective factors, resulting in unstable diagnostic results.

[0003] However, there are still some deficiencies in the existing technologies. For example, in the image enhancement link, it is often difficult for existing technologies to effectively retain the detailed information of small lesion areas. Especially in the super-resolution reconstruction process, due to the lack of a local enhancement mechanism for lesion areas, small lesion areas are likely to be missed. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a method for gastrointestinal tract auxiliary detection based on deep learning to solve the problem of missed detection of small lesion areas.

[0006] To solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for gastrointestinal tract auxiliary detection based on deep learning, which includes

[0008] Collecting original images of a patient's gastrointestinal tract through an endoscopic device based on an event-driven sampling strategy, and performing denoising, size adjustment, and quality improvement processing on the original images to obtain preprocessed images;

[0009] Extracting suspected lesion areas of the preprocessed images by using Canny edge detection;

[0010] Combining a super-resolution adversarial generative network and a self-attention mechanism to perform super-resolution reconstruction and local enhancement on the preprocessed images to obtain high-definition enhanced images;

[0011] Identifying lesion areas by performing color analysis on the high-definition enhanced images;

[0012] Using an active contour model to segment the lesion areas on the high-definition enhanced images to obtain lesion segmentation masks;

[0013] Classifying the lesion types based on the lesion segmentation masks and the high-definition enhanced images by using a pre-trained feature extraction network.

[0014] As a preferred solution of the deep learning-based digestive tract auxiliary detection method of the present invention, wherein: the original images of the patient's digestive tract are collected by an endoscope device based on an event-driven sampling strategy, and the original images are denoised, resized, and quality-improved to obtain preprocessed images. The specific steps are as follows:

[0015] When using the endoscope device to collect the original images of the patient's digestive tract, a preliminary judgment model is constructed based on a deep learning network to preliminarily judge the original images in real time;

[0016] Based on the judgment results of the preliminary judgment model on the originally collected images in real time, the acquisition frequency of the original images is adjusted;

[0017] The wavelet transform method is used to remove noise from the collected original images;

[0018] Through the bilinear interpolation method, the sizes of the original images after noise removal are unified, and the contrast of the original images with unified sizes is enhanced by histogram equalization;

[0019] The original images with enhanced contrast are normalized to obtain preprocessed images.

[0020] As a preferred solution of the deep learning-based digestive tract auxiliary detection method of the present invention, wherein: the suspected lesion areas of the preprocessed images are extracted by using Canny edge detection. The specific steps are as follows:

[0021] The Sobel operator is used to calculate the gradients of the preprocessed images in the horizontal and vertical directions to obtain the gradient magnitudes of the preprocessed images;

[0022] According to the gradients of the preprocessed images in the horizontal and vertical directions, the gradient directions of the preprocessed images are calculated;

[0023] Based on non-maximum suppression, when the gradient magnitude of the preprocessed image is not a local maximum in the gradient direction, the gradient magnitude of the preprocessed image is set to zero, otherwise, it is retained;

[0024] According to the gradient magnitudes of the preprocessed images, the histograms of all pixel points in the preprocessed images are statistically calculated to obtain the gradient distribution curves of all pixel points in the preprocessed images;

[0025] According to the gradient distribution curves, high and low thresholds for double-threshold filtering are set;

[0026] According to the comparison of the gradient magnitudes of the pixel points in the preprocessed images with the high and low thresholds, strong and weak edge points and non-edge points are distinguished;

[0027] Remove weak edge points and non-edge points without strong edge points in the neighborhood, and connect strong edge points and weak edge points with strong edge points in the neighborhood to obtain a suspected lesion area.

[0028] As a preferred solution of the deep learning-based gastrointestinal tract auxiliary detection method described in the present invention, wherein: combining a super-resolution adversarial generation network and a self-attention mechanism to perform super-resolution reconstruction and local enhancement on the preprocessed image to obtain a high-definition enhanced image, the specific steps are as follows.

[0029] In the super-resolution adversarial generation network, based on the lesion area attention mechanism, enhance the features of the suspected lesion area to obtain a dual-branch super-resolution generative adversarial network.

[0030] In the overall super-resolution reconstruction branch, the generator extracts local features of the preprocessed image in the convolutional layer and enhances the local features through residual blocks.

[0031] Perform upsampling through sub-pixel convolution to convert the preprocessed image into a high-resolution image.

[0032] The discriminator uses a convolutional neural network to perform discrimination by calculating the true probability of the high-resolution image.

[0033] In the lesion area enhancement branch, based on the gradient magnitude of the pixel points in the suspected lesion area, calculate the attention weight map of the suspected lesion area.

[0034] Enhance the local features of the suspected lesion area in the high-resolution image through the Transformer mechanism to obtain a high-definition enhanced image.

[0035] Use the gastrointestinal endoscope image database to train the dual-branch super-resolution generative adversarial network.

[0036] Obtain high-resolution gastrointestinal endoscope images from the gastrointestinal endoscope image database.

[0037] Crop the high-resolution gastrointestinal endoscope images input into the dual-branch super-resolution generative adversarial network to the same size and reduce the resolution by the nearest neighbor interpolation method to obtain paired high-resolution and low-resolution gastrointestinal endoscope images.

[0038] Input the paired gastrointestinal endoscope images into the dual-branch super-resolution generative adversarial network for alternating training.

[0039] Optimize the dual-branch super-resolution generative adversarial network by calculating the total loss function composed of the generative adversarial loss function, the edge-preserving loss function, and the medical feature perception loss function.

[0040] When the total loss function has converged before the paired gastrointestinal endoscope images are completely input into the dual-branch super-resolution generative adversarial network for training, the training is terminated prematurely;

[0041] Conversely, continue the alternating training until the dual-branch super-resolution generative adversarial network completes the training on all paired gastrointestinal endoscope images.

[0042] As a preferred embodiment of the gastrointestinal tract auxiliary detection method based on deep learning according to the present invention, wherein: the high-definition enhanced image is converted to the HSV color space to obtain the hue, saturation, and brightness of the high-definition enhanced image in the HSV color space;

[0043] According to the hue, saturation, and brightness ranges of healthy gastrointestinal tract images in the HSV color space, set the hue range threshold, saturation range threshold, and brightness range threshold;

[0044] When there are areas in the high-definition enhanced image where the hue, saturation, and brightness are not within the hue range threshold, saturation range threshold, and brightness range threshold, the lesion area of the high-definition enhanced image is identified.

[0045] As a preferred embodiment of the gastrointestinal tract auxiliary detection method based on deep learning according to the present invention, wherein: the active contour model is used to segment the lesion area on the high-definition enhanced image to obtain a lesion segmentation mask, and the specific steps are as follows.

[0046] Set an initial closed curve around the lesion area of the high-definition enhanced image;

[0047] Use the gradient descent method to optimize the ACM energy function, gradually shrink the initial closed curve, and obtain the initial boundary of the lesion area;

[0048] By calculating the shortest Euclidean distance from each pixel point in the high-definition enhanced image to the initial boundary of the lesion area, a lesion segmentation mask is obtained.

[0049] As a preferred embodiment of the gastrointestinal tract auxiliary detection method based on deep learning according to the present invention, wherein: based on the lesion segmentation mask and the high-definition enhanced image, a pre-trained feature extraction network is used to classify the lesion type, and the specific steps are as follows.

[0050] Based on the lesion segmentation mask, extract the lesion area on the high-definition enhanced image to obtain a high-definition enhanced lesion area image;

[0051] Input different lesion sample images and high-definition enhanced lesion area images in the gastrointestinal endoscope image database into the pre-trained feature extraction network, and calculate the lesion feature vectors respectively;

[0052] By calculating the cosine similarity of the lesion feature vectors between the lesion sample image and the high-definition enhanced lesion region image, the lesion type in the high-definition enhanced image is output.

[0053] In a second aspect, the present invention provides a deep learning-based digestive tract auxiliary detection device, including an image acquisition module, a suspected lesion module, an image enhancement module, a lesion recognition module, a lesion segmentation module, and a lesion classification module.

[0054] The image acquisition module is used to collect original images of the patient's digestive tract through an endoscopic device based on an event-driven sampling strategy, and perform denoising, size adjustment, and quality improvement processing on the original images to obtain preprocessed images.

[0055] The suspected lesion module is used to extract suspected lesion regions of the preprocessed images by using Canny edge detection.

[0056] The image enhancement module is used to combine a super-resolution adversarial generation network and a self-attention mechanism to perform super-resolution reconstruction and local enhancement on the preprocessed images to obtain high-definition enhanced images.

[0057] The lesion recognition module is used to identify lesion regions by performing color analysis on the high-definition enhanced images.

[0058] The lesion segmentation module is used to segment the lesion regions on the high-definition enhanced images by using an active contour model to obtain a lesion segmentation mask.

[0059] The lesion classification module is used to classify the lesion types based on the lesion segmentation mask and the high-definition enhanced images by using a pre-trained feature extraction network.

[0060] In a third aspect, the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program, and: when the computer program is executed by the processor, any step of the deep learning-based digestive tract auxiliary detection method described in the first aspect of the present invention is implemented.

[0061] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and: when the computer program is executed by the processor, any step of the deep learning-based digestive tract auxiliary detection method described in the first aspect of the present invention is implemented.

[0062] The beneficial effects of the present invention are as follows: By optimizing the image acquisition process based on an event-driven sampling strategy, the present invention reduces redundant data while increasing the attention to potential lesion areas, thereby improving the detection sensitivity and efficiency. It uses Canny edge detection to accurately extract suspected lesion areas, combines a super-resolution adversarial generation network and a self-attention mechanism to enhance image details, especially the clarity of tiny lesion areas, effectively reducing the missed detection rate. It accurately identifies the color features of lesion areas through HSV color space conversion and uses an active contour model to achieve fine segmentation, providing a reliable basis for quantitative analysis. It uses a pre-trained feature extraction network to perform high-precision classification of lesion types, supports the identification of multiple types of lesions, and provides support for formulating personalized treatment plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0064] Figure 1 It is a flowchart of a deep learning-based gastrointestinal tract auxiliary detection method in Embodiment 1;

[0065] Figure 2 It is a module diagram of a deep learning-based gastrointestinal tract auxiliary detection device in Embodiment 1. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0066] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will provide a detailed description of the specific embodiments of the present invention with reference to the accompanying drawings of the specification.

[0067] In the following description, many specific details are set forth to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0068] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that can be included in at least one implementation manner of the present invention. The appearances of "in one embodiment" in different places in this specification do not all refer to the same embodiment, nor are they separate or selectively exclusive embodiments from other embodiments.

[0069] Embodiment 1, referring to Figure 1 and Figure 2 This is the first embodiment of the present invention, which provides a deep learning-based gastrointestinal tract auxiliary detection method, including the following steps:

[0070] S1. Using an endoscopic device, based on an event-driven sampling strategy, collect original images of the patient's digestive tract, and perform denoising, size adjustment, and quality improvement processing on the original images to obtain preprocessed images.

[0071] When using an endoscopic device to collect original images of the patient's digestive tract, based on a deep learning network, construct a preliminary judgment model to preliminarily judge the original images in real time;

[0072] Furthermore, the construction and training process of the preliminary judgment model is as follows. Collect images with lesion area annotations and lesion comparison images in the same area from a digestive tract endoscopic image database (such as the HyperKvasir dataset);

[0073] After uniformly adjusting the size and normalizing the images with lesion area annotations and lesion comparison images in the same area, divide them into a preliminary training set and a preliminary validation set;

[0074] Input the preliminary training set into a preliminary judgment model based on MobileNetV2 for iterative training, and output the probability that the images in the preliminary training set have potential lesions;

[0075] After every 10 training epochs, perform performance verification on the preliminary judgment model on the preliminary validation set;

[0076] During the training process, by calculating the binary cross-entropy loss function of the preliminary judgment model, obtain the loss value of the preliminary judgment model, and use the Adam optimizer to optimize the learning rate of the preliminary judgment model. The expression is as follows:

[0077]

[0078] Among them, A is the binary cross-entropy loss value of the preliminary judgment model, N is the number of samples in the preliminary training set, y i is the true label of the i-th sample (1 for the presence of a lesion area and 0 for the absence of a lesion area), is the probability value that the preliminary judgment model predicts the i-th sample has a lesion area;

[0079] When the change rate of the loss value of the preliminary judgment model for ten consecutive validation sets does not exceed 2%, it indicates that the preliminary judgment model reaches the optimal state and the training of the preliminary judgment model is completed;

[0080] Based on the judgment result of the preliminary judgment model on the real-time collected original images, adjust the frequency of original image collection;

[0081] Specifically, if the probability value judged by the preliminary judgment model that there is a lesion area in the originally acquired real-time image increases, the acquisition frequency of the original image is increased. For example, when the probability that the preliminary judgment model judges that there is a lesion area in the originally acquired real-time image is greater than or equal to 0.7, the acquisition frequency of the original image is increased to 2 times the default acquisition frequency. When the probability that the preliminary judgment model judges that there is a lesion area in the originally acquired real-time image is greater than or equal to 0.3 and less than 0.7, the default acquisition frequency is maintained. When the probability that the preliminary judgment model judges that there is a lesion area in the originally acquired real-time image is less than 0.3, the acquisition frequency of the original image is reduced to 0.5 times the default acquisition frequency;

[0082] It should be noted that based on such an event-driven sampling strategy, the acquisition of original image data in the lesion-free area can be reduced. While improving the acquisition efficiency of the original images of the patient's digestive tract, the efficiency of further analyzing the original images is also improved. And MobileNetV2 is a network structure based on depthwise separable convolution, which has the characteristics of small computational complexity and few parameters, and is very suitable for running on edge devices such as endoscopes;

[0083] The wavelet transform method is used to remove the noise from the acquired original images;

[0084] Through the bilinear interpolation method, the sizes of the original images after noise removal are unified, and the histogram equalization is used to enhance the contrast of the original images with unified sizes;

[0085] The original images with enhanced contrast are normalized to obtain preprocessed images.

[0086] S2. Use Canny edge detection to extract the suspected lesion areas of the preprocessed images.

[0087] The Sobel operator is used to calculate the gradients of the preprocessed images in the horizontal and vertical directions to obtain the gradient magnitudes of the preprocessed images. The expressions are as follows:

[0088] a x a(i,j) = Sobel(I(i,j), dx = 1, dy = 0, kernel = 3);

[0089] a y a(i,j) = Sobel(I(i,j), dx = 0, dy = 1, kernel = 3);

[0090]

[0091] where a x and a yPreprocess the image into horizontal and vertical gradients respectively. Let (i, j) be the coordinates of the pixel points in the preprocessed image, I represent the preprocessed image, dx and dy represent the derivative calculations in the x and y directions respectively, kernel = 3 indicates that the calculation window size of the Sobel operator is 3×3, and A is the gradient magnitude of the preprocessed image;

[0092] Based on the horizontal and vertical gradients of the preprocessed image, calculate the gradient direction of the preprocessed image. The expression is as follows:

[0093]

[0094] where θ is the gradient direction of the preprocessed image;

[0095] Based on non-maximum suppression, when the gradient magnitude of the preprocessed image is not a local maximum in the gradient direction, set the gradient magnitude of the preprocessed image to zero; otherwise, retain it.

[0096] Through non-maximum suppression, non-edge similarities are removed, and pixels with local maximum gradients are retained, making the edge lines thinner and avoiding the phenomenon of edge blurring.

[0097] According to the gradient magnitude of the preprocessed image, count the histogram of all pixel points in the preprocessed image to obtain the gradient distribution curve of all pixel points in the preprocessed image.

[0098] According to the gradient distribution curve, set the high threshold (such as 90% of the gradient magnitude distribution) and low threshold (such as 40% of the gradient magnitude distribution) for double-threshold filtering.

[0099] According to the comparison of the gradient magnitude of the pixel points in the preprocessed image with the high threshold and low threshold, distinguish strong, weak edge points and non-edge points.

[0100] Specifically, when the gradient magnitude of the pixel point in the preprocessed image is greater than or equal to the high threshold, retain this pixel point as a strong edge point; when the gradient magnitude of the pixel point in the preprocessed image is less than the high threshold and greater than or equal to the low threshold, retain this pixel point as a weak edge point; when the gradient magnitude of the pixel point in the preprocessed image is less than the low threshold, set this pixel point as a non-edge point.

[0101] Remove weak edge points and non-edge points without strong edge points in the neighborhood, and connect strong edge points and weak edge points with strong edge points in the neighborhood to obtain the suspected lesion area.

[0102] It should be noted that the suspected lesion area is the image obtained from the set of all strong edge points and weak edge points with strong edge points in the neighborhood.

[0103] S3. Combine the super-resolution adversarial generation network and the self-attention mechanism to perform super-resolution reconstruction and local enhancement on the preprocessed image to obtain a high-definition enhanced image.

[0104] In the super-resolution adversarial generation network, based on the lesion area attention mechanism, enhance the features of the suspected lesion area to obtain a dual-branch super-resolution generative adversarial network;

[0105] Based on the lesion area attention mechanism, it is a deep learning enhancement method specifically for the lesion area by combining the attention weights of the lesion area and the Transformer mechanism. It can automatically focus on the key features of the lesion area and reduce background interference, thereby improving the detection ability of the dual-branch super-resolution generative adversarial network for small or blurred lesions;

[0106] In the overall super-resolution reconstruction branch, the generator extracts the local features of the preprocessed image in the convolutional layer and enhances the local features through residual blocks;

[0107] Perform upsampling through sub-pixel convolution to convert the preprocessed image into a high-resolution image;

[0108] The discriminator uses a convolutional neural network to perform discrimination by calculating the true probability of the high-resolution image;

[0109] In the lesion area enhancement branch, based on the gradient magnitude of the pixel points in the suspected lesion area, calculate the attention weight map of the suspected lesion area, and the expression is as follows:

[0110]

[0111] Among them, Y is the attention weight of the suspected lesion area, A ′ is the gradient magnitude of the pixel points in the suspected lesion area, and max(A(i,j)) represents the maximum gradient magnitude of the whole image;

[0112] Enhance the local features of the suspected lesion area in the high-resolution image through the Transformer mechanism to obtain a high-definition enhanced image;

[0113] Specifically, extract the image of the suspected lesion area to be enhanced from the high-resolution image. To ensure that the Transformer only focuses on the suspected lesion area, use the self-attention mechanism to calculate the query matrix, key matrix, and value matrix, and use the dot-product attention algorithm to calculate the attention score of the suspected lesion area to enhance the local features of the suspected lesion area in the high-resolution image;

[0114] Use the digestive tract endoscope image database to train the dual-branch super-resolution generative adversarial network;

[0115] Obtain high-resolution gastrointestinal endoscope images from the gastrointestinal endoscope image database;

[0116] Crop the high-resolution gastrointestinal endoscope images input into the dual-branch super-resolution generative adversarial network to the same size, and reduce the resolution by the nearest neighbor interpolation method to obtain low-resolution gastrointestinal endoscope images, thereby obtaining paired high-resolution and low-resolution gastrointestinal endoscope images, where the paired high-resolution and low-resolution gastrointestinal endoscope images represent high-resolution and low-resolution images of the same image content;

[0117] Input the paired gastrointestinal endoscope images into the dual-branch super-resolution generative adversarial network for alternating training;

[0118] Optimize the dual-branch super-resolution generative adversarial network by calculating the total loss function composed of the generative adversarial loss function, the edge-preserving loss function, and the medical feature perception loss function. The expression is as follows:

[0119]

[0120] L2 = ∑ i,j (A1(i,j) - A2(i,j)) 2 ;

[0121]

[0122] L = L1 + L2 + 0.1L3;

[0123] Among them, Equation 1 is the generative adversarial loss function, Equation 2 is the edge-preserving loss function, Equation 3 is the medical feature perception loss function, Equation 4 is the total loss function, L1 is the generative adversarial loss value, is the expected value, D(I2) represents the probability that the discriminator judges the high-resolution gastrointestinal endoscope image I2 as a real image, I1 is the high-resolution gastrointestinal endoscope image generated by the generator, I2 is the high-resolution gastrointestinal endoscope image, L2 is the edge-preserving loss value, A1 is the gradient magnitude of the high-resolution gastrointestinal endoscope image generated by the generator, A2 is the gradient magnitude of the high-resolution gastrointestinal endoscope image (the calculation method is the same as the gradient magnitude of the preprocessed image), L3 is the medical feature perception loss value, λ is the feature loss weight, usually set in [0.001, 0.01], n is the number of convolutional layers of the pre-trained medical imaging network (such as ResNet), m is the index of the convolutional layers of the pre-trained medical imaging network, is the feature vector extracted by the pre-trained medical imaging network at the m-th layer for the high-resolution gastrointestinal endoscope image generated by the generator, is the feature vector extracted by the pre-trained medical imaging network at the m-th layer for the high-resolution gastrointestinal endoscope image, and L is the total loss value;

[0124] When the paired gastrointestinal endoscope images are not fully input into the dual-branch super-resolution generative adversarial network for training and the total loss function has converged, the training is terminated prematurely;

[0125] On the contrary, continuous alternating training is carried out until the dual-branch super-resolution generative adversarial network completes the training of all paired gastrointestinal endoscope images;

[0126] Specifically, the convergence of the total loss function means that the change value of the total loss value for consecutive multiple training batches is less than the change threshold. For example, if the change value of the total loss value does not exceed 0.004 for 20 consecutive training rounds, it indicates that the total loss function has converged and the training is terminated prematurely;

[0127] It should be noted that during the super-resolution reconstruction of the preprocessed image, the details of the tiny lesion area are likely to be smoothed or lost, especially resulting in the missed detection of small lesions. The Transformer introducing the lesion area attention mechanism specifically enhances the details of the lesion area to ensure the clarity of the lesion area and the authenticity of the texture information, and improve the detection rate of small lesions.

[0128] S4. Identify the lesion area by performing color analysis on the high-definition enhanced image.

[0129] Convert the high-definition enhanced image to the HSV color space to obtain the hue, saturation, and brightness of the high-definition enhanced image in the HSV color space;

[0130] Specifically, since the high-definition enhanced image is an RGB image, its RGB values need to be normalized first, and the expression is as follows:

[0131]

[0132] where R, G, and B are the intensity values of red, green, and blue in the high-definition enhanced image, and R ′ , G ′ and B ′ are the intensity values of normalized red, green, and blue in the high-definition enhanced image respectively

[0133] Calculate the difference between the maximum value and the minimum value of the normalized intensity values of red, green, and blue in the high-definition enhanced image;

[0134] Calculate the hue, saturation, and brightness of the high-definition enhanced image in the HSV color space, and the expression is as follows:

[0135] Δ = C max (R ′ , G ′ , B ′ ) - C min (R ′ , G′ , B ′ );

[0136]

[0137] V = C max ;

[0138] where C max and C min are respectively the maximum and minimum values of the normalized red, green, and blue intensity values in the high-definition enhanced image, Δ is the difference between C max and C min , H is the hue, mod represents the modulo operation, S is the saturation, and V is the brightness;

[0139] Set the hue range threshold, saturation range threshold, and brightness range threshold according to the hue, saturation, and brightness ranges of the healthy digestive tract image in the HSV color space;

[0140] When there is an area in the high-definition enhanced image where the hue, saturation, and brightness are not within the hue range threshold, saturation range threshold, and brightness range threshold, the lesion area of the high-definition enhanced image is recognized; otherwise, it is a non-lesion area.

[0141] S5. Use the active contour model to segment the lesion area on the high-definition enhanced image to obtain a lesion segmentation mask.

[0142] Set an initial closed curve around the lesion area of the high-definition enhanced image;

[0143] Optimize the ACM energy function using the gradient descent method, gradually shrink the initial closed curve, and obtain the initial boundary of the lesion area;

[0144] Obtain the lesion segmentation mask by calculating the shortest Euclidean distance from each pixel point in the high-definition enhanced image to the initial boundary of the lesion area;

[0145] Specifically, set the initial closed curve around the lesion area of the high-definition enhanced image in the form of an ellipse or a rectangle;

[0146] Calculate the ACM energy function of the active contour model, and the expression is as follows:

[0147] E = E1 + E2;

[0148] E1 = ∫|Q ′ (s)| 2 ds + ∫|Q ″ (s)| 2 ds;

[0149]

[0150] Among them, E is the value of the ACM energy function, E1 is the internal energy value, which can constrain the smoothness of the curve and prevent the boundary from bending excessively. E2 is the external energy value, which guides the curve to fit the lesion area by using the image gradient, and Q ′ (s) is the first derivative of the initialized closed curve Q(s), representing the smoothness of the initialized closed curve, and Q ″ (s) is the second derivative of the initialized closed curve, representing the degree of curvature of the curve. s is the pixel point on the initialized closed curve, and g represents the Gaussian filter, is the gradient operator, representing the calculation of the gradient magnitude of the high-definition enhanced image after Gaussian smoothing. I3 is the high-definition enhanced image;

[0151] By using the gradient descent method, the ACM energy function is optimized to realize the iterative update of the initialized closed curve Q(s). The expression is as follows:

[0152]

[0153] Among them, Q β (s) is the initialized closed curve updated at the β-th time, and Q β+1 (s) is the initialized closed curve updated at the (β + 1)-th time, and η is the learning rate, represents the partial derivative, and Q(s) is the initialized closed curve;

[0154] When the average Euclidean distance between Q β+1 (s) and Q β (s) is less than 1 pixel, it means that the ACM energy function converges, and the initial boundary of the lesion area is output. Otherwise, continue to optimize until the ACM energy function converges or reaches the maximum number of iterations (such as 500 times), then stop the optimization and output the initial boundary of the lesion area;

[0155] Calculate the shortest Euclidean distance from each pixel point in the high-definition enhanced image to the initial boundary of the lesion area. The shortest Euclidean distance of the pixel points outside the initial boundary of the lesion area is positive, the shortest Euclidean distance of the pixel points inside the initial boundary of the lesion area is negative, and the shortest Euclidean distance of the pixel points on the initial boundary of the lesion area is 0;

[0156] Based on the calculation result of the shortest Euclidean distance, obtain the lesion segmentation mask. The expression is as follows:

[0157]

[0158] Among them, M is the lesion segmentation mask, φ is the shortest Euclidean distance from a pixel point to the initial boundary of the lesion area, M(i,j)=1 indicates that the pixel point (i,j) is in the lesion area, and M(i,j)=0 indicates that the pixel point (i,j) is not in the lesion area.

[0159] S6. Based on the lesion segmentation mask and the high-definition enhanced image, use the pre-trained feature extraction network to classify the lesion types.

[0160] Based on the lesion segmentation mask, extract the lesion area from the high-definition enhanced image to obtain the high-definition enhanced lesion area image, and the expression is as follows:

[0161] I5(x,y) = M(i,j) × I3(i,j);

[0162] Among them, I5(x,y) represents the high-definition enhanced lesion area image;

[0163] It should be noted that this calculation method is equivalent to performing bitwise multiplication pixel by pixel on the high-definition enhanced image. If M(i,j)=1, then the output I5(x,y)=I3(i,j), that is, this pixel retains the value of the original image. If M(i,j)=0, then the output I5(x,y)=0, and the value of this pixel is set to 0, indicating that this pixel point is masked, ensuring that only the pixel values of the lesion area are retained, while the pixels in other areas (such as non-lesion areas and the background) are filtered out, removing irrelevant information and improving the accuracy of lesion classification and feature extraction;

[0164] Input the different lesion sample images and the high-definition enhanced lesion area images in the digestive endoscope image database into the pre-trained feature extraction network, and calculate the lesion feature vectors respectively;

[0165] By calculating the cosine similarity of the lesion feature vectors of the lesion sample image and the high-definition enhanced lesion area image, output the lesion type in the high-definition enhanced image;

[0166] Specifically, use the ResNet50 pre-trained model as the feature extraction network, extract the visual features of the high-definition enhanced lesion area image through ResNet50 to obtain the lesion feature vector to be measured;

[0167] At the same time, extract the visual features of different lesion sample images through ResNet50 to obtain a set of sample lesion feature vectors;

[0168] By calculating the cosine similarity between the lesion feature vector to be measured and each sample lesion feature vector in the set of sample lesion feature vectors, and taking the lesion type to which the sample lesion feature vector with the highest cosine similarity value belongs as the lesion type output in the high-definition enhanced image.

[0169] This embodiment also provides a digestive tract auxiliary detection device based on deep learning, including: an image acquisition module, a suspected lesion module, an image enhancement module, a lesion recognition module, a lesion segmentation module, and a lesion classification module. The image acquisition module is used to collect original images of the patient's digestive tract through an endoscopic device based on an event-driven sampling strategy, and perform denoising, size adjustment, and quality improvement processing on the original images to obtain preprocessed images; the suspected lesion module is used to extract suspected lesion areas of the preprocessed images by using Canny edge detection; the image enhancement module is used to combine a super-resolution adversarial generation network and a self-attention mechanism to perform super-resolution reconstruction and local enhancement on the preprocessed images to obtain high-definition enhanced images; the lesion recognition module is used to identify lesion areas by performing color analysis on the high-definition enhanced images; the lesion segmentation module is used to segment the lesion areas on the high-definition enhanced images by using an active contour model to obtain lesion segmentation masks; the lesion classification module is used to classify the lesion types based on the lesion segmentation masks and the high-definition enhanced images by using a pre-trained feature extraction network.

[0170] This embodiment also provides a computer device applicable to the situation of the digestive tract auxiliary detection method based on deep learning, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the digestive tract auxiliary detection method based on deep learning as proposed in the above embodiment.

[0171] This computer device can be a terminal, and this computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of this computer device is used to provide computing and control capabilities. The memory of this computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of this computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a carrier network, NFC (near field communication), or other technologies. The display screen of this computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of this computer device can be a touch layer covered on the display screen, or a button, a trackball, or a touchpad provided on the shell of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0172] This embodiment also provides a storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the method for gastrointestinal tract assisted detection based on deep learning as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM for short), electrically erasable programmable read-only memory (EEPROM for short), erasable programmable read-only memory (EPROM for short), programmable read-only memory (PROM for short), read-only memory (ROM for short), magnetic memory, flash memory, magnetic disk or optical disc.

[0173] In summary, the present invention optimizes the image acquisition process based on an event-driven sampling strategy, reduces redundant data while increasing the attention to potential lesion areas, thereby improving the detection sensitivity and efficiency. It uses Canny edge detection to accurately extract suspected lesion areas, combines a super-resolution adversarial generation network and a self-attention mechanism to enhance image details, especially the clarity of tiny lesion areas, effectively reducing the missed detection rate. It accurately identifies the color features of lesion areas through HSV color space conversion, and uses an active contour model to achieve fine segmentation, providing a reliable basis for quantitative analysis. It uses a pre-trained feature extraction network to perform high-precision classification of lesion types, supports multi-class lesion recognition, and provides support for formulating personalized treatment plans.

[0174] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not restrictive. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A digestive tract assisted detection method based on deep learning, characterized in that: including, using an endoscopic device to collect original images of a patient's digestive tract based on an event-driven sampling strategy, and performing denoising, size adjustment, and quality improvement on the original images to obtain preprocessed images; using Canny edge detection to extract suspected lesion areas from the preprocessed images; combining a super-resolution generative adversarial network and a self-attention mechanism to perform super-resolution reconstruction and local enhancement on the preprocessed images to obtain high-definition enhanced images; identifying lesion areas by performing color analysis on the high-definition enhanced images; using an active contour model to segment the lesion areas on the high-definition enhanced images to obtain lesion segmentation masks; classifying the lesion types based on the lesion segmentation masks and the high-definition enhanced images using a pre-trained feature extraction network.

2. The deep learning-based digestive tract auxiliary detection method according to claim 1, wherein: The step of using an endoscopic device to collect original images of a patient's digestive tract based on an event-driven sampling strategy, and performing denoising, size adjustment, and quality improvement on the original images to obtain preprocessed images is as follows. When using an endoscopic device to collect original images of a patient's digestive tract, a preliminary judgment model is constructed based on a deep learning network to perform preliminary judgment on the original images in real time; Based on the judgment results of the preliminary judgment model on the original images collected in real time, adjust the frequency of original image collection; Adopt the wavelet transform method to remove noise from the collected original images; Through bilinear interpolation, unify the size of the original images after noise removal, and use histogram equalization to enhance the contrast of the original images with unified size; Perform normalization on the original images with enhanced contrast to obtain preprocessed images.

3. The method for auxiliary detection of digestive tract based on deep learning according to claim 2, wherein: The step of using Canny edge detection to extract suspected lesion areas from the preprocessed images is as follows. Use the Sobel operator to calculate the gradients of the preprocessed images in the horizontal and vertical directions to obtain the gradient magnitudes of the preprocessed images; Calculate the gradient directions of the preprocessed images based on the gradients of the preprocessed images in the horizontal and vertical directions; Based on non-maximum suppression, when the gradient magnitude of the preprocessed image is not a local maximum in the gradient direction, set the gradient magnitude of the preprocessed image to zero, otherwise, keep it; According to the gradient magnitudes of the preprocessed images, count the histograms of all pixel points in the preprocessed images to obtain the gradient distribution curves of all pixel points in the preprocessed images; Set the high threshold and low threshold of double-threshold filtering according to the gradient distribution curves; Distinguish strong, weak edge points, and non-edge points according to the comparison between the gradient magnitudes of pixel points in the preprocessed images and the high threshold and low threshold; Remove weak edge points and non-edge points without strong edge points in the neighborhood, and connect strong edge points and weak edge points with strong edge points in the neighborhood to obtain suspected lesion areas.

4. The digestive tract assisted detection method based on deep learning according to claim 3, wherein: The step of combining a super-resolution generative adversarial network and a self-attention mechanism to perform super-resolution reconstruction and local enhancement on the preprocessed images to obtain high-definition enhanced images is as follows. In the super-resolution generative adversarial network, enhance the features of the suspected lesion areas based on the lesion area attention mechanism to obtain a dual-branch super-resolution generative adversarial network; In the overall super-resolution reconstruction branch, the generator extracts local features of the preprocessed image in the convolutional layer and enhances the local features through residual blocks; Upsampling is performed through sub-pixel convolution to convert the preprocessed image into a high-resolution image; The discriminator uses a convolutional neural network to make a discrimination by calculating the true probability of the high-resolution image; In the lesion area enhancement branch, based on the gradient magnitude of the pixel points in the suspected lesion area, the attention weight map of the suspected lesion area is calculated; The local features of the suspected lesion area in the high-resolution image are enhanced through the Transformer mechanism to obtain a high-definition enhanced image; Using the gastrointestinal endoscope image database, the dual-branch super-resolution generative adversarial network is trained; High-resolution gastrointestinal endoscope images are obtained from the gastrointestinal endoscope image database; The high-resolution gastrointestinal endoscope images input into the dual-branch super-resolution generative adversarial network are cropped to the same size, and the resolution of the high-resolution gastrointestinal endoscope images is reduced by the nearest neighbor interpolation method to obtain paired high-resolution and low-resolution gastrointestinal endoscope images; The paired gastrointestinal endoscope images are input into the dual-branch super-resolution generative adversarial network for alternating training; The dual-branch super-resolution generative adversarial network is optimized by calculating the total loss function composed of the generative adversarial loss function, the edge-preserving loss function, and the medical feature perception loss function; When the total loss function has converged before the paired gastrointestinal endoscope images are completely input into the dual-branch super-resolution generative adversarial network for training, the training is terminated in advance; On the contrary, continue the alternating training until the dual-branch super-resolution generative adversarial network completes the training of all paired gastrointestinal endoscope images.

5. The deep learning-based digestive tract assisted detection method according to claim 4, characterized in that: The specific steps for identifying the lesion area by performing color analysis on the high-definition enhanced image are as follows. The high-definition enhanced image is converted to the HSV color space to obtain the hue, saturation, and brightness of the high-definition enhanced image in the HSV color space; According to the hue, saturation, and brightness ranges of healthy gastrointestinal images in the HSV color space, the hue range threshold, saturation range threshold, and brightness range threshold are set; When there is an area in the high-definition enhanced image where the hue, saturation, and brightness are not within the hue range threshold, saturation range threshold, and brightness range threshold, the lesion area of the high-definition enhanced image is identified.

6. The method for auxiliary detection of digestive tract based on deep learning according to claim 5, characterized in that: The specific steps for segmenting the lesion area on the high-definition enhanced image using the active contour model to obtain the lesion segmentation mask are as follows. An initial closed curve is set outside the lesion area of the high-definition enhanced image; The ACM energy function is optimized using the gradient descent method, and the initial closed curve is gradually shrunk to obtain the initial boundary of the lesion area; The lesion segmentation mask is obtained by calculating the shortest Euclidean distance from each pixel point in the high-definition enhanced image to the initial boundary of the lesion area.

7. The method for auxiliary detection of the digestive tract based on deep learning according to claim 6, wherein: The specific steps for classifying the lesion type using the pre-trained feature extraction network based on the lesion segmentation mask and the high-definition enhanced image are as follows. Based on the lesion segmentation mask, the lesion area is extracted from the high-definition enhanced image to obtain the high-definition enhanced lesion area image; Input different lesion sample images and high-definition enhanced lesion area images in the digestive tract endoscope image database into the pre-trained feature extraction network, and calculate the lesion feature vectors respectively; Output the lesion type in the high-definition enhanced image by calculating the cosine similarity of the lesion feature vectors of the lesion sample image and the high-definition enhanced lesion area image.

8. A digestive tract assisted detection device based on deep learning, based on the digestive tract assisted detection method based on deep learning according to any one of claims 1 to 7, characterized in that: It includes an image acquisition module, a suspected lesion module, an image enhancement module, a lesion recognition module, a lesion segmentation module, and a lesion classification module. The image acquisition module is used to collect the original images of the patient's digestive tract through the endoscope device based on the event-driven sampling strategy, and perform denoising, size adjustment, and quality improvement processing on the original images to obtain preprocessed images; The suspected lesion module is used to extract the suspected lesion areas of the preprocessed images by using Canny edge detection; The image enhancement module is used to combine the super-resolution adversarial generation network and the self-attention mechanism to perform super-resolution reconstruction and local enhancement on the preprocessed images to obtain high-definition enhanced images; The lesion recognition module is used to identify the lesion areas by performing color analysis on the high-definition enhanced images; The lesion segmentation module is used to segment the lesion areas on the high-definition enhanced images by using the active contour model to obtain lesion segmentation masks; The lesion classification module is used to classify the lesion types by using the pre-trained feature extraction network based on the lesion segmentation masks and the high-definition enhanced images.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the deep learning-based digestive tract auxiliary detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the deep learning-based digestive tract auxiliary detection method according to any one of claims 1 to 7.

Images