A deep learning-based gastrointestinal tract auxiliary detection method and device

Abstract

The application discloses a kind of based on deep learning's gastrointestinal tract auxiliary detection method and device, it is related to auxiliary detection technical field, including, through endoscope equipment, based on event-driven sampling strategy to patient gastrointestinal tract is carried out original image acquisition, to original image is denoised, size adjustment and quality promotion processing, obtain preprocessed image;Using Canny edge detection, the suspected lesion area of preprocessed image is extracted;Combining super-resolution generative adversarial network and self-attention mechanism, super-resolution reconstruction and local enhancement are carried out to preprocessed image, obtain high-definition enhanced image;Based on lesion segmentation mask and high-definition enhanced image, using pre-training feature extraction network classifies lesion type.The application is combined super-resolution generative adversarial network and self-attention mechanism, carries out super-resolution reconstruction and local enhancement to preprocessed image, enhances image detail, especially the definition of micro-lesion area, effectively reduces the rate of missed detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of auxiliary detection technology, and in particular to a digestive tract auxiliary detection method and device based on deep learning. BACKGROUND

[0002] In recent years, the detection technology of digestive tract diseases has developed rapidly, especially in the field of endoscopic image analysis. The introduction of deep learning technology has significantly improved the accuracy and efficiency of lesion area detection. Traditional methods of auxiliary detection of the digestive tract mainly rely on the experience of doctors and manual annotation tools. Such methods are not only time-consuming and laborious, but also susceptible to subjective factors, resulting in unstable diagnostic results.

[0003] However, the existing technology still has some deficiencies, for example, in the image enhancement link, the existing technology often fails to effectively preserve the details of small lesion areas, especially in the super-resolution reconstruction process. Due to the lack of local enhancement mechanism for lesion areas, it is easy to miss small lesions. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a digestive tract auxiliary detection method based on deep learning to solve the problem of missing small lesion areas.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a digestive tract auxiliary detection method based on deep learning, which comprises,

[0008] Through an endoscope device, raw images of the patient's digestive tract are collected based on an event-driven sampling strategy. The raw images are denoised, resized, and quality-improved to obtain preprocessed images.

[0009] The suspected lesion areas of the preprocessed images are extracted using Canny edge detection.

[0010] The preprocessed images are super-resolution reconstructed and locally enhanced using a super-resolution generative adversarial network and a self-attention mechanism to obtain high-definition enhanced images.

[0011] The lesion areas are identified by color analysis of the high-definition enhanced images.

[0012] The lesion areas on the high-definition enhanced images are segmented using an active contour model to obtain lesion segmentation masks.

[0013] Based on the lesion segmentation masks and the high-definition enhanced images, the lesion types are classified using a pre-trained feature extraction network.

[0014] As a preferred scheme of the digestive tract auxiliary detection method based on deep learning, wherein: the original image of the patient's digestive tract is collected based on the event-driven sampling strategy through the endoscope device, and the original image is denoised, sized and quality improved to obtain a preprocessed image, the specific steps are as follows,

[0015] When the original image of the patient's digestive tract is collected using the endoscope device, a preliminary judgment model based on a deep learning network is used to make a preliminary judgment on the original image in real time;

[0016] Based on the judgment result of the preliminary judgment model on the real-time collected original image, the frequency of original image collection is adjusted;

[0017] Wavelet transform method is used to remove noise from the collected original image;

[0018] The size of the original image after noise removal is unified by bilinear interpolation method, and the contrast of the original image with unified size is enhanced by histogram equalization;

[0019] The original image with enhanced contrast is normalized to obtain a preprocessed image.

[0020] As a preferred scheme of the digestive tract auxiliary detection method based on deep learning, wherein: the suspected lesion area of the preprocessed image is extracted by Canny edge detection, and the specific steps are as follows,

[0021] The gradient of the preprocessed image in the horizontal and vertical directions is calculated by using Sobel operator to obtain the gradient amplitude of the preprocessed image;

[0022] The gradient direction of the preprocessed image is calculated according to the gradient of the preprocessed image in the horizontal and vertical directions;

[0023] Based on non-maximum suppression, if the gradient amplitude of the preprocessed image is not a local maximum in the gradient direction, the gradient amplitude of the preprocessed image is set to zero, otherwise, it is retained;

[0024] According to the gradient amplitude of the preprocessed image, the histogram of all pixel points in the preprocessed image is calculated to obtain the gradient distribution curve of all pixel points in the preprocessed image;

[0025] According to the gradient distribution curve, the high threshold and low threshold of the double threshold filtering are set;

[0026] According to the comparison of the gradient amplitude of the pixel points in the preprocessed image with the high threshold and low threshold, the strong, weak edge points and non-edge points are distinguished;

[0027] Remove weak edge points and non-edge points without strong edge points in the neighborhood, connect strong edge points and weak edge points with strong edge points in the neighborhood, and obtain a suspected lesion area.

[0028] As a preferred scheme of the digestive tract auxiliary detection method based on deep learning, wherein: the combination of the super-resolution generative adversarial network and the self-attention mechanism is used to perform super-resolution reconstruction and local enhancement on the preprocessed image to obtain a high-definition enhanced image, and the specific steps are as follows,

[0029] In the super-resolution generative adversarial network, the lesion area attention mechanism is used to enhance the features of the suspected lesion area to obtain a double-branch super-resolution generative adversarial network.

[0030] In the overall super-resolution reconstruction branch, the generator extracts local features of the preprocessed image in the convolution layer and enhances the local features through the residual block;

[0031] The sub-pixel convolution is used for upsampling to convert the preprocessed image into a high-resolution image;

[0032] The discriminator adopts a convolutional neural network to calculate the real probability of the high-resolution image for discrimination;

[0033] In the lesion area enhancement branch, the gradient amplitude of the pixel points in the suspected lesion area is used to calculate the attention weight map of the suspected lesion area;

[0034] The Transformer mechanism is used to enhance the local features of the suspected lesion area in the high-resolution image to obtain a high-definition enhanced image;

[0035] The double-branch super-resolution generative adversarial network is trained using a digestive tract endoscopy image database;

[0036] High-resolution digestive tract endoscopy images are obtained from the digestive tract endoscopy image database;

[0037] The high-resolution digestive tract endoscopy images input into the double-branch super-resolution generative adversarial network are cropped to the same size, and the resolution is reduced by the nearest neighbor interpolation method to obtain pairs of high-resolution and low-resolution digestive tract endoscopy images;

[0038] The pairs of digestive tract endoscopy images are input into the double-branch super-resolution generative adversarial network for alternating training;

[0039] The total loss function composed of the generative adversarial loss function, the edge preservation loss function and the medical feature perception loss function is calculated to optimize the double-branch super-resolution generative adversarial network;

[0040] When the paired endoscopy images are not completely input into the dual-branch super-resolution generative adversarial network for training, and the total loss function has converged, the training is ended in advance;

[0041] Conversely, the training is continuously alternated until the dual-branch super-resolution generative adversarial network completes the training on all paired endoscopy images.

[0042] As a preferred scheme of the deep learning-based auxiliary detection method for the digestive tract, the high-definition enhanced image is converted to an HSV color space to obtain the hue, saturation and brightness of the high-definition enhanced image in the HSV color space.

[0043] According to the hue, saturation and brightness ranges of the healthy digestive tract image in the HSV color space, a hue range threshold, a saturation range threshold and a brightness range threshold are set.

[0044] When the regions in the high-definition enhanced image, in which the hue, saturation and brightness are all not within the hue range threshold, the saturation range threshold and the brightness range threshold, appear, it is identified that the high-definition enhanced image has a lesion region.

[0045] As a preferred scheme of the deep learning-based auxiliary detection method for the digestive tract, the high-definition enhanced image is converted to an HSV color space to obtain the hue, saturation and brightness of the high-definition enhanced image in the HSV color space.

[0046] An initialization closed curve is set outside the lesion region of the high-definition enhanced image.

[0047] A gradient descent method is used to optimize the ACM energy function, and the initialization closed curve is gradually contracted to obtain an initial boundary of the lesion region.

[0048] The shortest Euclidean distance of each pixel point in the high-definition enhanced image to the initial boundary of the lesion region is calculated to obtain a lesion segmentation mask.

[0049] As a preferred scheme of the deep learning-based auxiliary detection method for the digestive tract, the high-definition enhanced image is converted to an HSV color space to obtain the hue, saturation and brightness of the high-definition enhanced image in the HSV color space.

[0050] Based on the lesion segmentation mask, a lesion region on the high-definition enhanced image is extracted to obtain a high-definition enhanced lesion region image.

[0051] Different lesion sample images in the endoscopy image database and the high-definition enhanced lesion region image are input into the pre-trained feature extraction network, and lesion feature vectors are calculated respectively.

[0052] The lesion type in the high-definition enhanced image is output by calculating the cosine similarity of the lesion feature vectors of the lesion sample image and the high-definition enhanced lesion region image.

[0053] In a second aspect, the present application provides a deep learning-based digestive tract auxiliary detection device, comprising an image acquisition module, a suspected lesion module, an image enhancement module, a lesion recognition module, a lesion segmentation module, and a lesion classification module.

[0054] The image acquisition module is configured to acquire original images of a patient's digestive tract based on an event-driven sampling strategy through an endoscope device, and perform denoising, size adjustment, and quality improvement processing on the original images to obtain preprocessed images.

[0055] The suspected lesion module is configured to extract suspected lesion regions of the preprocessed images using Canny edge detection.

[0056] The image enhancement module is configured to combine a super-resolution generative adversarial network and a self-attention mechanism to perform super-resolution reconstruction and local enhancement on the preprocessed images to obtain high-definition enhanced images.

[0057] The lesion recognition module is configured to recognize lesion regions by performing color analysis on the high-definition enhanced images.

[0058] The lesion segmentation module is configured to segment the lesion regions on the high-definition enhanced images using an active contour model to obtain lesion segmentation masks.

[0059] The lesion classification module is configured to classify lesion types based on the lesion segmentation masks and the high-definition enhanced images using a pre-trained feature extraction network.

[0060] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the computer program, when executed by the processor, implements any step of the deep learning-based digestive tract auxiliary detection method according to the first aspect of the present application.

[0061] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any step of the deep learning-based digestive tract auxiliary detection method according to the first aspect of the present application.

[0062] The application has the beneficial effects that: the application optimizes the image acquisition process based on an event-driven sampling strategy, reduces redundant data while improving the attention of potential lesion areas, thereby improving detection sensitivity and efficiency, accurately extracts suspected lesion areas using Canny edge detection, enhances image details, especially the clarity of small lesion areas, using a super-resolution generative adversarial network and a self-attention mechanism, effectively reduces the missed detection rate, accurately identifies the color features of the lesion area through HSV color space conversion, and realizes fine segmentation with the help of an active contour model, provides reliable quantitative analysis basis, uses a pre-trained feature extraction network to classify the lesion types with high precision, supports multi-class lesion identification, and provides support for individualized treatment plan formulation. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0064] Figure 1 The flowchart of the deep learning-based digestive tract auxiliary detection method in embodiment 1;

[0065] Figure 2 The module diagram of the deep learning-based digestive tract auxiliary detection device in embodiment 1. DETAILED DESCRIPTION

[0066] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the specific embodiments of the application will be described in detail below with reference to the drawings of the specification.

[0067] In the following description, many specific details are set forth in order to provide a thorough understanding of the application, but the application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the application, therefore the application is not limited by the specific embodiments disclosed below.

[0068] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the application. "In one embodiment" appearing in different places in this specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0069] Embodiment 1, refer to Figure 1 and Figure 2 , the first embodiment of the application, the embodiment provides a deep learning-based digestive tract auxiliary detection method, including the following steps:

[0070] S1, through an endoscope device, based on an event-driven sampling strategy, raw images of a patient's digestive tract are collected, and the raw images are denoised, resized and quality improved to obtain preprocessed images.

[0071] When raw images of a patient's digestive tract are collected using an endoscope device, a preliminary judgment model is constructed based on a deep learning network to make real-time preliminary judgments on the raw images.

[0072] Further, the construction and training process of the preliminary judgment model is as follows: images with lesion region annotations and images with the same region lesion contrast are collected from a digestive tract endoscope image database (such as the HyperKvasir dataset);

[0073] After the images with lesion region annotations and images with the same region lesion contrast data are uniformly resized and normalized, they are divided into a preliminary training set and a preliminary validation set;

[0074] The preliminary training set is input into the preliminary judgment model based on MobileNetV2 for iterative training, and the probability of the images in the preliminary training set having potential lesions is output;

[0075] After every 10 training rounds, the performance of the preliminary judgment model is verified on the preliminary validation set;

[0076] During training, the loss value of the preliminary judgment model is obtained by calculating the binary cross-entropy loss function of the preliminary judgment model, and the learning rate of the preliminary judgment model is optimized using the Adam optimizer, as follows:

[0077]

[0078] Where A is the binary cross-entropy loss value of the preliminary judgment model, N is the number of samples in the preliminary training set, y i is the true label of the i-th sample (1 for lesion region, 0 for no lesion region), is the probability value of the preliminary judgment model predicting that the i-th sample has a lesion region;

[0079] When the loss value change rate of the preliminary judgment model for the last ten validation sets does not exceed 2%, it indicates that the preliminary judgment model has reached the optimal, and the training of the preliminary judgment model is complete.

[0080] Based on the judgment result of the preliminary judgment model on the real-time collected raw images, the frequency of raw image collection is adjusted;

[0081] Specifically, when the preliminary judgment model judges that the probability value of the existence of the lesion area in the real-time collected original image increases, the collection frequency of the original image is increased, for example: when the preliminary judgment model judges that the probability of the existence of the lesion area in the real-time collected original image is greater than or equal to 0.7, the collection frequency of the original image is increased to twice the default collection frequency, when the preliminary judgment model judges that the probability of the existence of the lesion area in the real-time collected original image is greater than or equal to 0.3 and less than 0.7, the default collection frequency is maintained, and when the preliminary judgment model judges that the probability of the existence of the lesion area in the real-time collected original image is less than 0.3, the collection frequency of the original image is reduced to 0.5 times the default collection frequency;

[0082] It should be noted that based on such an event-driven sampling strategy, the collection of original image data without lesion areas can be reduced, the collection efficiency of the patient's digestive tract original image is improved, and the efficiency of subsequent further analysis of the original image is also improved, and MobileNetV2 is a network structure based on deep separable convolution, which has the characteristics of small calculation amount and few parameters, and is very suitable for running on edge devices such as endoscopes;

[0083] The wavelet transform method is used to remove noise from the collected original image;

[0084] The size of the original image after noise removal is unified by the bilinear interpolation method, and the contrast of the original image with unified size is enhanced by histogram equalization;

[0085] The original image with enhanced contrast is normalized to obtain a preprocessed image.

[0086] S2, using Canny edge detection to extract the suspected lesion area of the preprocessed image.

[0087] The Sobel operator is used to calculate the gradient of the preprocessed image in the horizontal and vertical directions to obtain the gradient amplitude of the preprocessed image, and the expression is as follows:

[0088] a x (i,j) = Sobel(I(i,j), dx = 1, dy = 0, kernel = 3);

[0089] a y (i,j) = Sobel(I(i,j), dx = 0, dy = 1, kernel = 3);

[0090]

[0091] wherein a x and a yPreprocess the image in horizontal and vertical direction gradient respectively, (i, j) is the pixel coordinate of the preprocessed image, I represents the preprocessed image, dx and dy represent the derivative calculation in x and y direction respectively, kernel = 3 represents the calculation window size of Sobel operator is 3x3, A is the gradient amplitude of the preprocessed image;

[0092] According to the gradient of the preprocessed image in horizontal and vertical direction, the gradient direction of the preprocessed image is calculated, and the expression is as follows:

[0093]

[0094] Wherein, θ is the gradient direction of the preprocessed image;

[0095] Based on non-maximum suppression, when the gradient amplitude of the preprocessed image is not the local maximum value in the gradient direction, the gradient amplitude of the preprocessed image is zero, otherwise, it is reserved;

[0096] Through non-maximum suppression, non-edge similar is removed, and the pixels of local maximum gradient are reserved, so that the edge line is thinner, and the phenomenon of edge blur is avoided;

[0097] According to the gradient amplitude of the preprocessed image, the histogram of all pixel points in the preprocessed image is counted, and the gradient distribution curve of all pixel points in the preprocessed image is obtained;

[0098] According to the gradient distribution curve, the high threshold (such as 90% of the gradient amplitude distribution) and the low threshold (such as 40% of the gradient amplitude distribution) of the double threshold filtering are set;

[0099] According to the comparison of the gradient amplitude of the pixel points in the preprocessed image with the high threshold and the low threshold, the strong, weak edge points and non-edge points are distinguished;

[0100] Specifically, when the gradient amplitude of the pixel points in the preprocessed image is greater than or equal to the high threshold, the pixel points are reserved as strong edge points, when the gradient amplitude of the pixel points in the preprocessed image is less than the high threshold and greater than or equal to the low threshold, the pixel points are reserved as weak edge points, when the gradient amplitude of the pixel points in the preprocessed image is less than the low threshold, the pixel points are set as non-edge points;

[0101] Remove the weak edge points and non-edge points without strong edge points in the neighborhood, connect the strong edge points and the weak edge points with strong edge points in the neighborhood, and obtain the suspected lesion area;

[0102] It should be noted that the suspected lesion area is the image obtained by all strong edge points and weak edge points with strong edge points in the neighborhood.

[0103] S3, in combination with the super-resolution generative adversarial network and the self-attention mechanism, performs super-resolution reconstruction and local enhancement on the preprocessed image to obtain a high-definition enhanced image.

[0104] In the super-resolution generative adversarial network, the lesion region attention mechanism is used to enhance the features of the suspected lesion region to obtain a dual-branch super-resolution generative adversarial network.

[0105] The lesion region attention mechanism is a deep learning enhancement method that combines the attention weight of the lesion region and the Transformer mechanism and is specifically designed for the lesion region. It can automatically focus on the key features of the lesion region and reduce background interference, thereby improving the detection ability of the dual-branch super-resolution generative adversarial network for small or fuzzy lesions.

[0106] In the overall super-resolution reconstruction branch, the generator extracts local features of the preprocessed image through convolution layers and enhances the local features through residual blocks.

[0107] The preprocessed image is converted into a high-resolution image through sub-pixel convolution upsampling.

[0108] The discriminator uses a convolutional neural network to calculate the real probability of the high-resolution image and make a judgment.

[0109] In the lesion region enhancement branch, the attention weight map of the suspected lesion region is calculated based on the gradient amplitude of the pixels in the suspected lesion region, and the expression is as follows:

[0110]

[0111] where Y is the attention weight of the suspected lesion region, A ′ is the gradient amplitude of the pixels in the suspected lesion region, and max(A(i,j)) represents the maximum gradient amplitude of the entire image.

[0112] The local features of the suspected lesion region in the high-resolution image are enhanced through the Transformer mechanism to obtain a high-definition enhanced image.

[0113] Specifically, the suspected lesion region image to be enhanced is extracted from the high-resolution image. To ensure that the Transformer only focuses on the suspected lesion region, the self-attention mechanism is used to calculate the query matrix, key matrix, and value matrix. The dot product attention algorithm is used to calculate the attention score of the suspected lesion region to enhance the local features of the suspected lesion region in the high-resolution image.

[0114] The dual-branch super-resolution generative adversarial network is trained using a database of gastrointestinal endoscopy images.

[0115] obtaining high-resolution gastrointestinal endoscopy images from a gastrointestinal endoscopy image database;

[0116] cropping the high-resolution gastrointestinal endoscopy images input into the dual-branch super-resolution generative adversarial network to the same size, and reducing the resolution by the nearest neighbor interpolation method to obtain low-resolution gastrointestinal endoscopy images, thereby obtaining pairs of high-resolution and low-resolution gastrointestinal endoscopy images representing high-resolution and low-resolution images of the same image content;

[0117] inputting the pairs of gastrointestinal endoscopy images into the dual-branch super-resolution generative adversarial network for alternating training;

[0118] optimizing the dual-branch super-resolution generative adversarial network by calculating a total loss function composed of a generative adversarial loss function, an edge preservation loss function and a medical feature perception loss function, and the expression is as follows:

[0119]

[0120] L2=∑ i,j (A1(i,j)-A2(i,j)) 2 ;

[0121]

[0122] L=L1+L2+0.1L3;

[0123] wherein, formula 1 is a generative adversarial loss function, formula 2 is an edge preservation loss function, formula 3 is a medical feature perception loss function, formula 4 is a total loss function, L1 is a generative adversarial loss value, is an expected value, D(I2) represents the probability that the discriminator judges that the high-resolution gastrointestinal endoscopy image I2 is a real image, I1 is a high-resolution gastrointestinal endoscopy image generated by the generator, I2 is a high-resolution gastrointestinal endoscopy image, L2 is an edge preservation loss value, A1 is the gradient amplitude of the high-resolution gastrointestinal endoscopy image generated by the generator, A2 is the gradient amplitude of the high-resolution gastrointestinal endoscopy image (the calculation method is the same as that of the gradient amplitude of the preprocessed image), L3 is a medical feature perception loss value, λ is a feature loss weight, which is usually set to [0.001, 0.01], n is the number of convolution layers of the pre-trained medical image network (such as ResNet), and m is the index of the number of convolution layers of the pre-trained medical image network, is a feature vector extracted by the pre-trained medical image network from the high-resolution gastrointestinal endoscopy image generated by the generator at the mth layer, is a feature vector extracted by the pre-trained medical image network from the high-resolution gastrointestinal endoscopy image at the mth layer, and L is a total loss value;

[0124] When the paired endoscopy images are not fully input into the dual-branch super-resolution generative adversarial network for training, and the total loss function has converged, the training is ended in advance;

[0125] Conversely, the training is continuously alternated until the dual-branch super-resolution generative adversarial network completes the training on all paired endoscopy images;

[0126] Specifically, the total loss function converges, that is, the change value of the total loss value of the continuous multiple training batches is less than the change threshold, for example, the change value of the total loss value is not more than 0.004 for 20 consecutive training rounds, which indicates that the total loss function converges, and the training is terminated in advance;

[0127] It should be noted that in the process of super-resolution reconstruction of the preprocessed image, the details of the micro-lesion region are easy to be smoothed or lost, especially leading to missed detection of small lesions. The lesion region attention mechanism of the Transformer is introduced to specifically enhance the details of the lesion region, so as to ensure the clarity and authenticity of the texture information of the lesion region and improve the detection rate of small lesions.

[0128] S4, identifying the lesion region by color analysis on the high-definition enhanced image.

[0129] Converting the high-definition enhanced image to the HSV color space to obtain the hue, saturation and brightness of the high-definition enhanced image in the HSV color space;

[0130] Specifically, since the high-definition enhanced image is an RGB image, the RGB values thereof need to be normalized first, and the expression is as follows:

[0131]

[0132] Wherein, R, G and B are the intensity values of red, green and blue in the high-definition enhanced image, respectively, R ′ , G ′ and B ′ are the normalized intensity values of red, green and blue in the high-definition enhanced image

[0133] Calculate the difference between the maximum and minimum values of the normalized intensity values of red, green and blue in the high-definition enhanced image;

[0134] Calculate the hue, saturation and brightness of the high-definition enhanced image in the HSV color space, and the expression is as follows:

[0135] Δ=C max (R ′ ,G ′ ,B ′ )-C min (R ′ ,G′ B ′ );

[0136]

[0137] V=C max ;

[0138] wherein, C max and C min are maximum and minimum values of normalized intensity values of red, green and blue in the high-definition enhanced image, Δ is a difference between C max and C min , H is a hue, mod represents a modulo operation, S is a saturation, and V is a brightness;

[0139] According to a hue, a saturation and a brightness range of the healthy digestive tract image on the HSV color space, a hue range threshold, a saturation range threshold and a brightness range threshold are set;

[0140] When a region in which the hue, the saturation and the brightness are all not within the hue range threshold, the saturation range threshold and the brightness range threshold appears in the high-definition enhanced image, it is identified that the high-definition enhanced image is a lesion region, otherwise, it is a non-lesion region.

[0141] S5, a lesion region on the high-definition enhanced image is segmented by using the active contour model, to obtain a lesion segmentation mask.

[0142] An initialization closed curve is set on a periphery of the lesion region of the high-definition enhanced image;

[0143] A gradient descent method is used to optimize an ACM energy function, to gradually shrink the initialization closed curve, to obtain an initial boundary of the lesion region;

[0144] A lesion segmentation mask is obtained by calculating a shortest Euclidean distance from each pixel point in the high-definition enhanced image to the initial boundary of the lesion region;

[0145] Specifically, an initialization closed curve is set on a periphery of the lesion region of the high-definition enhanced image in the form of an ellipse or a rectangle;

[0146] An ACM energy function of the active contour model is calculated, and an expression is as follows:

[0147] E=E1+E2;

[0148] E1=∫|Q ′ (s)| 2 ds+∫|Q ″ (s)| 2 ds;

[0149]

[0150] wherein E is the ACM energy function value, E1 is the internal energy value, which can constrain the smoothness of the curve and prevent the boundary from over-bending, E2 is the external energy value, which guides the curve to fit the lesion region by using image gradient, Q ′ (s) is the first derivative of the initialized closed curve Q(s), which represents the smoothness of the initialized closed curve, Q ″ (s) is the second derivative of the initialized closed curve, which represents the bending degree of the curve, s is a pixel point on the initialized closed curve, and g represents a Gaussian filter, is a gradient operator, which represents calculating the gradient amplitude of the high-definition enhanced image after Gaussian smoothing, and I3 is the high-definition enhanced image;

[0151] The ACM energy function is optimized by the gradient descent method, so as to realize the iterative update of the initialized closed curve Q(s), and the expression is as follows:

[0152]

[0153] wherein Q β (s) is the initialized closed curve after the βth update, Q β+1 (s) is the initialized closed curve after the (β+1)th update, and η is a learning rate, represents a partial derivative, and Q(s) is the initialized closed curve;

[0154] When the average Euclidean distance between Q β+1 (s) and Q β (s) is less than 1 pixel, it indicates that the ACM energy function converges, and the initial boundary of the lesion region is output, otherwise, the optimization is continued until the ACM energy function converges or the maximum number of iterations (such as 500 times) is reached, and then the optimization is stopped, and the initial boundary of the lesion region is output;

[0155] The shortest Euclidean distance of each pixel point in the high-definition enhanced image to the initial boundary of the lesion region is calculated, the shortest Euclidean distance of the pixel point outside the initial boundary of the lesion region is a positive value, the shortest Euclidean distance of the pixel point inside the initial boundary of the lesion region is a negative value, and the shortest Euclidean distance of the pixel point on the initial boundary of the lesion region is 0;

[0156] Based on the calculation result of the shortest Euclidean distance, the lesion segmentation mask is obtained, and the expression is as follows:

[0157]

[0158] Wherein, M is a lesion segmentation mask, φ is the shortest Euclidean distance from a pixel point to the initial boundary of the lesion area, M(i,j) = 1 indicates that the pixel point (i,j) is in the lesion area, and M(i,j) = 0 indicates that the pixel point (i,j) is not in the lesion area.

[0159] S6, based on the lesion segmentation mask and the high-definition enhanced image, using a pre-trained feature extraction network to classify the lesion type.

[0160] Based on the lesion segmentation mask, the lesion area on the high-definition enhanced image is extracted to obtain a high-definition enhanced lesion area image, and the expression is as follows:

[0161] I5(x,y) = M(i,j) x I3(i,j);

[0162] Wherein, I5(x,y) represents the high-definition enhanced lesion area image;

[0163] It should be noted that this calculation method is equivalent to performing pixel-by-pixel bitwise multiplication on the high-definition enhanced image. If M(i,j) = 1, then I5(x,y) = I3(i,j), that is, the pixel retains the value of the original image. If M(i,j) = 0, then I5(x,y) = 0, the value of the pixel is set to 0, indicating that the pixel point is shielded, ensuring that only the pixel values of the lesion area are retained, while the pixels in other areas (such as non-lesion areas and background) are filtered out, removing irrelevant information and improving the accuracy of lesion classification and feature extraction.

[0164] Different lesion sample images in the digestive tract endoscopic image database and high-definition enhanced lesion area images are input into the pre-trained feature extraction network, and lesion feature vectors are calculated respectively;

[0165] By calculating the cosine similarity of the lesion feature vectors of the lesion sample images and the high-definition enhanced lesion area images, the lesion type in the high-definition enhanced image is output.

[0166] Specifically, a ResNet50 pre-trained model is used as a feature extraction network, and the visual features of the high-definition enhanced lesion area image are extracted by ResNet50 to obtain a to-be-measured lesion feature vector;

[0167] At the same time, the visual features of different lesion sample images are extracted by ResNet50 to obtain a sample lesion feature vector set;

[0168] By calculating the cosine similarity of the to-be-measured lesion feature vector and each sample lesion feature vector in the sample lesion feature vector set, and the lesion type to which the sample lesion feature vector with the highest cosine similarity value belongs is output as the lesion type in the high-definition enhanced image.

[0169] The embodiment also provides a deep learning-based digestive tract auxiliary detection device, which comprises an image acquisition module, a suspected lesion module, an image enhancement module, a lesion recognition module, a lesion segmentation module and a lesion classification module. The image acquisition module is configured to acquire original images of a patient's digestive tract based on an event-driven sampling strategy through an endoscope device, and perform denoising, size adjustment and quality improvement processing on the original images to obtain preprocessed images. The suspected lesion module is configured to extract suspected lesion regions of the preprocessed images by using Canny edge detection. The image enhancement module is configured to perform super-resolution reconstruction and local enhancement on the preprocessed images by combining a super-resolution generative adversarial network and a self-attention mechanism to obtain high-definition enhanced images. The lesion recognition module is configured to recognize lesion regions by performing color analysis on the high-definition enhanced images. The lesion segmentation module is configured to segment the lesion regions on the high-definition enhanced images by using an active contour model to obtain lesion segmentation masks. The lesion classification module is configured to classify lesion types by using a pre-trained feature extraction network based on the lesion segmentation masks and the high-definition enhanced images.

[0170] The embodiment also provides a computer device suitable for the deep learning-based digestive tract auxiliary detection method, which comprises a memory and a processor. The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions to implement the deep learning-based digestive tract auxiliary detection method proposed in the above embodiment.

[0171] The computer device can be a terminal, which comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is configured to perform wired or wireless communication with external terminals. The wireless communication can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device. In addition, the input device can be an external keyboard, touchpad or mouse, etc.

[0172] The embodiment also provides a storage medium on which a computer program is stored, the program being executed by a processor to implement the method for assisting detection of a digestive tract based on deep learning proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0173] To sum up, the present application optimizes the image acquisition process based on an event-driven sampling strategy, reduces redundant data while improving the attention of potential lesion areas, thereby improving detection sensitivity and efficiency, accurately extracts suspected lesion areas using Canny edge detection, enhances image details, especially the clarity of small lesion areas, using a super-resolution generative adversarial network and a self-attention mechanism, effectively reduces the missed detection rate, accurately identifies the color features of the lesion area through HSV color space conversion, and realizes fine segmentation with the help of an active contour model, provides reliable quantitative analysis basis, classifies lesion types with high precision using a pre-trained feature extraction network, supports multi-class lesion recognition, and provides support for individualized treatment plan development.

[0174] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A deep learning-based method for assisting in the detection of the digestive tract, characterized in that: include, Using an endoscopic device, raw images of the patient's digestive tract are acquired based on an event-driven sampling strategy. The raw images are then denoised, resized, and improved in quality to obtain preprocessed images. Canny edge detection was used to extract suspected lesion areas from the preprocessed image; By combining super-resolution generative adversarial networks and self-attention mechanisms, super-resolution reconstruction and local enhancement are performed on the preprocessed image to obtain a high-resolution enhanced image. The specific steps are as follows. In super-resolution adversarial generative networks, the features of suspected lesion areas are enhanced based on the lesion area attention mechanism, resulting in a two-branch super-resolution generative adversarial network. In the overall super-resolution reconstruction branch, the generator extracts local features of the preprocessed image in the convolutional layer, and the residual block is used to enhance the local features. Upsampling is performed through subpixel convolution to convert the preprocessed image into a high-resolution image; The discriminator uses a convolutional neural network to make distinctions by calculating the true probability of high-resolution images; In the lesion region enhancement branch, the attention weight map of the suspected lesion region is calculated based on the gradient magnitude of the pixels in the suspected lesion region. The Transformer mechanism is used to enhance the local features of suspected lesion areas in high-resolution images, resulting in high-definition enhanced images. A two-branch super-resolution generative adversarial network was trained using a database of gastrointestinal endoscopy images. Retriev high-resolution gastrointestinal endoscopy images from a gastrointestinal endoscopy image database; The high-resolution gastrointestinal endoscope images input to the dual-branch super-resolution generative adversarial network are cropped to the same size, and the resolution of the high-resolution gastrointestinal endoscope images is reduced by the nearest neighbor interpolation method to obtain pairs of high-resolution and low-resolution gastrointestinal endoscope images. Pairs of gastrointestinal endoscope images are input into a two-branch super-resolution generative adversarial network for alternating training. The dual-branch super-resolution generative adversarial network is optimized by calculating the total loss function, which consists of a generative adversarial loss function, an edge-preserving loss function, and a medical feature perception loss function. If the total loss function has converged before the paired gastrointestinal endoscope images are fully input into the two-branch super-resolution generative adversarial network for training, then the training will end prematurely. Conversely, training continues alternately until the two-branch super-resolution generative adversarial network has completed training on all pairs of gastrointestinal endoscope images. By performing color analysis on high-definition enhanced images, lesion areas can be identified; Active contour modeling is used to segment lesion regions in high-resolution enhanced images to obtain lesion segmentation masks; Based on lesion segmentation masks and high-resolution enhanced images, a pre-trained feature extraction network is used to classify lesion types.

2. The deep learning-based digestive tract assisted detection method according to claim 1, characterized in that: The process involves acquiring raw images of the patient's digestive tract using an endoscopic device based on an event-driven sampling strategy. These raw images are then subjected to noise reduction, resizing, and quality enhancement to obtain preprocessed images. The specific steps are as follows: When using endoscopic equipment to acquire raw images of a patient's digestive tract, a preliminary judgment model based on a deep learning network is constructed to make a preliminary judgment on the raw images in real time. Based on the judgment results of the preliminary judgment model on the real-time acquired raw images, the frequency of raw image acquisition is adjusted. The wavelet transform method is used to remove noise from the acquired raw images; The original image after noise removal is resized using bilinear interpolation, and histogram equalization is used to enhance the contrast of the resized original image. The original image with enhanced contrast is normalized to obtain a preprocessed image.

3. The deep learning-based digestive tract assisted detection method according to claim 2, characterized in that: The method of using Canny edge detection to extract suspected lesion areas from the preprocessed image involves the following steps. The Sobel operator is used to calculate the gradients in the horizontal and vertical directions of the preprocessed image to obtain the gradient magnitude of the preprocessed image; Calculate the gradient direction of the preprocessed image based on the horizontal and vertical gradients of the preprocessed image; Based on non-maximum suppression, if the gradient magnitude of the preprocessed image is not a local maximum in the gradient direction, the gradient magnitude of the preprocessed image is set to zero; otherwise, it is retained. Based on the gradient magnitude of the preprocessed image, the histogram of all pixels in the preprocessed image is statistically analyzed to obtain the gradient distribution curve of all pixels in the preprocessed image. Based on the gradient distribution curve, set the high and low thresholds for dual-threshold filtering; Based on the comparison of the gradient magnitude of pixels in the preprocessed image with high and low thresholds, strong, weak edge points and non-edge points are distinguished. Remove weak edge points and non-edge points without strong edge points in the neighborhood, and connect strong edge points with weak edge points with strong edge points in the neighborhood to obtain the suspected lesion area.

4. The deep learning-based digestive tract assisted detection method according to claim 1, characterized in that: The method involves color analysis of the high-definition enhanced image to identify the lesion area. The specific steps are as follows: Convert the high-definition enhanced image to the HSV color space to obtain the hue, saturation, and brightness of the high-definition enhanced image in the HSV color space; Based on the hue, saturation, and brightness range of healthy digestive tract images in the HSV color space, set hue range thresholds, saturation range thresholds, and brightness range thresholds. When a region appears in a high-definition enhanced image where the hue, saturation, and brightness are all outside the hue range threshold, saturation range threshold, and brightness range threshold, a lesion region in the high-definition enhanced image is identified.

5. The deep learning-based digestive tract assisted detection method according to claim 1, characterized in that: The method of using an active contour model to segment lesion regions in high-resolution enhanced images to obtain lesion segmentation masks involves the following specific steps. An initial closed curve is set around the lesion area in the high-definition enhanced image; The gradient descent method was used to optimize the ACM energy function, gradually shrinking the initial closed curve to obtain the initial boundary of the lesion region; The lesion segmentation mask is obtained by calculating the shortest Euclidean distance from each pixel in the high-resolution enhanced image to the initial boundary of the lesion region.

6. The deep learning-based digestive tract assisted detection method according to claim 1, characterized in that: The method of classifying lesion types based on lesion segmentation masks and high-resolution enhanced images using a pre-trained feature extraction network involves the following steps: Based on the lesion segmentation mask, the lesion region is extracted from the high-definition enhanced image to obtain a high-definition enhanced lesion region image; Different lesion sample images and high-definition enhanced lesion area images from the gastrointestinal endoscopy image database are input into a pre-trained feature extraction network to calculate lesion feature vectors respectively. The type of lesion in the high-definition enhanced image is output by calculating the cosine similarity of the lesion feature vectors between the lesion sample image and the high-definition enhanced lesion region image.

7. A deep learning-based auxiliary detection device for the digestive tract, based on the deep learning-based auxiliary detection method for the digestive tract according to any one of claims 1 to 6, characterized in that: It includes an image acquisition module, a suspected lesion module, an image enhancement module, a lesion recognition module, a lesion segmentation module, and a lesion classification module. The image acquisition module is used to acquire raw images of the patient's digestive tract using an endoscopic device based on an event-driven sampling strategy, and to perform noise reduction, size adjustment and quality improvement processing on the raw images to obtain preprocessed images. The suspected lesion module is used to extract suspected lesion areas from preprocessed images using Canny edge detection; The image enhancement module is used to combine super-resolution generative adversarial network and self-attention mechanism to perform super-resolution reconstruction and local enhancement on preprocessed image to obtain high-definition enhanced image; The lesion recognition module is used to identify lesion areas by performing color analysis on high-definition enhanced images; The lesion segmentation module is used to segment the lesion region on the high-definition enhanced image using an active contour model to obtain a lesion segmentation mask. The lesion classification module is used to classify lesion types based on lesion segmentation masks and high-definition enhanced images, using a pre-trained feature extraction network.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the deep learning-based digestive tract assisted detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the deep learning-based digestive tract assisted detection method according to any one of claims 1 to 6.

Images