Goblet cell semantic segmentation method and system based on boundary gradient attention network

Abstract

The application provides a goblet cell semantic segmentation method and system based on a boundary gradient attention network. The method adopts an improved boundary gradient attention U-Net network structure. By improving an existing deep neural network model, three boundary gradient attention mechanisms are introduced between an encoding layer and a decoding layer. A color gradient feature extraction layer is used to learn color feature changes around the goblet cell. On the basis of obtaining cell shape information, the semantic segmentation network can effectively capture color gradient features around the goblet cell, and accurate segmentation of a goblet cell region in a confocal laser endomicroscopy image is realized.

Description

Technical Field

[0001] This invention belongs to the field of medical image semantic segmentation technology, and particularly relates to a goblet cell semantic segmentation method and system based on boundary gradient attention network. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Gastric cancer has become the third leading cause of cancer death and the fifth leading cause of new cancer cases, seriously threatening human life and health. Current medical research indicates that early detection and diagnosis are key to reducing the incidence of gastric cancer and improving patient survival rates. It is generally believed that gastric cancer develops from chronic gastritis, atrophic gastritis, intestinal metaplasia, and intraepithelial neoplasia. This series of changes is known as the Correa sequence in gastric carcinogenesis, and gastric intestinal metaplasia (GIM), as an intermediate step in the multi-step hypothesis of gastric cancer development within the Correa sequence, is considered a precancerous lesion. GIM is defined as the replacement of surface, pitted, and / or glandular epithelium in the acid-secreting or antral mucosa by goblet cells. Therefore, the distribution and area occupied by goblet cells can effectively reflect the degree of lesion in gastric intestinal metaplasia.

[0004] Due to limitations in imaging technology, early clinical diagnosis struggled to obtain real-time medical images of cellular structures. The assessment of the severity of gastric mucosal intestinal metaplasia primarily relied on post-stained tissue structure observation. This method lacked microscopic data on cellular structure, hindering cellular-level quantitative analysis. Thanks to the rapid development of digital imaging over the past decade, researchers have successfully applied confocal microscopy to clinical diagnosis, developing confocal laser endomicroscopy (CLE). As a novel technology, CLE allows real-time observation of cellular state information magnified 1000 times, reflecting the severity of the disease at a microscopic level. As a diagnostic tool, CLE reveals more detailed changes in the disease, enabling more precise pathological assessments and making it possible to diagnose patient conditions at the cellular level. In recent years, CLE has driven advancements in numerous medical fields.

[0005] In the field of confocal laser microscopy for diagnosing the severity of intestinal metaplasia, goblet cells, as an important precancerous feature, play a crucial role in physician consultation, targeted treatment, and postoperative examination. Previous studies have shown that confocal laser microscopy for determining the distribution of goblet cells has high specificity and can effectively assess the severity of the disease. Clinically, goblet cell identification is a vital part of disease diagnosis and has received widespread attention. However, confocal laser microscopy offers higher image resolution within the same area, requires a wider observation range, and is time-consuming and laborious for manual observation, making diagnostic assessment more difficult. Furthermore, the presence of normal cell regions surrounding goblet cells creates a similar color and texture structure on the confocal laser microscopy screen. In addition, cytoplasmic leakage and ciliary distribution also affect the segmentation of goblet cell regions, making accurate segmentation of goblet cell regions impossible with traditional identification methods.

[0006] Currently, deep neural networks have become one of the most popular research directions in the medical field. However, due to the influence of sample size and image complexity, deep neural networks often fail to perform well in the semantic segmentation of goblet cells in confocal laser microscopy, and the following problems exist: First, deep neural networks cannot pay attention to the features of color gradient changes around the cells when the sample size is small; second, deep neural networks have difficulty achieving accurate semantic segmentation of the goblet cell region under confocal laser microscopy when the training sample size is small. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a semantic segmentation method and system for goblet cells based on a boundary gradient attention network. This method utilizes the U-Net semantic segmentation network model to achieve accurate segmentation of goblet cell regions in confocal laser microscopy images. By improving existing deep neural network models, three boundary gradient attention mechanisms are introduced between the encoding and decoding layers. A color gradient feature extraction layer learns the color feature changes around the goblet cells, enabling the semantic segmentation network to effectively capture the color gradient features around the goblet cells while acquiring cell shape information, thus improving the accuracy of cell semantic segmentation.

[0008] To achieve the above objectives, the present invention mainly includes the following aspects:

[0009] In a first aspect, Embodiment 1 of the present invention provides a goblet cell semantic segmentation method based on a boundary gradient attention network.

[0010] A goblet cell semantic segmentation method based on boundary gradient attention network includes:

[0011] Acquire confocal laser microscopy images;

[0012] Non-pathological regions are removed from the acquired images, and pathological regions are extracted from the images;

[0013] The three-channel color information of the pathological region of the image is read and data augmentation is performed to obtain the enhanced color feature information;

[0014] A goblet cell semantic segmentation model was established, and the loss function was defined using the cross-entropy function to train the goblet cell semantic segmentation model.

[0015] The enhanced color feature information is input into the trained goblet cell semantic segmentation model to obtain the semantic segmentation results of the goblet cell region;

[0016] The goblet cell semantic segmentation model adopts a U-Net network structure, which includes an input layer, an encoding layer, a decoding layer, a color gradient feature extraction layer, and an output layer connected in sequence. The color gradient feature extraction layer is used to learn the color feature changes around the goblet cell.

[0017] In one possible implementation, acquiring confocal laser microscopy images includes the following steps:

[0018] Acquiring clinical medical images using confocal laser microscopy and endoscopy;

[0019] The acquired confocal laser endoscopy images are extracted by sampling at fixed intervals.

[0020] The output layer is used to output the goblet cell region, and outputs the goblet cell region according to pixel probability information.

[0021] In one possible implementation, the method for extracting pathological regions from the image is as follows: reading confocal laser microscopy image information and non-pathological regions, eliminating image regions without pathological information using a rule-based approach, and extracting pathological regions from the confocal laser microscopy image by removing non-pathological regions.

[0022] In one possible implementation, the three-channel information is augmented using the Frankle-McCann Retinex algorithm.

[0023] In one possible implementation, the goblet cell semantic segmentation model includes an input layer, four encoding layers, four decoding layers, three boundary gradient attention-guided layers, and an output layer. Between the encoding and decoding layers, three boundary gradient attention mechanisms are introduced to learn the color feature changes around the goblet cell.

[0024] In one possible implementation, the input layer is used to receive three-channel information after Frankle-McCann Retinex data enhancement;

[0025] The coding layer is used to extract color features from the goblet cell image and capture deep features of the image;

[0026] The decoding layer is used to restore the image and convert the image's feature information into corresponding pixel probability information.

[0027] The boundary gradient attention mechanism is used to focus on color change information around goblet cells and exclude areas such as normal cells.

[0028] The output layer is used to output the goblet cell region, and outputs the goblet cell region according to pixel probability information.

[0029] In one possible implementation, during the encoding and decoding stages, the model extracts feature information from the input image, obtains deep feature information of the image through the encoding structure, and transforms the feature information into a probability map during the decoding stage.

[0030] During the decoding phase, the model restores the deep feature information to the original region probability feature information, and transforms the feature information into an image probability feature map through the decoding structure.

[0031] During the decoding phase, the model restores the deep feature information to the original region probability feature information, and transforms the feature information into an image probability feature map through the decoding structure.

[0032] Secondly, embodiments of the present invention provide a goblet cell semantic segmentation system based on a boundary gradient attention network.

[0033] A goblet cell semantic segmentation system based on a boundary gradient attention network includes:

[0034] The acquisition module acquires images from confocal laser microscopy endoscopes.

[0035] The extraction module removes non-pathological regions from the acquired image and extracts the pathological regions from the image; the enhancement module reads the three-channel color information of the pathological regions of the image and performs data enhancement to obtain enhanced color feature information.

[0036] The training module establishes a goblet cell semantic segmentation model, uses the cross-entropy function to define the loss function, and trains the goblet cell semantic segmentation model.

[0037] The segmentation module inputs the enhanced color feature information into the trained goblet cell semantic segmentation model to obtain the semantic segmentation results of the goblet cell region;

[0038] The goblet cell semantic segmentation model adopts a U-Net network structure, which includes an input layer, an encoding layer, a decoding layer, a color gradient feature extraction layer, and an output layer connected in sequence. The color gradient feature extraction layer is used to learn the color feature changes around the goblet cell.

[0039] Thirdly, embodiments of the present invention provide a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the goblet cell semantic segmentation method based on boundary gradient attention network as described in any possible implementation of the first aspect above are performed.

[0040] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the goblet cell semantic segmentation method based on boundary gradient attention network as described in any possible implementation of the first aspect above.

[0041] The beneficial effects of this invention are:

[0042] This invention utilizes deep learning to achieve semantic segmentation of goblet cells in confocal microscopic endoscopy images. It proposes a boundary gradient attention-based U-Net model, which improves existing deep neural networks by introducing three boundary gradient attention mechanisms between the encoding and decoding layers. This enables the model to focus on the region surrounding the goblet cells, effectively noting the color feature information around the cells and removing normal gastric mucosal cell regions, thus greatly improving the accuracy of goblet cell region segmentation and obtaining accurate goblet cell distribution information. Attached Figure Description

[0043] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0044] Figure 1 This is a flowchart of the goblet cell semantic segmentation method based on boundary gradient attention U-Net according to an embodiment of the present invention.

[0045] Figure 2 This is a structural diagram of the goblet cell semantic segmentation model based on boundary gradient attention U-Net in an embodiment of the present invention.

[0046] Figure 3 This is a structural diagram of the pixel attention mechanism in an embodiment of the present invention. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0048] Example 1

[0049] Due to limitations in sample size and image complexity, deep neural networks often fail to perform well in semantic segmentation of goblet cells under confocal laser microscopy. In particular, when the sample size is small, they cannot focus on the characteristics of color gradient changes around the cells, making it difficult to achieve accurate semantic segmentation of goblet cell regions under confocal laser microscopy.

[0050] Therefore, this embodiment improves the existing deep neural network model to focus on the area surrounding the goblet cell, and applies the improved model to semantic segmentation of goblet cells in confocal laser microscopy, thereby assisting doctors in diagnosis, reducing labor costs, and obtaining accurate goblet cell distribution information.

[0051] This embodiment provides a goblet cell semantic segmentation method based on a boundary gradient attention network, including the following steps:

[0052] Step 1: Acquire confocal laser endoscopy images;

[0053] Step 2: Remove non-pathological regions from the acquired images and extract the pathological regions from the images;

[0054] Step 3: Read the three-channel color information of the pathological region of the image and perform data enhancement to obtain enhanced color feature information;

[0055] Step 4: Establish a goblet cell semantic segmentation model, define the loss function using the cross-entropy function, and train the goblet cell semantic segmentation model;

[0056] Step 5: Input the enhanced color feature information into the trained goblet cell semantic segmentation model to obtain the semantic segmentation results of the goblet cell region;

[0057] The boundary gradient attention network adopts an improved U-Net network structure.

[0058] The goblet cell semantic segmentation model based on boundary gradient attention U-Net consists of an input layer, four encoding layers, three color gradient feature extraction layers, four decoding layers, and an output layer connected in sequence. Between the four encoding layers and the decoding layers, three color gradient feature extraction layers are introduced to learn the color feature changes around the goblet cell.

[0059] In step 1, confocal laser microscopy images are acquired using a confocal laser microscopy system.

[0060] In practice, the EC.3870CIK confocal laser microscopy system manufactured by Pentax Corporation of Japan is used. Its structure is basically similar to that of a conventional endoscope system, including: a confocal microscopy endoscope, a touch screen display for the confocal microscopy endoscope, a conventional endoscope image display, an image processor, an optical unit, and a confocal control unit.

[0061] The confocal endoscope has a distal diameter of 12.8 mm. Its distal end includes a water / air nozzle, two light guides, an auxiliary channel, and a 2.8 mm working channel. The distal end can bend vertically by 130° and horizontally by 120°. Two buttons on the operating handle allow for adjustment of the depth of the imaging plane.

[0062] Store clinical confocal data and construct a clinical confocal microendoscopy dataset D. For the original sample video to be identified in dataset D, extract experimental trials z at fixed frame intervals.

[0063] Stored patient information S and captured images S z ;

[0064] In confocal endoscopic microscopy image S z Extract and identify the diseased cell region within the identification unit;

[0065] A training sample set R for confocal laser microscopy and endoscopy was established, storing labeled information and pathological regions of confocal images, which were used to train a semantic segmentation network for goblet cells in confocal laser microscopy and endoscopy.

[0066] In step 2, to address the limitations of confocal laser microscopy imaging, a rule-based approach is used to eliminate image regions lacking pathological information. The specific steps are as follows:

[0067] By reading confocal laser microscopy image information and non-pathological regions, a rule-based method is used to extract pathological regions from the non-pathological regions of the confocal laser microscopy image, as shown in the following formula:

[0068]

[0069] Among them, image i To output image information, S z For confocal laser microscopy images, L Z For non-pathological regions, x is the x-coordinate of the corresponding pixel, y is the y-coordinate of the corresponding pixel, and C is the complement operator.

[0070] In step 3, data enhancement of the pathological region of the confocal laser endoscopic microscopy image is performed by reading the three-channel color information of the pathological region of the confocal laser endoscopic microscopy image, and each reconstructed image... ih×w×c Data augmentation is performed using the Frankle-McCann Retinex algorithm for ∈R, where h is the length of the image, w is the width of the image, c is the number of red, green and blue channels, and R is the training sample set.

[0071] The specific calculation steps are as follows:

[0072] (a) Extract the green channel feature map of the original image and convert the pixel values ​​of the feature map from the integer domain to the logarithmic domain to reduce the computational load of subsequent algorithms;

[0073] (b) Initialize a matrix of the same size as the original image S(x,y) with all element values ​​t, where t is the mean of the input feature map of the original image;

[0074] (c) Find the maximum distance S between the target point and the two comparison points:

[0075] S = 2fix[log2min(m,n)-1] (2) where m is the x-coordinate and n is the y-coordinate; fix is ​​the floor function.

[0076] (d) Calculate the pixels along the path using the following formula:

[0077]

[0078]

[0079] Where, r n (x, y) represents the result of the previous iteration, and the updated result r is obtained by calculating the path pixels. n '(x,y), then average the two, and finally get the output result r. n+1 (x,y), where △l is the brightness difference of the target point along this path;

[0080] (e) Let S = -2 / S;

[0081] (f) Repeat steps ce three times n times until |S|<1, and select a different initial comparison point each time;

[0082] (j) The target is linearly stretched, and the result of the linear stretching process is:

[0083]

[0084] Where max is r n The maximum value in (x,y) is min, and r is the minimum value. n The minimum value in (x,y);

[0085] (h) Repeat step aj for the red component and the blue component respectively;

[0086] (i) The processed green, red and blue components are spliced ​​together to form enhanced three-channel image features.

[0087] The three-channel information, after Frankle-McCann Retinex data augmentation, is used as image features and participates in subsequent steps.

[0088] In step 4, during implementation, a network for semantic segmentation is constructed specifically to address the characteristics of goblet cells, such as... Figure 2 As shown, this model consists of one input layer, four encoding layers, four decoding layers, three in-layer feature extraction layers, and one output layer. The input layer receives the three-channel information after Frankle-McCann Retinex data augmentation. The encoding layers extract color features from the goblet cell image, capturing deep image features. The decoding layers reconstruct the image, converting the image's feature information into corresponding pixel probability information. The color gradient feature extraction layer focuses on color changes around the goblet cells, excluding normal cells and other regions. The output layer outputs the goblet cell region based on the pixel probability information.

[0089] This embodiment designs the functions of each layer of the goblet cell semantic segmentation network based on boundary gradient attention U-Net:

[0090] (1) Input Layer (L1): The input layer L1 takes a Frankle-McCann Retinex augmented feature vector l1 as input. l1 is a 3D matrix W×H×3, where W×H is the width and height of the medical image, and 3 represents the 3D feature channels augmented by Frankle-McCann Retinex. l1 is convolved with a 3*3 kernel, a stride of 3, 3 input channels, and 64 output channels to obtain a feature vector l2. l2 is a 3D feature vector W×H×64, where 64 represents the number of feature channels.

[0091] (2) Encoding Layer (L2): The main function of this layer is to perform shallow extraction on the original image passed from the input layer. The input of the encoding layer L2 is the feature vector l2. l2 is passed through a convolution with a kernel size of 3*3, a stride of 1, 64 input channels and 128 output channels; a normalization function; an activation function; another convolution with a kernel size of 3*3, a stride of 1, 128 input channels and 128 output channels; a normalization function; and an activation function to obtain the feature vector l3, which is a three-dimensional feature vector W×H×128.

[0092] (3) Encoding Layer (L3): The main function of this layer is to integrate the new features of L2 and achieve the effect of capturing deeper structures. The input of the encoding layer L3 is the feature vector l3. After a 2x random downsampling operation, l3 is sequentially passed through a convolution with a kernel of 3*3, a stride of 1, 128 input channels and 256 output channels; a normalization function; an activation function; another convolution with a kernel of 3*3, a stride of 1, 256 input channels and 256 output channels; a normalization function; and an activation function to obtain the feature vector l4, which is a three-dimensional feature vector W / 2×H / 2×256.

[0093] (4) Encoding Layer (L4): The main function of this layer is to integrate the feature maps of L3 and achieve the effect of capturing deeper structures. The input of the encoding layer L3 is the feature vector l4. After a 2x random downsampling operation, l4 is sequentially passed through a convolution with a kernel size of 3*3, a stride of 1, 256 input channels, and 512 output channels; a normalization function; an activation function; another convolution with a kernel size of 3*3, a stride of 1, 512 input channels, and 512 output channels; a normalization function; and an activation function to obtain the feature vector l5, which is a three-dimensional feature vector W / 4×H / 4×512.

[0094] (5) Encoding Layer (L5): The main function of this layer is to integrate the feature maps of L4 and achieve the effect of capturing deeper structures. The input of the encoding layer L4 is the feature vector l5. After a 2x random downsampling operation, l5 is passed through a convolution with a kernel size of 3*3, a stride of 1, 512 input channels, and 1024 output channels; a normalization function; an activation function; another convolution with a kernel size of 3*3, a stride of 1, 1024 input channels, and 1024 output channels; a normalization function; and an activation function to obtain the feature vector l6; l6 is a three-dimensional feature vector W / 8×W / 8×1024.

[0095] (6) Decoding layer (L6): The input of this layer is the feature vector l6. l6 is passed through a series of convolutions with a kernel size of 3*3, a stride of 1, an input channel size of 1024, and an output channel size of 1024; a normalization function; an activation function; another convolution with a kernel size of 3*3, a stride of 1, an input channel size of 1024, and an output channel size of 1024; a normalization function; and an activation function to obtain the feature vector l7. l7 is a three-dimensional feature vector W / 8×H / 8×1024.

[0096] (7) Color gradient feature extraction layer (L10): such as Figure 3As shown, the function of layer L10 is to focus on the color gradient changes around the cells and exclude some non-goblet cell regions. The input features of L10 are feature vectors l7 and l5. l5 is extracted by a convolution operation with a kernel of 7*1, a stride of 1, 512 input channels, and 1024 output channels to obtain feature vector w5; and by a convolution operation with a kernel of 1*7, a stride of 1, 512 input channels, and 1024 output channels to obtain feature vector h5. Then w5 and h5 are fused through an additive attention mechanism, and the fused features are passed through an activation function and a normalization function to obtain feature vector O1. Feature vector l7 is extracted by a convolution operation with a kernel of 1*1, a stride of 1, 1024 input channels, and 1024 output channels, followed by an upsampling operation. The extracted features are fused with feature vector O1 using an additive attention mechanism. The fused features are then processed through a 3x3 convolution with a stride of 1, 1024 input channels, and 1 output channel; a normalization function; and an activation function to obtain feature vector O2. O2 is then multiplicatively fused with feature vector O1 using a multiplicative attention mechanism after data dimensionality expansion to obtain feature vector O3. O3 is a three-dimensional feature vector W / 4×H / 4×1024. The upsampling operation uses bilinear interpolation, and the upsampling process uses the following formula:

[0097]

[0098]

[0099]

[0100] (8) Decoding Layer (L7): The function of the decoding layer (L7) is to restore the deep features into probabilistic information. The input of this layer is feature vector O3 and feature vector l7. Feature vector l7 is upsampled and then concatenated with feature vector O3. The concatenated feature vector is then passed through a convolution with a stride of 1, 2048 input channels and 1024 output channels; a normalization function; an activation function; a convolution with a kernel of 3*3, a stride of 1, 1024 input channels and 512 output channels; a normalization function; and an activation function to obtain feature vector l8, which is a three-dimensional feature vector W / 4×H / 4×512.

[0101] (9) Color Gradient Feature Extraction Layer (L11): The function of the L11 layer is to focus on the color gradient changes around the cells and exclude some non-goblet cell regions. The input features of L11 are feature vectors l4 and l8. l4 is extracted by a convolution operation with a kernel of 7*1, a stride of 1, 256 input channels, and 512 output channels to obtain feature vector w6; and by a convolution operation with a kernel of 1*7, a stride of 1, 256 input channels, and 512 output channels to obtain feature vector h6. Then w6 and h6 are fused through an additive attention mechanism, and the fused features are passed through an activation function and a normalization function to obtain feature vector O4. Feature vector l8 is extracted by a convolution operation with a kernel of 1*1, a stride of 1, 512 input channels, and 512 output channels, and an upsampling operation. The extracted features are fused with feature vector O4 using an additive attention mechanism. The fused features are then processed through a 3x3 convolution with a stride of 1, 512 input channels, and 1 output channel; a normalization function; and an activation function to obtain feature vector O5. O5 is then multiplicatively fused with feature vector O4 using a multiplicative attention mechanism after data dimensionality expansion to obtain feature vector O6, which is a three-dimensional feature vector W / 2×H / 2×512. The upsampling operation uses bilinear interpolation, and the upsampling process uses the following formula:

[0102]

[0103]

[0104]

[0105] (10) Decoding Layer (L8): The function of the decoding layer (L8) is to restore the deep features into probabilistic information. The input of this layer is feature vector O6 and feature vector l8. Feature vector l8 is upsampled and then concatenated with feature vector O6. The concatenated feature vector is then passed through a convolution with a stride of 1, 1024 input channels and 512 output channels; a normalization function; an activation function; a convolution with a kernel of 3*3, a stride of 1, 512 input channels and 256 output channels; a normalization function; and an activation function to obtain feature vector l9, which is a three-dimensional feature vector W / 2×H / 2×256.

[0106] (11) Color Gradient Feature Extraction Layer (L12): The function of the L12 layer is to focus on the color gradient changes around the cells and exclude some non-goblet cell regions. The input features of L11 are feature vectors l3 and l9. l3 is used to extract feature vector w7 through a convolution operation with a kernel of 7*1, a stride of 1, 128 input channels, and 256 output channels; and to extract feature vector h7 through a convolution operation with a kernel of 1*7, a stride of 1, 128 input channels, and 256 output channels. Then w7 and h7 are fused through an additive attention mechanism, and the fused features are passed through an activation function and a normalization function to obtain feature vector O7. Feature vector l9 is extracted through a convolution operation with a kernel of 1*1, a stride of 1, 256 input channels, and 256 output channels, and an upsampling operation. The extracted features are fused with feature vector O7 using an additive attention mechanism. The fused features are then processed through a 3x3 convolution with a stride of 1, 256 input channels, and 1 output channel; a normalization function; and an activation function to obtain feature vector O8. O8 is then augmented and fused with feature vector O7 using a multiplicative attention mechanism to obtain feature vector O9, which is a three-dimensional feature vector W×H×256. The upsampling operation uses bilinear interpolation, and the upsampling process uses the following formula:

[0107]

[0108]

[0109]

[0110] (12) Decoding Layer (L9): The function of the decoding layer (L9) is to restore the deep features to the probability information of the corresponding pixels in the image. The input of this layer is feature vector O9 and feature vector l9. Feature vector l9 is upsampled and then concatenated with feature vector O9. The concatenated feature vector is then passed through a convolution with a stride of 1, 512 input channels, and 256 output channels; a normalization function; an activation function; a convolution with a kernel of 3*3, a stride of 1, 256 input channels, and 128 output channels; a normalization function; and an activation function to obtain feature vector l9. 10 , l 10 The feature vector is a three-dimensional feature vector W×H×128.

[0111] (13) Output Layer (L13): The output layer (L13) is responsible for achieving the final classification. The input to this layer is the feature vector l. 10 l 10After a convolutional layer with a 3x3 kernel, a stride of 1, 128 input channels, and 64 output channels, followed by an activation function, a normalization, and another convolutional layer with a 3x3 kernel, a stride of 1, 64 input channels, and 2 output channels, the final probability information, FET, is output. FET represents a probability information of W×H×2, where 2 indicates the number of channels. The first layer's channel count represents the probability that the corresponding pixel is a goblet cell region, and the second layer's feature channels represent the probability that the corresponding pixel is a background region. By comparing the values ​​of the first and second layer channels, the corresponding pixel categories can be determined.

[0112] In the model, the formulas for convolution in the input layer, encoder layer, and decoder layer are as follows:

[0113]

[0114] In the formula, m is the x-coordinate of the image, n is the y-coordinate of the image, i is the x-coordinate of the convolution kernel, j is the y-coordinate of the convolution kernel, g is the corresponding convolution kernel, x is the x-coordinate of the pixel in the resulting image, and y is the y-coordinate of the pixel in the resulting image.

[0115] The loss function constructed in this embodiment is defined as follows:

[0116] Loss=cross_entropy(p,t) (17)

[0117] Where p is the output value of the model, and t is the labeled data.

[0118] To accelerate network convergence, the weights and biases of each layer in the network are randomized to a normal distribution with a mean of 0 and a variance of 1. Ninput , where N input This represents the number of upper-layer features. The loss function is defined using the cross-entropy function. Simultaneously, the Adam optimizer and gradient descent algorithm are used in PyTorch to adjust the connection weights and biases. The maximum number of iterations is set to 1000, and the loss threshold is set to 0.01.

[0119] This embodiment uses a confocal microscopy system to capture confocal microscopy images. Non-interested regions are eliminated using a rule-based approach. Frankle-McCann Retinex data augmentation is used to reduce the influence of illumination. Finally, goblet cell semantic segmentation based on boundary gradient attention U-Net is performed. Boundary gradient attention U-Net can effectively notice the color feature information around the cells and remove normal gastric mucosal cell areas, which greatly improves the accuracy of goblet cell semantic segmentation in confocal microscopy images.

[0120] Example 2

[0121] Embodiment 2 of the present invention provides a goblet cell semantic segmentation system based on a boundary gradient attention network.

[0122] A goblet cell semantic segmentation system based on a boundary gradient attention network includes:

[0123] The acquisition module acquires images from confocal laser microscopy endoscopes.

[0124] The extraction module removes non-pathological regions from the acquired images and extracts the pathological regions from the images.

[0125] The enhancement module reads the three-channel color information of the pathological region of the image and performs data enhancement to obtain enhanced color feature information;

[0126] The training module establishes a goblet cell semantic segmentation model, uses the cross-entropy function to define the loss function, and trains the goblet cell semantic segmentation model.

[0127] The segmentation module inputs the enhanced color feature information into the trained goblet cell semantic segmentation model to obtain the semantic segmentation results of the goblet cell region;

[0128] Among them, the boundary gradient attention network is a U-Net network structure; the goblet cell semantic segmentation model includes an input layer, an encoding layer, a decoding layer, a color gradient feature extraction layer and an output layer connected in sequence, and the color gradient feature extraction layer is used to learn the color feature changes around the goblet cell.

[0129] Example 3

[0130] Embodiment 3 of the present invention provides a computer device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the goblet cell semantic segmentation method based on boundary gradient attention network as described in any possible embodiment of the first aspect above.

[0131] Example 4

[0132] Embodiment 4 of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the goblet cell semantic segmentation method based on boundary gradient attention network as described in any possible embodiment of the first aspect above.

[0133] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A goblet cell semantic segmentation method based on boundary gradient attention network, characterized in that, include: Acquire confocal laser microscopy images; Non-pathological regions are removed from the acquired images, and pathological regions are extracted from the images; The three-channel color information of the pathological region of the image is read and data augmentation is performed to obtain the enhanced color feature information. The Frankle-McCann Retinex algorithm is used to augment the three-channel information. A goblet cell semantic segmentation model was established, and the loss function was defined using the cross-entropy function to train the goblet cell semantic segmentation model. The enhanced color feature information is input into the trained goblet cell semantic segmentation model to obtain the semantic segmentation results of the goblet cell region; The goblet cell semantic segmentation model uses a U-Net network structure. The goblet cell semantic segmentation model includes an input layer, four encoding layers, four decoding layers, three boundary gradient attention-guided layers, and an output layer. Between the encoding and decoding layers, three boundary gradient attention mechanisms are introduced to learn the color feature changes around the goblet cell. The boundary gradient attention mechanisms include 7x1 convolution, 1x7 convolution, additive attention mechanism, and multiplicative attention mechanism.

2. The goblet cell semantic segmentation method based on boundary gradient attention network as described in claim 1, characterized in that, The acquisition of confocal laser endoscopy images includes the following steps: Acquiring clinical medical images using confocal laser microscopy and endoscopy; The acquired confocal laser endoscopy images are extracted by sampling at fixed intervals.

3. The goblet cell semantic segmentation method based on boundary gradient attention network as described in claim 1, characterized in that, The method for extracting pathological regions from images is as follows: read confocal laser microscopy image information and non-pathological regions, eliminate image regions without pathological information using a rule-based approach, and extract pathological regions from confocal laser microscopy images by removing non-pathological regions.

4. The goblet cell semantic segmentation method based on boundary gradient attention network as described in claim 1, characterized in that, The input layer is used to receive three-channel information after Frankle-McCann Retinex data enhancement; The coding layer is used to extract color features from the goblet cell image and capture deep features of the image; The decoding layer is used to restore the image and convert the image's feature information into corresponding pixel probability information. The boundary gradient attention mechanism is used to focus on color change information around goblet cells and exclude normal cell regions. The output layer is used to output the goblet cell region, and outputs the goblet cell region according to pixel probability information.

5. The goblet cell semantic segmentation method based on boundary gradient attention network as described in claim 1, characterized in that, During the encoding stage, the model extracts feature information from the input image and obtains deep feature information of the image through the encoding structure, which is then used to calculate probability information. During the decoding phase, the model restores the deep feature information to the original region probability feature information, and transforms the feature information into an image probability feature map through the decoding structure.

6. A goblet cell semantic segmentation system based on a boundary gradient attention network, characterized in that, include: The acquisition module acquires images from confocal laser microscopy endoscopes. The extraction module removes non-pathological regions from the acquired images and extracts the pathological regions from the images. The training module establishes a goblet cell semantic segmentation model, uses the cross-entropy function to define the loss function, and trains the goblet cell semantic segmentation model. The enhancement module reads the three-channel color information of the pathological region of the image and performs data enhancement to obtain enhanced color feature information. The Frankle-McCann Retinex algorithm is used to enhance the three-channel information. The segmentation module inputs the enhanced color feature information into the trained goblet cell semantic segmentation model to obtain the semantic segmentation results of the goblet cell region; The goblet cell semantic segmentation model uses a U-Net network structure. The goblet cell semantic segmentation model includes an input layer, four encoding layers, four decoding layers, three boundary gradient attention-guided layers, and an output layer. Between the encoding and decoding layers, three boundary gradient attention mechanisms are introduced to learn the color feature changes around the goblet cell. The boundary gradient attention mechanisms include 7x1 convolution, 1x7 convolution, additive attention mechanism, and multiplicative attention mechanism.

7. A computer device, characterized in that, include: The system includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the goblet cell semantic segmentation method based on a boundary gradient attention network as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the goblet cell semantic segmentation method based on a boundary gradient attention network as described in any one of claims 1 to 5.

Images