Gastroscopic image analysis method using image segmentation technology, and computer system

Abstract

The present disclosure relates to a gastroscopic image analysis method using image segmentation technology, and a computer system. The method comprises: segmenting a gastroscopic image into a plurality of blocks; inputting the blocks into an encoder to obtain an encoded feature map; inputting the encoded feature map into a decoder to obtain a decoded feature map; and inputting the decoded feature map into a masked focus modulation decoder, wherein the masked focus modulation decoder is used for inputting a first template feature into a focus modulation function to obtain a focus modulation feature, inputting the decoded feature map into a scaling function and then substituting the decoded feature map into a projection function to obtain a scaled feature, and generating a second template feature on the basis of a mask, the scaled feature and the focus modulation feature; and on the basis of the second template feature, generating a prediction mask, which is used for segmenting a gastric intestinal metaplasia region. In the present disclosure, a gastroscopic image is analyzed by means of image segmentation technology, thereby achieving the effect of segmenting a gastric intestinal metaplasia region in the gastroscopic image.

Description

Endoscopic image analysis method and computer system using image segmentation technique TECHNICAL FIELD

[0001] The present disclosure relates to a method for analyzing endoscopic images using deep learning techniques, in particular, a method for segmenting the intestinal metaplasia region in the images. BACKGROUND

[0002] Gastric cancer is the sixth most common cancer and the fourth leading cause of cancer death worldwide. Gastric cancer occurs after Helicobacter pylori infection, inflammation of the gastric body, atrophy of the stomach, and intestinal metaplasia of the gastric mucosa. Regular follow-up of patients with precancerous lesions, such as intestinal metaplasia of the gastric mucosa, is important for improving survival rates through early diagnosis. How to correctly diagnose and grade gastric body inflammation and intestinal metaplasia of the gastric mucosa during gastroscopy is an important clinical issue. Currently, to diagnose gastric body inflammation and intestinal metaplasia of the gastric mucosa, biopsy samples are taken from five to six sites in the stomach during gastroscopy, and then pathologists interpret the results. The advantage is that the diagnosis is correct, but the disadvantage is that it is time-consuming, has risks such as bleeding, and is not suitable for large-scale screening. Therefore, developing a technique that does not require invasive biopsy and can quickly analyze intestinal metaplasia of the gastric mucosa is an important and new topic in precision health.

[0003] SUMMARY

[0004] An embodiment of the present disclosure provides an endoscopic image analysis method using image segmentation techniques, which is suitable for a computer system. The analysis method includes: obtaining an endoscopic image and cutting the endoscopic image into a plurality of blocks; inputting one of the blocks into an encoder to obtain an encoded feature map; inputting the encoded feature map into a decoder to obtain a decoded feature map; and inputting the decoded feature map into a mask focal modulation decoder. The mask focal modulation decoder includes a plurality of mask focal modulation stages. One of the mask focal modulation stages receives the decoded feature map and a first template feature. The first template feature is input into a focal modulation function to obtain a focal modulation feature. The decoded feature map is input into a scaling function and then into a projection function to obtain a scaled feature. A second template feature is generated according to a mask, the scaled feature, and the focal modulation feature. The method further includes generating a predicted mask according to the second template feature. The predicted mask includes a plurality of pixels and is used to segment an intestinal metaplasia region of the gastric mucosa.

[0005] In some embodiments, the step of generating the predicted mask according to the second template feature includes: inputting the second template feature corresponding to the last stage of the mask focal modulation stages into a plurality of neural networks to obtain a mask feature and a class feature; performing element multiplication on the mask feature and the decoded feature map to obtain a binary mask; and performing element multiplication on the binary mask and the class feature to obtain the predicted mask.

[0006] In some embodiments, the step of generating the second template feature according to the mask, the scaled feature, and the focus modulation feature comprises element-wise multiplying the mask, the scaled feature, and the focus modulation feature to generate the second template feature.

[0007] In some embodiments, the mask comprises a plurality of values, and the method further comprises determining whether each value is less than a threshold to generate a first-level mask.

[0008] In some embodiments, the encoder comprises a plurality of focus modulation blocks, and the decoder is a pixel decoder.

[0009] In another aspect, embodiments of the disclosure provide a computer system, comprising a memory and a processor. The memory is configured to store a plurality of instructions, and the processor is communicatively coupled to the memory and configured to execute the instructions to perform a plurality of steps. The steps comprise obtaining an endoscopic image, and segmenting the endoscopic image into a plurality of patches; inputting one of the patches into an encoder to obtain an encoded feature map; inputting the encoded feature map into a decoder to obtain a decoded feature map; and inputting the decoded feature map into a mask focus modulation decoder. The mask focus modulation decoder comprises a plurality of mask focus modulation stages, one of the mask focus modulation stages is configured to receive the decoded feature map and a first template feature, input the first template feature into a focus modulation function to obtain a focus modulation feature, input the decoded feature map into a scaling function and then into a projection function to obtain a scaled feature, and generate a second template feature according to a mask, the scaled feature, and the focus modulation feature. The steps further comprise generating a predicted mask according to the second template feature, the predicted mask comprising a plurality of pixels configured to segment an intestinal metaplasia area of a gastric mucosa.

[0010] In some embodiments, the step of generating the predicted mask according to the second template feature comprises inputting the second template feature corresponding to a last one of the mask focus modulation stages into a plurality of neural networks to obtain a mask feature and a class feature; element-wise multiplying the mask feature and the decoded feature map to obtain a binary mask; and element-wise multiplying the binary mask and the class feature to obtain the predicted mask.

[0011] In some embodiments, the step of generating the second template feature according to the mask, the scaled feature, and the focus modulation feature comprises element-wise multiplying the mask, the scaled feature, and the focus modulation feature to generate the second template feature.

[0012] In some embodiments, the mask comprises a plurality of values, and the processor is further configured to determine whether each value is less than a threshold to generate a first-level mask.

[0013] In some embodiments, the encoder comprises a plurality of focus modulation blocks, and the decoder is a pixel decoder. Attached Figure Description

[0014] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings.

[0015] Figure 1 is a schematic diagram illustrating a computer system according to an embodiment;

[0016] Figure 2 is a schematic diagram illustrating a deep learning network according to an embodiment;

[0017] Figure 3 is a schematic diagram illustrating the architecture of a mask focus modulation block according to an embodiment;

[0018] Figure 4 is a schematic diagram illustrating a mask focusing modulation layer according to an embodiment;

[0019] Figure 5 is a schematic diagram illustrating the experimental results according to one embodiment;

[0020] Figure 6 is a flowchart illustrating a method for analyzing gastrointestinal endoscopic images using image segmentation technology according to an embodiment.

[0021] [Symbol Explanation]

[0022] 110: Endoscopic gastroscopy images

[0023] 120: Computer System

[0024] 121: Processor

[0025] 122: Memory

[0026] 130: Predictive Masking

[0027] 201: Steps

[0028] 210: Encoder

[0029] 211–214: Focused modulation levels

[0030] 220: Decoder

[0031] 221: Decoding Feature Map

[0032] 230: Masked Focus Modulation Decoder

[0033] 231: Template Features

[0034] 232-234: Masking Focus Modulation Level

[0035] 241-242: Neural Networks

[0036] 251: Masking Feature

[0037] 252: Category Features

[0038] 261 : binarized mask

[0039] 300: mask focus modulation block

[0040] 301 : mask

[0041] 302: template feature

[0042] 311-312: mask focus modulation layers

[0043] 321-323: residual normalization layers

[0044] 331 : feed-forward layer

[0045] 340: template feature

[0046] 410: focus modulation function

[0047] 411-414: convolution layers

[0048] 421-424: gates

[0049] 501-505: test data

[0050] 510: region

[0051] 601-605: steps DETAILED DESCRIPTION

[0052] As used herein, the terms "first", "second", and so on do not necessarily mean a specific order or sequence, but are merely used to distinguish elements or operations described in the same technical terms.

[0053] FIG. 1 is a schematic diagram illustrating a computer system according to an embodiment. Referring to FIG. 1, the computer system 120 includes a processor 121 and a memory 122 for processing a gastroscope image 110 to generate a predicted mask 130 for segmenting a location of gastric intestinal metaplasia. In another aspect, the predicted mask 130 can also be referred to as a segmentation result of the gastroscope image 110, and the gastric intestinal metaplasia region is to be segmented. Here, the gastroscope image 110 can be a gastroscope image of an antrum, a body, or a cardia, and the disclosure does not limit the location, resolution, and angle of the gastroscope image 110.

[0054] The computer system 120 can be a personal computer, a server, a medical related device, or various electronic devices with computing capability, and the present disclosure is not limited thereto. The processor 121 can be a central processing unit, a graphics processing unit, a microprocessor, a microcontroller, a tensor processing unit, a special application integrated circuit, or the like. The memory can be a random access memory, a read only memory, a flash memory, a floppy disk, a hard disk, an optical disk, a solid state disk, a magnetic tape, or a database accessible via a network, which stores a plurality of instructions to be executed by the processor to perform the method of analyzing the endoscopic image, which will be described below.

[0055] In this embodiment, a deep learning network is used to generate a prediction mask. FIG. 2 illustrates a schematic diagram of the deep learning network according to an embodiment. Referring to FIG. 2, first, in step 201, the endoscopic image 110 is divided into a plurality of patches. For example, the resolution of the endoscopic image 110 is H x W x 3, and in this embodiment, 4 patches are cut for both the horizontal and vertical axes, so that the size of each patch is Next, a trainable linear layer (e.g., a fully connected layer) is used to convert the patch into a feature map as input. The above H, W, and C are positive integers, for example, H = 224, W = 224, and C = 96, but the present disclosure is not limited to these values.

[0056] The converted feature map from the patch is input to an encoder 210, which is, for example, a focal modulation network, and includes a plurality of focal modulation stages 211-214. For details, please refer to the first paper YANG, Jianwei, et al. Focal modulation networks. Advances in Neural Information Processing Systems, 2022, 35: 4203-4217. In this embodiment, the focal modulation stage 211 includes 2 focal modulation blocks, and the size of the feature map output by the focal modulation stage 211 is The focal modulation stage 212 includes 2 focal modulation blocks, and the size of the feature map output by the focal modulation stage 212 is The focal modulation stage 213 includes 6 focal modulation blocks, and the size of the feature map output by the focal modulation stage 213 is The size of the feature map output by the focal modulation stage 214 is The feature map output by the encoder 210 is also called an encoded feature map. Through focal modulation, the entire network can learn visual features from small field of view to large field of view.

[0057] Next, the encoded feature map is inputted into a decoder 220, which comprises a plurality of stages, each stage generating a feature map with a different resolution. The decoder 220 is, for example, a pixel decoder, details of which can be found in the second paper CHENG, Bowen, et al. Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022. p. 1290-1299, which is not described in detail here. The feature map generated by the decoder 220 is referred to as a decoded feature map 221, which is also denoted as f s where s represents the stage of the pixel decoder, and the output resolution of the last stage is equivalent to one quarter of the original resolution, i.e.

[0058] Next, the decoded feature map f s is inputted into a mask focus modulation decoder 230, which comprises a plurality of mask focus modulation stages 232-234, each of which receives the decoded feature map f s and comprises three mask focus modulation blocks. In addition, the mask focus modulation stage 232 also receives a template feature 231, which is of size N x C, where N is a positive integer (for example, 256), which is denoted as p b where b represents the bth mask focus modulation block. In each mask focus modulation stage 232-234, the template feature p b and the decoded feature map f s are used to generate the template feature p b+1 in the b+1th mask focus modulation block.

[0059] FIG. 3 is a diagram illustrating an architecture of a mask focus modulation block according to an embodiment. Referring to FIG. 3, the mask focus modulation block 300 includes mask focus modulation layers 311-312, residual normalization layers 321-323, and a feed-forward layer 331. A mask 301 is input to the mask focus modulation layers 311-312. A template feature 302 is input to the mask focus modulation layer 311 and the residual normalization layer 321. A decoded feature map 221 is input to the mask focus modulation layers 311-312. Each of the residual normalization layers 321-323 includes a residual connection and a layer normalization. The mask focus modulation block 300 generates a template feature 340 for a next mask focus modulation block. The mask 301 provides a segmentation result, which is used to guide the model to pay attention to a specific region during training, and the mask focus modulation provides semantic texture features at different sizes.

[0060] The operation of the mask focus modulation layers 311-312 is described. FIG. 4 is a diagram illustrating the mask focus modulation layer according to an embodiment. Referring to FIG. 4, the operation of the mask focus modulation layer can be represented as the following mathematical expression 1.

[0061] [mathematical expression 1]

[0062] MFM(p b ,f s ,m b )=Z(p b )⊙F(R(f s ))⊙m b

[0063] where MFM() represents the overall operation of the mask focus modulation layer. Z() is a focus modulation function, which is represented as a focus modulation function 410 in FIG. 4. Z(p b ) is also referred to as a focus modulation feature. R() is a scaling function. F() is a projection function, which can be implemented using a linear transformation, for example. m b is the mask 301 in the b-th mask focus modulation block. The template feature 302 in FIG. 4 is p b in the mathematical expression 1. The decoded feature map 221 is f s in the mathematical expression 1. In addition, the operation symbol ⊙ is element-wise multiplication.

[0064] The patch feature 302 is linearly transformed (e.g., by a fully connected layer) and then provided to multiple convolutional layers 411-414 and multiple gates 421-424. The focus modulation function 410 can be referred to the first paper above and will not be described in detail here. Similarly, the decoded feature map 221 is scaled and then linearly transformed (i.e., a projection function) to obtain the scaled feature F(R(f s ))).

[0065] Here, the mask m b is a binary mask, and m b is generated from the (b-1)th mask focus modulation block. Specifically, the mask of the previous stage contains multiple values, which are denoted as M b-1 (x, y), where x and y are coordinates. Each of these values is then compared to a threshold value (e.g., 0.5) to generate the mask m b of the next stage, as shown in the following mathematical expression 2.

[0066] [mathematical expression 2]

[0067] The mask can force the model to focus on the pixels that are difficult to classify. In another aspect, the mathematical expression 1 above inputs the patch feature p b to a focus modulation function Z() to obtain the focus modulation feature Z(p b ), inputs the decoded feature map f s to a scaling function R() and then to a projection function F() to obtain the scaled feature F(R(f s )), and then element-wise multiplies the mask m b , the scaled feature F(R(f s )), and the focus modulation feature Z(p b ) to generate the patch feature MFM(p b , f s , m b ) for use in the subsequent network.

[0068] Referring to FIG. 2, the pattern features (also referred to as second pattern features) generated by the last mask focus modulation stage 234 after passing through the mask focus modulation decoding stage 230 can be used to generate the predicted mask 130. Specifically, the second pattern features are input to a plurality of neural networks 241-242, such as multilayer perceptrons (MLPs). The neural network 241 outputs mask features 251, and the neural network 242 outputs class features 252, where the mask features 251 have a size of NxC, and the class features 252 have a size of NxK, where K is a positive integer. Next, the mask features 251 are element-wise multiplied with the decoded feature map 221 of the maximum resolution generated by the decoder 220 to obtain a binary mask 261 having a size of NxHxW. Then, the binary mask 261 is element-wise multiplied with the class features 252 to obtain the predicted mask 130.

[0069] The loss function employed by the present disclosure is shown in the following mathematical expression 3.

[0070] [mathematical expression 3]

[0071] l = λ ce l ce + λ dice l dice + λ cls l cls

[0072] where l ce , l dice and l cls represent the binary cross-entropy loss, the dice loss, and the classification loss, respectively, and λ ce , λ dice , λ cls are the weights corresponding to the losses. Details can be referred to the second paper described above.

[0073] The predicted mask 130 has a dimension of HxWxK, which contains a plurality of pixels for segmenting the gastric mucosa enterification region, such as setting the gastric mucosa enterification class as a value of "1" and the background as a value of "0", but the present disclosure is not limited to such examples. FIG. 5 is a schematic diagram illustrating experimental results according to an embodiment. FIG. 5 illustrates test data 501-505, which are generated by superimposing the predicted mask on the original gastroscopic image. For example, the test data 501 segments the region 510 where the gastric mucosa enterification occurs, and the others are similar.

[0074] In the present disclosure, a mask-focused modulation network is proposed to analyze endoscopic images of the stomach to segment the area of gastric mucosa intestinal metaplasia occurrence. In the network proposed in the present disclosure, due to the cooperation of the encoder and the decoder, the mask-focused modulation decoder focuses on the overall features generated by different fields of view, which makes the network focus on the learnable mask and provides better segmentation results.

[0075] FIG. 6 is a flowchart illustrating a method of analyzing endoscopic images of the stomach using image segmentation technology according to an embodiment. Referring to FIG. 6, in step 601, endoscopic images of the stomach are obtained, and the endoscopic images of the stomach are cut into blocks. In step 602, the blocks are input into an encoder to obtain encoded feature maps. In step 603, the encoded feature maps are input into a decoder to obtain decoded feature maps. In step 604, the decoded feature maps are input into a mask-focused modulation decoder. The mask-focused modulation decoder includes multiple mask-focused modulation stages, wherein a first stage is configured to receive the decoded feature maps and a first template feature, input the first template feature into a focus modulation function to obtain a focus modulation feature, input the decoded feature maps into a scaling function and then into a projection function to obtain a scaled feature, and generate a second template feature according to a mask, the scaled feature, and the focus modulation feature. In step 605, a predicted mask is generated according to the second template feature, and the predicted mask includes multiple pixels for segmenting a gastric mucosa intestinal metaplasia area. The steps in FIG. 6 have been described in detail above, and thus will not be described again here. It is worth noting that the steps in FIG. 6 can be implemented as multiple program codes or circuits, and the present disclosure is not limited in this regard. In addition, the method of FIG. 6 can be used in combination with the above embodiments or can be used alone, that is, other steps can be added between the steps of FIG. 6.

[0076] Although the present disclosure has been disclosed with the above embodiments, it is not intended to limit the present disclosure, and those skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the appended claims.

Claims

1. A method for analyzing endoscopic images of the stomach using image segmentation techniques, suitable for a computer system, characterized by, The analysis method comprises: obtaining a gastroscopic image and cutting the gastroscopic image into a plurality of blocks; inputting one of the plurality of blocks to an encoder to obtain an encoded feature map; inputting the encoded feature map to a decoder to obtain a decoded feature map; inputting the decoded feature map to a mask focal modulation decoder, wherein the mask focal modulation decoder comprises a plurality of mask focal modulation stages, one of the plurality of mask focal modulation stages is used to receive the decoded feature map and a first template feature, input the first template feature into a focal modulation function to obtain a focal modulation feature, input the decoded feature map into a scaling function and then into a projection function to obtain a scaled feature, and generate a second template feature according to a mask, the scaled feature and the focal modulation feature; and generating a predicted mask according to the second template feature, the predicted mask comprises a plurality of pixels, and is used to segment a gastric mucosa intestinal metaplasia area.

2. The method of analysis of claim 1, wherein, The step of generating the predicted mask according to the second template feature comprises: inputting the second template feature corresponding to the last stage of the plurality of mask focal modulation stages into a plurality of neural networks to obtain a mask feature and a class feature; performing element-wise multiplication on the mask feature and the decoded feature map to obtain a binary mask; and performing element-wise multiplication on the binary mask and the class feature to obtain the predicted mask.

3. The method of analysis of claim 1, wherein, The step of generating the second template feature according to the mask, the scaled feature and the focal modulation feature comprises: performing element-wise multiplication on the mask, the scaled feature and the focal modulation feature to generate the second template feature.

4. The method of analysis of claim 1, wherein, The mask comprises a plurality of numerical values, and the analysis method further comprises: determining whether each of the plurality of numerical values is less than a threshold value to generate a first-level mask.

5. The method of analysis of claim 1 wherein, The encoder comprises a plurality of focal modulation blocks, and the decoder is a pixel decoder.

6. A computer system, characterized by It comprises: a memory storing a plurality of instructions; and a processor communicatively connected to the memory and configured to execute the plurality of instructions to perform a plurality of steps: obtaining a gastroscopic image and cutting the gastroscopic image into a plurality of blocks; inputting one of the plurality of blocks to an encoder to obtain an encoded feature map; inputting the encoded feature map to a decoder to obtain a decoded feature map; inputting the decoded feature map to a mask focal modulation decoder, wherein the mask focal modulation decoder comprises a plurality of mask focal modulation stages, one of the plurality of mask focal modulation stages is used to receive the decoded feature map and a first template feature, input the first template feature into a focal modulation function to obtain a focal modulation feature, input the decoded feature map into a scaling function and then into a projection function to obtain a scaled feature, and generate a second template feature according to a mask, the scaled feature and the focal modulation feature; and generating a predicted mask according to the second template feature, the predicted mask comprises a plurality of pixels, and is used to segment a gastric mucosa intestinal metaplasia area.

7. The computer system of claim 6, wherein, The step of generating the predicted mask according to the second template feature comprises: inputting the second template feature corresponding to the last stage of the plurality of mask focal modulation stages into a plurality of neural networks to obtain a mask feature and a class feature; performing element-wise multiplication on the mask feature and the decoded feature map to obtain a binary mask; and performing element-wise multiplication on the binary mask and the class feature to obtain the predicted mask. inputting the second template feature corresponding to a last one of the plurality of mask focus modulation stages into a plurality of neural networks to obtain a mask feature and a class feature; performing element-wise multiplication on the mask feature and the decoded feature map to obtain a binarized mask; and performing element-wise multiplication on the binarized mask and the class feature to obtain the predicted mask.

8. The computer system of claim 6, wherein, The step of generating the second template feature according to the mask, the scaled feature, and the focus modulation feature comprises: performing element-wise multiplication on the mask, the scaled feature, and the focus modulation feature to generate the second template feature.

9. The computer system of claim 6, wherein, The mask comprises a plurality of numerical values, and the processor is further configured to determine whether each of the plurality of numerical values is less than a threshold value to generate a next-level mask.

10. The computer system of claim 6, wherein, The encoder comprises a plurality of focus modulation blocks, and the decoder is a pixel decoder. The encoder comprises a plurality of focus modulation blocks, and the decoder is a pixel decoder.

Images