A fine-grained classification method for submucosal tumors based on EUS

Abstract

The present invention relates to a submucosal tumor fine-grained classification method based on EUS, characterized in that it includes the following steps: establishing a multi-scale relational attention network; training the relational attention network in two stages; inputting the EUS image obtained in real time into the trained relational attention network, and the relational attention network outputs the corresponding category label. Compared with the prior art, the present invention can effectively improve the recognition accuracy of submucosal tumors based on EUS, and effectively reduce the dependence on target-level data labels in the EUS recognition process, and can effectively identify the submucosal tumor area in the EUS image.

Description

Technical Field

[0001] The invention relates to a visual system calibration method for online detection of submucosal tumors, and belongs to the technical field of instance fine-grained image classification. Background Art

[0002] Gastrointestinal submucosal lesions (SMTs) refer to lesions originating from the various layers below the gastrointestinal mucosal layer. There are five common gastrointestinal SMTs, including gastrointestinal stromal tumors, ectopic pancreas, lipomas, leiomyomas, and neuroendocrine tumors. Because they all appear as normal raised lesions on the mucosa under endoscopy, ordinary endoscopic examination techniques cannot determine their source layer and nature. Therefore, after using white light endoscopy to find normal raised lesions on the mucosa, other imaging methods need to be combined to further determine the type of tumor.

[0003] Among them, EUS is currently one of the most accurate imaging methods for evaluating digestive tract SMT. During the EUS imaging process, the transducer sends short pulses, ultrasonic energy enters the tissue and receives tissue reflection signals to form an ultrasonic image of the tissue, thereby helping doctors further determine the nature of submucosal tumors. However, the image content of EUS is complex and varied, usually showing a 1-5-layer annular hierarchical structure, and showing destruction of the hierarchical structure in the tumor area. Clinicians usually rely on observing the origin level of the lesion in the area of ​​interest and its endoscopic ultrasound manifestation (such as shape, high and low echoes, echo patterns, etc.) to make a diagnosis.

[0004] With the continuous advancement of science and technology, the connection between computer technology and medicine has become closer and closer, and there are more and more examples of applying computer vision to the medical field. Facts such as cloud reading and robot contactless drug delivery have also proved that these technologies have indeed provided great assistance. In the medical process, manual diagnosis of SMTs based on EUS has a steep learning curve. Manual reading of images requires years of learning and practice to condense into empirical knowledge. By using computers to assist clinicians in diagnosis, the learning cost of EUS can be reduced and the work efficiency of clinicians can be improved. Therefore, it has important research significance.

[0005] However, there are several major challenges in EUS-based SMTs recognition. First, due to the high difficulty of ultrasound imaging operation and unstable imaging, EUS images are usually accompanied by a large amount of speckle noise, and sometimes artifacts. This high-noise and strong interference image feature poses a huge challenge to the extraction of effective information. Secondly, EUS images can reflect the hierarchical structure information of the gastrointestinal wall, and this information is used to determine the origin level of the lesion, thereby further characterizing the lesion. However, EUS imaging often has dense layers and unclear boundaries, and its hierarchical structure shape changes irregularly. This phenomenon is not significant, and the highly variable hierarchical characteristics further increase the difficulty of extracting hierarchical origin information. Finally, EUS images have very complex semantic relationships, including multiple semantics such as tumor shape in the lesion area, echo pattern, and tumor origin level. These semantics and the coupling relationship between them play different degrees of important roles in determining the type of tumor, but also further increase the difficulty of visual SMTs recognition. Summary of the invention

[0006] The purpose of the present invention is to achieve classification of SMTs based on image-level labels only.

[0007] In order to achieve the above object, the technical solution of the present invention is to provide a method for fine-grained classification of submucosal tumors based on EUS, characterized in that it comprises the following steps:

[0008] Step 1, establishing a multi-scale relational attention network, which includes an encoder implemented based on a convolutional neural network structure and a relational attention module, wherein the encoder extracts multiple scale features of the input EUS image, and the relational attention module obtains an attention map of the features, and multiplies the attention map by the features extracted by the encoder element by element to enhance the region of interest in the spatial domain, suppress irrelevant regions, and output category labels based on the region of interest, where the categories are pre-defined different tumor categories;

[0009] Step 2: Train the model in two stages. Use a self-supervised learning pre-training method based on image restoration to train the feature extractor for tumor classification, and add a multi-scale feature relationship attention network downstream to further train the model until convergence. Specifically, the following steps are included:

[0010] Step 201, obtaining EUS image samples of different categories;

[0011] Step 202, the EUS image samples of different categories obtained in step 201 are combined into an EUS image training sample set;

[0012] Step 203: using an EUS adaptive occlusion algorithm to process each EUS image sample in the EUS image training sample set to obtain an occluded image sample and a corresponding occluded area. The EUS adaptive occlusion algorithm utilizes the imaging characteristics of EUS to adaptively occlude part of the information on the EUS image hierarchy to achieve self-supervised learning.

[0013] Step 204: Take the occluded image sample as input and the corresponding occluded area as a label, and train the model structure using ContextEncoder to complete the first stage of training. The model structure includes an encoder, a decoder, and a discriminator implemented based on a convolutional neural network structure. The model structure completes the occluded area of ​​the input occluded image sample and outputs a complete EUS image sample.

[0014] Step 205: assign the weight of the encoder of the trained model structure to the encoder of the relational attention network, which is the feature extractor;

[0015] Step 206: Using the EUS image sample obtained in step 201 as input and the corresponding category as a label, the relational attention network after the encoder weights are updated is trained, thereby completing the second stage of training;

[0016] Step 3: Input the real-time EUS image into the trained relational attention network, and the relational attention network outputs the corresponding category label.

[0017] Preferably, the tumor types include gastrointestinal stromal tumors, ectopic pancreas, neuroendocrine tumors, lipomas and leiomyomas.

[0018] Preferably, the EUS adaptive occlusion algorithm comprises the following steps:

[0019] Step 2031, performing histogram equalization processing on the EUS image sample, and then performing a thresholding operation, so that the values ​​of all pixels in the histogram equalized image with a pixel value between [0.7, 0.8] are 1, the values ​​of all pixels in the pixel value between [0.85, 0.99] are 2, and the pixel values ​​of the remaining pixels are all 0, thereby obtaining a thresholded image;

[0020] Step 2032: performing an erosion operation on the thresholded image, so that the region contours of regions close to each other in the thresholded image are connected to each other, while the region contours of regions far from each other are separated from each other;

[0021] Step 2033, find the closed curves in the image obtained in the previous step, calculate the curve area of ​​the area enclosed by each closed curve, sort the curve area areas, select the first three closed curves with the largest curve area areas, and assume them as the outer contour curves of the image hierarchy structure;

[0022] Step 2034: collect points inside the outer contour curve, use the obtained points as the central coordinate points for generating a mask, generate a mask based on the central coordinate points, and use the mask to block the EUS image samples, thereby obtaining the blocked image samples and the corresponding blocked areas.

[0023] Preferably, in step 2034, when sampling points inside the outer contour curve, any point inside the outer contour curve is selected as the center coordinate point.

[0024] Preferably, in step 2034, the mask is a square mask.

[0025] Compared with the prior art, the present invention can effectively improve the recognition accuracy of submucosal tumors based on EUS, and effectively reduce the dependence on target-level data labels in the EUS recognition process, and can effectively identify submucosal tumor areas in EUS images. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is the overall framework diagram of a fine-grained classification method for submucosal tumors under EUS proposed in the present invention;

[0027] Figure 2 A schematic diagram of a self-supervised pre-training method for fine-grained classification of submucosal tumors under EUS proposed by the present invention;

[0028] Figure 3 Schematic diagram of a multi-scale relational attention network suitable for fine-grained classification of submucosal tumors under EUS proposed in the present invention;

[0029] Figure 4 Schematic diagram of a relational attention mechanism proposed in the present invention for fine-grained classification of submucosal tumors under EUS. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0031] The present invention provides a fine-grained classification method for submucosal tumors based on EUS, comprising the following steps:

[0032] Step 1: Establish a hierarchical structure perception module based on self-supervised learning. With the help of the auxiliary task of hierarchical region image restoration, learn the hierarchical structure information of the image with only image-level annotations, including the following steps:

[0033] Step 101: Collect EUS image samples of five categories to establish an EUS image training sample set. The five categories are gastrointestinal stromal tumor, ectopic pancreas, neuroendocrine tumor, lipoma and leiomyoma.

[0034] Step 102: For each EUS image sample in the EUS image training sample set, an EUS adaptive occlusion algorithm (LPM) is used to occlude the hierarchical area to obtain the occluded image sample and the corresponding occluded area as the input and label of the CNN model structure for training using the context encoder.

[0035] The EUS adaptive occlusion algorithm uses the imaging characteristics of EUS to adaptively block part of the information on the occlusion hierarchy for the auxiliary task of self-supervised learning. Specifically, it includes the following steps:

[0036] Step 1021: The Contrast Limited Adaptive Histogram Equalization algorithm is used to process the EUS image samples, and a thresholding operation is performed in combination, so that the pixel values ​​of all pixels in the histogram equalized image between [0.7, 0.8] are 1, the pixel values ​​of all pixels in the [0.85, 0.99] are 2, and the pixel values ​​of the remaining pixels are 0, thereby obtaining an image Y.

[0037] Step 1022, perform an erosion operation on the image Y so that the adjacent regions of the image Y are merged together, and find closed curves on the eroded image, each closed curve corresponds to a contour, thereby forming a contour set {C};

[0038] Step 1023: Calculate the area of ​​the region enclosed by each contour in the contour set {C}, and select three contours with the largest area as candidate regions to be masked.

[0039] Step 1024: Sample any point in the candidate area as the center coordinate point of the square mask to generate a mask of size 56×56. In the image, the pixel value of the pixel in the mask area is 0, and the pixel values ​​of the remaining parts are 1. Through the element-by-element multiplication method, the mask is overlaid on the original image to obtain the occluded image and the corresponding mask area. The occluded image is the occluded image sample, and the mask area is the corresponding occluded area.

[0040] Step 103: Take the occluded image sample as input and the corresponding occluded area as a label to train the model structure using Contextencoder. The model structure includes an encoder implemented based on a convolutional neural network structure, a decoder, and a discriminator, which is used to repair the occluded area in the occluded image sample, so that in the process of repairing the image, the encoder can learn the hierarchical structure information of the image.

[0041] Step 2: Establish a multi-scale relational attention network, which includes an encoder that shares weights with the encoder in the hierarchical perception module and a relational attention module. In step 1, after training using the model structure of Contextencoder, the weights of the encoder in the hierarchical perception module are shared with the encoder of the multi-scale relational attention network, so that the encoder of the multi-scale relational attention network updates its weights before training.

[0042] The multi-scale relational attention network models the relationships between spatial semantics at different scales, allowing the model to learn the different importance of these relationships for tumor recognition, thereby fully mining the complex semantic information in EUS images.

[0043] The multi-scale relational attention network includes an encoder and a relational attention module. The encoder is implemented based on a given convolutional neural network structure. The multi-scale relational attention network extracts features of different scales output by the encoder. Taking VGG-16 as an example, the CNN features obtained after each Maxpooling layer are extracted as the input of the relational attention module. Assume that the batch contains a total of B samples, and its features can be expressed as (B, C, W, H), where C represents the number of channels, W represents the width of the feature map, and H represents the height of the feature map.

[0044] The relational attention module takes the following steps to process the input features (B, C, W, H):

[0045] Step 201: For a CNN feature at a certain scale whose input is (B, C, W, H), firstly use the Averagepooling operation to obtain a feature map of (B, 1, W, H) according to the channel dimension;

[0046] Step 202: Sample the pixel pairs in the 3×3 spatial domain on the feature map obtained in step 201, and use the convolution kernel to aggregate the two pixels of the pixel pairs to obtain the relationship representation of the pixel pairs. This relationship representation of the pixel pairs reflects spatial semantic relationships of different granularities at different scales. For example, at a low scale, this relationship is represented as a relationship between points; at a high scale, this relationship representation of the pixel pairs is a relationship between abstract semantics.

[0047] Step 203: For the relational representations of pixel pairs at different positions in the same spatial domain, learn the weights of different relational representations. In this embodiment, a total of n pairs of pixel pair relational representations are generated according to all possible situations of different pixel pair combinations composed of all pixel points in the 3×3 spatial domain (in this embodiment, n is 36), and the attention weights corresponding to different relational representations are learned. This process can be expressed as the following function:

[0048]

[0049] In the formula, p′ represents the result of 3×3 spatial domain aggregation, w k is the weight of the relationship to be learned, are the two pixel points of the kth pixel pair, is the convolution kernel for aggregating the kth pair of pixels.

[0050] Step 204: Repeat steps 201 to 203 C times in different channel dimensions, perform relational attention learning in C different channel dimensions, and finally obtain a spatial attention map containing semantic relation knowledge of C channels.

[0051] Step 205: Use channel attention to purify the spatial attention maps of C channels, and superimpose the purified spatial attention maps of all C channels to obtain the attention map S of the current input feature (B, C, W, H).

[0052] Step 206: Multiply the attention map S by the input features (B, C, W, H) element by element to enhance the region of interest in the spatial domain and suppress irrelevant regions.

[0053] Step 207: Perform classification based on the enhanced region of interest and finally output a category label.

Claims

1. A fine-grained classification method for submucosal tumors based on EUS, characterized in that: The following steps are involved: Step 1, establishing a multi-scale relational attention network, which includes an encoder implemented based on a convolutional neural network structure and a relational attention module, wherein the encoder extracts multiple scale features of the input EUS image, and the relational attention module obtains an attention map of the features, and multiplies the attention map by the features extracted by the encoder element by element to enhance the region of interest in the spatial domain, suppress irrelevant regions, and output category labels based on the region of interest, where the categories are pre-defined different tumor categories; Step 2: Train the model in two stages. Use a self-supervised learning pre-training method based on image restoration to train the feature extractor for tumor classification, and add a multi-scale feature relationship attention network downstream to further train the model until convergence. Specifically, the following steps are included: Step 201, obtaining EUS image samples of different categories; Step 202, the EUS image samples of different categories obtained in step 201 are combined into an EUS image training sample set; Step 203: using an EUS adaptive occlusion algorithm to process each EUS image sample in the EUS image training sample set to obtain an occluded image sample and a corresponding occluded area. The EUS adaptive occlusion algorithm utilizes the imaging characteristics of EUS to adaptively occlude part of the information on the EUS image hierarchy to achieve self-supervised learning. The EUS adaptive occlusion algorithm includes the following steps: Step 2031, performing histogram equalization processing on the EUS image sample, and then performing a thresholding operation, so that the values ​​of all pixels in the histogram equalized image with a pixel value between [0.7, 0.8] are 1, the values ​​of all pixels in the pixel value between [0.85, 0.99] are 2, and the pixel values ​​of the remaining pixels are all 0, thereby obtaining a thresholded image; Step 2032: performing an erosion operation on the thresholded image, so that the region contours of regions close to each other in the thresholded image are connected to each other, while the region contours of regions far from each other are separated from each other; Step 2033, find the closed curves in the image obtained in the previous step, calculate the curve area of ​​the area enclosed by each closed curve, sort the curve area areas, select the first three closed curves with the largest curve area areas, and assume them as the outer contour curves of the image hierarchy structure; Step 2034: sampling points inside the outer contour curve, using the obtained points as the central coordinate points for generating a mask, generating a mask based on the central coordinate points, and using the mask to block the EUS image sample, thereby obtaining the blocked image sample and the corresponding blocked area; Step 204: Take the occluded image sample as input and the corresponding occluded area as a label, and train the model structure using ContextEncoder to complete the first stage of training. The model structure includes an encoder, a decoder, and a discriminator implemented based on a convolutional neural network structure. The model structure completes the occluded area of ​​the input occluded image sample and outputs a complete EUS image sample. Step 205: assign the weight of the encoder of the trained model structure to the encoder of the relational attention network, which is the feature extractor; Step 206: Using the EUS image sample obtained in step 201 as input and the corresponding category as a label, the relational attention network after the encoder weights are updated is trained, thereby completing the second stage of training; Step 3: Input the real-time EUS image into the trained relational attention network, and the relational attention network outputs the corresponding category label.

2. The EUS-based submucosal tumor fine-grained classification method according to claim 1, characterized in that: The tumor categories include gastrointestinal stromal tumors, ectopic pancreas, neuroendocrine tumors, lipomas, and leiomyoma.

3. The EUS-based submucosal tumor fine-grained classification method according to claim 1, characterized in that: In step 2034, when sampling points inside the outer contour curve, any point inside the outer contour curve is selected as the center coordinate point.

4. The EUS-based submucosal tumor fine-grained classification method according to claim 3, characterized in that: In step 2034, the mask is a square mask.

Images