A Transformer-based grading system for the severity of intestinal metaplasia

Abstract

This invention proposes a grading system for the severity of intestinal metaplasia based on the deformation Transformer. By dividing the endoscopic image into multiple feature units, matching feature unit groups are obtained based on the similarity of the feature units. The feature unit groups are then fused after weight calculation, realizing the deformation of the sampling region in the Transformer. This overcomes the shortcomings of the fixed region sampling in the Transformer, improves the feature extraction capability, and enhances the accuracy of grading the severity of intestinal metaplasia.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a grading system for the severity of intestinal metaplasia based on the Transformer. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] According to global statistics from the World Health Organization (WHO), stomach cancer ranks fourth in cancer-related mortality. Intestinal metaplasia (IM) is a common precancerous condition of stomach cancer, indicating that the patient's risk of developing cancer is significantly higher than that of the general population. Furthermore, some studies suggest that the severity of IM may be a key factor in determining this risk, with higher severity patients corresponding to a higher risk of stomach cancer.

[0004] In clinical practice, the Operative Link on Gastric Intestinal Metaplasia Assessment (OLGIM) and the Endoscopic Grading of Gastric Intestinal Metaplasia (EGGIM) are widely recognized as effective methods for assessing the severity of gastric metaplasia (IM) and predicting cancer risk. OLGIM, based on pathological diagnosis, accurately identifies IM severity and predicts gastric cancer risk, and its effectiveness has been extensively validated by various studies, making it considered the gold standard for IM severity identification and gastric cancer risk prediction. However, collecting gastric biopsies for assessment is time-consuming and labor-intensive, and may cause harm to patients during the biopsy process. In recent years, EGGIM has been widely used as a reliable method for IM severity grading and cancer risk assessment. By utilizing image-enhanced endoscopy (IEE) to observe the severity of IM in multiple regions of the stomach, EGGIM can achieve gastric cancer risk prediction accuracy approaching that of OLGIM. EGGIM eliminates the need for biopsies, allowing endoscopists to make real-time diagnostic assessments and minimizing harm to patients.

[0005] EGGIM-based IM severity assessment relies heavily on the expertise and skill of endoscopists. However, organizing highly trained endoscopists to rapidly conduct large-scale IM severity screenings is a challenging task, and human-led diagnoses can be influenced by various subjective factors, such as diagnostic experience and eye strain, leading to inconsistencies in diagnostic opinions among different endoscopists. Utilizing computer-aided diagnosis (CAD) systems to assist endoscopists in assessing IM severity can help reduce the influence of these factors and alleviate the burden on endoscopists. To the inventors' knowledge, there is currently no CAD system for classifying IM severity.

[0006] In recent years, deep learning-based methods have been widely applied in medical image analysis and have achieved good results. However, methods based on convolutional neural networks (CNNs) and Transformers are limited by fixed sampling regions, which simultaneously extract features from lesion and non-lesion regions, resulting in the inability to obtain salient features and thus affecting the model's final lesion recognition performance.

[0007] Although deep learning-based methods have achieved good results in the field of medical image analysis, there is currently no solution to assist doctors in detecting the severity of intestinal metaplasia and predicting the risk of gastric cancer under endoscopy.

[0008] The following technical problems exist in the existing technology: At present, Transformer-based models have the ability to establish self-attention mechanisms and develop long-distance image dependencies, which improves the efficiency of image processing, video processing and sound recognition tasks. However, the recognition accuracy of Transformer-based deep learning models is relatively low. The main reason is that, limited by the constraints of its own fixed rectangular sampling area, it is difficult to obtain complete semantic information of the lesion area, and thus it is impossible to obtain significant features and accurate classification results. Summary of the Invention

[0009] To overcome the shortcomings of the prior art, this invention provides a grading system for the severity of intestinal metaplasia based on the deformation Transformer. By using a fusion module to weight and fuse the feature units corresponding to the segmented endoscopic images according to their similarity, the system achieves deformation of the sampling region in the Transformer, overcomes the shortcomings of fixed-region sampling in the Transformer, improves feature extraction capability, and increases the accuracy of grading the severity of intestinal metaplasia.

[0010] To achieve the above objectives, a first aspect of the present invention provides a grading system for the severity of intestinal metaplasia based on a deformation Transformer, comprising:

[0011] The preprocessing module is used to acquire endoscopic images and perform preprocessing.

[0012] The grading module is used to input the preprocessed endoscopic images into a preset grading network model for recognition and grading; wherein, the preset grading network model is a learning model based on the Transformer structure, and the grading network model includes an input layer, a multi-head self-attention module, a matching module, a fusion module, a feature extraction module, and an output layer;

[0013] The input layer is used to divide the preprocessed endoscopic image into multiple feature units;

[0014] The multi-head self-attention module is used to learn and generate the relationships between various feature units and update the Token feature;

[0015] The multi-head self-attention module is used to learn and generate the relationships between various feature units and update the features of the feature units;

[0016] The matching module is used to calculate the similarity between each feature unit to obtain a matched feature unit group;

[0017] The fusion module is used to calculate the weight of each feature unit in the matched feature unit group and perform weighted fusion to deform the sampling area and obtain fused features;

[0018] The feature extraction module is used to extract deep semantic features from the fused features;

[0019] The output layer is used to output the severity grading results of intestinal metaplasia based on the deep semantic features.

[0020] The above one or more technical solutions have the following beneficial effects:

[0021] In this invention, endoscopic images are divided into multiple feature units. Matching feature unit groups are obtained based on the similarity of these feature units. These feature unit groups are then weighted and fused, achieving deformation of the sampling region in the Transformer. This overcomes the shortcomings of fixed-region sampling in the Transformer, improves feature extraction capabilities, and increases the accuracy of grading the severity of intestinal metaplasia. In clinical applications, this reduces the burden on physicians in identifying the severity of intestinal metaplasia lesions under endoscopy.

[0022] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0024] Figure 1 This is a network structure diagram of the hierarchical network model in Embodiment 1 of the present invention. Detailed Implementation

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0027] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0028] Example 1

[0029] like Figure 1 As shown, this embodiment discloses a grading system for the severity of intestinal metaplasia based on the Transformer, including:

[0030] The preprocessing module is used to acquire endoscopic images and perform preprocessing.

[0031] The grading module is used to input the preprocessed endoscopic images into a preset grading network model for recognition and grading. The preset grading network model is a learning model based on the Transformer structure, and the grading network model includes an input layer, a multi-head self-attention module, a matching module, a fusion module, a feature extraction module, and an output layer.

[0032] The input layer is used to divide the preprocessed endoscopic image into multiple feature units;

[0033] The multi-head self-attention module is used to learn and generate the relationships between tokens and update token features;

[0034] The matching module is used to calculate the similarity between each feature unit to obtain a matching feature unit group;

[0035] The fusion module is used to calculate the weights of each feature unit in the matched feature unit group and perform weighted fusion to shape the sampling region and obtain the fused features;

[0036] The feature extraction module is used to extract deep semantic features from the fused features;

[0037] The output layer is used to output the severity grading results of intestinal metaplasia based on the deep semantic features.

[0038] In the preprocessing module of this embodiment, the initial endoscopic images will have annotations such as shooting time, shooting mode, and thumbnail, and will have two resolutions: 1980×1080 and 1280×720. After cropping and removing the annotations, the image resolutions will be divided into more than ten sizes. During model training, all images will be uniformly scaled to a resolution of 224×224, and data augmentation will be performed using random rotation and random flipping, and the order will be shuffled by generating random indices.

[0039] In the hierarchical module of this embodiment, such as Figure 1 As shown, the hierarchical network model consists of one input layer, 12 Transformer blocks, and one output layer connected in sequence. The multi-head self-attention module, token matching module, token fusion module, and feature extraction module are sequentially connected to form a Transformer block, resulting in a total of 12 Transformer blocks.

[0040] The input layer receives the preprocessed color image of the endoscope and breaks it down into feature units (Tokens) of fixed size and shape.

[0041] The input layer's operations on the preprocessed endoscope include:

[0042] The endoscopic image was divided into 196 rectangular image patches of size 16×16 pixels, and the feature code corresponding to each patch was calculated using a linear mapping method.

[0043] Add a learnable vector as a positional encoding to each patch, and initialize an additional token as a classification header (Class_Token) on top of the original token to complete the initial token construction.

[0044] Specifically: the input is The three-dimensional matrix, where W1 and H1 are the width and height of the medical image, and 3 is the number of channels.

[0045] The l1 is divided into 196 patches of size 16×16 pixels, which are then sequentially passed through a convolutional kernel of size 3*3, stride of 16, input channels of 3, and output channels of 768. The convolution formula is as follows: p0 represents the current convolution point position, p n Let w be the surrounding points listed in R, w be the network weight, and x(·) be the feature of the corresponding point.

[0046] The role of the multi-head self-attention module is to learn and generate the relationships between various tokens and update token features, including:

[0047] Multiple sets of Q, K, V vectors are generated for each token using a linear mapping method, and the interaction between the sets of Q, K, V vectors is calculated using dot product.

[0048] Calculate the weights between multiple sets of relationships, and concatenate them according to the weights to obtain the relationships between each token, and update the token features.

[0049] Specifically, the input is The token features are N2, which is the number of token features input to the multi-head self-attention module, and D2 is the dimension of the token features input to the multi-head self-attention module. The token features input from the input layer pass through a LayerNorm layer of size 768 and a multi-head self-attention mechanism module in sequence.

[0050] One of the multi-head self-attention mechanism modules includes a fully connected layer with an input size of 768 and an output size of 2304, and another fully connected layer with an input size of 768 and an output size of 768.

[0051] The formulas for the multi-head self-attention mechanism include:

[0052] Q = xW q ,

[0053] K = xW k ,

[0054] V = xW v ,

[0055]

[0056] MultiHead(Q,K,V)=Concat(Attention1,…,Attention n W O ,

[0057] Where Q, K, and V represent the query key, index key, and content key, respectively, x represents the token feature, and W... q W k W v These represent the weights corresponding to Q, K, and V, respectively, and d kLet K represent the dimension, s represent the number of fused tokens contained in each deformed token, Attention(Q,K,V) represent the self-attention mechanism, MultiHead(Q,K,V) represent the multi-head self-attention mechanism, Concat(·) represents matrix concatenation, and W represents the multi-head self-attention mechanism. O This represents the weight corresponding to each self-attention.

[0058] The formula for the GELU activation function is:

[0059] GELU(x) = x·Φ(x),

[0060] in, erf(·) is the Gaussian error function, and x is the token feature; the formula for the softmax function is:

[0061]

[0062] Where x i For the currently selected token characteristics, x j For each token's characteristics.

[0063] The main function of the token matching module is to find the most similar r / 2 groups of tokens for matching based on the token features provided by the multi-head self-attention module, where r is the set number of matches and fusions, including:

[0064] Calculate the cosine similarity between the Class Token and the other Tokens, set the top r / 2 Tokens with the highest similarity as the foreground group, and set the remaining r / 2 Tokens as the background group;

[0065] The tokens in both the foreground and background groups are further divided into two parts, A and B, in an alternating manner. Each token in part A finds the token in part B with the highest cosine similarity and establishes a connection.

[0066] Sort all connections by cosine similarity from high to low. In the foreground group (connections established between parts A and B), the first r / 4 edges remain connected. In the background group (connections established between parts A and B), the first 3r / 4 edges remain connected, and the remaining connections are disconnected.

[0067] The token that remains connected is considered the matching token.

[0068] Specifically, the input is The token features are defined as follows: N3 is the number of token features input to the token matching module, and D3 is the dimension of the token features input to the token matching module. The token features input from the multi-head self-attention module sequentially undergo a foreground and background token partitioning process based on cosine similarity; a random token binary grouping process; and a token matching process based on cosine similarity. The formula for cosine similarity is... Similarity(·) represents the cosine similarity, x s1 With x s2 The token used for similarity calculation.

[0069] The main function of the Token fusion module in this embodiment is to calculate the weights corresponding to each matching Token and perform weighted fusion, thereby deforming the Transformer in the fixed sampling area. The input is... N4 represents the number of matched token features, and D4 represents the dimension of the matched token features. The matched tokens sequentially pass through an importance score prediction module consisting of a fully connected network with 2048 input channels and 32 output channels; and a token fusion process based on weight information. The formula for importance score prediction is p. score =ISM(x matc h), where p score This represents weight information; ISM(·) represents the importance score prediction module; x match Represents the matched token; the formula for the weighted fusion process is: Where x merge p represents the characteristics of the merged token. score_ i represents the weight information corresponding to the token matched by p, x match_i With x match_i+1 This represents a set of matching tokens.

[0070] In this embodiment, the feature extraction module extracts deep semantic feature information of the endoscopic image based on the irregular sampling region of the deformation Transformer, including:

[0071] Based on the token weight information and multilayer perceptron, deep features are accurately extracted, and each token and the classification head Class_Token are continuously adjusted during the feature extraction process.

[0072] Specifically, the input to the feature extraction module is... N5 represents the number of fused token features, and D5 represents the dimension of the fused token features. The fused token features are sequentially passed through a LayerNorm layer of size 768 and a multilayer perceptron module.

[0073] A multilayer perceptron module consists of a fully connected layer with an input size of 768 and an output size of 3072, a GELU activation function, and a fully connected layer with an input size of 3072 and an output size of 768.

[0074] The output layer in this embodiment outputs the final grading result of the severity of intestinal metaplasia based on the feature information in Class_Token, including:

[0075] A fully connected network is used to map the feature space calculated by Class_Token to the sample label space, thereby reducing the impact of feature location on classification results.

[0076] The SoftMax function is used to map the output of the fully connected network to the range [0,1] to obtain the probability of occurrence of each severity level in the image, thus completing the final classification.

[0077] Specifically, in this embodiment, the input to the output layer is the selected class header (Class_Token) feature dimension from the output of the 12th feature extraction module. D6 represents the dimension of Class_Token. The information is sequentially passed through a fully connected classifier with 768 input channels and 3 output channels; and a SoftMax function; the formula for the SoftMax function is... Where y i For the output of the currently selected fully connected layer classifier, y j These are the outputs of the fully connected layer classifier.

[0078] To accelerate network convergence, the weights and biases of each layer in the network are randomized to a normal distribution with a mean of 0 and a variance of 1 / N. input , where N input This represents the number of upper-layer features. The loss function is defined using the cross-entropy function. Simultaneously, the Adam optimizer and gradient descent algorithm are used in PyTorch to adjust the connection weights and biases. The maximum number of iterations is set to 50, and the loss threshold is set to 0.0001.

[0079] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A grading system for the severity of intestinal metaplasia based on deformation Transformer, characterized in that, include: The preprocessing module is used to acquire endoscopic images and perform preprocessing. The grading module is used to input the preprocessed endoscopic images into a preset grading network model for recognition and grading; wherein, the preset grading network model is a learning model based on the Transformer structure, and the grading network model includes an input layer, a multi-head self-attention module, a matching module, a fusion module, a feature extraction module, and an output layer; The input layer is used to divide the preprocessed endoscopic image into multiple feature units; The multi-head self-attention module is used to learn and generate the relationships between various feature units and update the features of the feature units; The matching module is used to calculate the similarity between each feature unit to obtain the matched feature units; The fusion module is used to calculate the weights of the matched feature units and perform weighted fusion to deform the sampling region and obtain fused features; The feature extraction module is used to extract deep semantic features from the fused features; The output layer is used to output the severity grading results of intestinal metaplasia based on the deep semantic features; In the hierarchical module, an additional feature unit is initialized based on the existing feature units as... ; The matching module is used to calculate the similarity between each feature unit to obtain a matching feature unit group, specifically: Calculate the cosine similarity between Class_Token and the feature unit, and divide the feature unit into foreground group and background group based on the calculated cosine similarity; The feature units of both the foreground group and the background group are divided into parts A and B in an alternating manner. The feature units in part A of the foreground group and the feature units with the highest cosine similarity in part B are connected. The feature units in part B of the background group and the feature units with the highest cosine similarity in part A are connected. Sort all connections in descending order of cosine similarity. Based on the sorting results, select the lines in the foreground group and the background group that remain connected, and disconnect the remaining connections. The feature units that maintain the connection are the matching feature unit groups.

2. The intestinal metaplasia severity grading system based on deformation Transformer as described in claim 1, characterized in that, In the hierarchical module, the input layer is used to divide the preprocessed endoscopic image into multiple feature units, specifically: The preprocessed endoscopic image is divided into multiple rectangular image blocks; Multiple rectangular image blocks are linearly mapped to obtain corresponding initial feature units; Encoding information is added to each initial feature unit to obtain the corresponding feature unit.

3. The intestinal metaplasia severity grading system based on deformation Transformer as described in claim 1, characterized in that, The multi-head self-attention module is used to learn and generate the relationships between various feature units and update the features of the feature units, specifically as follows: Multiple sets of vectors are generated for each feature unit using linear mapping, and the relationships between the sets of vectors are obtained using dot product. Calculate the weights between multiple sets of vectors and concatenate them according to the weights to update the features of the feature units.

4. The intestinal metaplasia severity grading system based on deformation Transformer as described in claim 1, characterized in that, In the hierarchical module, the fusion module is used to calculate the weights of the matched feature units and perform weighted fusion to obtain fused features, specifically: Calculate the weights corresponding to the matched feature units; Based on the weights of the matched feature units, feature fusion is performed on the corresponding feature units to obtain fused features.

5. A grading system for the severity of intestinal metaplasia based on deformation Transformer as described in claim 4, characterized in that, The weights of the matching feature units are obtained based on the fully connected network.

6. A grading system for the severity of intestinal metaplasia based on deformation Transformer as described in claim 1, characterized in that, In the hierarchical module, the feature extraction module is used to extract deep semantic features from the fused features, specifically: The multi-head self-attention mechanism is used to learn the relationships between various fused features and the weight information of feature units; Adjust each feature unit based on its weight information and the deep features extracted by the multilayer perceptron. .

7. A grading system for the severity of intestinal metaplasia based on deformation Transformer as described in claim 1, characterized in that, In the grading module, the output layer is used to output the intestinal metaplasia grading result based on the deep semantic features, specifically as follows: The deep semantic features are classified through a fully connected network to obtain the classification results; The classification results are mapped using an activation function to obtain the probability of occurrence of each severity level in the image, thus completing the final grading of intestinal metaplasia.

8. A grading system for the severity of intestinal metaplasia based on deformation Transformer as described in claim 1, characterized in that, In the preprocessing module, the endoscopic image is subjected to rotation and vertical flip sample enhancement transformation to obtain the preprocessed endoscopic image.

Images