Gradual correction medical image recognition system based on deformable attention mechanism

Abstract

The invention provides a gradual correction medical image recognition system based on a deformable attention mechanism. The gradual correction medical image recognition system comprises a feature extraction module, an encoder module, a decoder initialization module and a decoder module. According to the method, the deformable attention mechanism is combined, the significant features in the medical image can be adaptively extracted, the medical image detection and segmentation accuracy is effectively improved, compared with a global self-attention mechanism, the deformable attention mechanism does not need to fuse each feature in a sequence, the calculation cost is greatly reduced, and secondly, compared with the global self-attention mechanism, the calculation efficiency is greatly improved. The deformable attention mechanism can adaptively select features for fusion, and due to the fact that the shape of a target to be detected and segmented in medicine is usually irregular, the structure is very suitable for a medical image, and the features of the medical image can be well extracted; according to the method, a step-by-step iteration optimization strategy is used, namely, the prediction result of the front layer is corrected step by step, the positive position or the positive position contour can be accurately recognized, and smooth proceeding of subsequent work is guaranteed.

Description

Technical Field

[0001] The present invention relates to the general field of image data processing or generation technology, and in particular to a gradually corrected medical image recognition system based on a deformable attention mechanism. Background Art

[0002] Image detection and segmentation is a basic but crucial task in the medical field. The results of medical image detection and segmentation directly affect subsequent work. Due to the difficulty and high precision requirements for positive area or positive area contour detection and segmentation in medical images, the accuracy of existing medical image segmentation methods needs to be improved and the accuracy cannot meet the requirements.

[0003] Therefore, a method is needed to effectively improve the accuracy and precision of medical image segmentation. Summary of the invention

[0004] The present invention aims to solve the accuracy problem of medical image segmentation and provides a gradually corrected medical image recognition system based on a deformable attention mechanism. Combined with the deformable attention mechanism, it can adaptively extract significant features in medical images and effectively improve the accuracy of medical image detection and segmentation. In addition, since it is difficult and requires high accuracy to detect and segment positive areas or positive area contours in medical images, the present invention adopts a gradually corrected strategy, which can accurately identify positive positions or positive position contours and ensure the smooth progress of subsequent work.

[0005] The present invention provides a gradually corrected medical image recognition system based on a deformable attention mechanism, comprising a feature extraction module, an encoder module, a decoder initialization module and a decoder module connected in sequence; The feature extraction module extracts the multi-scale features of the medical image and outputs the res2 feature, res3 feature, res4 feature and res5 feature to the encoder module respectively, and the receptive fields of the res2 feature, res3 feature, res4 feature and res5 feature increase successively; the encoder module fuses the context features of the res2 feature, res3 feature, res4 feature and res5 feature and obtains the updated res3 feature, the updated res4 feature, the updated res5 feature and the mask feature, and outputs them to the decoder initialization module; the decoder initialization module initializes the content query and the position query according to the updated res3 feature, the updated res4 feature, the updated res5 feature and the mask feature, and outputs the initialized content query and the initialized position query to the decoder module; the decoder module detects, segments and classifies the initialized content query and the initialized position query, and uses a step-by-step iterative optimization strategy to gradually correct the prediction results of the previous layer and output the position and category of the positive area in the medical image; The encoder module includes a splicing module, an encoder, a reshape module and a mask feature module. The splicing module performs convolution and flatten operations on res3 features, res4 features and res5 features, and then serializes and splices them as content query inputs to the encoder; the encoder is composed of N encoder blocks, each of which is composed of a multi-scale deformable attention layer, a residual connection and a fully connected layer, a normalization layer and a feedforward neural network layer in sequence. The multi-scale deformable attention layer adaptively selects features for fusion, and the output of the previous encoder block is used as the input of the next encoder block; the reshape module performs a reshape operation on the serialized features output by the encoder to obtain updated features of res3, res4 and res5; the mask feature module upsamples the res3 feature and adds it to the res2 feature to obtain the mask feature; The decoder module consists of M decoder blocks, each of which consists of a self-attention layer, a deformable attention layer, a residual connection and a fully connected layer, a normalization layer, and a feedforward neural network layer. Each decoder block outputs the detection and classification prediction results of the medical image. Only the Mth decoder block performs mask prediction and outputs the final predicted mask. The decoder module uses a step-by-step iterative optimization strategy, where the position prediction result produced by the previous decoder block is used to update the position encoding of the next decoder block.

[0006] The present invention discloses a gradually corrected medical image recognition system based on a deformable attention mechanism. As a preferred embodiment, the feature extraction module extracts multi-scale features of medical images through a Backbone network.

[0007] The present invention describes a gradually corrected medical image recognition system based on a deformable attention mechanism. As a preferred embodiment, the Backbone network is a Resnet series algorithm based on a convolutional neural network or a ViT series algorithm based on a Transformer, or a combination of an algorithm based on a convolutional neural network and an algorithm based on a Transformer.

[0008] The present invention discloses a gradually corrected medical image recognition system based on a deformable attention mechanism. As a preferred embodiment, the spatial dimensions of the res2 feature, the res3 feature, the res4 feature and the res5 feature are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original size of the medical image, respectively. The feature dimension of res3 after flatten operation is: (H / 8×W / 8, C), the feature dimension of res4 after flatten operation is: (H / 16×W / 16, C), and the feature dimension of res5 after flatten operation is: (H / 32×W / 32, C), where H is the height of the original medical image size, W is the width of the original medical image size, and C is the number of channels of the medical image feature; The shapes of the updated features of res3, res4 and res5 are H / 8×W / 8×C, H / 16×W / 16×C and H / 32×W / 32×C respectively; the shape of the mask feature is H / 4×W / 4×C; The present invention describes a gradually corrected medical image recognition system based on a deformable attention mechanism. As a preferred embodiment, the input of the encoder also includes a position code, and the position code is a pre-set or learnable parameter.

[0009] The present invention discloses a gradually corrected medical image recognition system based on a deformable attention mechanism. As a preferred embodiment, the input of the multi-scale deformable attention layer is: query feature , normalized reference point And the multi-scale feature map of the input x。

[0010] The present invention discloses a gradually corrected medical image recognition system based on a deformable attention mechanism. As a preferred embodiment, the calculation method of the multi-scale deformable attention layer is: ; in, is the layer index, is the total number of layers, is the index of the attention head, is the number of attention heads, is the index of the sampling point, is the total number of sampling points, and In the Feature layer The first The position offset and attention score of the sampling points are obtained by Linear transformation yields, Used to restore normalized coordinates to On the feature map layer, and is a set of learnable weights.

[0011] In the gradually corrected medical image recognition system based on a deformable attention mechanism described in the present invention, as a preferred embodiment, the method of initializing the decoder initialization module is as follows: initializing the content query and initializing the position query; The method for initializing the content query is: predict based on the updated features of res3, res4, and res5, select the top k features according to the classification results, and record them as encoded features. The encoded feature is the initialization content query. The method to initialize the position query is: multiply the encoded feature by the mask feature to obtain the predicted mask, convert the predicted mask to a box, and then encode the box to obtain the initialized position query.

[0012] The present invention describes a gradually corrected medical image recognition system based on a deformable attention mechanism. As a preferred method, the step-by-step iterative optimization strategy is: during the iteration process of M decoder blocks, the position prediction result generated by the i-th decoder block and the previous position code are added to obtain the corrected position code to locate the medical target object, 0<i<M, i is a positive integer.

[0013] The present invention describes a gradually corrected medical image recognition system based on a deformable attention mechanism. As a preferred method, the Mth decoder block multiplies the decoder feature with the mask feature output by the encoder, and then upsamples the predicted mask and restores it to the original image size to obtain the final predicted mask.

[0014] The present invention has the following advantages: (1) The present invention combines the deformable attention mechanism to adaptively extract significant features in medical images and effectively improve the accuracy of medical image detection and segmentation. Compared with the global self-attention mechanism, the deformable attention mechanism does not need to fuse each feature in the sequence, and the computational cost is greatly reduced. Secondly, compared with the global self-attention mechanism, the deformable attention mechanism can adaptively select features for fusion. Since the shape of the target to be detected and segmented in medicine is usually irregular, this structure is very suitable for medical images and can well extract medical image features. (2) The actual application scenarios have high requirements for medical image detection and segmentation, and require fine depiction of the target edge contour. Taking into account the characteristics of medical images, such as the different shapes and sizes of target objects, irregular edges, and blurred boundaries, the present invention uses a step-by-step iterative optimization strategy, that is, gradually correcting the prediction results of the previous layer, which can accurately identify the positive position or the positive position contour, and ensure the smooth progress of subsequent work. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1Schematic diagram of the structure of a gradually revised medical image recognition system based on a deformable attention mechanism. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Example 1

[0017] A gradually corrected medical image recognition system based on deformable attention mechanism includes four modules, namely feature extraction module, encoder module, decoder initialization module and decoder module. Figure 1 , as follows: 1. Feature extraction module The feature extraction module is used to extract multi-scale features of medical images. It can extract features of large targets as well as features of small targets. Specifically, the width and height of a medical image are W and H respectively. After passing through the Backbone network, the Backbone network can select the Resnet series algorithm based on convolutional neural network, the ViT series algorithm based on Transformer, or combine the algorithm based on convolutional neural network with the algorithm based on Transformer. After passing through the Backbone network, 4 layers of features are obtained, recorded as res2 features, res3 features, res4 features, and res5 features. The receptive fields of these layers of features increase successively, and their spatial dimensions are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size respectively.

[0018] 2. Encoder module The encoder module is used to perform context feature fusion on the features extracted by the feature extraction module. First, the features of res3, res4, and res5 are serialized through convolutional layers and flatten operations. The feature dimensions after flattening are (H / 8×W / 8, C), (H / 16×W / 16, C), and (H / 32×W / 32, C). The serialized features of res3, res4, and res5 are concatenated and input into the encoder as content query. In addition, the input also includes position encoding, which can be pre-set or a learnable parameter.

[0019] The encoder consists of N blocks, each of which consists of a deformable attention layer, a residual connection and a fully connected layer, a LayerNorm (normalization) layer, and an FFN (feedforward neural network) layer. The deformable attention mechanism is an efficient choice. First, compared with the global self-attention mechanism, the deformable attention mechanism does not need to fuse each feature in the sequence, and the computational cost is greatly reduced. Secondly, compared with the global self-attention mechanism, the deformable attention mechanism can adaptively select features for fusion. Since the shape of the target to be detected and segmented in medicine is usually irregular, this structure is very suitable for medical images and can well extract medical image features. The calculation method of the deformable attention module is shown in the following formula (1): (1) Input includes , , in is the query feature, is the reference point, is the input feature map, Represents the index of the attention head, Indicates the number of attention heads, Represents the index of the sampling point, represents the total number of sampling points, and Indicates The first The position offset and attentionweight of the sampling points, and All through The linear transformation is obtained. and is a set of learnable weights.

[0020] For the segmentation and recognition of medical images, it is necessary to extract features from both the overall image, i.e., the larger receptive field, and the details, i.e., the smaller receptive field. The above-mentioned deformable attention mechanism can be naturally extended to multiple scales, as shown in the following formula (2): (2) Input is , , ,in is the query feature, is the multi-scale feature map of the input, Indicates the layer index, Indicates the total number of layers, represents the normalized reference point, Represents the index of the attention head, represents the number of attention heads, Represents the index of the sampling point, represents the total number of sampling points, and Indicated in Feature layer The first The position offset and attention weight of the sampling points, and All through The linear transformation is obtained. Used to restore normalized coordinates to On the feature map, and is a set of learnable weights.

[0021] After N blocks, the output of the previous block is used as the input of the next block, and the serialized features output by the encoder are reshaped to obtain the updated features of res3, res4, and res5. Their shapes are (H / 8, W / 8, C), (H / 16, W / 16, C), and (H / 32, W / 32, C), respectively. In addition, the encoder module also needs to construct a feature for auxiliary segmentation, denoted as mask feature, which is obtained by upsampling feature res3 and adding it to feature res2. Its dimension is (H / 4, W / 4, C).

[0022] 3. Decoder initialization module The initialization of the decoder has an important impact on the training stability and algorithm performance. The input of the decoder consists of two parts, one is the content query and the other is the location query. There are several ways to initialize them: Both content query and location query are initialized randomly; The position query comes from the encoder, and the content query is randomly initialized; Both the content query and the location query come from the encoder.

[0023] Methods 1) and 2) will lead to slower convergence and poorer performance in the training process. Therefore, the present invention adopts method 3) and uses method 3) for initialization. Both the content query and the position query come from the encoder. The features of the content query initialization come from the features of the encoder. There are many encoder-based position query initialization strategies. The present invention takes the following three position query initialization strategies as examples, specifically: Convert the encoder predicted mask into a box as the initial input of the decoder position query; Use the box predicted by the encoder as the initial input of the decoder position query; Combine 1) and 2).

[0024] The method for initializing the content query and position query of the decoder is as follows: first initialize the content query, make predictions based on the output features of the encoder, select the topk features according to the classification results, record them as encoded features, and use the encoded features as the content query initialized by the decoder. Then initialize the position query, multiply the encoded feature by the mask feature output by the encoder to obtain the predicted mask, and then use the first) position query initialization strategy to convert the predicted mask into a box, and then encode the box as the position query initialized by the decoder.

[0025] 4. Decoder module The decoder module is initialized using the above method, and then performs detection, segmentation, and classification tasks. The decoder consists of M blocks, each of which consists of a self-attention mechanism, a deformable attention mechanism, a Linear layer, a LayerNorm layer, and an FFN layer. The classification and detection results are predicted according to the output features of each block, and the position query of the previous step is corrected with the predicted detection results. The optimization is iterated step by step. The last layer not only predicts the classification and detection results, but also predicts the mask. The method of predicting the mask is consistent with the method of predicting the mask using the encoder. The predicted mask is then upsampled and restored to the original image size to obtain the final prediction of the mask. The final prediction results of classification and detection also come from the last layer of the decoder.

[0026] 5. Application The present invention takes microorganisms as an example, and inputs the microorganism microscope pictures into the medical image recognition system proposed by the present invention. Figure 1The microorganisms in the images under the microscope are in the form of thin strips. The deformable attention mechanism in the system can extract its features well through flexible position selection. At the same time, the gradual correction strategy mentioned in the present invention can identify microorganisms more accurately. In the early stage of iteration, there may be some problems with the detection box for identifying microorganisms, such as the detection box is too small or too large. Figure 1 As shown in the black detection box in the middle, after gradual iteration, the detection box, mask and category of the microorganism can be accurately located. The detection box and mask are shown in Figure 1 Final prediction results.

[0027] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A progressively corrected medical image recognition system based on a deformable attention mechanism, characterized by: It includes a feature extraction module, an encoder module, a decoder initialization module and a decoder module connected in sequence; The feature extraction module extracts multi-scale features of the medical image and outputs res2 features, res3 features, res4 features, and res5 features to the encoder module respectively, and the receptive fields of the res2 features, the res3 features, the res4 features, and the res5 features are increased in sequence; the encoder module performs context feature fusion on the res2 features, the res3 features, the res4 features, and the res5 features and obtains updated res3 features, updated res4 features, updated res5 features, and mask features and outputs them to the decoder initialization module; the decoder initialization module initializes the content query and the position query according to the updated res3 features, the updated res4 features, the updated res5 features, and the mask features, and outputs the initialized content query and the initialized position query to the decoder module; the decoder module detects, segments, and classifies the initialized content query and the initialized position query, and uses a step-by-step iterative optimization strategy to gradually correct the prediction results of the previous layer and output the position and category of the positive area in the medical image; The encoder module includes a splicing module, an encoder, a reshape module and a mask feature module, and the encoder is composed of N encoder blocks, each of which is composed of a multi-scale deformable attention layer, a residual connection and a fully connected layer, a normalization layer and a feedforward neural network layer in sequence; The decoder module consists of M decoder blocks, each of which is composed of a self-attention layer, a deformable attention layer, a residual connection and a fully connected layer, a normalization layer and a feedforward neural network layer in sequence. The Mth decoder block performs mask prediction and outputs the final predicted mask.

2. The gradually corrected medical image recognition system based on deformable attention mechanism according to claim 1, characterized in that: The concatenation module performs convolution and flatten operations on the res3 feature, the res4 feature, and the res5 feature, and then serializes and concatenates the features and inputs the features into the encoder as content query; The multi-scale deformable attention layer adaptively selects features for fusion, and the output of the previous encoder block is used as the input of the next encoder block; the reshape module reshapes the serialized features output by the encoder to obtain the res3 updated features, the res4 updated features and the res5 updated features; the mask feature module upsamples the res3 features and adds them to the res2 features to obtain the mask features; Each of the decoder blocks outputs detection and classification prediction results of the medical image; the decoder module uses the step-by-step iterative optimization strategy, and the position prediction result generated by the previous decoder block is used to update the position coding of the next decoder block.

3. The gradually corrected medical image recognition system based on deformable attention mechanism according to claim 1, characterized in that: The feature extraction module extracts multi-scale features of the medical image through a Backbone network; The Backbone network is a Resnet series algorithm based on a convolutional neural network or a ViT series algorithm based on a Transformer, or a combination of an algorithm based on a convolutional neural network and an algorithm based on a Transformer.

4. The gradually corrected medical image recognition system based on deformable attention mechanism according to claim 1, characterized in that: The spatial dimensions of the res2 feature, the res3 feature, the res4 feature and the res5 feature are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original size of the medical image respectively; The feature dimension of the res3 feature after the flatten operation is: (H / 8×W / 8, C), the feature dimension of the res4 feature after the flatten operation is: (H / 16×W / 16, C), and the feature dimension of the res5 feature after the flatten operation is: (H / 32×W / 32, C), where H is the height of the original medical image size, W is the width of the original medical image size, and C is the number of channels of the medical image feature; The shapes of the updated res3 feature, the updated res4 feature and the updated res5 feature are respectively: H / 8×W / 8×C, H / 16×W / 16×C and H / 32×W / 32×C; the shape of the mask feature is: H / 4×W / 4×C.

5. The gradually corrected medical image recognition system based on deformable attention mechanism according to claim 1, characterized in that: The input of the encoder also includes a position code, which is a pre-set or learnable parameter.

6. The gradually corrected medical image recognition system based on deformable attention mechanism according to claim 1, characterized in that: The input of the multi-scale deformable attention layer is: query feature , normalized reference point And the multi-scale feature map of the input x。 7. The gradually corrected medical image recognition system based on deformable attention mechanism according to claim 6, characterized in that: The calculation method of the multi-scale deformable attention layer is: ； in, is the layer index, is the total number of layers, is the index of the attention head, is the number of attention heads, is the index of the sampling point, is the total number of sampling points, and In the Feature layer The first The position offset and attention score of the sampling points are obtained by Linear transformation yields, Used to restore normalized coordinates to On the feature map layer, and is a set of learnable weights.

8. The gradually corrected medical image recognition system based on deformable attention mechanism according to claim 1, characterized in that: The method of initializing the decoder initialization module is as follows: initializing content query and initializing position query; The method for initializing the content query is: predicting based on the updated features of res3, res4 and res5, selecting the top k features according to the classification results, recorded as encoded features, and encodedfeature is the initialization content query; The method for initializing the position query is: multiplying the encoded feature by the mask feature to obtain a predicted mask, converting the predicted mask into a box, and then encoding the box to obtain the initialized position query.

9. The gradually corrected medical image recognition system based on deformable attention mechanism according to claim 1, characterized in that: The step-by-step iterative optimization strategy is: in the iterative process of M decoder blocks, the position prediction result generated by the i-th decoder block is added to the previous position code to obtain the corrected position code to locate the medical target, 0<i<M, i is a positive integer.

10. The gradually corrected medical image recognition system based on deformable attention mechanism according to claim 1, characterized in that: The Mth decoder block multiplies the decoder features with the mask features output by the encoder, and then upsamples the predicted mask and restores it to the original image size to obtain the final predicted mask.

Images