Medical image recognition method and device based on step feature fusion network

Abstract

The application relates to a medical image recognition method and device based on a ladder type feature fusion network, wherein the method designs a medical image recognition network based on ladder type feature fusion, the network is designed in a ladder type, and from left to right, the network comprises a stem layer, four level networks and a classification layer; the number of subnetworks included in each level network decreases in turn, each subnetwork of the first, second and fourth level networks comprises a plurality of continuously stacked MBC-GRN modules, and each subnetwork of the third level network comprises a plurality of stacked Inverted-Transformer modules; the network is used for recognizing a to-be-tested medical image, and greatly helps in recognizing a gastrointestinal endoscope medical image. The network uses a large receptive field and an attention mechanism to extract features, and meanwhile, a specific ladder type network architecture realizes cross-subnetwork feature fusion; the method reduces network parameters and calculation amount, and improves recognition precision.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of image recognition, in particular to a medical image recognition method and device based on a ladder type feature fusion network. BACKGROUND

[0002] Using computers to detect diseases is an important research field. Algorithm detection of gastrointestinal abnormalities, especially polyp pathological detection, is particularly popular. Detecting, identifying and evaluating pathological results helps optimize scarce medical resources. In order to make the healthcare system more efficient and scalable, research in the field of computer science and medicine must break through traditional medical identification and combine artificial intelligence and medical image recognition. This research is in this background. Deep learning is applied to the medical field to identify images of pathological findings in the medical field. The pathological category is efficiently and accurately identified to help the development of the medical field.

[0003] Since AlexNet, convolutional neural networks have dominated the field of deep learning, giving birth to a series of classic neural networks such as VGG, GoogleNet, ResNet, MobileNet, etc. Various variants of neural networks continue to expand in the direction of deeper, wider and higher resolution. The MBC Block in MObileNetV3 introduces depth separable convolution, which achieves satisfactory results in terms of computation and performance. Not only is it favored in lightweight convolutional neural networks, but many existing hybrid architectures with Vit also use this idea in their design. In recent years, the self-attention mechanism represented by Transformer has once again pushed computer vision to a new height. Its global attention mechanism with Self-Attention performs well and has achieved great success in the field of computer vision, giving rise to a series of classic high-performance networks such as Vit, SwinTransformer and MobileVit. However, in actual research, although Transformer has the advantage of Self-Attention global attention, many studies have shown that due to the lack of the inductive bias characteristic in CNN, it is difficult to achieve the best performance in many visual tasks. Therefore, how to effectively combine CNN and Transformer architecture has always been the focus of research in the field of deep learning. SUMMARY

[0004] Therefore, it is necessary to provide a medical image recognition method and device based on a ladder type feature fusion network in view of the above technical problems.

[0005] A medical image recognition method based on a ladder type feature fusion network, the method comprising:

[0006] Obtain a medical image, and use the labeled medical image as a training sample.

[0007] A medical image recognition network based on ladder feature fusion is constructed, which is designed in a ladder form and includes a stem layer, four level networks, and a classification layer from left to right; the number of subnetworks included in each level network decreases in turn, each subnetwork of the first, second, and fourth level networks includes a plurality of continuously stacked MBC-GRN modules, and each subnetwork of the third level network includes a plurality of stacked Inverted-Transformer modules; the MBC-GRN module is obtained by introducing a global response normalization module into an MBC module, and the Inverted-Transformer module is obtained by introducing an MBC inverted bottleneck idea and a relative position bias into a Transformer.

[0008] The medical image recognition network is trained using the training sample to obtain a trained medical image recognition network.

[0009] A to-be-tested medical image is input into the trained medical image recognition network to obtain an image recognition result.

[0010] A medical image recognition device based on a ladder feature fusion network, the device comprising:

[0011] A training sample determination module is configured to obtain a medical image and use the labeled medical image as a training sample.

[0012] A medical image recognition network construction module is configured to construct a medical image recognition network based on ladder feature fusion, which is designed in a ladder form and includes a stem layer, four level networks, and a classification layer from left to right; the number of subnetworks included in each level network decreases in turn, each subnetwork of the first, second, and fourth level networks includes a plurality of continuously stacked MBC-GRN modules, and each subnetwork of the third level network includes a plurality of stacked Inverted-Transformer modules; the MBC-GRN module is obtained by introducing a global response normalization module into an MBC module, and the Inverted-Transformer module is obtained by introducing an MBC inverted bottleneck idea and a relative position bias into a Transformer.

[0013] A medical image recognition network training module is configured to train the medical image recognition network using the training sample to obtain a trained medical image recognition network.

[0014] The medical image recognition module is configured to input a to-be-tested medical image into the trained medical image recognition network to obtain an image recognition result.

[0015] The medical image recognition method and device based on the ladder feature fusion network are characterized in that the medical image recognition network based on the ladder feature fusion is designed, the medical image recognition network is designed in a ladder form, and from left to right, the medical image recognition network includes a stem layer, four level networks, and a classification layer; the number of subnetworks included in each level network decreases in turn, each subnetwork of the first, second, and fourth level networks includes a plurality of continuously stacked MBC-GRN modules, and each subnetwork of the third level network includes a plurality of stacked Inverted-Transformer modules; the MBC-GRN module is obtained by introducing a global response normalization module into an MBC module, and the Inverted-Transformer module is obtained by introducing an MBC inverted bottleneck idea and a relative position bias into a Transformer; the network is used to recognize a to-be-tested medical image, and an image recognition result is obtained. The method has fewer network parameters and less calculation amount, improves recognition accuracy, and greatly helps in recognizing gastrointestinal endoscopic medical images. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 FIG. 1 is a flowchart of a medical image recognition method based on a ladder feature fusion network according to an embodiment;

[0017] Figure 2 FIG. 2 is a medical image recognition network architecture diagram based on ladder feature fusion according to another embodiment;

[0018] Figure 3 FIG. 3 is a Level2 structure design diagram according to another embodiment;

[0019] Figure 4 FIG. 4 is a MBC-GRN module structure design diagram according to another embodiment;

[0020] Figure 5 FIG. 5 is a Level3 layer structure design diagram according to another embodiment;

[0021] Figure 6 FIG. 6 is an Inverted-Transformer module structure design diagram according to another embodiment;

[0022] Figure 7 FIG. 7 is a Level4 structure design diagram according to another embodiment;

[0023] Figure 8 FIG. 8 is a Level1 structure design diagram according to another embodiment;

[0024] Figure 9 is a structural design diagram of the Classifier layer in another embodiment;

[0025] Figure 10 is a structural design diagram of the Stem layer in another embodiment;

[0026] Figure 11 is a structural block diagram of a medical image recognition device based on a ladder feature fusion network in an embodiment. DETAILED DESCRIPTION

[0027] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0028] In one embodiment, as shown in Figure 1 a medical image recognition method based on a ladder feature fusion network is provided, which comprises the following steps:

[0029] Step 100: Obtain a medical image, and use the labeled medical image as a training sample.

[0030] Specifically, the medical image includes images containing large targets such as polyps and pylorus, and images focusing on texture details such as esophagitis and ulcerative colitis,

[0031] The medical image can be, but is not limited to, images showing anatomical landmarks, pathological findings, and two types of images related to endoscopic polypectomy in the gastrointestinal tract. The anatomical landmarks include three categories of Z lines, pylorus and cecum; the pathological findings include three categories of esophagitis, polyps and ulcerative colitis; the two types of images related to endoscopic polypectomy are stained wrinkled polyps and stained resection edges, a total of eight categories.

[0032] Step 102: Construct a medical image recognition network based on ladder feature fusion; the medical image recognition network is designed in a ladder form, including a stem layer, four level networks and a classification layer from left to right; the number of subnetworks included in each level network decreases in turn, each subnetwork of the first, second and fourth level networks includes a plurality of continuously stacked MBC-GRN modules, and each subnetwork of the third level network includes a plurality of stacked Inverted-Transformer modules; the MBC-GRN module is obtained by introducing a global response normalization module into the MBC module, and the Inverted-Transformer module is obtained by introducing the MBC inverted bottleneck idea and relative position bias into the Transformer.

[0033] Specifically, the network architecture as a whole adopts a ladder design, including Stem layer, first-level network, second-level network, third-level network, fourth-level network, and Classifier layer from left to right, a total of six levels of design, so that the network gradually extracts rich semantic information from shallow to deep. From top to bottom, it can be divided into 4 sub-networks (the number of sub-networks is shown in Table 2), and the depth of each sub-network increases layer by layer. The network information extracted by the upper layer is multiplied by the learnable parameter a and then gradually added to the next layer, which can be used as a supplement to the information extraction of the lower layer. After the rich semantic information extracted is fused and transmitted to the classification layer, classification is performed.

[0034] The first-level network includes a plurality of first-level sub-networks (Level1), the second-level network includes a plurality of second-level sub-networks (Level2), the third-level network includes a plurality of third-level sub-networks (Level3), and the fourth-level network includes a plurality of fourth-level sub-networks (Level4).

[0035] Each Level1, Level2, and Level4 layer includes 2 continuously stacked MBC-GRN blocks, wherein the expansion factor of the DW convolution of the MBC-GRN module is different. Here, the basic configuration is adopted: the expansion factor d of the DW convolution of the first MBC-GRN module is 1, and the expansion factor d of the DW convolution of the second MBC-GRN module is 2. This has the advantage of being able to capture a larger receptive field and avoid the gridding problem. The feature information is added layer by layer from the uppermost sub-network to the lowermost sub-network. The overall network architecture of the medical image recognition network based on ladder feature fusion is shown in Figure 2 .

[0036] Step 104: training the medical image recognition network using the training samples to obtain a trained medical image recognition network.

[0037] Step 106: inputting the to-be-tested medical image into the trained medical image recognition network to obtain an image recognition result.

[0038] In the medical image recognition method of the ladder feature fusion network, the medical image recognition network based on the ladder feature fusion is designed, and the medical image recognition network is designed in a ladder form and includes a stem layer, four level networks and a classification layer from left to right; the number of subnetworks included in each level network decreases in turn, each subnetwork of the first, second and fourth level networks includes a plurality of continuously stacked MBC-GRN modules, and each subnetwork of the third level network includes a plurality of stacked Inverted-Transformer modules; the MBC-GRN module is obtained by introducing a global response normalization module into an MBC module, and the Inverted-Transformer module is obtained by introducing an MBC inverted bottleneck idea and a relative position bias into a Transformer; the network is used to recognize the to-be-tested medical image to obtain an image recognition result. The method has less network parameters and less calculation amount, improves the recognition accuracy, and greatly helps the medical image recognition of a gastrointestinal endoscope.

[0039] In one embodiment, step 104 specifically includes the following steps:

[0040] Step 200: inputting the training sample into the stem layer to obtain a convolution feature.

[0041] Step 202: inputting the convolution feature into the first level network to obtain the output of each first level subnetwork.

[0042] Step 204: multiplying the output of each first level subnetwork by a learnable parameter, then adding the output to the output of the first level subnetwork layer by layer as the input of the corresponding second level subnetwork, and inputting the output into the second level network to obtain the output of each second level subnetwork.

[0043] Step 206: multiplying the output of each second level subnetwork by a learnable parameter, then adding the output to the output of the second level subnetwork layer by layer as the input of the corresponding third level subnetwork, and inputting the output into the third level network to obtain the output of each third level subnetwork.

[0044] Step 208: multiplying the output of the first third level subnetwork by a corresponding learnable parameter, then adding the output to the output of the second third level subnetwork as the input of the corresponding fourth level subnetwork, and inputting the output into the fourth level network to obtain the output of the fourth level network.

[0045] Step 210: inputting the output of the fourth level network into the classification layer to obtain a predicted image recognition result.

[0046] Step 212: performing reverse training on the medical image recognition network according to the prediction image recognition result and the label of the training sample, to obtain a trained medical image recognition network.

[0047] In one of the embodiments, the first-level network includes four first-level sub-networks, and the second-level network includes three second-level sub-networks; step 204 includes: multiplying the output of the first first-level sub-network by a first learnable parameter, and then adding the output of the second first-level sub-network to obtain the input of the first second-level sub-network, and inputting the input into the first second-level sub-network to obtain the output of the first second-level sub-network; multiplying the input of the first second-level sub-network by a second learnable parameter, and then adding the output of the third first-level sub-network to obtain the input of the second second-level sub-network, and inputting the input into the second second-level sub-network to obtain the output of the second second-level sub-network; multiplying the input of the second second-level sub-network by a third learnable parameter, and then adding the output of the fourth first-level sub-network to obtain the input of the third second-level sub-network, and inputting the input into the third second-level sub-network to obtain the output of the third second-level sub-network.

[0048] In one of the embodiments, the second-level sub-network includes two continuously stacked MBC-GRN modules, the MBC-GRN module includes two 1x1 convolution layers, one depth separable convolution, and one GRN module; multiplying the output of the first first-level sub-network by a first learnable parameter, and then adding the output of the second first-level sub-network to obtain the input of the first second-level sub-network, and inputting the input into the first second-level sub-network to obtain the output of the first second-level sub-network is:

[0049] X out2,1 = MBC~GRN(hswish(BN(MBC~GRN(DownSample(X out1,2 + α0*X out1,1 ))) ) (1)

[0050] MBC~GRN(Z) = Conv(GRN(DPCONV(CONV(Z))) ) + Z (2)

[0051] wherein, X out2,1 is the output of the first second-level sub-network, Z is the input of the MBC-GRN module, X out1,2 is the output of the second first-level sub-network, X out1,1is the output of the first first-level subnetwork, hswish() is an activation function, BN() is batch normalization, DownSample() is down-sampling, MBC-GRN() is a MBC-GRN module, Conv() is convolution, DPCONV() is depthwise separable convolution, GRN() is a GRN module, and a0 is a learnable factor.

[0052] Specifically, Level2 follows the same parameter configuration as Level1, so that each Level has the same receptive field size without introducing additional noise information; the design purposes of the dilation factors d = 1 and d = 2 are also to increase the receptive field in the spatial dimension and capture more feature information; the input of each Level2 comes from the sum of the output of the same level of Level1 of the first level subnetwork and the output of the previous level of Level1 of the first level subnetwork multiplied by a learnable factor a. The "ladder" design of the network architecture also makes this parameter configuration more advantageous, with "progressive" feature information fusion from the upper subnetwork to the lower subnetwork, constantly supplementing the information extracted by the lower layer, so that the final information is more abundant. The detailed structure of Level2 is shown in Figure 3 .

[0053] Formula (1) gives the formula representation of the addition operation of each subnetwork feature map and its Level2 process. Assuming that the output of the current subnetwork Level1 is and the output of the previous subnetwork Level1 is The output after the Level2 layer is X out1,i and X out1,i-1 are added and summed, and the result has two directions, 1) one is directly input to the Level2 layer of the same level of the next level subnetwork; 2) the other direction is multiplied by a learnable factor a and then added to the output of the Level1 layer of the next subnetwork:

[0054] The MBC-GRN Block is a new inverted bottleneck module proposed by the present disclosure, which has better characteristics than the MBC Block, with better regularization characteristics and channel attention feature selection. Among them:

[0055] 1) The depthwise separable convolution in the traditional MBC Block is retained: including DW (channel-wise convolution) and PW (point-wise convolution) two processes, where the first PW convolution performs dimension conversion, and the DW convolution performs feature fusion;

[0056] 2) Introduce global response normalization (GRN), which is a regularization method that improves representation quality by enhancing feature diversity, can be used as feature selection, the GRN module itself has inherent regularization properties, does not need to be separately normalized by the BN layer, while also reducing the amount of calculation and parameter quantity. For the GRN regularization method, the detailed algorithm process is given here:

[0057] First, global feature aggregation. Aggregate each channel Xi in a spatial feature map with C channels by L2 norm regularization technique, assuming the feature map is Given the aggregation function is So it can be represented as:

[0058]

[0059] Then, normalize the features. The aggregated feature vector is Define the normalization function as It can be represented as:

[0060]

[0061] Finally, feature calibration. In the actual implementation process, two learnable parameters γ and β are introduced, initialized to 0 to alleviate the optimization problem; At the same time, residual connection is introduced, which can make GRN start from the identity mapping, gradually fit the learning features in the training process, which can bring better generalization to the network. This process can be represented by formula (5):

[0062]

[0063] In addition, unlike the traditional simple stacking MBC module, in the continuous stacking MBC-GRN Block, by setting different DW convolution dilation factors, it can capture larger receptive field information in the spatial dimension, and will not increase the additional amount of calculation; The introduction of GRN global response normalization also realizes the attention and feature selection in the channel dimension. Through the two mechanisms of space and GRN, combined with the specific "ladder" network architecture design, more rich feature extraction is realized, and the network effect is also better.

[0064] Experiments prove that GRN also has better generalization performance and feature selection ability than SE module, and the experimental results are better. Through the above analysis and experimental verification, a new inverted bottleneck MBC-GRN Block with GRN is thus born. Figure 4The detailed design of the MBC-GRN Block is given; Table 1 gives the ablation experiment comparison of the GRN module and the SE module:

[0065] Table 1 Ablation experiment comparison of GRN module and SE module

[0066] Module ACC (%) Computational cost (M) Parameter cost (M) SE 92.87 2132.86 4.32 GRN 94.00 2122.93 3.82

[0067] In one of the embodiments, the third-level network includes two third-level sub-networks, and each third-level sub-network includes four stacked Inverted-Transformer modules; each Inverted-Transformer module includes two 1x1 convolution layers and a Rel-Attention-Mat module; the output of each second-level sub-network is multiplied by a learnable parameter and then added to the output of the second-level sub-network layer by layer as the input of the corresponding third-level sub-network, and input into the third-level network, and the output of each third-level sub-network is:

[0068] X out3,i =(×4)Inverted~Transformer(DownSample(X out2,i+1 +α j *X out2,i )) (6)

[0069] Inverted~Transformer(y)=Conv(Conv(y)*Attention(Q,K,V))+y (7)

[0070]

[0071] wherein, X out3,i is the output of the i-th third-level sub-network, DownSample() is downsampling, Inverted~Transformer() is the Inverted-Transformer module, X out2,i+1 is the output of the i+1-th second-level sub-network, X out2,i is the output of the i-th second-level sub-network; α j is a learnable factor, and j=2,4 respectively represent Figure 2 the learnable factors α2 and α4 in the Inverted-Transformer module.

[0072] Specifically, the Level3 layer adopts the Inverted-Transformer module proposed in the present application which integrates the advantages of CNN and Transformer, and four Inverted-Transformer modules are stacked in each Level3 layer:

[0073] 1)Self-Attention mechanism itself has the characteristics of global receptive field, which can realize global receptive field information extraction, and the increase of calculation and parameter amount is not large;

[0074] 2) Since Multi-head self-attention (MHSA) is more suitable for deep semantic information feature extraction, after the preliminary feature extraction, the size of the Level3 feature map is 14x14, and the application of Inverted-Transformer Block for global feature information extraction is more conducive to the advantages of MHSA;

[0075] 3) Due to the multi-subnetwork design of LadderNet, the parallel double-layer subnetwork Transformer information extraction architecture helps to make up for the disadvantages brought by the lack of data set, and the information extraction of the upper layer Transformer can compensate for the information extraction of the lower layer Transformer, further making the extracted information more rich. The Level3 structure diagram is shown in Figure 5

[0076] In formula (6), "x4" represents that the learned information is given through 4 consecutive Inverted-Transformer modules, and the detailed process of Level3 is shown. Given the output of the current subnetwork Level2 as and the result after summing the Level2 layer of the previous subnetwork The output of the current Level3 is After adding and fusing the two learned information, the information flow is in two directions: 1) one is input to the current Level3 layer for feature extraction; 2) the other is input to the input layer of the last subnetwork Level3 for feature fusion.

[0077] The Inverted-Transformer module is a novel Inverted-Transformer module proposed by the present application:

[0078] 1) By introducing the MBC inverted bottleneck idea into the Transformer, the Self-Attention mechanism and the FFN module in the traditional Transformer architecture are integrated, so that the two residual connection modules are integrated into one, which reduces the training parameters and complexity, and also makes the whole architecture more efficient and concise;

[0079] ​2) When only the dimension of V is expanded, the computational complexity of the Inverted-Transformer Block increases only linearly, which is not much different from the original architecture, thus avoiding the increase in quadratic complexity in the Transformer.

[0080] 3) Introducing relative position bias. In Vit, absolute position is used. In this design, the introduction of relative position bias can make the Self-Attention mechanism have characteristics similar to CNN, which can make the network have better generalization. Formula (8) gives the Q, K, V attention calculation method, B represents the introduction of relative position bias; the detailed structure of the Inverted-Transformer module is as follows Figure 6 shown.

[0081] In one embodiment, the third-level network includes two third-level sub-networks, and the fourth-level network includes one fourth-level sub-network; the output of the first third-level sub-network is multiplied by the corresponding learnable parameter and then added to the output of the second third-level sub-network as the input of the corresponding fourth-level sub-network, and then inputted into the fourth-level network, so that the output of the fourth-level network is:

[0082] X out4,1 =MBC~GRN(hswish(BN(MBC~GRN(DownSample(X out3,2 +α5*X out3,1 ))))) (9)

[0083] Among them, X out4,1 is the output of the first fourth-level sub-network, X out3,1 is the output of the first third-level sub-network, X out3,2 is the output of the second third-level sub-network, DownSample() is downsampling, hswish() is the activation function, BN() is batch normalization, DownSample() is downsampling, MBC~CRN() is the MBC-GRN module, and α5 is the learnable factor.

[0084] Specifically, Level 4 does not use the Inverted-Transformer module stacking, but returns to the CNN architecture and stacks two MBC-GRN blocks for the following reasons:

[0085] 1) Lever 4 is a stage rich in semantic information. When the data set is relatively small, CNN is better than Transformer and is more conducive to information feature extraction and fusion;

[0086] 2) Level4 feature map size is 7x7, and the ingenious is to use our MBC-GRN module design, by setting different dilation factors, also can realize 7x7 large receptive field, is also the global receptive field information;

[0087] 3) Although the Transformer can capture the global receptive field, but in the actual effect is not as good as MBC-GRN module, the reason may be limited sample size of the dataset, single Transformer Block stack can not fully extract information features. Level4 layer as shown in Figure 7 Table 3 is the ablation experiment comparison of replacing Level4 with Inverted-Transformer Block:

[0088] Equation (9) gives the information change after Level4. Assuming that the feature information after the previous Level3 feature extraction is The output feature after Level4 layer is MBC-GRN module and Inverted-Transformer module ablation experiment comparison results as shown in Table 2.

[0089] Table 2 MBC-GRN and Inverted-Transformer ablation experiment comparison

[0090] Module ACC (%) Computational cost (M) Parameter cost (M) Inverted-Transformer Module 93.37 2126.59 3.90 MBC-GRN Module 94.00 2122.93 3.82

[0091] In one embodiment, the convolutional feature is input into the first level network, and the output of each first level subnetwork is obtained:

[0092] X out1,i = MBC-GRN (hswish (BN (NBC-GRN (X))) (10)

[0093] where, X out1,i is the output of the i-th first level subnetwork, X is the input of the first level network, hswish() is an activation function, BN() is batch normalization, MBC-GRN() is MBC-GRN module.

[0094] Specifically, at Level 1 layer, instead of downsampling separately, it is merged into Stem layer, which has the advantages of: 1) After the Stem layer, the image is downsampled by 4 times, and in order to focus on the fusion and feature extraction of spatial information in this layer, the resolution size of the entire feature map is kept unchanged, and the DW convolution with different size expansion factors can capture a larger receptive field; 2) Since more texture information is in the shallow layer, combined with the inductive bias inherent in convolution itself: locality and translation invariance, convolution has more advantages than Self-Attention mechanism in Transformer in feature extraction in the shallow layer, and is easier to train; 3) In addition, the downsampling layer in Level 1 is merged into the Stem layer here, which reduces the training parameters and computational complexity to a certain extent. After the Stem layer, the feature map size is The output of Level 1 layer in each subnetwork is obtained by inputting it into Level 1 layer (i∈1, 2, 3, 4, represents the i-th subnetwork from top to bottom), and the detailed process is given by formula (10); the Level 1 structure design is as shown in Figure 8 .

[0095] In one embodiment, the classification layer includes: an adaptive global average pooling layer, a layer normalization, and a fully connected layer; the fourth-level network includes one fourth-level subnetwork; the output of the fourth-level network is input into the classification layer to obtain a predicted image recognition result as:

[0096] X outclassifier = FC(LN(AdaptiveASvgPool(X out4,1 ))) (11)

[0097] Wherein, X outclassifier is the predicted image recognition result, X out4,1 is the output of the first fourth-level subnetwork, AdaptiveAvgPool() is an adaptive global average pooling layer, LN() is a layer normalization, and FC() is a fully connected layer.

[0098] Specifically, the last layer of the classification layer (Classifier layer) is used for semantic information classification. First, the feature map with rich semantic information of 7x7x192 is adaptively globally pooled (AdaptiveAvePool) to aggregate the features; then it is subjected to a layer normalization (LayerNorm, LN); finally, it is input into a fully connected layer (FC) containing 8 classification neurons for classification.

[0099] After feature extraction by the previous network, it is then input into the Classifier layer. Given the information after Level 4 layer feature extraction The output features after the Classifier layer are Equation (11) gives the information change after the classification layer; the detailed structure diagram of the Classifier layer is shown in Figure 9 .

[0100] In one embodiment, the stem layer includes two convolutional layers; the training sample is input into the stem layer, and the convolutional features are obtained as follows:

[0101] X = Conv(hswish(BN(Conv(X in ) (12)

[0102] wherein X is the output of the stem layer, X in is the input of the medical image recognition network, hswish() is an activation function, BN() is batch normalization, and Conv() is convolution.

[0103] Specifically, Stem adopts two consecutive 3x3 convolutional layers with a stride of 2 for down-sampling, which can ensure a certain information superposition and fusion between each 3x3 window. In recent years, the common practice of hybrid networks is to directly down-sample 4 times through a 4x4 convolution kernel with a stride of 4 in the Stem layer, but this has certain defects and can easily cause information loss. As can be seen from the network architecture diagram, although the design this time is also down-sampling 4 times, the down-sampling strategy is changed, and the original down-sampling 2 times in each Level1 layer is merged into the Stem layer, so there is the design of down-sampling 4 times this time. This essentially also reduces a certain amount of parameters and calculation. Equation (12) gives the feature map change process after the Stem layer. Assuming a given standard input the output is obtained The detailed design of the Stem layer is shown in Figure 10 .

[0104] It should be understood that although each step in the flowchart of Figure 1 is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless explicitly stated in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 at least part of the steps in may include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0105] In one verification embodiment, the Kvasir dataset is used as the training sample. The Kvasir dataset is a multi-class image dataset for computer-aided gastrointestinal disease detection, and the images can be used for different tasks such as image retrieval, machine learning, deep learning, and transfer learning, etc. The images are annotated and verified by doctors (experienced endoscopy physicians). The dataset includes images showing anatomical landmarks in the gastrointestinal tract, pathological findings, and two types of images related to endoscopic polyp resection. The anatomical landmarks include Z-line, pylorus, and cecum, three categories; the pathological findings include esophagitis, polyps, and ulcerative colitis, three categories; two types of images related to endoscopic polyp resection: stained wrinkled polyps, and stained resection margins, a total of 8 categories, 500 pictures per category, a total of 4000 pictures, with picture resolutions ranging from 720x576 to 1920x1072 pixels. Among them, 3200 are used as the training set, and 800 are used as the training set.

[0106] (1) Experimental environment

[0107] The experiment uses the Pytorch framework, uses Python language, and the model is deployed on an NVIDIA GeForce RTX2080Ti server.

[0108] (2) Experimental results

[0109] In order to evaluate the network performance of LadderNet, four representative networks are selected for model comparison, including the classic network ResNet50, the NeurlPS2021 convolution + attention network CoAtNet, the CVPR2022 recursive gating network HorNet, and the latest network InceptionNeXt of CVPR2023. All experiments are run under the same variable settings and environment, and the experimental results show that the newly proposed LadderNet network architecture not only has higher accuracy than the above networks, but also has absolute advantages in computational complexity and parameter quantity. Table 3 gives the experimental results of different models on the Kvasir dataset:

[0110] Table 3 Comparison of experimental results of different models on Kvasir dataset

[0111] Model Best acc (%) Computational cost (M) Parameter cost (M) Epoch Description ResNet50 90.87 4131.71 23.52 400 Batchsize = 16 CoAtNet 92.00 3351.59 17.76 400 Batchsize = 16 InceptionNeXt 90.87 4198.25 25.76 400 Batchsize = 16 HorNet 80.13 3967.88 21.86 400 Batchsize = 16 LadderNet (ours) 94.00 2122.93 3.82 400 Batchsize = 16

[0112] In one embodiment, as shown in Figure 11 , a medical image recognition device based on a ladder feature fusion network is provided, comprising: a training sample determination module, a medical image recognition network construction module, a medical image recognition network training module, and a medical image recognition module, wherein:

[0113] The training sample determination module is configured to acquire a medical image and take the labeled medical image as a training sample.

[0114] The medical image recognition network construction module is configured to construct a medical image recognition network based on ladder-type feature fusion. The medical image recognition network is designed in a ladder type and includes a stem layer, four level networks, and a classification layer from left to right. The number of subnetworks included in each level network decreases successively. Each subnetwork of the first, second, and fourth level networks includes a plurality of MBC-GRN modules stacked in succession. Each subnetwork of the third level network includes a plurality of Inverted-Transformer modules stacked in succession. The MBC-GRN module is obtained by introducing a global response normalization module into an MBC module. The Inverted-Transformer module is obtained by introducing an MBC inverted bottleneck idea and a relative position bias into a Transformer.

[0115] The medical image recognition network training module is configured to train the medical image recognition network using the training sample to obtain a trained medical image recognition network.

[0116] The medical image recognition module is configured to input a to-be-tested medical image into the trained medical image recognition network to obtain an image recognition result.

[0117] In one of the embodiments, the medical image recognition network training module is further configured to input the training sample into the stem layer to obtain a convolutional feature; input the convolutional feature into the first level network to obtain an output of each first level subnetwork; multiply the output of each first level subnetwork by a learnable parameter and add the output to the output of the first level subnetwork layer by layer as an input of a corresponding second level subnetwork, and input the input into the second level network to obtain an output of each second level subnetwork; multiply the output of each second level subnetwork by a learnable parameter and add the output to the output of the second level subnetwork layer by layer as an input of a corresponding third level subnetwork, and input the input into the third level network to obtain an output of each third level subnetwork; multiply the output of the first third level subnetwork by a corresponding learnable parameter and add the output to the output of the second third level subnetwork as an input of a corresponding fourth level subnetwork, and input the input into the fourth level network to obtain an output of the fourth level network; input the output of the fourth level network into the classification layer to obtain a predicted image recognition result; perform back training on the medical image recognition network according to the predicted image recognition result and the label of the training sample to obtain a trained medical image recognition network.

[0118] In one of the embodiments, the first-level network comprises four first-level sub-networks, and the second-level network comprises three second-level sub-networks; the medical image recognition network training module is further configured to multiply the output of the first first-level sub-network by a first learnable parameter, and then add the output to the output of the second first-level sub-network to obtain an input of a first second-level sub-network, and input the input into the first second-level sub-network to obtain an output of the first second-level sub-network; multiply the input of the first second-level sub-network by a second learnable parameter, and then add the input to the output of the third first-level sub-network to obtain an input of a second second-level sub-network, and input the input into the second second-level sub-network to obtain an output of the second second-level sub-network; multiply the input of the second second-level sub-network by a third learnable parameter, and then add the input to the output of the fourth first-level sub-network to obtain an input of a third second-level sub-network, and input the input into the third second-level sub-network to obtain an output of the third second-level sub-network.

[0119] In one of the embodiments, the second-level sub-network comprises two continuously stacked MBC-GRN modules, and each MBC-GRN module comprises two 1x1 convolution layers, one depth separable convolution, and one GRN module; the medical image recognition network training module is further configured to multiply the output of the first first-level sub-network by a first learnable parameter, and then add the output to the output of the second first-level sub-network to obtain an input of a first second-level sub-network, and input the input into the first second-level sub-network to obtain an output of the first second-level sub-network, as shown in equation (1) and equation (2).

[0120] In one of the embodiments, the third-level network comprises two third-level sub-networks, and each third-level sub-network comprises four stacked Inverted-Transformer modules; each Inverted-Transformer module comprises two 1x1 convolution layers and one Rel-Attention-Mat module; the medical image recognition network training module is further configured to multiply the output of each second-level sub-network by a learnable parameter, and then add the output to the output of the second-level sub-network layer by layer to obtain an input of a corresponding third-level sub-network, and input the input into the third-level network to obtain an output of each third-level sub-network, as shown in equation (6) to equation (8).

[0121] In one of the embodiments, the third-level network comprises two third-level sub-networks, and the fourth-level network comprises one fourth-level sub-network; the medical image recognition network training module is further configured to multiply the output of the first third-level sub-network by a corresponding learnable parameter, and then add the output to the output of the second third-level sub-network to obtain an input of a corresponding fourth-level sub-network, and input the input into the fourth-level network to obtain an output of the fourth-level network, as shown in equation (9).

[0122] In one of the embodiments, the medical image recognition network training module is further configured to input the convolutional features into the first-level network to obtain an output of each first-level sub-network as shown in equation (10).

[0123] In one of the embodiments, the classification layer comprises an adaptive global average pooling layer, a layer normalization, and a fully connected layer; the fourth-level network comprises one fourth-level sub-network; and the medical image recognition network training module is further configured to input an output of the fourth-level network into the classification layer to obtain a predicted image recognition result as shown in equation (11).

[0124] In one of the embodiments, the stem layer comprises two convolutional layers; and the medical image recognition network training module is further configured to input the training sample into the stem layer to obtain the convolutional features as shown in equation (12).

[0125] The specific limitations of the medical image recognition device based on the ladder-type feature fusion network can refer to the limitations of the medical image recognition method based on the ladder-type feature fusion network described above, which will not be repeated here. Each module in the medical image recognition device based on the ladder-type feature fusion network described above can be realized by software, hardware, and combinations thereof, in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0126] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure.

[0127] The above-described embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are within the scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A medical image recognition method based on a step-by-step feature fusion network, characterized in that: The method comprises: Acquire a medical image and use the annotated medical image as a training sample; A medical image recognition network based on step-by-step feature fusion was constructed. The network was designed in a step-by-step manner and comprised, from left to right, a stem layer, four level networks, and a classification layer. The number of subnetworks included in each level network decreased sequentially. Each subnetwork of the first, second, and fourth level networks included several continuously stacked MBC-GRN modules, and each subnetwork of the third level network included several stacked Inverted-Transformer modules. The MBC-GRN module was obtained by introducing a global response normalization module into the MBC module, and the Inverted-Transformer module was obtained by introducing the MBC inverted bottleneck concept and relative position bias into the Transformer. Using the training samples to train the medical image recognition network to obtain a trained medical image recognition network; The medical image to be tested is input into the trained medical image recognition network to obtain the image recognition result.

2. The method according to claim 1, characterized in that The medical image recognition network is trained using the training samples to obtain a trained medical image recognition network, including: Inputting the training sample into the stem layer to obtain convolutional features; Inputting the convolutional features into the first-level network to obtain the output of each first-level sub-network; The output of each first-level sub-network is multiplied by a learnable parameter and then added layer by layer to the output of the first-level sub-network as the input of the corresponding second-level sub-network, and then inputted into the second-level network to obtain the output of each second-level sub-network; The output of each second-level sub-network is multiplied by a learnable parameter and then added layer by layer to the output of the second-level sub-network as the input of the corresponding third-level sub-network, and then inputted into the third-level network to obtain the output of each third-level sub-network; The output of the first third-level sub-network is multiplied by the corresponding learnable parameter and then added to the output of the second third-level sub-network as the input of the corresponding fourth-level sub-network, and the output is input into the fourth-level network to obtain the output of the fourth-level network; Inputting the output of the fourth-level network into the classification layer to obtain a predicted image recognition result; The medical image recognition network is reversely trained according to the predicted image recognition results and the annotations of the training samples to obtain a trained medical image recognition network.

3. The method according to claim 2, characterized in that The first-level network includes four first-level sub-networks, and the second-level network includes three second-level sub-networks; The output of each first-level sub-network is multiplied by the corresponding learnable parameter and then added to the output of the first-level sub-network layer by layer as the input of the corresponding second-level sub-network, and then inputted into the second-level network to obtain the output of each second-level sub-network, including: The output of the first first-level sub-network is multiplied by the first learnable parameter and then added to the output of the second first-level sub-network as the input of the first second-level sub-network, and then inputted into the first second-level sub-network to obtain the output of the first second-level sub-network; The input of the first second-level sub-network is multiplied by the second learnable parameter and then added to the output of the third first-level sub-network as the input of the second second-level sub-network, and then inputted into the second second-level sub-network to obtain the output of the second second-level sub-network; The input of the second second-level sub-network is multiplied by the third learnable parameter and then added to the output of the fourth first-level sub-network as the input of the third second-level sub-network, and then input into the third second-level sub-network to obtain the output of the third second-level sub-network.

4. The method according to claim 3, characterized in that The second-level sub-network includes two consecutively stacked MBC-GRN modules, and the MBC-GRN module includes two Convolutional layer, 1 depth-wise separable convolution and a GRN module; The output of the first first-level sub-network is multiplied by the first learnable parameter and then added to the output of the second first-level sub-network as the input of the first second-level sub-network, which is then input into the first second-level sub-network. The output of the first second-level sub-network is: in, is the output of the first second-level sub-network, Z is the input of the MBC-GRN module, is the output of the second first-level sub-network, is the output of the first-level sub-network, is the activation function, is batch normalization, is downsampling, For the MBC-GRN module, is convolution, is a depth-wise separable convolution, For the GRN module, is a learnable factor.

5. The method according to claim 2, characterized in that The third-level network includes two third-level sub-networks, each of which includes four stacked Inverted-Transformer modules; the Inverted-Transformer module includes two Convolutional layer and a Rel-Attention-Mat module; The output of each second-level sub-network is multiplied by the learnable parameters and then added to the output of the second-level sub-network layer by layer as the input of the corresponding third-level sub-network, and then input into the third-level network. The output of each third-level sub-network is obtained as follows: in, For the i The output of the third-level sub-network, is downsampling, is the Inverted-Transformer module, For the i +1 output of the second-level sub-network, For the i The output of the second-level sub-network, is the learnable factor, and (×4) is the Inverted-Transformer module stacked 4 times in a row.

6. The method according to claim 2, characterized in that The third-level network includes two third-level sub-networks, and the fourth-level network includes one fourth-level sub-network; The output of the first third-level sub-network is multiplied by the corresponding learnable parameter and then added to the output of the second third-level sub-network as the input of the corresponding fourth-level sub-network, which is then input into the fourth-level network. The output of the fourth-level network is: in, is the output of the first fourth-level sub-network, is the output of the first third-level sub-network, is the output of the second third-level sub-network, is downsampling, is the activation function, is batch normalization, is downsampling, For the MBC-GRN module, is a learnable factor.

7. The method according to claim 2, characterized in that The convolutional features are input into the first-level network, and the output of each first-level sub-network is: in, For the i The output of the first-level sub-network, X is the input of the first level network, is the activation function, is batch normalization, This is the MBC-GRN module.

8. The method according to claim 2, characterized in that The classification layer includes: an adaptive global average pooling layer, a layer normalization layer, and a fully connected layer; the fourth level network includes a fourth level subnetwork The output of the fourth-level network is input into the classification layer, and the predicted image recognition result is: in, To predict image recognition results, is the output of the first fourth-level sub-network, is an adaptive global average pooling layer, is layer normalization, is the fully connected layer.

9. The method according to claim 2, characterized in that The stem layer includes two convolutional layers; The training sample is input into the stem layer, and the convolution feature is obtained as follows: in, X is the output of the stem layer, is the input of the medical image recognition network, is the activation function, is batch normalization, is convolution.

10. A medical image recognition device based on a step-by-step feature fusion network, characterized in that: The device comprises: A training sample determination module is used to obtain medical images and use the annotated medical images as training samples; A medical image recognition network construction module is used to construct a medical image recognition network based on step-by-step feature fusion. The medical image recognition network adopts a step-by-step design and includes, from left to right, a stem layer, four level networks, and a classification layer. The number of subnetworks included in each level network decreases successively. Each subnetwork of the first, second, and fourth level networks includes several continuously stacked MBC-GRN modules, and each subnetwork of the third level network includes several stacked Inverted-Transformer modules. The MBC-GRN module is obtained by introducing a global response normalization module into the MBC module, and the Inverted-Transformer module is obtained by introducing the MBC inverted bottleneck concept and relative position bias into the Transformer. a medical image recognition network training module, configured to train the medical image recognition network using the training samples to obtain a trained medical image recognition network; The medical image recognition module is used to input the medical image to be tested into the trained medical image recognition network to obtain the image recognition result.

Images