Method for classifying brain slices by combining improved capsule network and residual network

Abstract

The invention discloses a method for classifying brain slices by combining an improved capsule network and a residual network. The method comprises the following steps: acquiring a magnetic resonance image of a to-be-measured target; inputting the magnetic resonance image into a trained residual capsule network to obtain a reconstructed image; and obtaining a classification result of the brain slices by using the reconstructed image. Wherein the residual capsule network comprises an encoder and a decoder, the encoder comprises a residual network and a capsule network, the decoder comprises a deconvolution layer, and the residual network is used for extracting feature maps of different depths from an input image and transmitting the feature maps to the capsule network; the capsule network carries out vectorized weighted merging on the feature maps of the different depths, and the decoder reconstructs the image by using the longest vector in the connected capsules of the capsule network. According to the method, the definition of the reconstructed image is improved, the image reconstruction time is shortened, and the method is particularly suitable for processing small, unbalanced and large DE-MRI data sets.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and more specifically, to a method for classifying brain slices by combining an improved capsule network with a residual network. Background Art

[0002] Stroke (CS), also known as "stroke", is an acute cerebrovascular disease. It is a group of diseases that cause brain tissue damage due to sudden rupture of brain blood vessels or blood vessel blockage that prevents blood from flowing into the brain, including ischemic and hemorrhagic strokes. Since radiologists need to manually select the most important CT slices, the current stroke diagnosis method is very subjective and prone to errors. Therefore, in order to improve the prognosis of patients, it is particularly important to seek more accurate and reliable automated stroke diagnosis and localization methods. For example, delayed enhancement magnetic resonance imaging (DE-MRI) of brain slices is an effective method for evaluating the characteristics and extent of stroke. This type of scan has the characteristics of high resolution and low motion blur. Because manual detection is usually time-consuming, in recent years, brain MRI machine learning has gradually been applied to the examination of asymptomatic cerebral infarction, brain atrophy, cerebral hemorrhage, subarachnoid hemorrhage, deformities and lesions and other early stroke risks.

[0003] At present, convolutional neural networks (CNNs) are widely used in the field of image processing. CNN uses convolution kernels to extract image information, reduces training parameters through pooling layers, and finally outputs classification results through classifiers. CNN achieves invariance to interpretation and rotation through weight sharing and pooling operations, but CNN cannot identify image spatial relationships that will have a significant impact on the results. For example, in medical images, even if the spatial position of organs is disrupted, CNN will still identify them as normal samples. During the training process, CNN requires a large data set to form accurate patterns. Deep networks are prone to gradient vanishing and explosion, and pooling layers may cause the extracted feature information to be lost, thus affecting the final classification accuracy.

[0004] To address the above-mentioned shortcomings of CNN, Sabour et al. proposed a new deep network called Capsule Network (CapsNet) (Sabour, S., N. Frosst, and G. E Hinton, Dynamic Routing Between Capsules. arXiv e-prints, 2017: p. arXiv: 1710.09829.). CapsNet considers the spatial relationship between features and uses a dynamic routing algorithm to encode the relationship between objects and results, thereby reflecting the overall characteristics of the image. Capsules can store vectors describing specific objects, where the length of the vector represents the probability of the object's existence and the direction represents the description parameter. The stored description parameter vector can be flattened and reversely rendered to form the original image. Compared with CNN, CapsNet is friendly to small data sets. CapsNet does not set a pooling layer that risks losing information. Although CapsNet performs well on the small data set MNIST, it has low accuracy in classifying complex data sets such as MRI scans. This is because CapsNet is a shallow network, mainly including convolutional layers for feature extraction, capsule layers for encoding mapping, and an optional decoder for reconstructing images. CapsNet usually has only two convolutional layers, which makes it unable to capture the deeper features contained in MRI scans. In addition, if the image is too large and the decoder regenerates the high-resolution image from only a few vectors, the reconstruction error of the image will increase, further reducing the classification accuracy, and it is prone to gradient disappearance and explosion, which will have an adverse effect on training speed and performance. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for classifying brain slices by combining an improved capsule network with a residual network. The method comprises the following steps:

[0006] Acquiring a magnetic resonance image of the target to be measured;

[0007] Inputting the magnetic resonance image into a trained residual capsule network to obtain a reconstructed image;

[0008] obtaining classification results of brain slices using the reconstructed image;

[0009] The residual capsule network includes an encoder and a decoder, the encoder includes a residual network and a capsule network, the decoder includes a deconvolution layer, the residual network is used to extract feature maps of different depths from the input image and pass them to the capsule network, the capsule network performs vectorized weighted merging on the feature maps of different depths, and the decoder uses the longest vector in the connected capsules of the capsule network to reconstruct the image.

[0010] Compared with the prior art, the advantage of the present invention is that a new type of capsule network is proposed, which replaces the convolution layer with a residual block. The residual block can extract deeper semantic information from the image, and then combine the shallow, middle and deep features extracted by different residual blocks to help build a more efficient capsule network. In addition, in order to avoid the vanishing and explosion of gradients caused by high-resolution input, scaling reconstruction is used in the decoder. Compared with the existing residual network and capsule network, the residual capsule network model proposed in the present invention improves the clarity of the reconstructed image, especially for large DE-MRI scans, and has better performance.

[0011] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0013] Figure 1 is a flowchart of a method for classifying brain slices by combining an improved capsule network with a residual network according to an embodiment of the present invention;

[0014] Figure 2 2 is a schematic diagram of the overall structure of a residual capsule network according to an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram of an original residual block and pre-activation according to an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram of the structure of an improved capsule network according to an embodiment of the present invention;

[0017] Figure 5 is a schematic diagram of a compression and excitation module according to an embodiment of the present invention;

[0018] Figure 6 is a structural diagram of a deconvolution network layer according to an embodiment of the present invention;

[0019] Figure 7 is a schematic diagram of a positive sample and a negative sample of a DE-MRI image according to an embodiment of the present invention;

[0020] Figure 8 is a comparison chart of test loss and test accuracy of three models according to an embodiment of the present invention;

[0021] Fig. 9 3 is a comparison chart of test loss and accuracy of capsule network under different data amounts according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0023] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0024] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0025] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0026] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0027] To improve the ability of CapsNet to process large-size medical image inputs, the present invention proposes a novel residual capsule network (or ResCapsNet), which replaces the convolutional layer with an improved residual network to compress features, reduces the number of primary capsules, and prevents gradient vanishing and explosion. The residual network (ResNet) introduces residual blocks, which enable the network to pass more information at each layer through shortcut connections. By applying a residual network and improving the structure of the capsule network, the number of weights in the network can be reduced. During the image reconstruction process, the original image is scaled to reduce the amount of computation. In addition, a new loss function using a scaling reconstruction method is proposed to further reduce the computational burden.

[0028] See also Figure 1 As shown, the provided method for classifying brain slices by combining an improved capsule network with a residual network includes the following steps.

[0029] Step S110: construct a residual capsule network, where the residual capsule network includes an encoder and a decoder, wherein the encoder includes a residual network and a capsule network.

[0030] See also Figure 2 As shown in Figure 1, the residual capsule network mainly includes an encoder and a decoder. First, the encoder encodes the input image, and then the decoder reconstructs the image. Figure 2In the example, the encoder includes a residual network and a capsule network, wherein the capsule network includes primary capsules, digital capsules, and connection capsules, and the decoder includes multiple deconvolution layers, for example, 3 or 5 deconvolution layers. The residual network is responsible for extracting feature information. The capsule network is responsible for vectorizing the features. The decoder is used to reconstruct the image and obtain the reconstruction loss by comparing it with the scaled input.

[0031] In the following, the improvements to the residual network, capsule network, and loss function will be specifically introduced.

[0032] 1) Improved residual network

[0033] In the provided residual capsule network, an improved residual network is used as the convolution layer, and the pooling layer in the residual network is removed because it loses image information. The residual network contains multiple residual blocks to preserve the spatial location information of the feature map and prevent gradient disappearance and explosion.

[0034] The residual network is mainly used to extract deep features and semantic information from images. For example, the setting includes 4 residual blocks. First, the image is input into the first residual block to obtain a feature map of 96×96×128. Then, after 3 residual layers (convolution kernel is 3×3, stride is 2), a shallow feature map of size 48×48×256 is obtained, and the extracted shallow feature map is transferred to PrimaryA (the first primary capsule). After the same residual block with convolution kernel size of 3×3 and stride of 2, a middle-level feature map of size 24×24×512 is obtained, and the extracted middle-level feature map is transferred to PrimaryB (the second primary capsule). Then, the middle-level features are input into the last residual block to obtain a deep feature map of size 8×8×256, and the extracted deep feature map is transferred to PrimaryC (the third primary capsule).

[0035] Preferably, a new residual block structure is introduced like Figure 3 As shown in the figure, the left figure is the structure of the original residual block, and the right figure is the pre-activated structure, which uses a 1×1 convolution layer for downsizing. In the new residual block structure, the activation function is moved to the residual part. By changing the activation function to an identity mapping, the accuracy of the model is improved. Figure 3 The new residual block includes a regularization layer (BN), an activation layer (such as using a ReLU activation function) and a convolutional layer.

[0036] 2) Improved capsule network

[0037] In the provided residual capsule network, the improved residual network is used as the convolutional layer of the capsule network, which can not only solve the gradient vanishing and exploding problems in the capsule network of large-size medical images, but also preserve the spatial position relationship between the features of the capsule layer, while reducing the number of primary capsules for easy deployment and calculation. The deep network has a larger receptive field and a stronger ability to represent semantic information, but a weaker ability to represent spatial information. Deep features can represent richer image information and can separate complex target areas. The shallow network has a smaller receptive field and a weaker ability to represent semantic information, but a stronger ability to represent spatial information. Shallow features represent less information, but can separate some simple target areas. The fusion of shallow and deep features can compensate each other, thereby improving the performance of the model.

[0038] In order to better integrate shallow features and deep features, the capsule network is improved. Figure 4 As shown in the figure, the capsule network includes three capsule layers or three primary capsules (labeled as PrimaryA, Primary B and Primary C), the respective digital capsule layers corresponding to the three capsule layers and a connected digital layer. The convolution layer outputs the features to three separate capsule layers. Then, the results of the three capsule layers are merged to obtain the final result. Specifically, the SE (Compression and Excitation Block) module will obtain the weights of different digital layers during the merging process. The shallow features ConvA, the middle features ConvB and the deep features ConvC extracted by the previous residual network are transmitted to the capsule layers PrimaryA, PrimaryB and PrimaryC respectively. These layers all contain compression and excitation (SE) modules, and the weight coefficients a1, a2 and a3 of the shallow, middle and deep features come from the corresponding SE modules. The three weight coefficients are multiplied by the three digital capsule layers to obtain three digital capsule layers with weights, which are labeled as DigitA, DigitB and DigitC respectively. Finally, the three digital layers with weight coefficients are connected into one digital layer. The outputs of DigitA, DigitB, and DigitC are two vectors representing two categories, and each output is an 8-dimensional capsule. The final connected capsule fusion layer is represented by D, which is a 24-dimensional capsule, expressed as:

[0039]

[0040] Among them, D1, D2 and D3 represent the outputs of DigitA, DigitB and DigitC respectively. Represents string concatenation.

[0041] In the encoder, the loss function uses the marginal loss function, which is defined as follows:

[0042]

[0043] Among them, T k Set to 1, represents the prediction vector, λ is a set constant, m + Represents the upper bound, for example, set to 0.9, m - represents the lower bound, for example, set to 0.1. If the length of the vector is greater than m + , meaning that the input belongs to this category, and L k = 1. On the contrary, if the length of the vector is less than m - , it means that the input does not belong to this category, and L k = 0. In a binary classification problem, there is only one T k =1 capsule, and another capsule T k = 0. Therefore, there is no imbalance problem, so the λ introduced to balance the edge loss is set to 1.

[0044] 3) Compression and excitation module

[0045] The Squeeze and Excitation Network (SENet) introduces an attention mechanism to adaptively weight the feature maps of each channel. This attention mechanism is implemented by the Squeeze and Excitation module (SE).

[0046] Adding the compression and excitation module to the provided residual capsule network helps to extract features better, thereby improving the accuracy of the network. The compression and excitation module, also known as compression and excitation, is a combination of two key operations that enrich the channel attention mechanism. For the primary capsule, one dimension is the vector array and the other dimension is the number of capsules. The attention mechanism provided by the compression and excitation module adds weights to the importance of each capsule and enhances or suppresses the corresponding capsules for different tasks to extract useful features and suppress useless features.

[0047] See also Figure 5 As shown in the figure, the compression and excitation module includes a global pooling layer and two fully connected layers. The operation of the compression and excitation module can be divided into three steps:

[0048] (1) Compression operation aims to compress capsule information, keep the number of capsules unchanged, and perform global information pooling to convert each capsule block into a number, whose value is the sum of the elements in the primary capsule layer divided by the number of elements, expressed as:

[0049]

[0050] Where n is the number of primary capsules, p is the length of the capsule, μ ij Represents the element inside the capsule.

[0051] (2) The excitation operation is implemented using two fully connected layers and a Sigmoid function, expressed as:

[0052] s=σ(W2δ(W1U k )) (5)

[0053] Among them, s is the output of the excitation operation, σ is the activation function Sigmoid, W2 and W1 are the corresponding weight parameters of the two fully connected layers, and δ is the activation function ReLU.

[0054] (3) Reweighting operation. When connecting digital capsules, weights are assigned to the digital capsules block by block. This completes the redistribution of block-level raw features, allowing the model to decide which level of features are important.

[0055] 4) Dynamic routing algorithm

[0056] The algorithm for calculating digital capsules based on primary capsules is called dynamic routing algorithm. The iteration r of dynamic routing is set to 3 in this invention. In each iteration, the initial capsule u is predicted. i Belong to digital capsule v j , expressed as:

[0057] c ij =softmax(W ji u i ·v j ) (6)

[0058] Among them, W ij is the transformation matrix, c ij is the prediction result. ij changes in each iteration. Therefore, the calculated v j (i.e. c ij W ij u i The sum of ) also varies in different iterations. The Softmax function ensures that c ij The sum is always equal to 1.

[0059] In the output, the final calculated v must be j To perform the extrusion:

[0060] v j =squash(v j ) (7)

[0061] in:

[0062]

[0063] From formula (8), we can see that the squeezing operation will transform the vector v j The norm of is compressed to the range of (0,1) without changing the direction of the vector. Since the direction of the vector represents the feature, the final result v jThe features corrected according to the primary capsule are retained. During the training process, multiple iterations are required to update the parameters multiple times, and the result of the last iteration is selected as the final output result of the digital capsule.

[0064] It is worth noting that the output of the dynamic routing algorithm is the digital capsule, not the prediction coefficient c ij Therefore, the algorithm can be regarded as a hidden layer with complex processes. At present, the effective method for training deep networks is back propagation, which requires more parameters for capsule networks to improve their accuracy during training. Therefore, the transformation matrix W is introduced ij , which not only transforms the primary capsules into different shapes, but also ensures the ability of the capsule network to capture information from the vector.

[0065] 5) Decoder

[0066] The decoder includes a deconvolution module. The original capsule network uses reconstruction loss as a regularization method to encourage the capsules in the digital capsule layer to encode as much useful information as possible. Reconstruction is done by simply feeding the output of the digital capsule layer to the decoder consisting of three fully connected layers. While this method works well on simple images, its reconstruction performance is not suitable for complex datasets such as DE-MRI scans, where the reconstructed images are blurry and difficult to distinguish.

[0067] To solve the above problems, in one embodiment, deconvolution is used to reconstruct the image to improve the reconstruction performance and classification accuracy of complex data sets. Figure 6 As shown in the figure, the two 24-dimensional capsule vectors obtained from the encoder are reformatted into vector images of size 4×4×3. Then, the vectors are deconvolved with three convolution kernels of size 3×3, with step sizes of 1, 1, 2, 3, and 1. The number of channels is changed from 3 to 1. Finally, a reconstructed image with a channel number of 1 and a size of 24×24 is obtained.

[0068] Step S120, training the residual capsule network using the set loss function.

[0069] In step S110, the large-size input image is scaled, and mean pooling with a convolution kernel size of 8×8 and a stride of 8 is used to reduce the image size. After scaling, the input image of size 224×224 can be reduced to 24×24. Different from the reconstruction loss of the traditional capsule network, in one embodiment, the sum of the squares of the difference between the scaled original image and the reconstructed image is calculated, and the sum of the squares is used as the reconstruction loss based on the scaled reconstruction, which is expressed as follows:

[0070]

[0071] Among them, y m is a true value, is the estimated value, and M is the total number of samples. reconstruction The larger it is, the worse the effect is.

[0072] In one embodiment, the overall loss function for training the residual capsule network is set to:

[0073]

[0074] Among them, L k Represents the edge loss of the encoder. Since there is a gap between the values ​​of the reconstruction loss and the edge loss, the weight of the reconstruction loss needs to be reduced when calculating the overall loss. For example, the weight coefficient α is set to 0.0005.

[0075] Step S130, using the trained residual capsule network to enhance the target image, and then analyzing the degree of stroke based on the enhanced image.

[0076] After the training is completed, the optimization parameters of the residual capsule network can be obtained, and then applied to the actual target image reconstruction, including: obtaining the target image to be enhanced; inputting the target image into the residual capsule network to obtain a clearer reconstructed image; and then using the reconstructed image to analyze the degree of stroke for clinical indication. The target image can be magnetic resonance imaging, delayed enhancement magnetic resonance imaging or other types.

[0077] To verify the effect of the present invention, the performance of the proposed residual capsule network, traditional CapsNet and ResNet were compared through experiments. The structures of the three models are shown in Table 1. First, experiments were carried out on a large medical dataset to evaluate the classification effectiveness of the three models in terms of accuracy, confusion matrix, sensitivity, specificity and AUC value. Then the models were compared under different data amounts.

[0078] Table 1: Structure of the three models

[0079]

[0080] During the training of the model, the Adam optimization algorithm was used for optimization, and the learning rate was set to 0.001. The number of iterations of the model training was set to 2000 to prevent overfitting. The number of iterations of the capsule layer dynamic routing algorithm was set to 3. Since the capsule network is prone to overfitting, the capsule layer was deactivated during each iteration of training, and the weights of the deactivated neurons were not updated in that iteration to reduce the complexity of the model.

[0081] 1) Dataset

[0082] The dataset in the experiment comes from Xiangya Hospital and contains 580 subjects. It was acquired using a Siemens Prisma, 1.5 Tesla, model-syngo MRE11 scanner. The echo time was set to 1.33 and the repetition time was 321.79. The size of the original image was 256×192. The diagnosis was made by a professional encephalologist. Written consent was obtained from all subjects participating in the experiment. Figure 7 An example of two types of samples is shown, where Figure 7 (a) is the scanned image of a normal sample. Figure 7 (b) is a scan image of a sample with stroke.

[0083] 2) Preprocessing

[0084] MRI data is a grayscale image. However, the file is saved as a three-channel PNG color image during the dump process. To reduce the amount of calculation, the image is converted to a single-channel format. There are a lot of black areas around the image that do not contain any organ images, so the image is cropped into a square image of size 192×192 to make the proportion of the brain slice image in the image as large as possible. In addition, to solve the problem of low contrast of the original image, an image enhancement method based on sparrow search is proposed to make the image grayscale distribution more uniform, thereby retaining more detail information and improving image quality. It has been verified that the algorithm can be used in different situations.

[0085] 3) Evaluation indicators

[0086] In the experiment, several commonly used evaluation indicators are introduced. The prediction results can be divided into 4 categories according to the actual situation and the predicted label: true positive (TP), false positive (FP), true negative (TN) and false negative (FN). The evaluation criteria include recall (P), precision (R) and F1 score. Accuracy represents the overall performance of the model, that is, the ratio of correctly classified samples, expressed as:

[0087]

[0088] Sensitivity is the ratio of correctly classified positive samples to all positive samples, representing the ability of the model to diagnose stroke based on background information, expressed as:

[0089]

[0090] 4) Experimental results

[0091] To compare the performance of the three models on a large dataset, all three models were trained for 100 iterations on the server. To improve the accuracy of the model, a step-wise learning rate was used. The initial learning rate was set to 0.0001, and the learning rate was reduced to 0.4 times the original learning rate after every 40 iterations. The relationship between the total number of iterations and the test loss and accuracy can be found in Figure 8 As shown, Figure 8 (a) is the test loss, Figure 8 (b) is the test accuracy.

[0092] like Figure 8 As shown in the figure, all three models converged after the 40th iteration. To further compare the differences in computational time among the three models, the number of parameters of the models is listed in Table 2.

[0093] Table 2: Training results of three models

[0094]

[0095] As shown in Table 2, ResCapsNet has an accuracy of 74.34% and a loss of 0.2583. CapsNet has an accuracy of 68.19% and a loss of 0.3136, both of which are better than the accuracy and loss of ResNet (63.27% and 0.3247, respectively). In terms of computing time, the ResCapsNet network has more than 9.4 million parameters, which is roughly equivalent to CapsNet (9 million parameters) and far less than ResNet (14.8 million parameters). Although the parameters of the two capsule networks are similar, ResCapsNet converges faster and has fewer parameters. Among the three models, ResCapsNet has the best accuracy and number of parameters.

[0096] To further compare the classification performance of the three models, the model after the last iteration was selected and its performance was verified. The test set contained 54 positive samples and 54 negative samples.

[0097] In order to diagnose stroke, we hope to diagnose as many patients with true features as possible, so high sensitivity shows that the model is practical. Experiments show that among stroke patients, ResCapsNet successfully detected 45 patients with a sensitivity of 83.33%. In contrast, CapsNet detected 42 patients with positive features with a sensitivity of 77.78%. ResNet detected 36 patients with a sensitivity of 66.67%. From this perspective, ResCapsNet has the best performance among the three models.

[0098] The AUC value can be calculated from the ROC graph and used to evaluate the three models because it is less affected by the imbalance of the positive and negative sample ratios. In the experiment, the AUC value of ResCapsNet is 0.9010, the AUC value of CapsNet is 0.8702, and the AUC value of ResNet is 0.8574. Therefore, in terms of AUC value, ResCapsNet has the best performance among the three models.

[0099] In addition, the performance of ResCapsNet and the original CapsNet under different data amounts was tested. While the test set remained unchanged, 10% of the training set data was randomly extracted to obtain the test loss and test accuracy of ResCapsNet and CapsNet. Then, 10% of the data was randomly added to the training set each time, and the accuracy and loss of each experiment were recorded until the data size increased to the original size. When the training set is small, the model often stops training before reaching convergence, resulting in inaccurate classification. In order to obtain the actual performance of the model under small data amounts, the number of training iterations is set to 400 when the training data amount is less than 50%, and it remains 100 in other cases.

[0100] The accuracy and loss of the two models on the test set can be found in Fig. 9 As shown, Fig. 9 (a) is the test loss, Fig. 9 (b) is the test accuracy. The overall trend shows that as the amount of data increases in both models, the accuracy increases and the loss decreases. Detailed results are shown in Table 3. When trained with 10% of the data, the accuracy of both models is less than 50%. However, when the amount of data increases to 20%, the accuracy of ResCapsNet exceeds 50%, reaching 54.67%, while the accuracy of CapsNet only exceeds 50% when the amount of data increases to 30%. This shows that the application of the residual network improves the ability of feature extraction and helps the residual capsule network model proposed in the present invention to achieve better performance than the original CapsNet with a small amount of data.

[0101] Table 3: Comparison of accuracy and loss of ResCapsNet and CapsNet under different data amounts

[0102]

[0103] In summary, the present invention proposes a residual capsule network for extracting multiple features for classifying high-resolution brain slice DE-MRI scans. In view of complex DE-MRI image problems, multiple residual blocks (e.g., 4) are used to replace the convolutional layer in the original capsule network to extract feature information of deeper layers of the image. In addition, in the provided residual capsule network, the shallow, middle, and deep features extracted by the residual network are fused into the capsule network to achieve full recognition of image information. In view of the problem that large-size inputs cause the gradient of the capsule network to disappear and explode, the present invention effectively alleviates the problem by incorporating an improved residual network. At the same time, the use of scaled reconstruction reduces the calculation time. The present invention is particularly suitable for processing small, unbalanced, and large DE-MRI datasets.

[0104] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0105] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0106] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0107] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0108] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0110] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0111] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0112] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for classifying brain slices by combining an improved capsule network with a residual network, comprising the following steps: Acquiring a magnetic resonance image of the target to be measured; Inputting the magnetic resonance image into a trained residual capsule network to obtain a reconstructed image; obtaining classification results of brain slices using the reconstructed image; The residual capsule network includes an encoder and a decoder, the encoder includes a residual network and a capsule network, the decoder includes a deconvolution layer, the residual network is used to extract feature maps of different depths from the input image and pass them to the capsule network, the capsule network performs vectorized weighted merging on the feature maps of different depths, and the decoder uses the longest vector in the connected capsules of the capsule network to reconstruct the image.

2. The method according to claim 1, characterized in that The residual network is configured to include four residual blocks for extracting shallow feature maps, middle feature maps and deep feature maps, and the capsule network is configured to include a first primary capsule, a second primary capsule and a third primary capsule, wherein the shallow feature map is transmitted to the first primary capsule, the middle feature map is transmitted to the second primary capsule, and the deep feature map is transmitted to the third primary capsule.

3. The method according to claim 2, characterized in that The residual block includes a regularization layer, a Relu activation layer and a convolution layer, and the residual block outputs the sum of the direct path and the residual path.

4. The method according to claim 1, characterized in that: The capsule network includes a plurality of primary capsules, a plurality of corresponding digital capsules and a connection capsule, wherein each primary capsule includes a convolution module and a compression and excitation module, and the compression and excitation module obtains the weight of the digital capsule during the feature merging process.

5. The method according to claim 4, characterized in that The compression and excitation module includes a global pooling layer, a first fully connected layer, a ReLU activation layer, a second fully connected layer and a Sigmoid activation layer, wherein the global pooling layer is used to perform compression operations, and the first fully connected layer, the ReLU activation layer, the second fully connected layer and the Sigmoid activation layer are used to perform excitation operations.

6. The method according to claim 1, characterized in that The overall loss function for training the residual capsule network is set to: L=Σ(L k +α·L reconstruction ) Among them, L k represents the edge loss of the encoder, L reconstruction represents the reconstruction loss, y m represents the true value, represents the estimated value, M is the total number of samples, and the reconstruction loss is the loss between the image scaled from the original input image and the reconstructed image.

7. The method according to claim 6, characterized in that The edge loss of the encoder is set to: Among them, T k Set to 1, represents the prediction vector, λ is a set constant, m + represents the upper bound, m - Represents the lower bound.

8. The method according to claim 1, characterized in that: The magnetic resonance image is a delayed enhancement magnetic resonance image.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images