Cardiac MRI image segmentation algorithm based on ShiftViT and deformable attention

Abstract

Compared with the prior art, the cardiac magnetic resonance image segmentation algorithm based on ShiftViT and deformable attention disclosed in the present invention pays more attention to the problems of the difficulty in fine segmentation of the edge of the myocardium and the difficulty in detecting the right ventricle area in cardiac segmentation. For the first time, ShiftViT and deformable attention technology are introduced into the automatic segmentation of the heart and the two are organically combined. The ShiftViT module obtains more contextual information through offset operation to depict the small areas of the heart; the deformable attention selects a reference area that can be learned to move to depict the edge areas between organs; combined with multi-scale input and deep supervision technology, the segmentation accuracy and efficiency of cardiac segmentation are effectively improved. Experiments have shown that in the ACDC data set, the prediction results of the present invention are: the overall segmentation Dice is 90.22, the right ventricle is 88.84, the myocardium is 87.12, and the left ventricle is 94.77. And the entire network has a greatly reduced number of parameters compared to other networks based on the attention mechanism, and the computational efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to a deep neural network, and in particular to an algorithm for improving the progress and speed of cardiac magnetic resonance image segmentation by using ShiftViT and a deformable attention module, and belongs to the field of semantic segmentation. Background Art

[0002] The "China Cardiovascular Health and Disease Report 2022" released by the authoritative National Center for Cardiovascular Diseases shows that the prevalence of cardiovascular disease in China has been on the rise. At the same time, it gives an estimated number of people suffering from cardiovascular disease, which is as high as 330 million. Not only is the number of patients high, but the mortality rate of cardiovascular disease ranks first in 2020, higher than other diseases such as tumors and respiratory diseases. The mortality rate of cardiovascular disease has remained high between 2000 and 2020, and is still rising.

[0003] In addition, among urban and rural residents who died of diseases, the number of deaths from cardiovascular diseases was higher than that from other diseases. In 2020, cardiovascular diseases accounted for 48.00% and 45.86% of deaths in rural and urban areas respectively. Two out of every five deaths were caused by cardiovascular diseases. It can be seen that cardiovascular diseases are the primary threat to people's lives and health. Prevention and treatment of cardiovascular diseases are urgent tasks. Rapid and effective diagnosis of whether a patient has cardiovascular diseases is the prerequisite for preventive treatment and is also one of the current hot medical issues.

[0004] The main method for detecting cardiovascular diseases in medical institutions is to perform magnetic resonance imaging of the heart to obtain a cardiac magnetic resonance image (MRI). The imaging principle is to draw a magnetic resonance image by detecting the attenuation degree of nuclear magnetic energy of different results of the human body. This imaging not only has the characteristics of high resolution and high imaging quality, but also MRI imaging is generally three-dimensional imaging, which can analyze the image from more angles and more accurately outline the internal structure of organ tissues. In addition to MRI, there are the following widely accepted and used imaging technologies in medical institutions: Doppler echocardiography (DE), computed tomography (CT), electrocardiogram (ECG), etc. The principle of computer-level tomography is to scan the patient's part with X-rays, and the detector detects the X-rays and converts them into electrical signals after the X-rays pass through the patient's part. X-rays have a certain effect on the human body, and the patient is not completely still during the imaging process. For example, actions such as breathing will have a great impact on the imaging results, and the test results may not be accurate. Doppler echocardiography uses ultrasound to penetrate the test site and images are formed based on the reflection of ultrasound waves from human tissues. This technology is more dependent on the skills of the medical personnel who are operating it, and bones and gas also affect the imaging, which limits the imaging part. Compared with these technologies, MRI imaging has almost no effect on the human body, and the imaging quality is high, with a high degree of organ differentiation. In the field of cardiovascular disease, cardiac magnetic resonance imaging technology is considered to be the gold standard imaging diagnostic method for diagnosing cardiovascular disease.

[0005] Segmentation based on cardiac MRI is a prerequisite for the diagnosis of cardiovascular diseases. Segmentation of the target area can provide some parameters of cardiac ventricular function, such as ejection fraction (EF), ventricular end-diastolic volume (VEDV), ventricular end-systolic volume (VESV), and ventricular myocardial mass (VMW). These are very important for determining whether the heart is abnormal. Therefore, cardiac organ segmentation is a prerequisite and an important step in the diagnosis and treatment of cardiovascular diseases. In actual clinical medicine, segmentation of various cardiac tissues is performed by experienced radiologists through visual judgment and manual segmentation. Generally speaking, a cardiac cycle contains about 300 cardiac MRI images. According to a survey, it takes at least 20 minutes for an experienced radiologist to manually segment a group of cases. In most cases, the segmentation results are subjective, and the segmentation results between different doctors are different and cannot be replicated. It takes a long time. These problems are unacceptable for clinical practice. Automatic segmentation methods are very important for quantitative index analysis of the heart.

[0006] In recent years, the field of artificial intelligence has gradually entered various fields, among which computer vision has shined. The deep learning method can well learn and express the features of complex images. There have been many research results that use deep learning methods in medical image processing, including automatic segmentation of cardiac MRI images. The existing mainstream method is to use convolutional neural networks to extract features from MRI images, realize the expression ability of deep networks for cardiac segmentation, and then complete automatic segmentation of instances. In recent years, the wide application of attention mechanisms in computer vision has also introduced the use of attention mechanisms in the field of organ segmentation. There are also models and ideas that have achieved good results in the field of automatic cardiac segmentation, but some problems in this field still exist: 1. It is difficult to achieve excellent results in the detailed segmentation of organs, because the myocardium and ventricle are intertwined in the area of ​​cardiac segmentation, and the edge division of the myocardium is one of the biggest difficulties today; 2. The segmentation effect of the right ventricle is poor. In 2D segmentation, the 3D MRI image is sliced ​​according to a single surface to complete the segmentation. Most methods now segment on the horizontal plane. When slicing, the distribution area of ​​the right ventricle on some horizontal planes is much smaller than that of other segmented areas or even does not exist, which puts high demands on the learning ability of neural networks; 3. High requirements on computing resources. The popular attention-based segmentation networks and even some large model-based segmentation networks in recent years have made great progress in segmentation effects, but the number of parameters, network size, and computing time of the network itself have also increased dramatically, and there is still a long way to go in practical applications. Summary of the invention

[0007] In order to solve the current problems in cardiac segmentation, the present invention integrates deformable attention and ShiftViT modules. This method can effectively improve the accuracy of cardiac MRI segmentation, has good segmentation performance, and reduces the running requirements compared to other models. The specific scheme of the present invention is as follows:

[0008] The cardiac MRI segmentation algorithm via deformable attention and ShiftViT modules includes the following steps:

[0009] S1. Data processing;

[0010] S2. Build a newly developed cardiac MRI segmentation algorithm based on ShiftViT and deformable attention network;

[0011] S3, optimize the loss function and train the semantic segmentation algorithm model, and save the optimal model;

[0012] S4, use the optimal model to make predictions, save the prediction results, obtain evaluation indicators, and finally compare the results;

[0013] Furthermore, the experimental data in step S1 is the ACDC (Automated Cardiac Diagnosis Challenge) dataset, which was obtained from real clinical examinations by the Dijon University Hospital. The dataset covers examination results from 150 different patients, evenly divided into 4 pathological groups and 1 healthy subject group. Each patient's examination result is also accompanied by additional information, including weight, height, and end-diastolic and end-systolic records. The database is provided to participants through two datasets on a dedicated online evaluation website after personal registration: (1) The training dataset is analyzed by clinical experts and includes data from 100 patients and other reference data; (2) The test dataset consists of 50 new patients without manual annotations, but contains the above patient information. The original images are saved in Nifti format. The purpose of this challenge dataset is to compare the performance of automatic methods for segmenting the left ventricular endocardium and epicardium into the right ventricular endocardium in both end-diastolic and end-systolic conditions.

[0014] Furthermore, the content of step S2 will mainly include the encoder multi-scale input, the module design of the deformable attention module for semantic segmentation, the design of enhanced feature detail extraction based on the ShiftViT module, and the optimization of the hybrid loss function. Specifically:

[0015] S21. By adding original images of different sizes to the high-level encoder, the model's ability to understand the original details of the features is improved, so that the model will not lose the original important semantic information during deep learning;

[0016] S22. Use the ShiftViT module to characterize the extracted features at different scales. Through the offset of different numbers of channels and perceptron learning, the model can further extract features at different fine-grained scales. The results of different scales are combined, so that the model can better take into account both overall features and detailed features, solving the problem of resolving subtle and invisible parts of the right ventricle in deep networks.

[0017] S23. The deformable attention module is used in heart segmentation. Compared with the ordinary attention mechanism, it greatly reduces the number of parameters and the size of the model. In addition, the deformable attention module can better depict the edge details of the segmented area and improve the edge understanding ability of the model.

[0018] S24. Use the CE and Dice mixed loss function to find a balance between classification accuracy and edge localization, and improve the accuracy and robustness of edge segmentation.

[0019] Furthermore, in step S21, the multi-scale input refers to the addition of the original training images of the corresponding scales before the pooling convolution operation is performed in the first few layers of the U-Net encoder. Constructing the input image of the pyramid structure makes the difference between classes more obvious, and obtains more appropriate receptive field features at different scales. Adding deep supervision in the latter layers of the decoder accelerates the convergence of the network and improves the performance of the model.

[0020] Furthermore, in step S22, the ShiftViT module is used to depict small areas of the heart. Through multiple channel offset options with different scaling ratios, a small portion of the image block is horizontally and vertically translated and passed through a multi-layer perceptron learning to understand the translation relationship in the image at multiple scales. The translation operation allows ShiftViT to capture the global and local information of the image in the attention mechanism. In the segmentation task, the network can better grasp the overall features and contextual relationships in the image under the global segmentation effect without losing detailed information such as the small right ventricular part. In addition, the image undergoes multiple downsampling and upsampling in U-Net, and the image size and segmentation area will continue to change. The translation operation introduced by ShiftViT can help the model establish long-range dependencies at different locations, which helps to better capture semantic information at different scales.

[0021] Furthermore, in step S23, the deformable attention selects the position of the key and value pairs in the self-attention in a data-dependent manner, so that the self-attention module can focus on the relevant area and capture more information. The offset generation network also has the perception of local features to learn a reasonable offset, operate in a more focused area and move the window, which can effectively detect the intertwined edges between complex organs. The difference from the ordinary attention-based segmentation model is that the ordinary attention calculation involves too many keys in each block query, resulting in a much higher computational cost than the convolution mode, and the model converges slowly. The deformable attention mechanism achieves a more detailed segmentation effect while consuming much less computing resources.

[0022] Furthermore, in step S24, the CE and Dice mixed loss functions are used to calculate the difference between the predicted output of the model during training and the actual target. The edge segmentation task requires accurate capture of the edge information of the structure in the image, and mixing these two losses can find a balance between classification accuracy and edge positioning, and improve the accuracy and robustness of edge segmentation. Cross entropy loss focuses on classification tasks, and Dice Loss pays more attention to the overlap of pixel-level positions than cross entropy loss, which can improve the accuracy of the model's positioning of edge positions. By mixing these two losses, the model can pay more attention to the edge part and improve the accurate segmentation of the edge. Secondly, the mixed loss can improve the stability of the model, avoid overfitting or underfitting, and improve the generalization ability of the model for different data sets and scenarios.

[0023] Furthermore, a heart segmentation algorithm is constructed for model training, the data set is loaded for training, and then the optimal model is saved, and the trained model is used to predict the test set of the data.

[0024] The beneficial effects of the present invention are:

[0025] Compared with the prior art, the cardiac magnetic resonance image segmentation algorithm based on ShiftViT and deformable attention disclosed in the present invention pays more attention to the problems of the difficulty in fine segmentation of the edge of the myocardium and the difficulty in detecting the right ventricular area in cardiac segmentation. It is the first time to introduce ShiftViT and deformable attention technology into the automatic segmentation of the heart and organically combine the two. The ShiftViT module obtains more contextual information through offset operation to depict small areas of the heart; the deformable attention selects a reference area that can be learned to move to depict the edge area between organs; combined with multi-scale input and deep supervision technology, the segmentation accuracy and efficiency of cardiac segmentation are effectively improved. In addition, the hardware requirements of the entire network are greatly reduced compared to other parts of the network that are not based on the attention mechanism, the number of parameters is greatly reduced, and the computing efficiency is greatly improved.

[0026] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0028] Figure 1 Schematic diagram of the improved overall network architecture;

[0029] Figure 2 Schematic diagram of the ShiftViT module;

[0030] Figure 3 is a comparison of the ACDC dataset labels and network segmentation results. DETAILED DESCRIPTION

[0031] The following is further explained in detail through specific implementation methods.

[0032] In order to solve the problems of low segmentation accuracy and high computing resource requirements of existing automatic heart methods, the present invention integrates ShiftViT and deformable attention module. This method can effectively improve segmentation accuracy, greatly reduce the number of parameters and model size. The specific scheme of the present invention is as follows:

[0033] The cardiac MRI image segmentation algorithm based on ShiftViT and deformable attention module includes the following steps:

[0034] S1. Data processing;

[0035] In step S1, the ACDC dataset is saved in Nifti format. The dataset is divided into two parts: a labeled training set consisting of 100 cases and an unlabeled test set consisting of 50 cases. After converting the 3D Nifti format image into a tensor structure and reading it, the 3D data of each training set and test set case are sliced ​​according to the horizontal plane. After processing, there are 1902 slices in the training set and 1076 slices in the test set. The 2D slice images are preprocessed by random horizontal and vertical translation, random affine transformation, etc.

[0036] S2. Build a newly developed cardiac MRI segmentation algorithm based on ShiftViT and deformable attention module;

[0037] S21. By adding original images of different sizes to the high-level encoder, the model's ability to understand the original details of the features is improved, so that the model will not lose the original important semantic information during deep learning;

[0038] The U-Net network structure is used in most semantic segmentation networks, that is, a multi-level network composed of an encoder, a bottleneck layer, a decoder, and a jump connection. In medical segmentation, especially in cardiac segmentation, this network structure has become a paradigm, but in actual application, it is found that a simple one-time image input will lose detailed information such as the edges between small segmented areas and organs during multiple downsampling and feature extraction as the network deepens. The present invention adds additional original image inputs to the first few input layers of the network, so that the model still has the ability to retain the details of the original image in the deep network. Specifically, the segmentation network has a total of five layers, of which the last layer is a bottleneck layer for feature compression and semantic information transmission, and the other four layers are distributed in the decoder and encoder, symmetrically constructed, downsampled and upsampled between layers, and there are jump connections between the symmetrical decoder and encoder for better understanding of image context information. In the present invention, the first three layers of the encoder have original image inputs of different sizes. Since the image size of the network below is already very small, the features are too small relative to the image, and it is not appropriate to add original image input. The size of the first layer input is The pixel size is 960x960. The input of the second layer is reduced by half through an average pooling based on the original image. The third layer further pools the input image of the second layer to realize the input of original images of different sizes.

[0039] In addition, deep supervision is added to the last three layers of the decoder. Deep supervision refers to the introduction of supervisory signals at different levels of the network, so that the network can generate predictions at different levels and combine these predictions for training. By calculating the loss of multi-layer predictions and further adjusting the optimization of the model, the network convergence can be accelerated and the performance of the model can be improved. Supervisory signals at different levels force the network to learn multi-level feature representations. Each layer of the network can contribute to the final segmentation result, allowing the network to understand image features at different levels.

[0040] S22. Use the ShiftViT module to characterize the extracted features at different scales. Through the offset of different numbers of channels and perceptron learning, the model can further extract features at different fine-grained scales. The results of different scales are combined, so that the model can better take into account both overall features and detailed features, solving the problem of resolving subtle and invisible parts of the right ventricle in deep networks.

[0041] The translation operation was proposed as a visual operation to replace convolution very early on. The early approach was to use two Add translation in the middle of the convolution to approximate The effect of large and small convolutions greatly reduces the computational complexity, but with the rise of visual attention, translation operations can also be combined and can achieve results that are no less than ViT without calculating key-value pairs.

[0042] Specifically, before the first layer of original image is input, the Patch Partition operation is performed to divide the image into smaller pixel blocks. The present invention divides the image into 4 blocks, and then Patch Embedding is performed to divide the original size into smaller pixel blocks. The image is projected into multiple dimensions. The specific operation is to calculate the size of each pixel block, that is, , the pixel block size is used as the kernel size and stride of the convolution, while changing the number of channels , and then it is passed into the ShiftViT module. ShiftViT completes the feature extraction through simple channel translation and fully connected network. After the image has multiple channels, it first translates some channels of the image, selects a small part of the channels, and shifts them in four spatial directions. The four directions are left, right, up and down, and the rest of the channels remain unchanged. For the translated image, any excess part is discarded and the missing part is filled with 0. The translated image is normalized and passed to the full connection and added to the image before regularization to form a residual structure. As a result, the image is passed to the next module for processing.

[0043] In the present invention, the ShiftViT module is used in parallel to control the number of offset channels and offset pixel values ​​to complete feature extraction at different fine-grained scales. At the high level of the network, since the features are more complex and numerous, in order to better understand the connection information between image spaces, the number of offset channels is controlled to be relatively large. At this time, the image size is also relatively large, and the offset pixels are more, which can make more pixels mutually informative. For example, the maximum number of offset channels in the first layer is the number of medium channels. , the maximum number of pixels shifted is 3. As the number of network layers increases, the image features are more obvious, and the correlation between the related information is less. If the image size itself is shifted significantly, the feature impact is greater. For example, the maximum number of channels shifted in the deepest layer is 1 / 2 of the number of channels in the middle layer. , the maximum number of pixels to be offset is 1. Each ShiftViT module is operated in parallel and consists of 3 basic ShiftViT modules. The number of offset channels and offset pixels of the three basic ShiftViT modules are different. For example, in the first layer, the number of offset channels of the three basic ShiftViT modules are , the offset pixels are 3, 2, and 1 respectively. Such a feature extraction module builds a pyramid structure, and keeps the information capture at different scales in each layer of feature extraction. The outputs of the three ShiftViT modules are combined with the original image to form a residual structure, which is then passed through a The convolution of completes the processing of the number of channels, and the result is passed to the deformable attention module.

[0044] The number of channels in the present invention varies as follows: 32, 64, 128, 256, 512, 256, 128, 64, 32.

[0045] S23. The deformable attention module is used in heart segmentation. Compared with the ordinary attention mechanism, it greatly reduces the number of parameters and the size of the model. In addition, the deformable attention module can better depict the edge details of the segmented area and improve the edge understanding ability of the model.

[0046] In the existing segmentation model, most of the computational complexity is relatively high. The reason is that the number of key-value pairs in the attention calculation method is a power of the image size. For example, in an attention calculation, the image height and width are and , the number of channels is ,use When extracting features through convolution, the computational complexity is When using ordinary attention calculation, a key-value pair must be extracted for each pixel in the image to compare similarity, and the calculation complexity is ,in and Represent the number of queries and key values ​​in each attention calculation. When the image size is reached, the computational complexity is , such computational complexity is unacceptable with the input image size being larger or the image size being further increased in the network. The approach taken is to abandon the use of all pixels to extract key-value pairs, and instead select some pixels with features and surrounding adjacent pixels to calculate attention. In deep networks, some key-value pairs are not very useful for feature extraction. It can be said that these key-value pairs and query calculations are redundant. A part of the reference points is selected, and these reference points are learned and moved during the network training process. For each picture, the selected reference points are different, realizing data-dependent attention calculation.

[0047] Specifically, deformable attention is used as a feature extraction module of U-Net. As a reusable module, it is combined with ShiftViT in each layer to complete the task. Before entering the feature extraction, the original image is convolved twice with a convolution kernel size of , after ShiftViT, it is passed into the deformable attention module. Taking the feature extraction of the first layer as an example, the initial input image size is ,In the initial stage, the reference point of the deformable attention is randomly selected, and the query value Generated by the query matrix, , query value The reference point generation network is first passed into the reference point generation network, which is of size of Through a step size of Depthwise separable convolution, is the scaling factor, and the size of the reference area is , processed Shape , regularized and activated by The convolution of , the output size is , which is the distance the reference point moves , This is to complete a reference point movement. The method of calculating attention in the reference area is: bilinear sampling of the reference area image is performed to obtain the information used for calculation The actual , recorded as , , the obtained and Calculate self-attention, the single-head attention calculation formula is ,in is the dimension of the key-value pair, It is the relative position deviation, which is also obtained by bilinear sampling. Finally, the multi-head attention combination is used to obtain the final result. At this point, one layer of feature extraction is completed.

[0048] S24. Use the CE and Dice mixed loss function to find a balance between classification accuracy and edge localization, and improve the accuracy and robustness of edge segmentation.

[0049] The cross entropy loss function is one of the commonly used loss functions in medical image segmentation. This loss function is used to measure the difference between two probability distributions and is usually used for classification problems. In image segmentation, each pixel can be regarded as a category, and the cross entropy loss function is used to measure the difference between the model's classification prediction and the actual label for each pixel.

[0050] Specifically, for the cross entropy loss function in medical image segmentation, assuming that there are C categories, the predicted distribution of each pixel is represented by the probability vector output by the model, and the actual label is a one-hot encoded vector (corresponding to the category to which the pixel belongs). The calculation formula of the cross entropy loss function is as follows:

[0051] ;

[0052] in is the total number of pixels, is the number of categories, which is 4 in the present invention, and the 4 categories are left ventricle, right ventricle, myocardium and background area, is the actual label The pixels belong to The probability of the categories, is the model prediction signature The pixels belong to The optimization goal of the cross entropy loss function is to make the model's predicted probability distribution as close as possible to the distribution of the actual label, so as to encourage the model to learn to correctly classify each pixel into the corresponding category. It promotes the model to learn the correct category classification by comparing the difference between the model's category prediction and the actual label.

[0053] The Dice loss function is a commonly used loss function in medical image segmentation tasks. It is used to measure the degree of spatial overlap between the model's predicted segmentation results and the true labels. The mathematical definition of the Dice loss function is as follows:

[0054] ;

[0055] Where Intersection is the intersection of the model prediction result and the true label, and Union is their union. is a smoothing term used to avoid the denominator being zero. Compared with the cross entropy loss function, Dice loss is more suitable for unbalanced datasets because it focuses on the similarity of segmentation results rather than simple pixel classification. The Dice loss function focuses on the overlap between the model-predicted region and the true label, and pays more attention to the similarity of the overall segmentation results.

[0056] The combination of the two is used as the final loss function, which is defined as follows:

[0057] ;

[0058] in The factor for controlling the weight of the two is set to 0.65 in the present invention.

[0059] S3, optimize the loss function and train the target detection algorithm model, and save the optimal model;

[0060] The evaluation index of the present invention uses Dice to measure the overlap between the segmented area and the real segmented area of ​​the label, and the range is [0,1]. The larger the value, the closer the tumor segmentation result is to the labeling result, and the better the segmentation effect. The specific calculation method is as described above.

[0061] The present invention conducts ablation experiments on the ACDC dataset, adds different modules to the U-Net model in turn, proves the effectiveness of the modules, and uses the same parameter settings for experiments. The experimental results are shown in the table, where DAT represents the deformable attention mechanism, Avg represents the average segmentation results of the three regions, RV represents the right ventricle segmentation result, MYO represents the myocardium segmentation result, LV represents the right ventricle segmentation result, Params represents the parameter quantity of the model, and the maximum value is bolded.

[0062] Table 1 Comparison of experimental results of adding different modules to U-Net

[0063] ;

[0064] It can be seen from Table 1 that each improved module has improved the segmentation accuracy based on the original network, especially the segmentation effect of the myocardial area has been significantly improved, indicating that the design of the privileged extraction module of the present invention can better distinguish the edge between the myocardium and the left and right ventricles. For the right ventricle part that is sometimes difficult to detect, the segmentation effect of the model has also been slightly improved, and the number of parameters of the model has been reduced by 30%. Under the premise of improving the segmentation accuracy, the number of model operations has been greatly reduced.

[0065] It can be seen that the cardiac MRI image segmentation model based on ShiftViT and deformable attention of the present invention is superior to the cardiac MRI image segmentation model based on the ordinary attention mechanism, and the model size is further reduced. This method solves the problems of insufficient accuracy, lack of edge segmentation, and high hardware requirements in the field of automatic cardiac segmentation.

[0066] S4, use the optimal model to make predictions, save the prediction results, obtain evaluation indicators, and finally compare the results;

[0067] In Figure 3 we show a visualization of the test results for the different detectors mentioned above.

Claims

1. A cardiac magnetic resonance image segmentation method based on the ShiftViT module and deformable attention, the method comprising: S1. Data processing; S2. Build a newly developed cardiac MRI segmentation algorithm based on ShiftViT and deformable attention network; S3, optimize the loss function and train the semantic segmentation algorithm model, and save the optimal model; S4, use the optimal model to make predictions, save the prediction results, obtain evaluation indicators, and finally compare the results; The step S2 includes encoder multi-scale input, which refers to adding the original training images of the corresponding scale before the pooling convolution operation is performed on the encoders of the first few layers of U-Net, constructing the input image of the pyramid structure to make the difference between classes more obvious, and obtaining more suitable receptive field features at different scales, the module design of the deformable attention module for semantic segmentation, and the design of strengthening feature detail extraction based on the ShiftViT module, as well as the optimization of the hybrid loss function, specifically: S21. By adding original images of different sizes to the high-level encoder, the model's ability to understand the original details of the features is improved, so that the model will not lose the original important semantic information during deep learning; Specifically, the first three layers of the encoder have original image inputs of different sizes. Since the image size of the network below is already very small, the features are too small relative to the image, so it is not appropriate to add the original image input. The input size of the first layer is 224*224 pixels. The input of the second layer is reduced by half through an average pooling based on the original image. The third layer further pools the input image of the second layer to achieve original image input of different sizes. The segmentation network has five layers, among which the last layer is the bottleneck layer for feature compression and semantic information transmission. The other four layers are distributed in the decoder and encoder, with a symmetrical structure. The layers are downsampled and upsampled. There are jump connections between the symmetrical decoder and encoder, and the jump connections are used to better understand the image context information. In addition, deep supervision is added to the last three layers of the decoder. Deep supervision refers to the introduction of supervisory signals at different levels of the network, so that the network can generate predictions at different levels and combine these predictions for training. The model optimization is further adjusted by calculating the loss of multi-layer predictions, which accelerates the convergence of the network and improves the performance of the model. Supervisory signals at different levels force the network to learn multi-level feature representations. Each layer of the network contributes to the final segmentation result, allowing the network to understand image features at different levels. S22, use the ShiftViT module to characterize the extracted features at different scales, through the offset of different channel numbers, perceptron learning, specifically, before the first layer of original image input, first perform PatchPartition operation to divide the image into 4 blocks, and then perform PatchEmbedding to project the original image of size H*W into multiple dimensions; the specific operation is to calculate the size of each pixel block, that is, The pixel block size is used as the kernel size and step size of the convolution, and the number of channels C is changed at the same time, and then it is passed to the ShiftVit module; after the image has multiple channels, some channels of the image are first translated, and a small part of the channels are selected and offset by n pixels in four spatial directions, namely left, right, up and down. The other channels remain unchanged. If there is any excess part of the image after translation, it is discarded, and the missing part is filled with 0. The regularization of the image after translation is only enough to be passed into the full connection and the image before regularization is added to form a residual structure as a result, and the image is passed to the next module for processing; By using ShiftViT modules in parallel, the number of offset channels and offset pixel values ​​are controlled to complete feature extraction at different fine-grained scales. Three basic ShiftViT modules are used for parallel operation. The number of offset channels and offset pixels of the three basic ShiftViT modules are different. In the first layer, the number of offset channels of the three basic ShiftViT modules are respectively The offset pixels are 3, 2, and 1 respectively. This feature extraction module builds a pyramid structure, and keeps the information capture at different scales in each layer of feature extraction. The outputs of the three ShiftViT modules are combined with the original image to form a residual structure. The number of channels is processed through a 3*3 convolution, and the result is passed to the deformable attention module. S23. The deformable attention module is used in heart segmentation. Specifically, the deformable attention is used as a feature extraction module of U-Net. As a reusable module, it is combined with ShiftViT in each layer to complete the task. Before entering the feature extraction, the original image is convolved twice with a convolution kernel size of 3*3. After ShiftViT, it is passed to the variable attention module. The feature extraction of the first layer, the initial input image size is 224*224. At the beginning, the reference point of the deformable attention is randomly selected, and the query value q is generated by the query matrix, q=xW q , the query value q is passed into the reference point generation network. The reference point generation network first passes q of size H*W*C through a depth-separable convolution with a step size of r, where r is the scaling factor and the size of the reference area is H G =H / r,W G =W / r, the processed q shape is H G ×W G ×C, regularized, activated, and then passed through a 1*1 convolution, the output size is H G ×W G ×2, which is the distance Δp moved by the reference point, and p+Δp is the distance of completing one reference point movement. The way to calculate attention in the reference area is: perform bilinear sampling on the reference area image to obtain the actual x used to calculate v and k, recorded as Conclusion The self-attention is calculated with q, and the single-head attention calculation formula is: Where d is the dimension of the key-value pair, is the relative position deviation, which is also obtained by bilinear sampling. Finally, the multi-head attention combination is used to obtain the final result. At this point, the feature extraction of one layer is completed. S24, using CE and Dice mixed loss function to find a balance between classification accuracy and edge localization, and improve the accuracy and robustness of edge segmentation; In image segmentation, each pixel is considered as a category, and the cross entropy loss function is used to measure the difference between the model's classification prediction for each pixel and the actual label; Specifically, for the cross entropy loss function in medical image segmentation, assuming there are C categories, the predicted distribution of each pixel is represented by the probability vector output by the model, and the actual label is a one-hot encoded vector corresponding to the category to which the pixel belongs. The calculation formula of the cross entropy loss function is as follows: Where N is the total number of pixels, C is the number of categories, and the categories are left ventricle, right ventricle, myocardium, and background area. ij is the probability that the ith pixel of the actual label belongs to the jth category, is the probability that the i-th pixel of the model prediction belongs to the j-th category; The Dice loss function is a commonly used loss function in medical image segmentation tasks. It is used to measure the degree of spatial overlap between the model's predicted segmentation results and the true labels. The mathematical definition of the Dice loss function is as follows: Where Intersection is the intersection of the model prediction result and the true label, Union is their union, and ∈ is a smoothing term used to avoid the denominator being zero; The two are combined as the final loss function, defined as follows: Loss=λCE Loss+(1-λ)Dice Loss Among them, λ is the factor that controls the weight of the two.

Images