A stroke lesion segmentation method based on combined image registration and segmentation

Abstract

The present invention relates to the field of image processing technology, and particularly to a stroke lesion segmentation method combining image registration and segmentation. In the method, a joint network segmentation model is constructed, which includes an image registration network and a 3DMI-UNet segmentation network. Medical images of patients and fixed template images are obtained to form training sample pairs, and the training sample pairs are used to train the joint network segmentation model. The trained joint network segmentation model is used to perform stroke lesion segmentation on the medical images of the patients to be examined. The present invention combines image registration and segmentation to design a joint framework, and uses a joint loss function to achieve optimization to complete stroke lesion segmentation.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a stroke lesion segmentation method combining image registration and segmentation. Background Art

[0002] Stroke is one of the most common cerebrovascular diseases in the world. It is universal, highly dangerous, and has a high incidence rate. The main cause of stroke is the interruption or reduction of blood supply to the brain. If not treated in time, brain cells die rapidly, which may seriously damage the nervous system and even lead to severe disability or death.

[0003] Professional stroke rehabilitation can effectively reduce these long-term disabilities. Lesion size, location, and overlap with existing brain regions have been successfully used as predictors in stroke recovery and rehabilitation prediction. However, this has only been conducted in smaller studies, and the results may conflict between large-scale studies or be limited to each sample. Neuroimaging can measure brain structure, which helps physicians understand structural changes in the brain after stroke and thus predict functional changes. In the chronic stroke stage, high-resolution T1-weighted structural magnetic resonance imaging can accurately and noninvasively determine structural changes in the brain after stroke. The overlap between stroke lesions and brain regions can effectively reflect the relationship between brain changes and behavioral manifestations, which requires the segmentation of stroke lesions. Therefore, achieving accurate lesion segmentation is a key obstacle to correctly analyzing large-scale stroke datasets and completing outcome predictions. Manual labels drawn by professional radiologists are considered the gold standard. However, stroke lesion boundaries are generally fuzzy, and even manual labeling can be inaccurate and unreliable. Considering that manual labeling is overly labor-intensive and introduces subjective differences between observers, manual segmentation is no longer suitable for a wide range of data needs. Automatic segmentation algorithms have the advantages of being repeatable, scalable, and not requiring manual parameter setting. With the development of artificial intelligence, they are becoming increasingly popular in medical image processing.

[0004] In recent years, deep learning algorithms have garnered widespread attention for their success in solving numerous medical image registration and segmentation problems, achieving advanced results. Deep learning algorithms, through layered networks, extract abstract features from data at varying levels. These algorithms not only enable machines to extract features from data independently, but also achieve greater objectivity and efficiency than traditional machine learning algorithms. Neural networks are a prime example, demonstrating their success in numerous tasks. Convolutional neural networks (CNNs) are specialized for processing 2D and 3D data, including images. CNNs are primarily trained and optimized based on the backpropagation algorithm. Through a series of convolutions, pooling, and other structures, CNNs map feature representations of sample distributions to a sample label space. Furthermore, the shared weights of CNNs, along with the introduction of local receptive fields and downsampling techniques during feature extraction, allow CNNs to adapt to a wide range of image data types and sizes. Consequently, CNNs have become one of the most representative algorithms in deep learning.

[0005] Image registration is a challenging task in medical image analysis, particularly since the size and location of stroke lesions vary significantly between individuals. The goal of image registration is to determine a transformation that maps two images (moving and fixed) to the same coordinate system. In practice, we seek an image mapping function that changes the coordinate system of the moving image to that of the fixed image. Deformable registration or elastic registration is the most commonly used approach. Linear methods are an alternative, but in this case, a linear global transformation of the entire image is required. Recently, several deep learning-based methods have been proposed that significantly reduce computational time while maintaining registration accuracy and robustness. In particular, Dalca et al. proposed a deep learning framework for atlas-based registration of brain MR images, and Christodulidis et al. proposed a parallel scheme for linear and deformable registration of lung MR images. However, these methods struggle with abnormal anatomical structures, such as tumor regions, because they lack similarity between the tumor region and normal brain regions, making them unable to register images at specific locations. This often results in distortion of the tumor region and its surroundings in the deformed image. Summary of the Invention

[0006] The present invention aims to achieve accurate segmentation of stroke lesions through the synergistic relationship between the two tasks of registration and segmentation, and an end-to-end network framework. To this end, a stroke lesion segmentation method combining image registration and segmentation is provided. In this method, a joint network segmentation model is constructed, which includes an image registration network and a 3D MI-UNet segmentation network. A medical image of a patient and a fixed template image are obtained to form a training sample pair, and the training sample pair is used to train the joint network segmentation model. The trained joint network segmentation model is used to perform stroke lesion segmentation on the medical image of the patient to be examined.

[0007] The training process of the joint network segmentation model includes the following steps:

[0008] S1. Select a patient's medical image and a fixed template image and input them into a shared encoder to obtain an encoded medical image and an encoded template image;

[0009] S2. Input the encoded medical image and the encoded template image independently into the 3D MI-UNet segmentation network to obtain the medical image mask and template image mask, respectively. The network is trained using a combined dice and cross-entropy loss function.

[0010] S3. Projecting the encoded medical image onto the encoded fixed image in the image registration network to obtain a deformation field; and training the network using a smoothing regularization function;

[0011] S4. Apply the deformation field of step S3 to the medical image mask in step S2, compare the transformed medical image mask with the template image mask, and use the anatomical similarity loss function for training optimization.

[0012] Furthermore, the joint loss function of the joint network segmentation model is expressed as:

[0013] L=L seg +βL reg +L a

[0014] Among them, L seg Denotes the combined loss function of dice and cross entropy, L reg represents the smooth regularization function, L a represents the anatomical similarity loss function, and β is the influence weight of the network joint optimization.

[0015] Furthermore, the combined loss function of dice and cross entropy is expressed as:

[0016] L seg =L Dice +L CE

[0017]

[0018]

[0019] Among them, L Dice Represents the loss function based on the dice similarity coefficient, L CE represents the cross entropy loss function, N represents the total number of pixels, s represents the smoothing term, N represents the total number of pixels in the medical image, and p n represents the predicted medical image mask, i.e. the nth pixel of the predicted segmentation map of the medical image; g n Represents the binary ground truth value corresponding to the n-th pixel of the medical image mask.

[0020] Furthermore, the image registration network aims to calculate the optimal deformation field corresponding to the smooth regularization hyperparameters and constructs the objective function, which is expressed as:

[0021]

[0022] in, represents the deformation field, L sim (·,·) represents the dissimilarity function, L reg (·) represents the smoothing regularization function, F represents the fixed template image, represents the medical image of the patient after the deformation field, λ p Indicates uniform sampling within a predefined range, and according to experience, λ p The predefined range of is set to [0, 10].

[0023] Furthermore, the joint network segmentation model adopts a deep Laplacian pyramid image registration network, in which a multi-level Laplacian pyramid framework is built. Based on the objective function, the optimization problem of each level in the multi-level Laplacian pyramid framework is established, which can be expressed as:

[0024]

[0025] in, Deformation field Smoothing regularization, NCC ω () represents the local normalized cross-correlation with a window size of ω, F i represents the template image of the i-th level input in the Laplacian pyramid framework, represents the moving image of the i-th level input in the Laplacian pyramid framework, and l represents the total number of levels of the Laplacian pyramid framework.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention provides a stroke lesion segmentation method that combines image registration and segmentation. Previous methods perform prior knowledge and lesion segmentation in steps. The present invention effectively combines the two tasks by utilizing the synergistic relationship between the two tasks, and solves stroke lesion segmentation in an end-to-end manner. Ultimately, the two tasks supervise and promote each other, thereby achieving more accurate segmentation performance than a single segmentation task or independent registration and segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a framework diagram of a joint network segmentation model according to an embodiment of the present invention;

[0029] Figure 2 This is a training flow chart of the joint network segmentation model according to an embodiment of the present invention;

[0030] Figure 3 This is a diagram showing the structure of a deep Laplacian pyramid image registration network according to an embodiment of the present invention;

[0031] Figure 4 This is a structural diagram of a conditional image registration module according to an embodiment of the present invention;

[0032] Figure 5 This is a diagram of the 2D MI-UNet segmentation network structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] The present invention provides a stroke lesion segmentation method combining image registration and segmentation, in which a Figure 2 The joint network segmentation model shown in FIG. comprises a shared encoder, a segmentation decoder, and a registration decoder. The segmentation decoder also represents a 3D MI-UNet segmentation network, and the registration decoder also represents an image registration network. The present invention trains the joint network segmentation model by acquiring patient medical images and fixed template images to form training sample pairs. The trained joint network segmentation model is then used to segment stroke lesions in the medical images of the patient being examined.

[0035] In one embodiment, if Figure 1 、 Figure 2 As shown in Figure 2, the training process of the joint network segmentation model includes the following steps:

[0036] S1. Select a patient's medical image M and a fixed template image F and input them into a shared encoder to obtain an encoded medical image and an encoded template image.

[0037] S2. Input the encoded medical image and the encoded template image independently into the 3D MI-UNet segmentation network (SegDecoder) to obtain the medical image mask and template image mask And the combined loss function of dice and cross entropy is used for training;

[0038] S3. Project the encoded medical image onto the encoded fixed image in the image registration network (Reg Decoder) to obtain the deformation field And use smooth regularization function for training;

[0039] S4. The deformation field of step S3 Medical image mask used in step S2 Mask the transformed (Warp) medical image With template image mask Comparison, using the anatomical similarity loss function L a Perform training optimization.

[0040] Specifically, in the 3DMI-UNet segmentation network, a combined loss function of dice and cross entropy is used for training, which is expressed as:

[0041] L seg =L Dice +L CE

[0042] The volume imbalance between stroke lesions and normal brain tissue makes segmentation challenging. To avoid the influence of a large number of background pixels, the present invention introduces a loss based on the dice similarity coefficient that has achieved satisfactory performance in the prostate segmentation task.

[0043] The dice similarity coefficient is mainly used to measure the similarity between different samples. It is a similarity measurement function between sets. The larger the coefficient, the better the separation efficiency. The calculation expression of the dice coefficient is as follows:

[0044]

[0045] V seg represents the predicted segmentation volume, V gtRepresents the actual segmentation volume. In medical image segmentation tasks, there is often a strong size imbalance between foreground and background, with the background typically occupying over 95% of the entire volume. However, the foreground is of greater concern to us; it is the object of interest. Therefore, the target voxel here is the foreground. The dice score ranges from 0 to 1, where 0 indicates no spatial overlap and 1 indicates complete overlap.

[0046] The loss function based on the dice similarity coefficient is expressed as:

[0047]

[0048] Where, · represents the scalar product, N represents the total number of pixels in the medical image, and P = {p1,…,p n ,…,p N}, p n represents the medical image mask, i.e., the nth pixel of the predicted segmentation map of the medical image; G = {g1,…,g n ,…,g N}, g n Represents the binary ground truth value corresponding to the nth pixel of the medical image mask. In order to avoid division by 0, that is, both P and G are empty, the smoothing term s=10 -8 Used for dice losses.

[0049] The cross entropy loss function used in this example is defined as:

[0050]

[0051] In one embodiment, the 3D MI-UNet segmentation network is an improved version based on the 2D MI-Unet network framework. Figure 5As shown in the figure, it has a U-shaped structure, consisting of a traditional feature extraction path (referred to as the compression path) and a restoration path (referred to as the expansion path). Each convolutional layer is followed by batch normalization (BN) and rectified linear units (ReLU). The compression path, considered as the encoder, consists of four convolutional blocks, each followed by a 2×2 max pooling layer to halve the resolution of the corresponding feature map. The expansion path, considered as the decoder, consists of four 2×2 upsampling layers, each followed by a convolutional block to restore the resolution and generalize the feature map. Skip connections are used between the encoder and decoder parts of the same resolution. To obtain the final binary segmentation, a 1×1 convolution is used in the final layer to map the 32-component feature vector to two class labels. A similar encoder-decoder structure is used for the 3D MI-UNet, consisting of seven convolutional blocks, three max pooling layers, and three upsampling layers. Due to the smaller input size of the 3D MI-UNet, the max pooling and upsampling layers in the 3D MI-UNet are one layer smaller than those in the 2D MI-UNet. This is also the reason why fewer feature channels are used in 3D MI-UNet.

[0052] In one embodiment, the image registration network aims to calculate the optimal deformation field corresponding to the smoothing regularization hyperparameter and constructs an objective function, which is expressed as:

[0053]

[0054] in, represents the deformation field, L sim (·,·) represents the dissimilarity function, L reg (·) represents the smoothing regularization function, F represents the fixed template image, represents the medical image of the patient after the deformation field, λ p Represents a predefined range for uniform sampling.

[0055] In this embodiment, λ p The predefined range of is set to [0, 10] so that in most cases, the p The optimal deformation field is a diffeomorphic one. The difference between the objectives of common deep learning-based unsupervised image registration methods and those of this embodiment is that this embodiment learns to optimize the objective function within a predefined hyperparameter range, rather than fixing the hyperparameter values. To illustrate the learning paradigm proposed in this embodiment, we follow the Laplacian pyramid network-based large deformation differential image registration and instantiate the objective function using pyramid-like and diffusion regularizers on the spatial gradient of the deformation field. We also employ a progressive training scheme to train the network in a coarse-to-fine manner.

[0056] Solving the image registration problem using an intensity-based similarity measure at the finest resolution typically results in a local minimum solution. By leveraging the fact that perfectly aligned image pairs will produce high similarity values ​​at all resolutions, this embodiment proposes a similarity pyramid framework to address this challenge. For simplicity, it is expressed using local normalized cross correlation (NCC). The proposed similarity pyramid formula is:

[0057]

[0058] Among them S K (·,·) represents a similarity pyramid with K levels, NCC ω represents the local normalized cross-correlation with a window size of ω, and (F i ,M i ) represents an image in the image pyramid (F1 is the image with the lowest resolution). Lower weights are assigned to similarity values ​​with lower resolutions to prevent similarity from dominating at lower levels. The proposed similarity pyramid captures similarity in a multi-resolution manner. Since similarity metrics are smoother and less sensitive to noise at coarse resolutions, integrating similarity metrics from lower levels helps avoid local minima in high-resolution optimization problems.

[0059] Specifically, an L-level Laplacian pyramid framework is built in the deep Laplacian pyramid image registration network, and CRN (CNN-based registration network) is used to solve the optimization problem of each level in the L-level Laplacian pyramid framework. The optimization problem of each level is defined as:

[0060]

[0061] where λ p uniformly sampled in [0, 10] for each iteration, NCC ω (·,·) represents the local normalized cross-correlation (NCC) with a window size of ω, where ω is set to 1+2i. It is worth noting that the learning paradigm proposed in this embodiment does not impose additional computational cost on the original objective function and can be easily transferred to various deep learning-based image registration applications.

[0062] The architecture of CRN is the same in all pyramid levels. CRN consists of three parts: feature encoder, a set of (N) conditional image registration modules and feature decoder. Figure 3As shown in , the feature encoder consists of two 3D convolutional layers of 3×3×3 with a stride of 1 and one 3D convolutional layer of 3×3×3 with a stride of 2; the conditional image registration module takes the necessary low-level feature maps and regularization hyperparameters extracted by the feature encoder for deformable image registration as input, and outputs hidden features with shifted feature statistics based on conditional instance normalization (CIN); specifically, the proposed conditional image registration module adopts a pre-activation structure, including two conditional instance normalization (CIN, Conditional Instance Norm) layers, each followed by a LeakyReLU activation with a negative slope of 0.2 and a convolutional layer with 28 filters, as shown in Figure 4 As shown, skip connections are added to preserve feature identity. The feature encoder extracts the necessary low-level features for deformable image registration, while the feature decoder upsamples and outputs the target deformation field. The present invention only conditions the hidden layer in each conditional image registration module as a hyperparameter of the smoothing regularization. In the implementation, L and N are set to 3 and 5, respectively.

[0063] In order to maintain the diversified conditional representation of the hyperparameters of each hidden layer, the conditional image registration module of the present invention adopts a distributed mapping network, such as Figure 4 As shown, the network learns a separate intermediate nonlinear latent variable for each conditional image registration module, which is shared among all CIN layers. Formally, the normalization regularization hyperparameter is given by Distributed mapping network g: First, λ is mapped to a latent code z∈Z. Then, the CIN layer learns a set of parameters that specialize Z as a regularized smoothness. The distributed mapping network is parameterized using a 4-layer multilayer perceptron (MLP). For simplicity, the number of perceptrons in each MLP layer and the dimension of the latent space are set to 64. The intermediate layers of the distributed mapping network use LeakyReLU activation to further introduce nonlinearity into the latent code.

[0064] Each feature map (hidden feature) h i The CIN operation is defined as:

[0065]

[0066] where γ θ,i , β θ,i ∈R are the affine parameters learned from the latent code z, μ(h i ),σ(h i )∈R is the feature map h of channel i i In other words, the smoothing regularization is controlled by using the corresponding affine parameter γ for each channel in the hidden feature map h.θ,i and β θ,i The feature statistics of the feature map are normalized and transformed to learn.

[0067] Specifically, given a fixed 3D scan F (fixed template image) and a moving 3D scan M (medical image of the patient), a diffeomorphic deformation field is estimated in a Laplacian pyramid network. Make the distorted moving scan Align with F and smooth the predicted velocity field v as the goal, and use the Laplacian pyramid network to parameterize the deformable registration problem as a function where θ represents the learning parameters in the network.

[0068] Specifically, if Figure 4 As shown in FIG, an L-level Laplacian pyramid architecture is used to implement the Laplacian pyramid network to simulate the traditional multi-resolution strategy. For simplicity, L is set to 3 in this embodiment. First, the input image pyramid is created by downsampling the input image (fixed 3D scan F and moving 3D scan M) using trilinear interpolation to obtain F i ∈{F1,F2,F3} and M i ∈{M1, M2, M3}, where F i Indicates a scale factor of 0.5 (L-i) Downsampling, and F3=F.

[0069] For the first level in the pyramid framework, CRN captures nonlinear misalignment from the concatenated input scans with the coarsest resolution (F1 and M1) and outputs a 3-channel dense vector field v1 and a deformation field

[0070] For the i-th level in the pyramid framework, i>1, first set the scale factor to 2 for the deformation field output by the previous pyramid level Upsampling is performed to obtain And use Twisted M i Obtaining a distorted moving image Then use the same scaling factor to adjust the velocity field v output by the previous pyramid level. i-1 Upsampling and will With input scan (F i and ) are connected to form a 5-channel input of a stage i CRN.

[0071] Combine the output of level i (velocity field) with the upsampled Add to get the velocity field v i , for the velocity field vi Integrate to produce the final deformation field at pyramid level i The features of the lower-level CRN embeddings are added to the next level via skip connections, which greatly increases the receptive field as well as the nonlinearity of the network to learn complex nonlinear correspondences at finer levels.

[0072] Training strategy: The proposed method was evaluated on the ATLAS dataset consisting of 229 subjects. All images were resampled and normalized to 1mm3 voxel resolution, with a volume size of 233×197×189. All experiments were performed in a five-fold cross-validation manner. For data partitioning, all 229 cases were first indexed by name, and then the training data and test data were separated according to the index. More specifically, data with indices {1, 6, 11,…, 226}, {2, 7, 12,…, 227}, {3, 8, 13,…, 228}, {4, 9, 14,…, 229}, {5, 10, 15,…, 225} were used to test partitions 1 to 5, respectively, and the others were used for training. The test data in each partition covered different lesion sizes. Through five-fold cross-validation, each data was used as test data once, and the experimental results were compared with a separate segmentation network and a traditional registration plus segmentation network.

[0073] In the present invention, unless otherwise clearly stipulated and limited, the terms "installation", "setting", "connection", "fixation", "rotation" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0074] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A stroke lesion segmentation method combining image registration and segmentation, characterized in that: A joint network segmentation model is constructed, comprising an image registration network and a 3D MI-UNet segmentation network. Medical images of the patient and fixed template images are used to form training sample pairs, which are then used to train the joint network segmentation model. The trained joint network segmentation model is used to segment stroke lesions in the medical images of the patient being examined. The training process of the joint network segmentation model includes the following steps: S1. Select a patient's medical image and a fixed template image and input them into a shared encoder to obtain an encoded medical image and an encoded template image; S2. Input the encoded medical image and the encoded template image independently into the 3D MI-UNet segmentation network to obtain the medical image mask and template image mask, respectively. The network is trained using a combined dice and cross-entropy loss function. S3. Projecting the encoded medical image onto the encoded fixed image in the image registration network to obtain a deformation field; and training the network using a smoothing regularization function; The image registration network aims to calculate the optimal deformation field corresponding to the smooth regularization hyperparameters and constructs the objective function, which is expressed as: in, represents the deformation field, L sim (·,·) represents the dissimilarity function, L reg (·) represents the smoothing regularization function, F represents the fixed template image, represents the medical image of the patient after the deformation field, λ p Represents a predefined range of uniform sampling, and according to experience, λ p The predefined range of is set to [0, 10]; The joint network segmentation model adopts a deep Laplacian pyramid image registration network, in which a multi-level Laplacian pyramid framework is built. Based on the objective function, the optimization problem of each level in the multi-level Laplacian pyramid framework is established, which can be expressed as: in, Deformation field Smoothing regularization, NCC ω () represents the local normalized cross-correlation with a window size of ω, F i represents the template image of the i-th level input in the Laplacian pyramid framework, represents the moving image of the i-th level input in the Laplacian pyramid framework, and l represents the total number of levels of the Laplacian pyramid framework; A CNN-based registration network is used to solve the optimization problem at each level in the L-level Laplacian pyramid framework. The CNN-based registration network consists of three parts: a feature encoder, a conditional image registration module, and a feature decoder. The feature encoder extracts the necessary low-level features for deformable image registration. The conditional image registration module takes the necessary low-level feature maps extracted by the feature encoder for deformable image registration and regularization hyperparameters as input and outputs hidden features with shifted feature statistics based on conditional instance normalization. The feature decoder upsamples and outputs the target deformation field. S4. Apply the deformation field of step S3 to the medical image mask in step S2, compare the transformed medical image mask with the template image mask, and use the anatomical similarity loss function for training optimization.

2. The stroke lesion segmentation method combining image registration and segmentation according to claim 1, characterized in that: The joint loss function of the joint network segmentation model is expressed as: L=L seg +βL reg +L a Among them, L seg Denotes the combined loss function of dice and cross entropy, L reg represents the smooth regularization function, L a represents the anatomical similarity loss function, and β is the influence weight of the network joint optimization.

3. The stroke lesion segmentation method combining image registration and segmentation according to claim 1 or 2, characterized in that: The combined loss function of dice and cross entropy is expressed as: L seg =L Dice +L CE Among them, L Dice Represents the loss function based on the dice similarity coefficient, L CE represents the cross entropy loss function, N represents the total number of pixels in the medical image, s represents the smoothness term, and p n represents the medical image mask, i.e., the nth pixel of the predicted segmentation map of the medical image, g n Represents the binary ground truth value corresponding to the n-th pixel of the medical image mask.

Images