Medical image segmentation and imaging methods for ischemic stroke based on large model fine-tuning

Abstract

The present invention discloses a method for ischemic stroke medical image segmentation based on large model fine-tuning, including obtaining existing ischemic stroke medical images and pre-processing to construct a training data set; constructing an ischemic stroke medical image segmentation primary model including an image symmetry processing network, an image coding network, an image embedding network and an image segmentation network and training to obtain an ischemic stroke medical image segmentation model; obtaining actual ischemic stroke medical images and inputting them into the ischemic stroke medical image segmentation model to complete the segmentation of actual ischemic stroke medical images. The present invention also discloses an imaging method including the ischemic stroke medical image segmentation method based on large model fine-tuning. Based on the construction and processing of the image symmetry processing network to generate symmetric data, and based on the symmetric data and the constructed image embedding network and image segmentation network, not only the segmentation of ischemic stroke medical images is finally achieved, but also the reliability of the segmentation is higher and the accuracy is better.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and in particular relates to an ischemic stroke medical image segmentation method and an imaging method based on large model fine-tuning. Background Art

[0002] Medical image segmentation is the basis of various medical image applications and plays an extremely important role in the clinical process of medicine and the basic research process of medicine. Brain images play an important role in both the clinical stage and the basic research stage of ischemic stroke; therefore, the segmentation of medical images of ischemic stroke is of great significance for clinical medicine and basic medicine.

[0003] Traditional medical image segmentation schemes for ischemic stroke usually adopt a scheme that directly performs end-to-end hierarchical feature representation learning, such as the scheme designed by Deniz Alis et al. based on U-Net, residual modules and convolutional long-short cycle memory modules, and the scheme based on multimodal CT images and dedicated residual U-Net network designed by Albert Clèrigues et al. Such U-Net schemes and their variants of the U-type CNN encoder-decoder architecture have achieved certain segmentation effects. However, such schemes have limited representation and generalization capabilities and are usually limited to specific small data sets. Recently, medical image segmentation models based on fine-tuning of the SAM large model have achieved stronger representation and generalization capabilities. For example, the MSA model adopts the scheme of fine-tuning the SAM encoder and decoder by Adapter, and the MedSAM model adopts the scheme of fine-tuning the SAM decoder. However, these schemes do not introduce specific clinical prior knowledge in the field of stroke segmentation (for example, bilateral asymmetry is often used in the clinical stage to judge the suspected ischemic stroke area), nor do they model the continuity of lesions between adjacent slices. Therefore, such schemes still have the problems of low reliability and poor accuracy. Summary of the invention

[0004] One of the purposes of the present invention is to provide an ischemic stroke medical image segmentation method based on large model fine-tuning with high reliability and good accuracy.

[0005] A second object of the present invention is to provide an imaging method including the ischemic stroke medical image segmentation method based on large model fine-tuning.

[0006] The ischemic stroke medical image segmentation method based on large model fine-tuning provided by the present invention comprises the following steps:

[0007] S1. Obtain existing medical images of ischemic stroke;

[0008] S2. Preprocessing the medical image data obtained in step S1 to construct a training data set; the image preprocessing includes removing the skull area, setting the window width and window position, image cropping, image resampling and image standardization;

[0009] S3. Construct an image symmetry processing network and an image encoding network based on convolutional layers, construct an image embedding network based on Transformer blocks, and construct an image segmentation network based on the attention mechanism. Finally, a primary model of ischemic stroke medical image segmentation is constructed, which includes an image symmetry processing network, an image encoding network, an image embedding network, and an image segmentation network.

[0010] S4. Using the training data set obtained in step S2, the primary model for ischemic stroke medical image segmentation constructed in step S3 is trained to obtain an ischemic stroke medical image segmentation model;

[0011] S5. Acquire an actual ischemic stroke medical image and input it into the ischemic stroke medical image segmentation model obtained in step S4 to complete the segmentation of the actual ischemic stroke medical image.

[0012] The image preprocessing described in step S2 specifically includes the following steps:

[0013] The skull in the image is removed using a convex optimization method;

[0014] According to the set requirements, set the window width and window position to improve the accuracy of image segmentation;

[0015] Crop the image to the area containing brain tissue to improve segmentation efficiency;

[0016] Resample the cropped image to unify the spatial resolution of the image;

[0017] The Z-Score method was used to standardize the images;

[0018] Finally, the image preprocessing process is completed.

[0019] The step S3 comprises the following steps:

[0020] The primary model for ischemic stroke medical image segmentation includes an image symmetry processing network, an image encoding network, an image embedding network and an image segmentation network connected in series in sequence;

[0021] An image symmetry processing network is constructed based on the convolution module; the image symmetry processing network is used to convert the input image into a bilateral quasi-symmetric image in the axial view for subsequent image segmentation;

[0022] An image coding network is constructed based on the convolutional layer; the image coding network is used to embed and positionally encode the image output by the image symmetry processing network in blocks to maintain the relative position information between image blocks;

[0023] An image embedding network is constructed based on the Transformer block; the image embedding network is used to process the position-encoded image to obtain image embedding data;

[0024] An image segmentation network is constructed based on the attention mechanism; the image segmentation network is used to process the image embedding data to obtain the final segmentation results of ischemic stroke medical images.

[0025] The image symmetry processing network based on the convolution module is constructed, which specifically includes the following steps:

[0026] In the input image, three consecutive slices are randomly selected as a group and input into the image symmetry processing network;

[0027] The constructed image symmetry processing network includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, an average pooling layer, a fully connected layer and an activation function layer which are connected in series in sequence;

[0028] The first convolution module, the second convolution module, the third convolution module and the fourth convolution module are all used to extract features; the average pooling layer is used to reduce the dimension of the features; the fully connected layer is used to map the features to the affine transformation parameters; the activation function layer is used to limit the output range to -1 to 1; the structures of the first convolution module, the second convolution module, the third convolution module and the fourth convolution module are all the same; the convolution module includes a downsampling 3D convolution layer, a first group normalization layer, a first ReLU layer, a first 3D convolution layer, a second group normalization layer, a second ReL layer, which are connected in series in sequence. The U layer and the second 3D convolution layer have residual connections from input to output in the process of the first group normalization layer, the first ReLU layer, the first 3D convolution layer, the second group normalization layer, the second ReLU layer and the second 3D convolution layer; the convolution kernel of the downsampling 3D convolution layer is 3, the step size of the depth dimension convolution is 1, the step size of the height dimension convolution is 2, and the step size of the width dimension convolution is 2; the first group normalization layer and the second group normalization layer have the same structure, and are both used to solve the internal covariate shift in the training process; the calculation formula of the group normalization layer is expressed as , where Y is the normalized feature and X is the input feature after grouping by channel. is the mean of the feature graph of this group, is the variance of the feature map of this group, A constant to prevent division by zero errors. is the first parameter to be learned, is the second parameter to be learned; the first ReLU layer and the second ReLU layer have the same structure and are both used to enhance the nonlinear representation ability of the model; the calculation formula of the ReLU layer is expressed as ,in is the value after being processed by the ReLU layer, is the i-th value in the feature map; the first 3D convolutional layer and the second 3D convolutional layer have the same structure and are both used for feature extraction; the convolution kernel of the 3D convolutional layer is 3, the stride is 1, and the padding is 1;

[0029] The activation function layer uses the tanh function as the activation function; the calculation formula of the tanh function is expressed as , where is the value after the activation function, is the i1th value in the image;

[0030] The processing process of the image symmetry processing network is expressed as Where View is the output of the image symmetry processing network; Tanh represents the processing process of the activation function layer; FC represents the processing process of the fully connected layer; Avpool represents the processing process of the average pooling layer; Fea is the output of the fourth convolution module;

[0031] Convert the View to The affine matrix M and the corresponding inverse matrix ; Perform affine transformation on the three slices X of the input image through the affine matrix M to obtain three symmetrical image slices ; Symmetrical image slices The image is bilaterally quasi-symmetrical in the axial view.

[0032] The image coding network constructed based on the convolutional layer specifically includes the following steps:

[0033] Slice the 3 symmetrical images As a three-channel image of R, G, B of a natural image;

[0034] Flattened into a 1-dimensional sequence through Patch Embedding operation;

[0035] For the obtained 1D sequence, position encoding is added to maintain the relative position information between image blocks;

[0036] For symmetric image slicing The operation is represented by In the formula is the output of the image encoding network; Pat represents a 2D convolution with a convolution kernel of 16 and a stride of 16; Adapter represents a 2D convolution with a convolution kernel of 3 and a stride of 2; is the position encoding processing function, and , Indicates the encoded value of the 2i2th feature dimension at the posth position, where pos is the order of the current Token among all Tokens (i.e., the position index), i2 is the feature dimension, and i2 ranges from 0 to integer, is the embedding dimension of Token, Represents the encoded value of the 2i2+1th feature dimension at the posth position.

[0037] The image embedding network based on the Transformer block is constructed, and specifically includes the following steps:

[0038] The constructed image embedding network consists of 12 Transformer blocks and Neck blocks connected in series;

[0039] The Transformer block includes a first normalization layer, a multi-head attention layer, a second normalization layer, and a feedforward network layer. The input features of the Transformer block are processed by the first normalization layer and the multi-head attention layer, and then added to the input features to obtain the attention features. The attention features are processed by the second normalization layer and the feedforward network layer, and then added to the attention features to obtain the output features of the Transformer block.

[0040] The first normalization layer and the second normalization layer have the same structure; the processing of the normalization layer is expressed as In the formula is the normalized feature; The input features of the layer normalization layer; is the mean of all channels in the sample; is the variance of all channels in the sample; A constant set to prevent division by zero errors; is the third parameter to be learned; is the fourth parameter to be learned;

[0041] The multi-head attention layer includes several self-attention layers. The processing process of each self-attention layer is expressed as Where Q is the query vector of the self-attention layer, and , is the frozen query vector projection matrix, x is the input vector; K is the key vector of the self-attention layer, and , The frozen key vector projection matrix; V is the value vector of the self-attention layer, and , is the frozen value vector projection matrix; Softmax is the Softmax function, and , The ii-th value in the input vector, is the jjth value in the input vector; is the first vector to be learned; is the second vector to be learned; is element-wise multiplication; is matrix multiplication;

[0042] The processing of the multi-head attention layer is expressed as Where MHSA-FT is the output of the multi-head attention layer; is a linear projection function; Concat is a concatenation operation along the feature dimension; is the output of the Hth self-attention layer;

[0043] The processing of the feedforward network layer is expressed as In the formula is the output of the feedforward network layer; is the second linear processing function, and , is the frozen second linear projection weight value, is the frozen second bias value, zz is the input; is the first linear processing function, and , is the frozen first linear projection weight value, is the first offset value to be frozen; is the third vector to be learned; GELU is the Gaussian error linear unit, and , erf is an intermediate function and ;

[0044] The processing of a single Transformer block is expressed as In the formula is the input feature of the Transformer block; For attention characteristics; is the output feature of the Transformer block; LN is the processing function of the layer normalization layer;

[0045] The Neck block is used to reduce the dimension of the input features; the processing process of the Neck block is expressed as In the formula is the output of the Neck block; It is a 2D convolution with a convolution kernel of 3; It is a 2D convolution with a convolution kernel of 1; is the output feature of the last Transformer block.

[0046] The image segmentation network based on the attention mechanism is constructed, which specifically includes the following steps:

[0047] Generate manual position hints from the Ground Truth; the manual position hints include point hints, box hints, and coarse mask hints; set the height, width, and coordinate origin of the image to obtain the coordinates of the manual position hints and the corresponding coordinates of the symmetry hints;

[0048] The processing process of the constructed image segmentation network is expressed as Where Tokens is the overall abstract vector describing the mask and prompt; Concat is the concatenation operation along the feature dimension; is the mask to be output; is the predicted mask quality; Hints for sparse dots or boxes; is the symmetric point or box hint of the sparse point or box; Src is the image embedding after adding the dense hint; is the output of the image embedding network; Hint for dense coarse mask; Symmetrical cues for dense coarse mask cues; It is Src after interacting with Tokens; Tokens after interaction with Src; and All passed The Transformer processing includes two consecutive cue self-attention processes, cue-to-image cross-attention processes, and image-to-cue cross-attention processes. Encode the position of the image; Output an abstract vector for the interacted mask; Output an abstract vector for the mask quality after interaction; and All by Get; Split represents the splitting operation along the first dimension of the tensor; Masks is the final mask of the output; Upscale is an intermediate function, and the processing process is , It is a 2D transposed convolution with a kernel of 2 and a stride of 2; is a fully connected network for outputting Masks; Predict the quality score of the output mask; For output A fully connected network; Expressed as matrix multiplication.

[0049] The output probability values ​​of the image segmentation network are thresholded to obtain the final binary segmentation mask.

[0050] The training described in step S4 specifically includes the following steps:

[0051] The first loss is calculated using the following formula: : Where TP is the true positive value; FP is the false positive value; FN is the false negative value;

[0052] The second loss is calculated using the following formula: : Where N is the total number of samples; is the true label of the mth sample; is the predicted label of the mth sample;

[0053] Finally, the following formula is used as the total loss function L of the training process: In the formula is the first weight; is the second weight; is the third weight; is the fourth weight; It is the first L1 loss function of the image symmetry processing network; It is the second L1 loss function of the image symmetry processing network;

[0054] During the training process, the Adam optimizer is used to optimize the model parameters, and the poly strategy is used to adjust the learning rate.

[0055] The present invention also provides an imaging method including the ischemic stroke medical image segmentation method based on large model fine-tuning, further comprising the following steps:

[0056] S6. The segmentation result obtained in step S5 is marked and re-imaged on the actual ischemic stroke medical image to obtain the ischemic stroke medical image with the segmentation result.

[0057] The ischemic stroke medical image segmentation method and imaging method based on large model fine-tuning provided by the present invention generate symmetrical data based on the construction and processing of an image symmetry processing network, and based on the symmetrical data and the constructed image embedding network and image segmentation network, ultimately not only achieves the segmentation of ischemic stroke medical images, but also has higher reliability and better accuracy in the segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of the process flow of the segmentation method of the present invention.

[0059] Figure 2The figure is a schematic diagram of the method flow of the imaging method of the present invention. DETAILED DESCRIPTION

[0060] like Figure 1 The figure shows a flow chart of the segmentation method of the present invention: the ischemic stroke medical image segmentation method based on large model fine-tuning provided by the present invention comprises the following steps:

[0061] S1. Obtain existing medical images of ischemic stroke;

[0062] S2. Performing image preprocessing on the medical image data obtained in step S1 to construct a training data set; the image preprocessing includes removing the skull area, setting the window width and window position, image cropping, image resampling and image standardization; specifically includes the following steps:

[0063] The skull in the image is removed using a convex optimization method;

[0064] According to the set requirements, set the window width and window level to better observe the brain tissue and improve the accuracy of image segmentation; for CT images, the window level can be set to 40 HU and the window width to 80HU;

[0065] Cropping the image to the area containing brain tissue to improve segmentation efficiency and further reduce computational cost;

[0066] Since the pixel spacing between images may be different, the cropped images are resampled to unify the spatial resolution of the images, which is beneficial to the feature learning of the network and accelerates the convergence of the model;

[0067] The Z-Score method was used to standardize the images;

[0068] Finally, the image preprocessing process is completed;

[0069] S3. Construct an image symmetry processing network and an image encoding network based on the convolutional layer, construct an image embedding network based on the Transformer block, and construct an image segmentation network based on the attention mechanism. Finally, a primary model of ischemic stroke medical image segmentation including an image symmetry processing network, an image encoding network, an image embedding network and an image segmentation network is constructed; the steps are as follows:

[0070] The primary model for ischemic stroke medical image segmentation includes an image symmetry processing network, an image encoding network, an image embedding network and an image segmentation network connected in series in sequence;

[0071] An image symmetry processing network is constructed based on the convolution module; the image symmetry processing network is used to convert the input image into a bilateral quasi-symmetric image in the axial view for subsequent image segmentation;

[0072] An image coding network is constructed based on the convolutional layer; the image coding network is used to embed and positionally encode the image output by the image symmetry processing network in blocks to maintain the relative position information between image blocks;

[0073] An image embedding network is constructed based on the Transformer block; the image embedding network is used to process the position-encoded image to obtain image embedding data;

[0074] An image segmentation network is constructed based on the attention mechanism; the image segmentation network is used to process the image embedding data to obtain the final segmentation result of ischemic stroke medical images;

[0075] In specific implementation, the image symmetry processing network based on the convolution module is constructed, which specifically includes the following steps:

[0076] In the input image, three consecutive slices are randomly selected as a group and input into the image symmetry processing network;

[0077] The constructed image symmetry processing network includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, an average pooling layer, a fully connected layer and an activation function layer which are connected in series in sequence;

[0078] The first convolution module, the second convolution module, the third convolution module and the fourth convolution module are all used to extract features; the average pooling layer is used to reduce the dimension of the features; the fully connected layer is used to map the features to the affine transformation parameters; the activation function layer is used to limit the output range to -1 to 1; the structures of the first convolution module, the second convolution module, the third convolution module and the fourth convolution module are all the same; the convolution module includes a downsampling 3D convolution layer, a first group normalization layer, a first ReLU layer, a first 3D convolution layer, a second group normalization layer, a second ReL layer, which are connected in series in sequence. The U layer and the second 3D convolution layer have residual connections from input to output in the process of the first group normalization layer, the first ReLU layer, the first 3D convolution layer, the second group normalization layer, the second ReLU layer and the second 3D convolution layer; the convolution kernel of the downsampling 3D convolution layer is 3, the step size of the depth dimension convolution is 1, the step size of the height dimension convolution is 2, and the step size of the width dimension convolution is 2; the first group normalization layer and the second group normalization layer have the same structure, and are both used to solve the internal covariate shift in the training process; the calculation formula of the group normalization layer is expressed as , where Y is the normalized feature and X is the input feature after grouping by channel. is the mean of the feature graph of this group, is the variance of the feature map of this group, A constant to prevent division by zero errors. is the first parameter to be learned, is the second parameter to be learned; the first ReLU layer and the second ReLU layer have the same structure and are both used to enhance the nonlinear representation ability of the model; the calculation formula of the ReLU layer is expressed as ,in is the value after being processed by the ReLU layer, is the i-th value in the feature map; the first 3D convolutional layer and the second 3D convolutional layer have the same structure and are both used for feature extraction; the convolution kernel of the 3D convolutional layer is 3, the step size is 1, and the padding is 1;

[0079] The activation function layer uses the tanh function as the activation function; the calculation formula of the tanh function is expressed as , where is the value after the activation function, is the i1th value in the image;

[0080] The processing process of the image symmetry processing network is expressed as Where View is the output of the image symmetry processing network; Tanh represents the processing process of the activation function layer; FC represents the processing process of the fully connected layer; Avpool represents the processing process of the average pooling layer; Fea is the output of the fourth convolution module;

[0081] Convert the View to The affine matrix M and the corresponding inverse matrix ; Perform affine transformation on the three slices X of the input image through the affine matrix M to obtain three symmetrical image slices ; Symmetrical image slices The image is left-right quasi-symmetrical in the axial view;

[0082] In specific implementation, the image coding network constructed based on the convolutional layer specifically includes the following steps:

[0083] Slice the 3 symmetrical images As a three-channel image of R, G, B of a natural image;

[0084] Flattened into a 1-dimensional sequence through Patch Embedding operation;

[0085] For the obtained 1D sequence, position encoding is added to maintain the relative position information between image blocks;

[0086] For symmetric image slicing The operation is represented by In the formula is the output of the image coding network; Pat represents a 3D convolution with a convolution kernel of 16 and a stride of 16; Adapter represents a 3D convolution with a convolution kernel of 3 and a stride of 2; is the position encoding processing function, and , Indicates the encoded value of the 2i2th feature dimension at the posth position, where pos is the order of the current Token among all Tokens (i.e., the position index), i2 is the feature dimension, and i2 ranges from 0 to integer, is the embedding dimension of Token, Represents the encoding value of the 2i2+1th feature dimension at the posth position;

[0087] In specific implementation, the image embedding network is constructed based on the Transformer block, and specifically includes the following steps:

[0088] The constructed image embedding network consists of 12 Transformer blocks and Neck blocks connected in series;

[0089] The Transformer block includes a first normalization layer, a multi-head attention layer, a second normalization layer, and a feedforward network layer. The input features of the Transformer block are processed by the first normalization layer and the multi-head attention layer, and then added to the input features to obtain the attention features. The attention features are processed by the second normalization layer and the feedforward network layer, and then added to the attention features to obtain the output features of the Transformer block.

[0090] The first normalization layer and the second normalization layer have the same structure; the processing of the normalization layer is expressed as In the formula is the normalized feature; The input features of the layer normalization layer; is the mean of all channels in the sample; is the variance of all channels in the sample; A constant set to prevent division by zero errors; is the third parameter to be learned; is the fourth parameter to be learned;

[0091] The multi-head attention layer is a multi-head self-attention layer with efficient parameter fine-tuning. It includes several self-attention layers. The processing process of each self-attention layer is expressed as Where Q is the query value of the self-attention layer, and , is the frozen query vector projection matrix, x is the input vector; K is the key value of the self-attention layer, and , The frozen key vector projection matrix; V is the value of the self-attention layer, and , is the frozen value vector projection matrix; Softmax is the Softmax function, which is used to convert the input into a probability distribution with a sum of 1, and , The ii-th value in the input vector, is the jjth value in the input vector; is the first vector to be learned; is the second vector to be learned; is element-wise multiplication; is matrix multiplication;

[0092] The processing of the multi-head attention layer is expressed as Where MHSA-FT is the output of the multi-head attention layer; is a linear projection function; Concat is a concatenation operation along the feature dimension; is the output of the Hth self-attention layer;

[0093] The feedforward network layer is a feedforward network layer with efficient parameter fine-tuning, and its processing process is expressed as In the formula is the output of the feedforward network layer; is the second linear processing function, and , is the frozen second linear projection weight value, is the frozen second bias value, zz is the input; is the first linear processing function, and , is the frozen first linear projection weight value, is the first offset value to be frozen; is the third vector to be learned; GELU is the Gaussian error linear unit, and , erf is an intermediate function and ;

[0094] The processing of a single Transformer block is expressed as In the formula is the input feature of the Transformer block; For attention characteristics; is the output feature of the Transformer block; LN is the processing function of the layer normalization layer;

[0095] The feedforward network layer and the multi-head attention layer are Efficient parameter fine-tuning: Efficient parameter fine-tuning can achieve the same performance as full parameter fine-tuning with very few trainable parameters (0.01% of trainable parameters);

[0096] The Neck block is used to reduce the dimension of the input features; the processing process of the Neck block is expressed as In the formula is the output of the Neck block; It is a 2D convolution with a convolution kernel of 3; It is a 2D convolution with a convolution kernel of 1; and The corresponding step size and fill value are set; is the output feature of the last Transformer block;

[0097] In specific implementation, the image segmentation network based on the attention mechanism is constructed, which specifically includes the following steps:

[0098] Generate manual position hints from Ground Truth; the manual position hints include hints of points, boxes and coarse masks; set the height, width and coordinate origin of the image to obtain the coordinates of the manual position hints and the corresponding symmetric hints; take the point as an example, set the image height and width to H and W respectively, the lower left corner to be the coordinate origin, and set the coordinates of the hint P of the manual point to , then the symmetric point of P suggests The coordinates are expressed as ; because the asymmetric original input X has been transformed into axial quasi-symmetric by the symmetry processing network , so the hint P and its symmetric hint Symmetrical hemispheric regions on both sides of the brain are indicated;

[0099] Based on the clinical prior knowledge that the contralateral brain of the ischemic side is generally healthy tissue, the visual cue P is combined with the symmetry cue They are respectively regarded as foreground hints and background hints. The foreground hints are given by the user, while the background hints are automatically generated by the algorithm. In this way, when performing semi-automatic hint segmentation (whether it is points, boxes or coarse masks), the model can automatically compare the information on the left and right sides.

[0100] Finally, the processing of the constructed image segmentation network is expressed as Where Tokens is the overall abstract vector describing the mask and prompt; Concat is the concatenation operation along the feature dimension; is the mask to be output; is the predicted mask quality; Hints for sparse dots or boxes; is the symmetric point or box hint of the sparse point or box; Src is the image embedding after adding the dense hint; is the output of the image embedding network; Hint for dense coarse mask; Symmetrical cues for dense coarse mask cues; It is Src after interacting with Tokens; Tokens after interaction with Src; and All passed The Transformer processing includes two consecutive cue self-attention processes, cue-to-image cross-attention processes, and image-to-cue cross-attention processes. Encode the position of the image; Output an abstract vector for the interacted mask; Output an abstract vector for the mask quality after interaction; and All by Get; Split represents the splitting operation along the first dimension of the tensor; Masks is the final mask of the output; Upscale is an intermediate function, and the processing process is , It is a 2D transposed convolution with a kernel of 2 and a stride of 2; is a fully connected network for outputting Masks; Predict the quality score of the output mask; For output A fully connected network; Expressed as matrix multiplication;

[0101] Threshold the output probability value of the image segmentation network (the threshold is set to 0) to obtain the final binary segmentation mask;

[0102] S4. Using the training data set obtained in step S2, the primary model for ischemic stroke medical image segmentation constructed in step S3 is trained to obtain an ischemic stroke medical image segmentation model;

[0103] In specific implementation, the training process includes the following steps:

[0104] The first loss is calculated using the following formula: : Where TP is the true positive value; FP is the false positive value; FN is the false negative value;

[0105] The second loss is calculated using the following formula: : Where N is the total number of samples; is the true label of the mth sample; is the predicted label of the mth sample;

[0106] Finally, the following formula is used as the total loss function L of the training process: In the formula is the first weight; is the second weight; is the third weight; is the fourth weight; It is the first L1 loss function of the image symmetry processing network; It is the second L1 loss function of the image symmetry processing network;

[0107] During the training process, the Adam optimizer is used to optimize the model parameters, and the poly strategy is used to adjust the learning rate;

[0108] S5. Acquire an actual ischemic stroke medical image and input it into the ischemic stroke medical image segmentation model obtained in step S4 to complete the segmentation of the actual ischemic stroke medical image.

[0109] The effect of the segmentation method of the present invention is described below in conjunction with the embodiments:

[0110] Compare the segmentation method of the present invention with the existing scheme:

[0111] Existing solutions include:

[0112] SAM scheme: This scheme is the first 2D visual base model trained on large-scale natural images. Its Base version uses 12 basic Transformer Block stacks and designs a hint encoder based on manual position hints and a mask decoder based on Transformer and MLP. To ensure the fairness of the experiment, this example fine-tunes the decoder of SAM on the stroke dataset.

[0113] MedSAM scheme: This scheme fine-tunes SAM based on hints on a large-scale 2D medical image dataset to bridge the gap between natural images and medical images. The fine-tuning method of MedSAM is to freeze all parameters except the mask decoder. To ensure the fairness of the experiment, we continue to fine-tune on the stroke dataset according to the fine-tuning method of MedSAM.

[0114] Medical SAM Adapter (MSA) solution: This solution uses Adapter to fine-tune the encoder and decoder of SAM. The specific method is to insert a series of learnable Adapters into SAM and freeze other parameters. To ensure the fairness of the experiment, we continue to fine-tune on the stroke dataset according to the fine-tuning method of Medical SAM Adapter.

[0115] In the experiment, Dice, Precision, Sensitivity, HD95, and ASSD evaluation indicators are used to evaluate the performance of the method.

[0116] The Dice index is often used to evaluate the quality of medical image segmentation. The closer its value is to 1, the more accurate the segmentation result is.

[0117] Precision and Sensitivity are also commonly used to evaluate the performance of image segmentation; the higher the value of the index, the more accurate the segmentation result;

[0118] The HD95 index and ASSD index represent the distance between two sets, and the smaller the value, the better the segmentation performance;

[0119] The existing acute ischemic stroke dataset was used; to ensure the fairness of the comparison, the present method and the comparison method were evaluated on the same test set using 5 positive point prompts, and the same random seed was used (to ensure that the positions of the 5 random point prompts generated were the same). The experimental results are shown in Table 1. The numbers in () are standard deviations.

[0120] Table 1 Schematic diagram of segmentation results

[0121]

[0122] As can be seen from Table 1, the proposed method achieves the best performance among all methods in terms of Dice and ASSD. Model evaluation on the test set yields the average Dice, Precision, Sensitivity, HD95, and ASSD of all test samples of 58.42 (27.88), 70.21 (27.84), 52.63 (30.61), 35.52 (35.11), and 12.86 (13.05), respectively.

[0123] At the same time, it can be seen from the table that the method proposed by the present invention has achieved the best results in all indicators. It can be found from the table that the segmentation results of SAM and MSA are not ideal, which may be due to the fact that the pre-trained weights of these models are based on natural images, lacking the modeling of knowledge such as blurred medical image boundaries, different lesion shapes and sizes, and low contrast. The weight of MedSAM is obtained through training of large-scale medical images. Although only the decoder is fine-tuned, a relatively ideal segmentation result is obtained. The segmentation method of the present invention not only performs efficient fine-tuning of parameters of the encoder of SAM, fine-tuning of all parameters of the decoder and prompt encoder, but also considers modeling of the continuity of the three slices, and symmetry prompts attention to the ischemic side and the healthy side of the brain. This shows that the segmentation method designed by the present invention can more effectively extract the features of stroke lesions, thereby obtaining a more accurate segmentation result.

[0124] like Figure 2 The figure shows a schematic diagram of the method flow of the imaging method of the present invention: the imaging method disclosed in the present invention, which includes the ischemic stroke medical image segmentation method based on large model fine-tuning, comprises the following steps:

[0125] S1. Obtain existing medical images of ischemic stroke;

[0126] S2. Preprocessing the medical image data obtained in step S1 to construct a training data set; the image preprocessing includes removing the skull area, setting the window width and window position, image cropping, image resampling and image standardization;

[0127] S3. Construct an image symmetry processing network and an image encoding network based on convolutional layers, construct an image embedding network based on Transformer blocks, and construct an image segmentation network based on the attention mechanism. Finally, a primary model of ischemic stroke medical image segmentation is constructed, which includes an image symmetry processing network, an image encoding network, an image embedding network, and an image segmentation network.

[0128] S4. Using the training data set obtained in step S2, the primary model for ischemic stroke medical image segmentation constructed in step S3 is trained to obtain an ischemic stroke medical image segmentation model;

[0129] S5. Acquire the actual ischemic stroke medical image and input it into the ischemic stroke medical image segmentation model obtained in step S4 to complete the actual ischemic stroke medical image segmentation;

[0130] S6. The segmentation result obtained in step S5 is marked and re-imaged on the actual ischemic stroke medical image to obtain the ischemic stroke medical image with the segmentation result.

[0131] The imaging method provided by the present invention can be directly applied to existing ischemic stroke medical imaging equipment (such as a CT machine or an MRI machine), or directly applied to a terminal (such as a computer); in specific application, the existing scheme is used to obtain actual ischemic stroke medical images, and then the obtained data is input into the corresponding machine equipment (CT machine, MRI machine) or terminal. At this time, the machine equipment or terminal can obtain the actual segmentation result of the ischemic stroke medical image according to the imaging method disclosed in the present invention, and display the segmentation result on the original image through different types of representation (such as color), and then perform secondary imaging and output; at this time, the output image is an image with the segmentation result of the ischemic stroke medical image, and the image can reflect the actual segmentation result of the ischemic stroke medical image, thereby greatly facilitating clinical medical personnel and laboratory experimenters to carry out subsequent work.

Claims

1. A medical image segmentation method for ischemic stroke based on large model fine-tuning, characterized in that The steps include: S1. Obtain existing medical images of ischemic stroke; S2. Performing image preprocessing on the medical image data obtained in step S1 to construct a training data set; the image preprocessing includes removing the skull area, setting the window width and window position, image cropping, image resampling and image standardization; specifically includes the following steps: The skull in the image is removed using a convex optimization method; According to the set requirements, set the window width and window position to improve the accuracy of image segmentation; Crop the image to the area containing brain tissue to improve segmentation efficiency; Resample the cropped image to unify the spatial resolution of the image; The Z-Score method was used to standardize the images; Finally, the image preprocessing process is completed; S3. Construct an image symmetry processing network and an image encoding network based on the convolutional layer, construct an image embedding network based on the Transformer block, and construct an image segmentation network based on the attention mechanism. Finally, a primary model of ischemic stroke medical image segmentation including an image symmetry processing network, an image encoding network, an image embedding network and an image segmentation network is constructed; the steps are as follows: The primary model for ischemic stroke medical image segmentation includes an image symmetry processing network, an image encoding network, an image embedding network and an image segmentation network which are connected in series in sequence; An image symmetry processing network is constructed based on a convolution module; the image symmetry processing network is used to convert the input image into a bilateral quasi-symmetric image in an axial view for image segmentation; specifically, the steps include: In the input image, three consecutive slices are randomly selected as a group and input into the image symmetry processing network; The constructed image symmetry processing network includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, an average pooling layer, a fully connected layer and an activation function layer which are connected in series in sequence; The first convolution module, the second convolution module, the third convolution module and the fourth convolution module are all used to extract features; the average pooling layer is used to reduce the dimension of the features; the fully connected layer is used to map the features to the affine transformation parameters; the activation function layer is used to limit the output range to -1 to 1; the structures of the first convolution module, the second convolution module, the third convolution module and the fourth convolution module are all the same; the convolution module includes a downsampling 3D convolution layer, a first group normalization layer, a first ReLU layer, a first 3D convolution layer, a second group normalization layer, a second ReL layer, which are connected in series in sequence. The U layer and the second 3D convolution layer have residual connections from input to output in the process of the first group normalization layer, the first ReLU layer, the first 3D convolution layer, the second group normalization layer, the second ReLU layer and the second 3D convolution layer; the convolution kernel of the downsampling 3D convolution layer is 3, the step size of the depth dimension convolution is 1, the step size of the height dimension convolution is 2, and the step size of the width dimension convolution is 2; the first group normalization layer and the second group normalization layer have the same structure, and are both used to solve the internal covariate shift in the training process; the calculation formula of the group normalization layer is expressed as , where Y is the normalized feature and X is the input feature after grouping by channel. is the mean of the feature graph of this group, is the variance of the feature map of this group, A constant to prevent division by zero errors. is the first parameter to be learned, is the second parameter to be learned; the first ReLU layer and the second ReLU layer have the same structure and are both used to enhance the nonlinear representation ability of the model; the calculation formula of the ReLU layer is expressed as ,in is the value after being processed by the ReLU layer, is the i-th value in the feature map; the first 3D convolutional layer and the second 3D convolutional layer have the same structure and are both used for feature extraction; the convolution kernel of the 3D convolutional layer is 3, the stride is 1, and the padding is 1; The activation function layer uses the tanh function as the activation function; the calculation formula of the tanh function is expressed as , where is the value after the activation function, is the i1th value in the image; The processing process of the image symmetry processing network is expressed as Where View is the output of the image symmetry processing network; Tanh represents the processing process of the activation function layer; FC represents the processing process of the fully connected layer; Avpool represents the processing process of the average pooling layer; Fea is the output of the fourth convolution module; Convert the View to The affine matrix M and the corresponding inverse matrix ; The three slices X of the input image are affine transformed by the affine matrix M to obtain three symmetrical image slices ; Symmetrical image slices The image is left-right quasi-symmetrical in the axial view; An image coding network is constructed based on a convolutional layer; the image coding network is used to perform block embedding and position encoding on the image output by the image symmetry processing network to maintain the relative position information between image blocks; specifically, the steps include: Slice the 3 symmetrical images As a three-channel image of R, G, B of a natural image; Flattened into a 1-dimensional sequence through Patch Embedding operation; For the obtained 1D sequence, position encoding is added to maintain the relative position information between image blocks; For symmetric image slicing The operation is represented by In the formula is the output of the image encoding network; Pat represents a 2D convolution with a convolution kernel of 16 and a stride of 16; Adapter represents a 2D convolution with a convolution kernel of 3 and a stride of 2; is the position encoding processing function, and , Indicates the encoded value of the 2i2th feature dimension at the posth position, where pos is the order of the current Token among all Tokens, i2 is the feature dimension, and i2 ranges from 0 to integer, is the embedding dimension of Token, Represents the encoding value of the 2i2+1th feature dimension at the posth position; An image embedding network is constructed based on the Transformer block; the image embedding network is used to process the position-encoded image to obtain image embedding data; An image segmentation network is constructed based on the attention mechanism; the image segmentation network is used to process the image embedding data to obtain the final segmentation result of ischemic stroke medical images; S4. Using the training data set obtained in step S2, the primary model for ischemic stroke medical image segmentation constructed in step S3 is trained to obtain an ischemic stroke medical image segmentation model; S5. Acquire an actual ischemic stroke medical image and input it into the ischemic stroke medical image segmentation model obtained in step S4 to complete the segmentation of the actual ischemic stroke medical image.

2. The ischemic stroke medical image segmentation method based on large model fine-tuning according to claim 1 is characterized in that The image embedding network based on the Transformer block is constructed, and specifically includes the following steps: The constructed image embedding network consists of 12 Transformer blocks and Neck blocks connected in series; The Transformer block includes the first normalization layer, the multi-head attention layer, the second normalization layer, and the feedforward network layer. The input features of the Transformer block are processed by the first normalization layer and the multi-head attention layer, and then added to the input features to obtain the attention features. The attention features are processed by the second normalization layer and the feedforward network layer, and then added to the attention features to obtain the output features of the Transformer block; The first normalization layer and the second normalization layer have the same structure; the processing of the normalization layer is expressed as In the formula is the normalized feature; The input features of the layer normalization layer; is the mean of all channels in the sample; is the variance of all channels in the sample; A constant set to prevent division by zero errors; is the third parameter to be learned; is the fourth parameter to be learned; The multi-head attention layer includes several self-attention layers. The processing process of each self-attention layer is expressed as Where Q is the query vector of the self-attention layer, and , is the frozen query vector projection matrix, x is the input vector; K is the key vector of the self-attention layer, and , The frozen key vector projection matrix; V is the value vector of the self-attention layer, and , is the frozen value vector projection matrix; Softmax is the Softmax function, and , The ii-th value in the input vector, is the jjth value in the input vector; is the first vector to be learned; is the second vector to be learned; is element-wise multiplication; is matrix multiplication; The processing of the multi-head attention layer is expressed as Where MHSA-FT is the output of the multi-head attention layer; is a linear projection function; Concat is a concatenation operation along the feature dimension; is the output of the Hth self-attention layer; The processing of the feedforward network layer is expressed as In the formula is the output of the feedforward network layer; is the second linear processing function, and , is the frozen second linear projection weight value, is the frozen second bias value, zz is the input; is the first linear processing function, and , is the frozen first linear projection weight value, is the first offset value to be frozen; is the third vector to be learned; GELU is the Gaussian error linear unit, and , erf is an intermediate function and ; The processing of a single Transformer block is expressed as In the formula is the input feature of the Transformer block; For attention characteristics; is the output feature of the Transformer block; LN is the processing function of the layer normalization layer; The Neck block is used to reduce the dimension of the input features; the processing process of the Neck block is expressed as In the formula is the output of the Neck block; It is a 2D convolution with a convolution kernel of 3; It is a 2D convolution with a convolution kernel of 1; is the output feature of the last Transformer block.

3. The ischemic stroke medical image segmentation method based on large model fine-tuning according to claim 2 is characterized in that The image segmentation network based on the attention mechanism is constructed, which specifically includes the following steps: Generate manual location hints from Ground Truth; the manual location hints include point hints, box hints, and coarse mask hints; Set the height, width and coordinate origin of the image to obtain the coordinates of the manual position prompt and the corresponding symmetry prompt; The processing process of the constructed image segmentation network is expressed as Where Tokens is the overall abstract vector describing the mask and prompt; Concat is the concatenation operation along the feature dimension; is the mask to be output; is the predicted mask quality; Hints for sparse dots or boxes; Symmetrical point or box hints for sparse points or boxes; Src is the image embedding after adding dense hints; is the output of the image embedding network; Hint for dense coarse mask; Symmetrical cues for dense coarse mask cues; It is Src after interacting with Tokens; Tokens after interaction with Src; and All passed get; The Transformer processing includes two consecutive cue self-attention processes, cue-to-image cross-attention processes, and image-to-cue cross-attention processes; Encode the position of the image; Output an abstract vector for the interacted mask; Output an abstract vector for the mask quality after interaction; and All by Get; Split represents the splitting operation along the first dimension of the tensor; Masks is the final mask of the output; Upscale is an intermediate function, and the processing process is , It is a 2D transposed convolution with a kernel of 2 and a stride of 2; is a fully connected network for outputting Masks; Predict the quality score of the output mask; For output A fully connected network; Expressed as matrix multiplication; The output probability values ​​of the image segmentation network are thresholded to obtain the final binary segmentation mask.

4. The ischemic stroke medical image segmentation method based on large model fine-tuning according to claim 3 is characterized in that The training described in step S4 specifically includes the following steps: The first loss is calculated using the following formula: : Where TP is the true positive value; FP is the false positive value; FN is the false negative value; The second loss is calculated using the following formula: : Where N is the total number of samples; is the true label of the mth sample; is the predicted label of the mth sample; Finally, the following formula is used as the total loss function L of the training process: In the formula is the first weight; is the second weight; is the third weight; is the fourth weight; It is the first L1 loss function of the image symmetry processing network; It is the second L1 loss function of the image symmetry processing network; During the training process, the Adam optimizer is used to optimize the model parameters, and the poly strategy is used to adjust the learning rate.

5. An imaging method comprising the ischemic stroke medical image segmentation method based on large model fine-tuning according to any one of claims 1 to 4, characterized in that The following steps are also included: S6. The segmentation result obtained in step S5 is marked and re-imaged on the actual ischemic stroke medical image to obtain the ischemic stroke medical image with the segmentation result.

Images