Method and system for segmenting focus area in medical image

Abstract

The invention relates to a segmentation method and system for a focus area in a medical image, and the method comprises the steps: carrying out the segmentation of a target medical image through a first image segmentation model, and obtaining an intermediate segmentation result; determining a region of interest from the target medical image according to the intermediate segmentation result; and calculating the region of interest according to a second image segmentation model to obtain a predicted value of each voxel in the region of interest, determining a segmentation result according to a mean value of the predicted values, and determining the uncertainty of the segmentation result according to the mean value and the variance of the predicted values.

Description

Technical Field

[0001] The present application relates to the technical field of medical image processing, and in particular to a method and system for segmenting a lesion area in a medical image. Background Art

[0002] In the medical field, reliable detection and segmentation of lesion areas from magnetic resonance imaging is a prerequisite for effective surgical planning and treatment evaluation. Currently, most images are manually segmented by medical experts, which is time-consuming and overly dependent on the subjective experience of experts. In recent years, with the rapid development of deep learning technology in the field of medical imaging, computer-assisted diagnosis and treatment technology based on deep learning technology has gradually been widely used.

[0003] However, the application of deep learning methods in actual clinical settings remains challenging. When segmenting pathological structures, even the most advanced models may make mistakes when encountering difficult-to-distinguish lesion boundary tissues. In medical systems that rely on computers for judgment output, inaccurate predictions can be life-threatening. Existing models only provide a deterministic segmentation result that cannot express the uncertainty or reliability of the model's predictions, and the segmentation performance of existing models still has room for improvement. Summary of the invention

[0004] The embodiments of the present application provide a method for segmenting lesion areas in medical images, so as to at least solve the problem that the uncertainty or reliability of model prediction cannot be expressed in the related art, and the segmentation performance still has room for improvement.

[0005] In a first aspect, an embodiment of the present application provides a method for segmenting a lesion area in a medical image, comprising:

[0006] Segmenting the target medical image by using a first image segmentation model to obtain an intermediate segmentation result;

[0007] determining a region of interest from the target medical image according to the intermediate segmentation result;

[0008] The region of interest is calculated according to the second image segmentation model to obtain a predicted value of each voxel in the region of interest, a segmentation result is determined according to a mean of the predicted values, and an uncertainty of the segmentation result is determined according to the mean and variance of the predicted values.

[0009] In one embodiment, determining the uncertainty of the segmentation result according to the mean and variance of the predicted value includes:

[0010] u=σ(P m )(1-σ(P m ))P v

[0011] The sliding window sampling of the region of interest is performed by the second image segmentation model to perform multiple predictions on each voxel to obtain a predicted value, σ(·) represents the activation function, P m represents the mean of the predicted values ​​of each voxel in the region of interest, P v represents the variance of the predicted value for each voxel in the region of interest.

[0012] In one embodiment, the method further comprises: visualizing the uncertainty of the segmentation result in the form of a heat map to obtain an uncertainty map.

[0013] In one embodiment, determining the segmentation result according to the mean of the predicted values ​​includes:

[0014] The mean is processed by an activation function, and the processed data is binarized;

[0015] Whether it is a tumor area is determined according to the binarization result of the voxel.

[0016] In one embodiment, the method further comprises preprocessing the target medical image, wherein the preprocessing comprises:

[0017] Register multimodal medical images, register T1 modality images and T2 modality images to the corresponding T1ce modality;

[0018] Correcting the registered image through a bias field correction algorithm;

[0019] Filter invalid data in the image according to the voxel value of the corrected image;

[0020] Normalize the filtered image;

[0021] The images of T1 modality, T2 modality and T1ce modality are concatenated in the channel dimension.

[0022] In a second aspect, an embodiment of the present application provides a method for acquiring a medical image segmentation model, which is used to implement the method for segmenting a lesion area in a medical image described in the first aspect, including:

[0023] Acquire training set medical images and lesion labels, and preprocess the training set medical images;

[0024] Constructing a medical image segmentation model, the medical image segmentation model comprising a first image segmentation model and a second image segmentation model, the second image segmentation model comprising an encoder, a decoder, a segmentation head module and an uncertainty head module, the keys and values ​​in the encoder being connected to the decoder through jump connections;

[0025] Constructing a loss function of the medical image segmentation model;

[0026] The medical image segmentation model is trained using the preprocessed training set medical images and the loss function.

[0027] In one embodiment, obtaining a second training set image of a second image segmentation model includes:

[0028] Segmenting the first training set image by using the first image segmentation model;

[0029] A region of interest data set is determined according to the segmentation result, and the region of interest data set is used as a second training set image.

[0030] In one embodiment, the loss functions of the first image segmentation model and the second image segmentation model are configured as:

[0031] The overall loss function is constructed by using the Dice Loss function and the binary cross entropy loss function BCE Loss.

[0032] In one embodiment, the first image segmentation model and the second image segmentation model have the same model structure, and the parameters obtained by training the first image segmentation model are used as initial parameters of the second image segmentation model.

[0033] In one embodiment, the method further includes: a sliding sampling window size of the second image segmentation model is smaller than a sliding sampling window size of the first image segmentation model.

[0034] In a third aspect, an embodiment of the present application provides a system for segmenting a lesion area in a medical image, comprising:

[0035] The first segmentation module is used to segment the target medical image through the first image segmentation model to obtain an intermediate segmentation result;

[0036] A second input module: used for determining a region of interest from the target medical image according to the intermediate segmentation result;

[0037] The second segmentation module is used to calculate the region of interest according to the second image segmentation model to obtain a predicted value for each voxel in the region of interest, determine a segmentation result according to a mean of the predicted values, and determine the uncertainty of the segmentation result according to the mean and variance of the predicted values.

[0038] In a fourth aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for segmenting the lesion area in the medical image described in the first aspect and the method for acquiring the medical image segmentation model described in the second aspect are implemented.

[0039] In the fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, it implements the method for segmenting the lesion area in the medical image described in the first aspect and the method for obtaining the medical image segmentation model described in the second aspect.

[0040] The embodiment of the present application provides a method and system for segmenting lesion areas in medical images, which have at least the following technical effects.

[0041] The present application performs a rough segmentation of the target medical image through the first image segmentation model, and performs a fine segmentation of the rough segmentation result through the second image segmentation model. Through the two segmentations, the segmentation performance of the model is improved. In addition, the uncertainty of the segmentation result is output by the second segmentation model as supplementary information of the segmentation result, thereby improving the integrity of the segmentation result.

[0042] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 is a flow chart of a method for segmenting a lesion area in a medical image according to an exemplary embodiment;

[0045] Figure 2 is a schematic structural diagram of a second image segmentation model according to an exemplary embodiment;

[0046] Figure 3 It is a schematic diagram of the structure of the encoding block and the decoding block of the layered visual transformer based on the shifting window;

[0047] Figure 4 is a structural block diagram of a system for segmenting a lesion area in a medical image according to an embodiment;

[0048] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0050] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0051] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0052] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0053] In a first aspect, the present application provides a method for segmenting a lesion area in a medical image. Figure 1 is a flow chart of a method for segmenting a lesion area in a medical image according to an exemplary embodiment. Figure 1 As shown, the method includes:

[0054] Step S101, segmenting the target medical image using a first image segmentation model to obtain an intermediate segmentation result.

[0055] Optionally, the first image segmentation model and the second image segmentation model in the present application are both built based on the PyTorch deep learning framework. Optionally, the first image segmentation model can be a medical image segmentation model such as U-Net. Preferably, the first image segmentation model is a model combining the U-Net structure, the Swin-Transformer structure and the multi-head attention mechanism, and the model includes an encoder, a decoder, a segmentation head module and an uncertainty head module, and the keys and values ​​in the encoder are connected to the decoder by jumps. Consistent with the structure of the subsequent second image segmentation model.

[0056] Figure 2 is a schematic diagram of the structure of a second image segmentation model according to an exemplary embodiment. Figure 2As shown in the figure, the structure of the model specifically includes: a sliding sampling layer, a patch partitioning layer, a linear embedding layer, an encoding end, a bottleneck layer, a decoding end, and an upsampling layer, a linear projection layer, a segmentation head module and an uncertainty head module after the input end and before the encoding end.

[0057] Specifically, step S101 includes:

[0058] refer to Figure 2 , the image is input into the first image segmentation model, and after sliding sampling, several image blocks of size (D×H×W×3) are obtained as input to the model. After a patch partitioning layer, the image blocks are divided into non-overlapping patches of size 4×4×4×3, and then a linear embedding layer maps the patches to a space of dimension 96. The size of each image block data becomes The cropping size of the image can be set according to actual application conditions.

[0059] At the encoding end of the model, three Swin-TransformerEncoder blocks based on shifted windows are used for feature extraction. After each Swin-Transformer Encoder block, a Patch Merging Layer is used to downsample the data, keeping the dimension D unchanged, halving the dimensions H and W, and doubling the channel C dimension. The bottleneck layer consists of a Swin-Transformer Encoder.

[0060] At the decoder end of the model, three Swin-Transformer Decoder blocks are used to decode the features. Before each Swin-Transformer Encoder block, a Patch Expanding Layer is used to upsample the data, keeping the dimension D unchanged, doubling the dimensions H and W, and halving the channel C dimension.

[0061] in, Figure 3 This is a schematic diagram of the structure of the encoding block and decoding block of the layered visual transformer based on the shift window. The structure of Swin-Transformer Encoder and Swin-Transformer Decoder is as follows Figure 3As shown in the figure, each Swin-Transformer has two consecutive layers to calculate self-attention in a local manner. The first layer is a standard Transformer structure, and the second layer replaces the standard multi-head self-attention mechanism with a multi-head attention mechanism with a shift window (Multi-Head Self-Attention, MSA).

[0062] In addition, the spatial information on the encoder side is fused with the semantic information on the decoder side through jump connections on the encoder and decoder sides. At the same time, in order to realize the interaction of hierarchical information, the key and value of the Swin-TransformerEncoder Block on the encoder side are passed to the Swin-Transformer Decoder Block on the decoder side. This improves the accuracy of the model in extracting image information and improves the segmentation performance of the model. Among them, the specific formula for calculating the self-attention feature on the decoder side is as follows:

[0063]

[0064] in K represents the query, key and value of the decoder layer l respectively. (l) , V (l) They represent the key and value of the encoder layer l respectively. Atten(·) is the attention calculation, and the calculation formula is expressed as:

[0065]

[0066] where d represents the dimension of query and key, and B represents the trainable relative position bias.

[0067] Continue to refer Figure 2 After decoding at the decoder end, the image block data is restored to the size after sliding sampling through a four-fold upsampling layer. Then a linear projection layer is used to generate the prediction value of the model. The linear projection layer includes a 3×3×3 convolution layer, a batch normalization layer, an activation function, a 3×3×3 convolution layer, a batch normalization layer, an activation function, a residual link, and a 1×1×1 convolution layer.

[0068] After the input image data is sampled by the sliding window, the voxels of the image may be sampled multiple times. The first image segmentation model performs sliding window sampling on the target medical image to perform multiple predictions on each voxel to obtain a predicted value. In the segmentation head, the mean of the predicted values ​​obtained after each voxel passes through the encoding and decoding networks is calculated, and the image blocks after sliding sampling are spliced ​​back to the size of the original input.

[0069] The mean of the predicted value is binarized by the Sigmoid activation function. Values ​​greater than or equal to 0.5 are set to 1, indicating a tumor area; values ​​less than 0.5 are set to 0, indicating a non-tumor area. Thus, the intermediate segmentation result is determined based on the tumor area. The division threshold of the binarization result can be set according to actual needs.

[0070] Before step S101, the method further includes preprocessing the target medical image, specifically including:

[0071] (1) Align the multimodal images in the data set, and align the images of the T1 mode and the images of the T2 mode to the corresponding T1ce mode. Optionally, the data set of the first image segmentation model includes tumor three-dimensional image data, and each data contains three modes: T1, T1ce and T2. Use the ANTs toolkit to align the T1 mode and T2 mode of the magnetic resonance image (MRI) of the tumor to the T1ce mode. It should be noted that this operation is not performed if it is single-modality data.

[0072] (2) Correcting the registered image by a bias field correction algorithm. Optionally, an ANTs N4 bias field correction algorithm is used.

[0073] (3) Filter invalid data in the image according to the voxel values ​​of the corrected image. Optionally, filter out invalid data whose voxel values ​​are all zero.

[0074] (4) Standardize the filtered image. Optionally, the z-score method is used to standardize the data, and the calculation formula is:

[0075]

[0076] Among them, X is the data before standardization, μ is the mean of the voxel values ​​in the image, and σ is the standard deviation of the voxel values ​​in the image.

[0077] (5) Connecting the images of the T1 modality, the T2 modality, and the T1ce modality in the channel dimension. Optionally, generating three-channel three-dimensional image data, each data modality corresponds to one channel.

[0078] Continue to refer to Figure 1 , after step S101, step S102 is performed, as follows.

[0079] Step S102: determining a region of interest from the target medical image according to the intermediate segmentation result.

[0080] Optionally, the target medical image is subjected to ROI cropping according to the intermediate segmentation result to determine the region of interest. When cropping, the intermediate segmentation result can be directly used as the region of interest. Preferably, a length of 40 is added on each axis based on the intermediate segmentation result. Assuming that the range of the lesion obtained by the first image segmentation model is (X min , Y min , Z mi n)~(X max , Y max , Z max ), then the area of ​​interest is (X min -20, Y min -20, Z min -20)~(X max +20, Y max +20, Z max +20). When the region of interest is determined based on the intermediate segmentation result, the length added on each coordinate axis can be set according to the specific application requirements. In this way, it is avoided that some lesions are not segmented and are cropped, and the accuracy and integrity of the model segmentation are improved. In addition, the input of the second image segmentation model is determined according to the segmentation result of the first image segmentation model, so as to perform a second fine segmentation, which is conducive to improving the accuracy of the segmentation method in the present application.

[0081] Step S103, calculating the region of interest according to the second image segmentation model to obtain a predicted value of each voxel in the region of interest, determining a segmentation result according to a mean of the predicted values, and determining the uncertainty of the segmentation result according to the mean and variance of the predicted values.

[0082] Optionally, after the input region of interest is sampled by the sliding window, the voxels of the image are sampled multiple times. The mean and variance of the predicted values ​​obtained after each voxel passes through the encoder and decoder in the second image segmentation model are calculated, and the uncertainty of the segmentation result is determined according to the mean and variance of the predicted values.

[0083] The structure of the second image segmentation model has been described in detail in step S101. Figure 2 and Figure 3 .

[0084] In step S103, the region of interest is calculated according to the second image segmentation model to obtain a predicted value of each voxel in the region of interest. The detailed steps have been described in detail when the target medical image is segmented by the first image segmentation model in step S101.

[0085] In one example, the segmentation result is determined according to the mean of the predicted values ​​in step S103, which specifically includes: processing the mean of each voxel through an activation function, binarizing the processed data, and determining whether it is a tumor area according to the binarization result of the voxel.

[0086] Optionally, the mean of the predicted value is binarized by a Sigmoid activation function, and a value greater than or equal to 0.5 is set to 1, indicating a tumor area; a value less than 0.5 is set to 0, indicating a non-tumor area. Thus, the segmentation result is determined based on the tumor area. The segmentation threshold of the binarization result can be set according to actual needs.

[0087] In one example, determining the uncertainty of the segmentation result according to the mean and variance of the predicted value in step S103 specifically includes:

[0088] u=σ(P m )(1-σ(P m ))P v

[0089] The sliding window sampling of the region of interest is performed by the second image segmentation model to perform multiple predictions on each voxel to obtain a predicted value, σ(·) represents the Sigmoid activation function, P m represents the mean of the predicted values ​​for each voxel in the region of interest, P v represents the variance of the predicted value for each voxel in the region of interest.

[0090] In one example, the method further includes: visualizing the uncertainty of the segmentation result in the form of a heat map to obtain an uncertainty map. In this way, the uncertainty map of the segmentation result is output, and the uncertainty result is used as supplementary information of the segmentation result to improve the integrity of the segmentation result.

[0091] In summary, the present application performs a rough segmentation of the target medical image through the first image segmentation model, and performs a fine segmentation of the rough segmentation result through the second image segmentation model. The segmentation performance of the model is improved through two segmentations. In addition, the uncertainty of the segmentation result is output by the second segmentation model as supplementary information of the segmentation result to improve the integrity of the segmentation result. The training result of the first image segmentation model is used as the initial parameter of the second image segmentation model to improve the training speed and performance of the model.

[0092] In a second aspect, an embodiment of the present application provides a method for acquiring a medical image segmentation model, which is used to implement the method for segmenting a lesion area in a medical image described in the first aspect, including:

[0093] Step S201, obtain training set medical images and lesion labels, and pre-process the training set medical images. Optionally, the training set medical images include multi-modal data and single-modal data. The images used in this application are magnetic resonance images.

[0094] In one example, obtaining a first training set image of a first image segmentation model includes:

[0095] Step S2011, align the multimodal images in the data set, and align the images of the T1 mode and the images of the T2 mode to the corresponding T1ce mode. Optionally, the data set of the first image segmentation model includes three-dimensional image data of the tumor, and each data contains three modes of T1, T1ce and T2. Use the ANTs toolkit to align the T1 mode and T2 mode of the magnetic resonance image (MRI) of the tumor to the T1ce mode. It should be noted that this operation is not performed if it is single-modality data.

[0096] Step S2012, correcting the registered image by using a bias field correction algorithm. Optionally, the ANTs N4 bias field correction algorithm is used.

[0097] Step S2013: Filter invalid data in the image according to the voxel values ​​of the corrected image. Optionally, filter out invalid data whose voxel values ​​are all zero.

[0098] Step S2014, standardize the filtered image. Optionally, the z-score method is used to standardize the data, and the calculation formula is:

[0099]

[0100] Among them, X is the data before standardization, μ is the mean of the voxel values ​​in the image, and σ is the standard deviation of the voxel values ​​in the image.

[0101] Step S2015, connect the images of the T1 mode, the T2 mode and the T1ce mode in the channel dimension. Optionally, three-channel three-dimensional image data is generated, and each data modality corresponds to one channel.

[0102] In addition, after the data set data is processed as above, the connected images can be divided according to the preset size, and the divided images are data enhanced to obtain the first training set images. Optionally, the images are cropped into data blocks of 3×128×128×128 size, and then the images are randomly flipped and randomly transformed for data enhancement. Thereby increasing the number of images in the first training set and enhancing the accuracy and generalization performance of the first image segmentation model. Among them, the cropping size of the image can be set according to the actual application.

[0103] In one example, obtaining a second training set image of a second image segmentation model includes:

[0104] The first training set image is segmented by the first image segmentation model, and the region of interest data set is determined according to the segmentation result, and the region of interest data set is used as the second training set image. In this way, the region of interest data set determined by the first image segmentation model is used as the second training set image, which is more time-saving and labor-saving than the manual segmentation method. At the same time, it is equivalent to the pre-segmentation result, which reduces the amount of calculation during the second segmentation.

[0105] Specifically, when determining the region of interest, the intermediate segmentation result can be directly selected as the region of interest. Preferably, a length of 40 is added on each axis based on the intermediate segmentation result. Assuming that the range of the lesion obtained by the first image segmentation model is (X min , Y min , Z mi n)~(X max , Y max , Z max ), then the area of ​​interest is (X min -20, Y min -20, Z min -20)~(X max +20, Y max +20, Z max +20). When the region of interest is determined based on the intermediate segmentation result, the length added on each coordinate axis can be set according to the specific application requirements. In this way, it is avoided that some lesions are not segmented and are cropped, and the accuracy and integrity of the model segmentation are improved. In addition, the input of the second image segmentation model is determined according to the segmentation result of the first image segmentation model, so as to perform a second fine segmentation, which is conducive to improving the accuracy of the segmentation method in the present application.

[0106] In one example, the method further includes: a sliding sampling window size of the second image segmentation model is smaller than a sliding sampling window size of the first image segmentation model.

[0107] Optionally, the sliding sampling window size of the first image segmentation model and the second image segmentation model can be set according to actual needs. In the present application, the sliding sampling window size of the second image segmentation model is set to half of the sliding sampling window size of the first image segmentation model. The size of the sliding sampling window of the second image segmentation model is specifically set to 3×64×64×64. By setting the sliding sampling window size of the second image segmentation model to half of the sliding sampling window of the first image segmentation model, the second image segmentation model extracts smaller and finer feature areas for fine segmentation through the second image segmentation model.

[0108] Step S202, constructing a medical image segmentation model, the medical image segmentation model includes a first image segmentation model and a second image segmentation model, the second image segmentation model includes an encoder, a decoder, a segmentation head module and an uncertainty head module, and the keys and values ​​in the encoder are connected to the decoder through jumps.

[0109] Optionally, the first image segmentation model can be a medical image segmentation model such as U-Net. Preferably, the first image segmentation model is a model combining U-Net structure, Swin-Transformer structure and multi-head attention mechanism, and the model includes an encoder, a decoder, a segmentation head module and an uncertainty head module, and the keys and values ​​in the encoder are connected to the decoder through jumps. It is consistent with the structure of the subsequent second image segmentation model.

[0110] The structures of the first image segmentation model and the second image segmentation model in the preferred embodiment are specifically shown in Figure 2 The structure of the model specifically includes: a sliding sampling layer, a patch partitioning layer, a linear embedding layer, an encoding end, a bottleneck layer, a decoding end, and an upsampling layer, a linear projection layer, a segmentation head module, and an uncertainty head module after the decoding end.

[0111] Step S203, constructing a loss function of the medical image segmentation model. Optionally, the loss function during training includes at least one of existing loss functions such as a cross entropy loss function, a Manhattan loss function, and a Chebyshev distance loss.

[0112] In one example, the loss functions of the first image segmentation model and the second image segmentation model are configured as follows: an overall loss function is constructed by using a Dice Loss function and a binary cross entropy loss function BCE Loss, which is specifically expressed as:

[0113]

[0114] L total (G,Y)=0.7L Dice (G,Y)+0.3L BCE (G,Y)

[0115] Among them, N represents the number of voxels in the image, G and Y represent the true value and predicted value of the image, G i and Y i They represent the true value and predicted value of the i-th voxel respectively. The weight value of the loss function can be set according to specific application requirements.

[0116] Step S204, training the medical image segmentation model using the preprocessed training set medical images and the loss function.

[0117] Optionally, when training the first image segmentation model, a sliding window is used for sampling, the size of the sampling window is set to 3×128×128×128, and the sliding step size is set to 64. The weight parameters of each layer are optimized by gradient back propagation. The training epoch is set to 300, and the Adam optimizer is used during training. The learning rate lr is 1e-4, and the learning rate adjustment algorithm uses the cosine annealing algorithm. Among them, all parameters, loss functions, and learning rate adjustment algorithms involved in the actual training of the model can be set according to the model characteristics and convergence speed, and are not limited to the parameters used in this application.

[0118] When training the second image segmentation model, the sliding sampling window size of the second image segmentation model is set to half the sliding sampling window size of the first image segmentation model. The size of the sliding sampling window of the second image segmentation model is specifically set to 3×64×64×64. By setting the sliding sampling window size of the second image segmentation model to half the sliding sampling window of the first image segmentation model, the second image segmentation model extracts smaller and finer feature areas, so as to perform fine segmentation through the second image segmentation model.

[0119] Considering that the first training result is used as the pre-training parameter during the second training, and the result of the first training is used as the data set during the second training, the model convergence speed during the second training is faster. Therefore, the training epoch is set to 100, the Adam optimizer is used during training, the learning rate lr is 1e-4, and the learning rate adjustment algorithm adopts the cosine annealing algorithm. Among them, all parameters, loss functions, and learning rate adjustment algorithms involved in the actual training of the model can be set according to the model characteristics and convergence speed, and are not limited to the parameters used in this application.

[0120] The first image segmentation model and the second image segmentation model were trained according to the above parameters and algorithms. The segmentation performance of the model was evaluated using indicators such as Dice similarity coefficient, Hausdorff distance, average symmetric surface distance, relative volume error, sensitivity and specificity. The parameters obtained from the last training were used as the parameters of the medical image segmentation model.

[0121] In one example, step S204 includes: using the parameters obtained by training the first image segmentation model as initial parameters of the second image segmentation model. Optionally, using the trained first image segmentation model as a pre-trained model of the second image segmentation model. This effectively improves the training speed and model performance of the second image segmentation model.

[0122] In summary, the present application performs a rough segmentation of the target medical image through the first image segmentation model, and performs a fine segmentation of the rough segmentation result through the second image segmentation model. The segmentation performance of the model is improved through two segmentations. In addition, the uncertainty of the segmentation result is output by the second segmentation model as supplementary information of the segmentation result to improve the integrity of the segmentation result. The training result of the first image segmentation model is used as the initial parameter of the second image segmentation model to improve the training speed and performance of the model.

[0123] In a third aspect, the present application provides a system for segmenting a lesion area in a medical image. Figure 4 is a structural block diagram of a system for segmenting a lesion region in a medical image according to an embodiment. Figure 4 As shown, the system includes:

[0124] The first segmentation module 100 is used to segment the target medical image using a first image segmentation model to obtain an intermediate segmentation result.

[0125] Optionally, the first image segmentation model in the first segmentation module 100 and the second image segmentation model in the second segmentation module 300 are both built based on the PyTorch deep learning framework. Optionally, the first image segmentation model can be a medical image segmentation model such as U-Net. Preferably, the first image segmentation model is a model combining the U-Net structure, the Swin-Transformer structure and the multi-head attention mechanism, and the model includes an encoder, a decoder, a segmentation head module and an uncertainty head module, and the keys and values ​​in the encoder are connected to the decoder by jumping. Consistent with the structure of the subsequent second image segmentation model.

[0126] The structural diagram of the second image segmentation model is as follows Figure 2 As shown in the figure, the structure of the model specifically includes: a sliding sampling layer, a patch partitioning layer, a linear embedding layer, an encoding end, a bottleneck layer, a decoding end, and an upsampling layer, a linear projection layer, a segmentation head module and an uncertainty head module after the input end and before the encoding end.

[0127] Specifically, the first segmentation module 100 is specifically used for:

[0128] refer to Figure 2 , the image is input into the first image segmentation model, and after sliding sampling, several image blocks of size (D×H×W×3) are obtained as input to the model. After a patch partitioning layer, the image blocks are divided into non-overlapping patches of size 4×4×4×3, and then a linear embedding layer maps the patches to a space of dimension 96. The size of each image block data becomes The cropping size of the image can be set according to actual application conditions.

[0129] At the encoding end of the model, three Swin-TransformerEncoder blocks based on shifted windows are used for feature extraction. After each Swin-Transformer Encoder block, a Patch Merging Layer is used to downsample the data, keeping the dimension D unchanged, halving the dimensions H and W, and doubling the channel C dimension. The bottleneck layer consists of a Swin-Transformer Encoder.

[0130] At the decoder end of the model, three Swin-Transformer Decoder blocks are used to decode the features. Before each Swin-Transformer Encoder block, a Patch Expanding Layer is used to upsample the data, keeping the dimension D unchanged, doubling the dimensions H and W, and halving the channel C dimension.

[0131] in, Figure 3 This is a schematic diagram of the structure of the encoding block and decoding block of the layered visual transformer based on the shift window. The structure of Swin-Transformer Encoder and Swin-Transformer Decoder is as follows Figure 3 As shown in the figure, each Swin-Transformer has two consecutive layers to calculate self-attention in a local manner. The first layer is a standard Transformer structure, and the second layer replaces the standard multi-head self-attention mechanism with a multi-head attention mechanism with a shift window (Multi-Head Self-Attention, MSA).

[0132] In addition, the spatial information on the encoder side is fused with the semantic information on the decoder side through jump connections on the encoder and decoder sides. At the same time, in order to realize the interaction of hierarchical information, the key and value of the Swin-TransformerEncoder Block on the encoder side are passed to the Swin-Transformer Decoder Block on the decoder side. This improves the accuracy of the model in extracting image information and improves the segmentation performance of the model. Among them, the specific formula for calculating the self-attention feature on the decoder side is as follows:

[0133]

[0134] in K represents the query, key and value of the decoder layer l respectively. (l), V (l) They represent the key and value of the encoder layer l respectively. Atten(·) is the attention calculation, and the calculation formula is expressed as:

[0135]

[0136] where d represents the dimension of query and key, and B represents the trainable relative position bias.

[0137] Continue to refer Figure 2 After decoding at the decoder end, the image block data is restored to the size after sliding sampling through a four-fold upsampling layer. Then a linear projection layer is used to generate the prediction value of the model. The linear projection layer includes a 3×3×3 convolution layer, a batch normalization layer, an activation function, a 3×3×3 convolution layer, a batch normalization layer, an activation function, a residual link, and a 1×1×1 convolution layer.

[0138] After the input image data is sampled by the sliding window, the voxels of the image may be sampled multiple times. The first image segmentation model performs sliding window sampling on the target medical image to perform multiple predictions on each voxel to obtain a predicted value. In the segmentation head, the mean of the predicted values ​​obtained after each voxel passes through the encoding and decoding networks is calculated, and the image blocks after sliding sampling are spliced ​​back to the size of the original input.

[0139] The mean of the predicted value is binarized through the sigmoid activation function, and the value greater than or equal to 0.5 is set to 1, indicating the tumor area; the value less than 0.5 is set to 0, indicating the non-tumor area. In this way, the intermediate segmentation result is determined based on the tumor area. Among them, the division threshold of the binarization result can be set according to actual needs.

[0140] Before the first segmentation module 100, the system also includes a module for preprocessing the target medical image, specifically including:

[0141] (1) The multimodal images in the data set are registered, and the images of the T1 modality and the images of the T2 modality are registered to the corresponding T1ce modality. Optionally, the data set of the first image segmentation model includes tumor three-dimensional image data, and each data contains three modalities of T1, T1ce and T2. The T1 modality and T2 modality of the magnetic resonance imaging (MRI) of the tumor are registered to the T1ce modality using the ANTs toolkit.

[0142] (2) Correcting the registered image by a bias field correction algorithm. Optionally, an ANTs N4 bias field correction algorithm is used.

[0143] (3) Filter invalid data in the image according to the voxel values ​​of the corrected image. Optionally, filter out invalid data whose voxel values ​​are all zero.

[0144] (4) Standardize the filtered image. Optionally, the z-score method is used to standardize the data, and the calculation formula is:

[0145]

[0146] Among them, X is the data before standardization, μ is the mean of the voxel values ​​in the image, and σ is the standard deviation of the voxel values ​​in the image.

[0147] (5) Connecting the images of the T1 modality, the T2 modality, and the T1ce modality in the channel dimension. Optionally, generating three-channel three-dimensional image data, each data modality corresponds to one channel.

[0148] The second input module 200 is used to determine a region of interest from a target medical image according to the intermediate segmentation result.

[0149] Optionally, the target medical image is subjected to ROI cropping according to the intermediate segmentation result to determine the region of interest. When cropping, the intermediate segmentation result can be directly used as the region of interest. Preferably, a length of 40 is added on each axis based on the intermediate segmentation result. Assuming that the range of the lesion obtained by the first image segmentation model is (X min , Y min , Z mi n)~(X max , Y max , Z max ), then the area of ​​interest is (X min -20, Y min -20, Z min -20)~(X max +20, Y max +20, Z max +20). When the region of interest is determined based on the intermediate segmentation result, the length added on each coordinate axis can be set according to the specific application requirements. In this way, it is avoided that some lesions are not segmented and are cropped, and the accuracy and integrity of the model segmentation are improved. In addition, the input of the second image segmentation model is determined according to the segmentation result of the first image segmentation model, so as to perform a second fine segmentation, which is conducive to improving the accuracy of the segmentation method in the present application.

[0150] The second segmentation module 300 is used to calculate the region of interest according to the second image segmentation model to obtain a predicted value for each voxel in the region of interest, determine a segmentation result according to a mean of the predicted values, and determine the uncertainty of the segmentation result according to the mean and variance of the predicted values.

[0151] Optionally, after the input region of interest is sampled by the sliding window, the voxels of the image are sampled multiple times. The mean and variance of the predicted values ​​obtained after each voxel passes through the encoder and decoder in the second image segmentation model are calculated, and the uncertainty of the segmentation result is determined according to the mean and variance of the predicted values.

[0152] The structure of the second image segmentation model has been described in detail in the first segmentation module 100. Figure 2 and Figure 3 .

[0153] The detailed steps have been described in detail when the target medical image is segmented by the first image segmentation model in the first segmentation module 100 .

[0154] In one example, the segmentation result is determined according to the mean of the predicted values, specifically including: processing the mean of each voxel through an activation function, binarizing the processed data, and determining whether it is a tumor area according to the binarization result of the voxel.

[0155] Optionally, the mean of the predicted value is binarized by a sigmoid activation function, and a value greater than or equal to 0.5 is set to 1, indicating a tumor area; a value less than 0.5 is set to 0, indicating a non-tumor area. Thus, the segmentation result is determined based on the tumor area. The segmentation threshold of the binarization result can be set according to actual needs.

[0156] In one example, the uncertainty of the segmentation result is determined according to the mean and variance of the predicted values, including:

[0157] u=σ(P m )(1-σ(P m ))P v

[0158] Among them, the sliding window sampling of the region of interest is performed through the second image segmentation model to perform multiple predictions on each voxel to obtain the predicted value, σ(·) represents the activation function, P m represents the mean of the predicted values ​​for each voxel in the region of interest, P v represents the variance of the predicted value for each voxel in the region of interest.

[0159] In one example, the system further includes: a method for visualizing the uncertainty of the segmentation result in the form of a heat map to obtain an uncertainty map. In this way, the uncertainty map of the segmentation result is output, and the uncertainty result is used as supplementary information of the segmentation result to improve the integrity of the segmentation result.

[0160] In summary, the present application performs a rough segmentation of the target medical image through the first image segmentation model, and performs a fine segmentation of the rough segmentation result through the second image segmentation model. The segmentation performance of the model is improved through two segmentations. In addition, the uncertainty of the segmentation result is output by the second segmentation model as supplementary information of the segmentation result to improve the integrity of the segmentation result. The training result of the first image segmentation model is used as the initial parameter of the second image segmentation model to improve the training speed and performance of the model.

[0161] In a fourth aspect, an embodiment of the present application provides an electronic device, Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for segmenting the lesion area in the medical image provided in the first aspect and the method for acquiring the medical image segmentation model provided in the second aspect are implemented. Figure 5 The electronic device 60 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0162] The electronic device 60 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 60 may include but are not limited to: at least one processor 61, at least one memory 62, and a bus 63 connecting different system components (including the memory 62 and the processor 61).

[0163] The bus 63 includes a data bus, an address bus, and a control bus.

[0164] The memory 62 may include a volatile memory, such as a random access memory (RAM) 621 and / or a cache memory 622 , and may further include a read-only memory (ROM) 623 .

[0165] The memory 62 may also include a program / utility 625 having a set (at least one) of program modules 624, such program modules 624 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0166] The processor 61 executes various functional applications and data processing by running the computer program stored in the memory 62, such as the method for segmenting the lesion area in the medical image provided in the first aspect of the present application and the method for acquiring the medical image segmentation model provided in the second aspect.

[0167] The electronic device 60 may also communicate with one or more external devices 64 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 65. Furthermore, the model-generated device 60 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 66. As shown, the network adapter 66 communicates with other modules of the model-generated device 60 via a bus 63. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the model-generated device 60, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (RAID) systems, tape drives, and data backup storage systems.

[0168] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0169] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for segmenting the lesion area in the medical image provided in the first aspect and the method for acquiring the medical image segmentation model provided in the second aspect are implemented.

[0170] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0171] In a possible embodiment, the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the method for segmenting the lesion area in the medical image provided by the first aspect and the method for acquiring the medical image segmentation model provided by the second aspect.

[0172] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on a user device, partially on a user device, as an independent software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0173] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for segmenting a lesion area in a medical image, characterized in that: include: Segmenting the target medical image by using a first image segmentation model to obtain an intermediate segmentation result; determining a region of interest from the target medical image according to the intermediate segmentation result; The region of interest is calculated according to the second image segmentation model to obtain a predicted value of each voxel in the region of interest, a segmentation result is determined according to a mean of the predicted values, and an uncertainty of the segmentation result is determined according to the mean and variance of the predicted values.

2. The method for segmenting a lesion area in a medical image according to claim 1, characterized in that: The step of determining the uncertainty of the segmentation result according to the mean and variance of the predicted value includes: u=σ(P m )(1-σ(P m ))P v The sliding window sampling of the region of interest is performed by the second image segmentation model to perform multiple predictions on each voxel to obtain a predicted value, σ(·) represents the activation function, P m represents the mean of the predicted values ​​of each voxel in the region of interest, P v represents the variance of the predicted value for each voxel in the region of interest.

3. The method for segmenting a lesion area in a medical image according to claim 2, characterized in that: The method further comprises: The uncertainty of the segmentation result is visualized in the form of a heat map to obtain an uncertainty map.

4. The method for segmenting a lesion area in a medical image according to claim 1, characterized in that: The determining the segmentation result according to the mean of the predicted values ​​comprises: The mean is processed by an activation function, and the processed data is binarized; Whether it is a tumor area is determined according to the binarization result of the voxel.

5. The method for segmenting a lesion area in a medical image according to claim 1, characterized in that: The method further comprises preprocessing the target medical image, wherein the preprocessing comprises: Register multimodal medical images, register T1 modality images and T2 modality images to the corresponding T1ce modality; Correcting the registered image through a bias field correction algorithm; Filter invalid data in the image according to the voxel value of the corrected image; Normalize the filtered image; The images of T1 modality, T2 modality and T1ce modality are concatenated in the channel dimension.

6. A method for acquiring a medical image segmentation model, characterized in that: A method for segmenting a lesion area in a medical image according to any one of claims 1 to 5, comprising: Acquire training set medical images and lesion labels, and preprocess the training set medical images; Constructing a medical image segmentation model, the medical image segmentation model comprising a first image segmentation model and a second image segmentation model, the second image segmentation model comprising an encoder, a decoder, a segmentation head module and an uncertainty head module, the keys and values ​​in the encoder being connected to the decoder through jump connections; Constructing a loss function of the medical image segmentation model; The medical image segmentation model is trained using the preprocessed training set medical images and the loss function.

7. The method for obtaining a medical image segmentation model according to claim 6, characterized in that: Obtaining a second training set image for a second image segmentation model includes: Segmenting the first training set image by using the first image segmentation model; A region of interest data set is determined according to the segmentation result, and the region of interest data set is used as a second training set image.

8. The method for obtaining a medical image segmentation model according to claim 6, characterized in that: The loss functions of the first image segmentation model and the second image segmentation model are configured as: The overall loss function is constructed by using the Dice Loss function and the binary cross entropy loss function BCE Loss.

9. The method for obtaining a medical image segmentation model according to claim 6, characterized in that: The model structure of the first image segmentation model is the same as the model structure of the second image segmentation model, and the parameters obtained by training the first image segmentation model are used as initial parameters of the second image segmentation model.

10. The method for obtaining a medical image segmentation model according to claim 6, characterized in that: The method further comprises: The sliding sampling window size of the second image segmentation model is smaller than the sliding sampling window size of the first image segmentation model.

11. A system for segmenting lesion areas in medical images, characterized in that: include: The first segmentation module is used to segment the target medical image through the first image segmentation model to obtain an intermediate segmentation result; A second input module: used for determining a region of interest from the target medical image according to the intermediate segmentation result; The second segmentation module is used to calculate the region of interest according to the second image segmentation model to obtain a predicted value for each voxel in the region of interest, determine a segmentation result according to a mean of the predicted values, and determine the uncertainty of the segmentation result according to the mean and variance of the predicted values.

12. An electronic device, characterized in that: include Memory, processor, and A computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for segmenting the lesion area in the medical image according to any one of claims 1 to 5 and the method for acquiring the medical image segmentation model according to any one of claims 6 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for segmenting a lesion area in a medical image according to any one of claims 1 to 5 and the method for acquiring a medical image segmentation model according to any one of claims 6 to 10 are implemented.

Images