Medical image segmentation method, device, storage medium and electronic device

Abstract

The present invention provides a medical image segmentation method, device, storage medium and electronic device. A medical image segmentation method comprises: obtaining a medical image to be segmented; segmenting the medical image to be segmented using a trained image segmentation model, obtaining a first segmentation result of a target organ and a background and a second segmentation result of a target organ lesion and a background, respectively, and superimposing the first segmentation result and the second segmentation result and performing feature fusion to obtain a third segmentation result of the background, the target organ and the target organ lesion. The present invention integrates two different neural network structures in a neural network framework to segment organs and lesions respectively, and fuses the segmentation results of the two neural network structures to obtain the final overall segmentation result, thereby solving the image segmentation problem caused by the uneven size of organs and lesions in the organs.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation technology, and in particular to a medical image segmentation method, device, storage medium and electronic equipment. Background Art

[0002] In recent years, the rapid development of minimally invasive surgery has brought hope to patients with poor tolerance. With the advancement of artificial intelligence technology, in the field of multimodal hot and cold ablation, users have an increasing demand for automatic planning systems for ablation needles. Since the automatic planning of ablation needles relies on the three-dimensional reconstruction of human organs, and the basis of three-dimensional reconstruction is the accurate positioning of the liver and its lesion area in each CT image, in order to achieve accurate and rapid automatic planning of ablation needles, it is necessary to accurately segment the liver and lesion area in each CT image.

[0003] In 2015, Ronneberger et al. proposed the concept of U-Net, which made a breakthrough in deep learning in medical image segmentation. However, the traditional U-Net performs well on CT slices containing only the liver. When CT slices without the liver are also input into the neural network for training, the effect of the traditional U-Net becomes extremely poor, and even the liver cannot be segmented correctly. When the training reaches about 8,000 times, the traditional U-Net will have the problem of gradient disappearance, which directly leads to model collapse. Later, people proposed various neural network models for medical image segmentation based on the U-Net structure. However, using a single U-Net or improved U-Net cannot segment the liver and liver lesion area well. This is mainly caused by the uneven size of the background area, liver area and lesion area. Therefore, it is necessary to propose a new technical solution to solve the image segmentation problem caused by the uneven size of organs and lesions in organs. Summary of the invention

[0004] In order to solve the image segmentation problem caused by uneven sizes of organs and lesions in the organs, the present invention provides a medical image segmentation method, device, storage medium and electronic device.

[0005] In a first aspect, an embodiment of the present invention provides a medical image segmentation method, comprising:

[0006] Acquire a medical image to be segmented;

[0007] Segmenting the medical image to be segmented using the trained image segmentation model to obtain a first segmentation result of the target organ and the background and a second segmentation result of the target organ lesion and the background, respectively, and superimposing the first segmentation result and the second segmentation result and performing feature fusion to obtain a third segmentation result of the background, the target organ, and the target organ lesion;

[0008] The image segmentation model includes: a first neural network model whose input is the medical image to be segmented and whose output is a first segmentation result, and a second neural network model whose input is the medical image to be segmented and whose output is a second segmentation result.

[0009] In some embodiments, the first neural network model includes a first Res-UNet Plus model formed by replacing the convolutional layer of the U-Net model with a residual neural network module and inserting a jump connection portion into the residual neural network module.

[0010] In some embodiments, the second neural network model includes a second Res-UNetPlus model and a third Res-UNet Plus model that are jump-connected, and some feature graphs of the first Res-UNet Plus model are jump-connected to the second Res-UNet Plus model and the third Res-UNet Plus model;

[0011] Among them, the second Res-UNet Plus model and the third Res-UNet Plus model are models formed by replacing the convolutional layer of the U-Net model with a residual neural network module and inserting the jump connection part into the residual neural network module; the second Res-UNet Plus model and the third Res-UNet Plus model have less than 4 pooling layers.

[0012] In some embodiments, the shallow feature map of the down-sampling part in the first Res-UNet Plus model is jump-connected to the pooling layer of the second Res-UNet Plus model, and the deep feature map of the up-sampling part in the first Res-UNet Plus model is jump-connected to the deconvolution layer of the third Res-UNet Plus model.

[0013] In some embodiments, the training process of the image segmentation model includes:

[0014] Get medical image datasets;

[0015] Dividing the medical image dataset into a training set, a validation set and a test set according to a preset ratio;

[0016] Inputting N consecutive medical images in the training set into the image segmentation model for model training, and using a mask image of a middle medical image among the N consecutive medical images as a true label;

[0017] Inputting the medical images in the verification set into the image segmentation model for model verification;

[0018] If the loss between the output obtained by model verification and the true label is less than the first set threshold, or the Dice coefficient of the output obtained by model verification reaches the second set threshold, the trained image segmentation model is output.

[0019] In some implementations, the training process of the image segmentation model further includes:

[0020] Inputting the medical images in the test set into the trained image segmentation model for model testing;

[0021] If the Dice coefficient outputted from the model test reaches the third set threshold, the image segmentation model is retained;

[0022] If the Dice coefficient outputted from the model test does not reach the third set threshold, the parameters of the image segmentation model are adjusted for training again, and the N consecutive medical images in the training set are input into the image segmentation model for model training.

[0023] In some embodiments, the loss function of the image segmentation model is expressed as follows:

[0024] loss = α × loss 1 +β×loss 2 +γ×loss all

[0025] Among them, loss represents the loss of the image segmentation model, loss 1 Represents the first loss between the first segmentation result and the true label, loss 2 Represents the second loss between the second segmentation result and the true label, loss all represents the third loss between the third segmentation result and the true label, α, β, and γ represent the weights of the first loss, the second loss, and the third loss, respectively.

[0026] In a second aspect, an embodiment of the present invention provides a medical image segmentation device, comprising:

[0027] An acquisition module, used for acquiring a medical image to be segmented;

[0028] a segmentation module, used to segment the medical image to be segmented using a trained image segmentation model, to obtain a first segmentation result of a target organ and a background and a second segmentation result of a target organ lesion and a background, and to superimpose the first segmentation result and the second segmentation result and perform feature fusion to obtain a third segmentation result of the background, the target organ and the target organ lesion;

[0029] The image segmentation model includes: a first neural network model whose input is the medical image to be segmented and whose output is a first segmentation result, and a second neural network model whose input is the medical image to be segmented and whose output is a second segmentation result.

[0030] In a third aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon. When the computer program is executed by one or more processors, the method described in the first aspect is implemented.

[0031] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a memory and one or more processors, wherein the memory stores a computer program, and when the computer program is executed by the one or more processors, the method described in the first aspect is implemented.

[0032] Compared with the prior art, one or more embodiments of the present invention can at least bring the following beneficial effects:

[0033] The method provided in this embodiment obtains a first segmentation result of the target organ and the background, and a second segmentation result of the target organ lesion and the background through a first neural network model and a second neural network model, respectively, and integrates two different neural network structures in one neural network framework to segment the organs and the lesions respectively, and integrates the segmentation results of the two neural network structures to obtain the final overall segmentation result, thereby solving the image segmentation problem caused by the uneven size of the organs and the lesions in the organs, and avoiding the situation where the lesions may disappear when the organs and lesions are segmented. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 is a flow chart of a medical image segmentation method provided by an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of an image segmentation model provided by an embodiment of the present invention;

[0037] Figure 3 is a schematic diagram of a first neural network model provided by an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of a second neural network model provided by an embodiment of the present invention;

[0039] Figure 5 A residual neural network module using a BasicBlock structure provided by an embodiment of the present invention;

[0040] Figure 6 It is a schematic diagram of the training process of the image segmentation model provided by an embodiment of the present invention;

[0041] Figure 7 is another schematic diagram of the training process of the image segmentation model provided by an embodiment of the present invention;

[0042] Figure 8 is a schematic diagram of a medical image segmentation device provided by an embodiment of the present invention;

[0043] Fig. 9 is a schematic diagram of an electronic device provided by an embodiment of the present invention;

[0044] Fig.10 It is a comparison between the segmentation result of the image to be segmented and the true label of the image segmentation model provided in the embodiment of the present invention, wherein (a) to (h) are the comparison results corresponding to different organs and lesions respectively. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0046] Since the sizes of the background, organs (such as the liver) and lesions in the organs vary greatly, using a single network structure to segment the organs and lesions may cause the lesions to disappear. Therefore, an embodiment of the present invention integrates two different neural network structures in a neural network framework to segment the organs and lesions respectively, and integrates the segmentation results of the two neural network structures to obtain the final overall segmentation result.

[0047] Embodiment 1

[0048] Figure 1 A flow chart of a medical image segmentation method is shown in FIG. Figure 1 As shown, this embodiment provides a medical image segmentation method, including steps S110 to S120:

[0049] Step S110, obtaining a medical image to be segmented;

[0050] Step S120, using the trained image segmentation model to segment the medical image to be segmented, respectively obtaining a first segmentation result of the target organ and the background and a second segmentation result of the target organ lesion and the background, and superimposing the first segmentation result and the second segmentation result and performing feature fusion to obtain a third segmentation result of the background, the target organ, and the target organ lesion;

[0051] The image segmentation model includes: a first neural network model whose input is the medical image to be segmented and whose output is the first segmentation result, and a second neural network model whose input is the medical image to be segmented and whose output is the second segmentation result.

[0052] The method provided in this embodiment obtains a first segmentation result of the target organ and the background, and a second segmentation result of the target organ lesion and the background through a first neural network model and a second neural network model, respectively, and integrates two different neural network structures in one neural network framework to segment the organs and the lesions respectively, thereby realizing the simultaneous segmentation of the organs and the lesions inside the organs, and integrating the segmentation results of the two neural network structures to obtain the final overall segmentation result, thereby solving the image segmentation problem caused by the uneven size of the organs and the lesions inside the organs, and avoiding the situation where the lesions may disappear when the organs and lesions are segmented.

[0053] In some implementations, the first neural network model includes a first Res-UNetPlus model formed by replacing the convolutional layer of the U-Net model with a residual neural network module and inserting a jump connection portion into the residual neural network module.

[0054] This embodiment replaces all convolution blocks with residual neural network modules on the basis of the traditional U-type neural network, and also inserts the jump connection part into the residual neural network module. By replacing the convolution layer with the residual neural network module, the complexity of the first Res-UNet Plus model is improved. By improving the traditional U-type neural network structure and organically combining it with the residual neural network, the degradation and gradient vanishing problems of the traditional U-type neural network model are improved, and the Dice accuracy of the first Res-UNet Plus model and the segmentation accuracy of the organ region are improved. At the same time, the residual neural network module is added to the jump connection of the U-type neural network, which deepens the memory of the shallow features of the segmented image and further improves the segmentation accuracy of the organ region. Experiments show that the traditional U-type neural network has gradient vanishing after training 8,000 times, and the image segmentation model of the present invention still has no signs of gradient vanishing after training 50,000 times.

[0055] Since the model size of the traditional U-shaped neural network is relatively small, it is impossible to train all CT slices. The image segmentation model of this embodiment adopts a heart-shaped composite neural network structure, which not only realizes the simultaneous training of all CT slices, but also realizes the simultaneous segmentation of the liver and the lesions inside the liver.

[0056] In some implementations, the second neural network model includes a second Res-UNetPlus model and a third Res-UNet Plus model with jump connections, and some feature maps of the first Res-UNet Plus model are jump-connected to the second Res-UNetPlus model and the third Res-UNet Plus model. Wherein, the second Res-UNet Plus model and the third Res-UNet Plus model are both models formed by replacing the convolutional layer of the U-Net model with a residual neural network module and partially inserting the jump connection into the residual neural network module; the second Res-UNet Plus model and the third Res-UNet Plus model both have less than 4 pooling layers.

[0057] Since the size of liver lesions is relatively small, four pooling processes will cause the features of tiny lesions to disappear, which is not conducive to the segmentation of liver lesions. Therefore, the second neural network model of this embodiment is based on the Res-UNet Plus neural network, and two Res-UNetPlus models are connected end to end to form an M-shaped double U-shaped network structure, and by reducing the pooling layers of the second Res-UNet Plus model and the third Res-UNet Plus model to less than 4, for example, to 2 pooling layers, a relatively shallow network structure is formed, the depth and complexity of a single Res-UNetPlus model in the M-shaped structure are reduced, and the computing speed is improved; at the same time, since the number of network layers is relatively shallow, it is easy to cause the inability to extract sufficient features for lesion segmentation, and this embodiment connects two Res-UNet Plus models with reduced pooling layers end to end to form an M shape to achieve the effect of increasing the depth of the neural network.

[0058] In some implementations, the shallow feature map of the downsampled part in the first Res-UNet Plus model is jump-connected to the pooling layer of the second Res-UNet Plus model, and the deep feature map of the upsampled part in the first Res-UNet Plus model is jump-connected to the deconvolution layer of the third Res-UNet Plus model.

[0059] The medical images to be segmented in this embodiment include CT images. Since there are too many background pixels in CT images, it is very difficult to extract organ lesions at the global level. Therefore, this embodiment jumps part of the shallow information and part of the deep information in the first Res-UNet Plus model to the shallow feature map and deep feature map of the M-shaped network structure of the second neural network model to enhance information fusion, increase the weight of the liver area, and improve the segmentation effect of liver lesions. In some cases, the second Res-UNetPlus model and the third Res-UNet Plus model of the second neural network model can also be connected through a residual neural network module to perform residual processing to improve model accuracy.

[0060] In some cases, using a residual neural network module to superimpose the first segmentation result and the second segmentation result and perform feature fusion can make the fused result more consistent with the actual situation.

[0061] The image segmentation model of this embodiment is implemented by a deep learning method, which is improved on the basis of the traditional U-shaped neural network to form a composite neural network architecture (Heart Net) with a heart-shaped structure. Figure 2 As shown in Figure 2, the neural network structure mainly consists of two parts: Figure 3 The first neural network model shown and Figure 4 The second neural network model shown; wherein the first neural network model is a Res-UNet model in which the U-Net convolutional layer is replaced with a residual neural network module, and the skip connection part is also inserted with a residual neural network module to form a first Res-UNet Plus model; the second neural network model is to connect two Res-UNet Plus models (the second Res-UNet Plus model and the third Res-UNet Plus model) with reduced downsampling times end to end to form an M-shaped neural network structure. The first Res-UNetPlus model is used for semantic segmentation of organs (such as the liver), and the M-shaped second neural network model is used for semantic segmentation of lesions (such as liver tumors). At the same time, the skip connection method is also used to connect some shallow feature maps and some deep feature maps in the first neural network model to the M-shaped second neural network model, providing more reference information for liver lesion segmentation, so that it can better segment the liver and liver lesions. Among them, the image segmentation model of this embodiment uses the sum of the cross entropy loss CE loss and the set similarity metric function loss Dice loss as the loss function algorithm of the neural network.

[0062] The residual neural network module in this embodiment can adopt the BasicBlock structure, such as Figure 5As shown, two 3*3 convolutional layers are used, but are not limited to the BasicBlock structure.

[0063] In some implementations, such as Figure 6 As shown, the training process of the above image segmentation model includes:

[0064] Step S610: Acquire a medical image dataset.

[0065] For example, a medical image dataset of 70 cases can be collected and organized, including part of the 3Dircadb dataset and part of the LiTS17 dataset.

[0066] Step S620: divide the medical image dataset into a training set, a validation set, and a test set according to a preset ratio.

[0067] Furthermore, taking the preset ratio of 8:1:1 as an example, the medical image data set of 70 cases is divided into a training set, a validation set and a test set in a ratio of 8:1:1, wherein the training set has 56 items, the validation set has 7 items and the test set has 7 items. The training set is used for training the image segmentation model, the validation set is used to optimize the training parameters of the image segmentation model, and the test set is used to test the final tuning result of the image segmentation model.

[0068] Step S630: input N consecutive medical images in the training set into the image segmentation model for model training, and use the mask image of the middle medical image among the N consecutive medical images as the true label.

[0069] In practical applications, before model training, the input N consecutive medical images are first zero-meaned and normalized, where N can be 3, that is, three consecutive CT images are superimposed together as the input of the image segmentation model, and the mask image of the background, liver, and liver lesions in the middle image of the three consecutive CT images is used as the true label to calculate the loss of the image segmentation model output. At this time, if only CT slices containing the liver are input for training, then slices that do not contain the liver are prone to mis-segmentation, for example, organs such as the heart will be identified as the liver.

[0070] In order to improve the robustness of the segmentation results, the CT slices containing the liver and the CT slices not containing the liver are input into the image segmentation model of this embodiment for training together. Due to its specific model structure, the image segmentation model of this embodiment has a complexity sufficient to carry the task of inputting the CT slices containing the liver and the CT slices not containing the liver into the model for training together, so that the trained model can be well used to segment organs, lesions in organs, and backgrounds, and the segmentation precision and accuracy are significantly improved.

[0071] In some cases, a mask image of a middle medical image among N consecutive medical images is used as a true label, including: One-Hot encoding the background, target organ, and target organ lesion mask image of the middle medical image among N consecutive medical images as the true label.

[0072] Step S640, inputting the medical images in the verification set into the image segmentation model for model verification;

[0073] Step S650: If the loss between the output obtained by model verification and the true label is less than the first set threshold, or the Dice coefficient of the output obtained by model verification reaches the second set threshold, the trained image segmentation model is output.

[0074] In this embodiment, the acquired medical image data set is divided into a training set, a validation set and a test set according to a preset ratio; then N consecutive medical images in the training set are input into an image segmentation model for model training, wherein a mask image of a middle medical image among the N consecutive medical images is used as a true label; the medical images in the validation set are input into the image segmentation model for model verification to verify whether the loss between the current model output and the true label is within an allowable range, and then determine whether to output the trained image segmentation model; if the loss between the output obtained by the model verification and the true label is less than a first set threshold, or the Dice coefficient of the output obtained by the model verification reaches a second set threshold, then the trained image segmentation model is output; on the contrary, if the loss between the output obtained by the model verification and the true label is not less than the first set threshold, or the Dice coefficient of the output obtained by the model verification does not reach the second set threshold, then the training continues.

[0075] In practical applications, the expression of Dice coefficient (DSC) can be as follows:

[0076]

[0077] It can be understood that the larger the Dice coefficient is, the smaller the loss value calculated by the loss function of the set similarity measurement function is.

[0078] In some implementations, such as Figure 7 As shown, the training process of the image segmentation model further includes:

[0079] Step S660: input the medical images in the test set into the trained image segmentation model for model testing;

[0080] Step S670: if the Dice coefficient outputted from the model test reaches a third set threshold (for example, the Dice coefficient reaches a maximum), retain the image segmentation model;

[0081] Step S680: If the Dice coefficient outputted from the model test does not reach the third set threshold, the parameters of the image segmentation model are adjusted for training again, and step S630 of inputting N consecutive medical images in the training set into the image segmentation model for model training is executed.

[0082] exist Figure 3 In the first neural network model shown, the original CT image 110 to be segmented is input into the first neural network model 120 to obtain the image output 140 (predicted label), the convolutional layers of the downsampling part 1201 and the upsampling part 1203 are replaced with residual neural network modules, and the skip connection part 1202 is also inserted with a residual neural network module.

[0083] exist Figure 4 In the second neural network model shown, the original CT image 110 to be segmented is input into the M-shaped second neural network model to obtain the corresponding image output, wherein the pooling layers of the second Res-UNet Plus model and the third Res-UNet Plus model with jump connections are reduced to 2 layers, and the jump connection part between the second Res-UNet Plus model and the third Res-UNetPlus model is also inserted with a residual neural network module.

[0084] In some implementations, model training uses SGD (stochastic gradient descent) as an optimizer to optimize the image segmentation model parameters, with a learning rate of 0.01. The loss function of the image segmentation model consists of three parts:

[0085] One is the loss function of comparing the segmentation result output by the first neural network model with the true label. The loss function takes the form of the sum of the loss function (Dice) of the set similarity measurement function and the cross entropy loss function (CE).

[0086] The loss function formula of the set similarity measurement function is:

[0087]

[0088] Where n represents the total number of samples, x represents the sample, the set sum A(x) is the true label, the set sum B(x) is the predicted label obtained by the generator segmentation, |A(x)∩B(x)| represents the common elements between the set sum A(x) and the set sum B(x), |A(x)| represents the number of elements in A, |B(x)| represents the number of elements in B, and s represents the smoothing coefficient. Preferably, the value is s=1e -5 The Dice calculated at this time is more accurate.

[0089] The cross entropy loss function formula is:

[0090]

[0091] Among them, n represents the total number of samples, x represents the sample, the set sum A(x) is the true label, and the set sum B(x) is the predicted label obtained by the generator segmentation.

[0092] Furthermore, the loss function expression of the first neural network model is as follows:

[0093] loss 1 =loss Dice_1 +loss CE_1 (3)

[0094] Among them, loss Dice_1 Represents the loss of the set similarity measurement function of the organ region, loss CE_1 Represents the cross entropy loss of the organ region.

[0095] The second is the loss function of comparing the segmentation result output by the second neural network model with the true label, which is expressed as follows:

[0096] loss 2 =loss Dice_2 +loss CE_2 (4)

[0097] Among them, loss Dice_2 Represents the loss of the set similarity measurement function of the organ lesion area, loss CE_2 Represents the cross entropy loss of the organ lesion area.

[0098] The third is the loss function of comparing the overall segmentation result (third segmentation result) after the fusion of the organ and organ lesion segmentation results with the true label, which is expressed as follows:

[0099] loss all =loss Dice_all +loss CE_all (5)

[0100] Among them, loss Dice_all Represents the loss of the set similarity measurement function of organs and organ lesion regions, loss CE_all Represents the cross entropy loss between organs and organ lesion regions.

[0101] The final loss function expression of the image segmentation model of this embodiment is obtained as follows:

[0102] loss = α × loss 1 +β×loss 2 +γ×loss all (6)

[0103] Among them, loss represents the loss of the image segmentation model, loss1 Represents the first loss loss1 between the first segmentation result and the true label, loss 2 Represents the second loss loss2 between the second segmentation result and the true label, loss all represents the third loss loss3 between the third segmentation result and the true label, α, β, and γ represent the weights of the first loss loss1, the second loss loss2, and the third loss loss3, respectively; in practical applications, α=1.0, β=1.0, and γ=1.1 can be set according to experience, and the trained image segmentation model has a better effect.

[0104] In an example, under the same image preprocessing conditions, the above image segmentation model (Heart Net), the first Res-UNet Plus model (Res-UNet Plus) and the U-Net neural network structure were tested using the test set, respectively. The test indicators included the set similarity measurement index (Dice Similariy Coefficient, DSC), the intersection over Union index (Intersection over Union), the true positive rate index (True Positive Rate, TPR) and the positive predictive value index (Positive Predictive Value, PPV). The test results of each average index are shown in Table 1.

[0105] Table 1 Test results of average indicators of different network structures

[0106]

[0107] It can be seen that the image segmentation model of this embodiment is superior to the other two network structures in all indicators. The average set similarity metric index of the liver of the Heart Net neural network test set proposed by the present invention is 91.52%, and the average set similarity metric index of the liver lesions is 55.04%. However, the traditional U-type neural network cannot correctly detect the liver and liver lesions, and the average set similarity metric index is less than 10%.

[0108] The image segmentation model of this embodiment compares the segmentation result of the image to be segmented with the true label as follows: Fig.10 As shown, (a) to (h) are the comparison results corresponding to different organs and lesions. It can be seen that the image segmentation model of this embodiment has good segmentation accuracy and effect.

[0109] Embodiment 2

[0110] Corresponding to the first embodiment, this embodiment provides a medical image segmentation device, such as Figure 8 As shown, including:

[0111] An acquisition module 810 is used to acquire a medical image to be segmented;

[0112] The segmentation module 820 is used to segment the medical image to be segmented using the trained image segmentation model, obtain a first segmentation result of the target organ and the background and a second segmentation result of the target organ lesion and the background, and superimpose the first segmentation result and the second segmentation result and perform feature fusion to obtain a third segmentation result of the background, the target organ and the target organ lesion;

[0113] Among them, the image segmentation model includes a first neural network model and a second neural network model. The input of the first neural network model includes the medical image to be segmented, and the output includes the segmentation result of the target organ and the background. The input of the second neural network model includes the medical image to be segmented, and the output includes the segmentation result of the target organ lesion and the background.

[0114] The device provided in this embodiment obtains a first segmentation result of a target organ and a background, and a second segmentation result of a target organ lesion and a background through a first neural network model and a second neural network model, respectively, and integrates two different neural network structures in one neural network framework to segment organs and lesions respectively, and integrates the segmentation results of the two neural network structures to obtain a final overall segmentation result, thereby solving the image segmentation problem caused by uneven sizes of organs and lesions in the organs, and avoiding the situation where the lesions may disappear when segmenting the organs and lesions.

[0115] In some implementations, the first neural network model includes a first Res-UNetPlus model formed by replacing the convolutional layer of the U-Net model with a residual neural network module and inserting a jump connection portion into the residual neural network module.

[0116] This embodiment replaces all convolution blocks with residual neural network modules on the basis of the traditional U-type neural network, and inserts the jump connection part into the residual neural network module. By replacing the convolution layer with the residual neural network module, the complexity of the first Res-UNetPlus model is improved, thereby avoiding gradient vanishing, improving the Dice accuracy of the first Res-UNet Plus model and the segmentation accuracy of the organ region. At the same time, the residual neural network module is added to the jump connection of the U-type neural network, which deepens the memory of the segmented image for shallow features and further improves the segmentation accuracy of the organ region.

[0117] In some implementations, the second neural network model includes a second Res-UNetPlus model and a third Res-UNet Plus model with jump connections, and some feature maps of the first Res-UNet Plus model are jump-connected to the second Res-UNetPlus model and the third Res-UNet Plus model; wherein the second Res-UNet Plus model and the third Res-UNet Plus model are models formed by replacing the convolutional layers of the U-Net model with residual neural network modules and partially inserting the jump connection into the residual neural network modules; the second Res-UNet Plus model and the third Res-UNet Plus model have less than 4 pooling layers.

[0118] In some implementations, the shallow feature map of the downsampled part in the first Res-UNet Plus model is jump-connected to the pooling layer of the second Res-UNet Plus model, and the deep feature map of the upsampled part in the first Res-UNet Plus model is jump-connected to the deconvolution layer of the third Res-UNet Plus model.

[0119] In some cases, using a residual neural network module to superimpose the first segmentation result and the second segmentation result and perform feature fusion can make the fused result more consistent with the actual situation.

[0120] The residual neural network module in this embodiment can adopt the BasicBlock structure and use two 3*3 convolutional layers, but is not limited to the BasicBlock structure.

[0121] In some implementations, the device further includes a training module for training the image segmentation model, including:

[0122] Get medical image datasets;

[0123] Divide the medical image dataset into training set, validation set and test set according to a preset ratio;

[0124] Input N consecutive medical images in the training set into the image segmentation model for model training, and use the mask image of the middle medical image in the N consecutive medical images as the true label;

[0125] Inputting the medical images in the validation set into the image segmentation model for model validation; and

[0126] If the loss between the output obtained by model verification and the true label is less than the first set threshold, or the Dice coefficient of the output obtained by model verification reaches the second set threshold, the trained image segmentation model is output.

[0127] In some cases, a mask image of a middle medical image among N consecutive medical images is used as a true label, including: One-Hot encoding the background, target organ, and target organ lesion mask image of the middle medical image among N consecutive medical images as the true label.

[0128] In some implementations, the training module trains the image segmentation model, and further includes:

[0129] Input the medical images in the test set into the trained image segmentation model for model testing;

[0130] If the Dice coefficient outputted from the model test reaches a third set threshold (for example, the Dice coefficient reaches a maximum), retaining the image segmentation model; and

[0131] If the Dice coefficient outputted from the model test does not reach the third set threshold, the parameters of the image segmentation model are adjusted and trained again, and N consecutive medical images in the training set are input into the image segmentation model for model training.

[0132] The loss function of the image segmentation model of this embodiment is expressed as follows:

[0133] loss = α × loss 1 +β×loss 2 +γ×loss all

[0134] Among them, loss represents the loss of the image segmentation model, loss 1 Represents the first loss between the first segmentation result and the true label, loss 2 Represents the second loss between the second segmentation result and the true label, loSs all represents the third loss between the third segmentation result and the true label, α, β, and γ represent the weights of the first loss, the second loss, and the third loss, respectively. In practical applications, α = 1.0, β = 1.0, and γ = 1.1 can be set according to experience, and the effect of the trained image segmentation model is better.

[0135] Those skilled in the art should understand that the above modules or steps can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any limited combination of hardware and software.

[0136] Embodiment 3

[0137] This embodiment provides a storage medium, on which a computer program is stored. When the computer program is executed by one or more processors, the medical image segmentation method of the first embodiment is implemented.

[0138] In this embodiment, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk. The content of the method is detailed in Example 1 and will not be repeated here.

[0139] Embodiment 4

[0140] This embodiment provides an electronic device, including a memory and one or more processors. The memory stores a computer program, and when the computer program is executed by the one or more processors, the medical image segmentation method of the first embodiment is implemented.

[0141] like Fig. 9 As shown, the medical image segmentation method provided by the present invention can be executed by an electronic device 200. The electronic device 200 can be a terminal, such as a tablet computer, a personal computer, a smart phone, etc.; the electronic device can also be a server, such as a cloud server, a network server, etc.

[0142] Exemplarily, the electronic device 200 may include: a communication module 210 for connecting and communicating with the outside world, different types of memory 220, a processor 230 for executing program instructions, and a bus 240. The memory 220 may be a read-only memory, a random access memory, or any combination of the two. The bus 240 includes a data bus, an address bus, and a control bus.

[0143] Exemplarily, the processor 230 can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components to execute the methods in the above embodiments.

[0144] Exemplarily, computer programs are stored in the memory 220. The processor 230 can call and execute these programs from the memory 220, so as to perform the medical image segmentation method of the first embodiment. By executing the medical image segmentation method, the processor 230 can segment the medical image, thereby outputting the segmented medical image.

[0145] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed system and method can also be implemented in other ways. The system and method embodiments described above are merely illustrative.

[0146] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0147] Although the embodiments disclosed in the present invention are as above, the above contents are only embodiments adopted for facilitating the understanding of the present invention and are not intended to limit the present invention. Any technician in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present invention, but the patent protection scope of the present invention shall still be subject to the scope defined in the attached claims.

Claims

1. A medical image segmentation method, characterized in that: include: Acquire a medical image to be segmented; Segmenting the medical image to be segmented using the trained image segmentation model to obtain a first segmentation result of the target organ and the background and a second segmentation result of the target organ lesion and the background, respectively, and superimposing the first segmentation result and the second segmentation result and performing feature fusion to obtain a third segmentation result of the background, the target organ, and the target organ lesion; The image segmentation model includes: a first neural network model whose input is the medical image to be segmented and whose output is a first segmentation result, and a second neural network model whose input is the medical image to be segmented and whose output is a second segmentation result; The first neural network model includes a first Res-UNet Plus model formed by replacing the convolutional layer of the U-Net model with a residual neural network module and inserting a jump connection part into the residual neural network module; The second neural network model includes a second Res-UNet Plus model and a third Res-UNet Plus model connected by jump connections, and some feature graphs of the first Res-UNet Plus model are jump-connected to the second Res-UNet Plus model and the third Res-UNet Plus model; Among them, the second Res-UNet Plus model and the third Res-UNet Plus model are models formed by replacing the convolutional layer of the U-Net model with a residual neural network module and inserting the jump connection part into the residual neural network module; the second Res-UNet Plus model and the third Res-UNet Plus model have less than 4 pooling layers.

2. The medical image segmentation method according to claim 1, characterized in that: The shallow feature map of the down-sampling part in the first Res-UNet Plus model is jump-connected to the pooling layer of the second Res-UNet Plus model, and the deep feature map of the up-sampling part in the first Res-UNet Plus model is jump-connected to the deconvolution layer of the third Res-UNetPlus model.

3. The medical image segmentation method according to claim 1, characterized in that: The training process of the image segmentation model includes: Get medical image datasets; Dividing the medical image dataset into a training set, a validation set and a test set according to a preset ratio; Inputting N consecutive medical images in the training set into the image segmentation model for model training, and using a mask image of a middle medical image among the N consecutive medical images as a true label; Inputting the medical images in the verification set into the image segmentation model for model verification; If the loss between the output obtained by model verification and the true label is less than the first set threshold, or the Dice coefficient of the output obtained by model verification reaches the second set threshold, the trained image segmentation model is output.

4. The medical image segmentation method according to claim 3, characterized in that: The training process of the image segmentation model also includes: Inputting the medical images in the test set into the trained image segmentation model for model testing; If the Dice coefficient outputted from the model test reaches the third set threshold, the image segmentation model is retained; If the Dice coefficient outputted from the model test does not reach the third set threshold, the parameters of the image segmentation model are adjusted for training again, and the N consecutive medical images in the training set are input into the image segmentation model for model training.

5. The medical image segmentation method according to claim 3, characterized in that: The loss function of the image segmentation model is expressed as follows: loss=α×loss1+β×loss2+γ×loss all Among them, loss represents the loss of the image segmentation model, loss1 represents the first loss between the first segmentation result and the true label, loss2 represents the second loss between the second segmentation result and the true label, and loss all represents the third loss between the third segmentation result and the true label, α, β, and γ represent the weights of the first loss, the second loss, and the third loss, respectively.

6. A medical image segmentation device, characterized in that: include: An acquisition module, used for acquiring a medical image to be segmented; a segmentation module, used to segment the medical image to be segmented using a trained image segmentation model, to obtain a first segmentation result of a target organ and a background and a second segmentation result of a target organ lesion and a background, and to superimpose the first segmentation result and the second segmentation result and perform feature fusion to obtain a third segmentation result of the background, the target organ and the target organ lesion; The image segmentation model includes: a first neural network model whose input is the medical image to be segmented and whose output is a first segmentation result, and a second neural network model whose input is the medical image to be segmented and whose output is a second segmentation result; The first neural network model includes a first Res-UNet Plus model formed by replacing the convolutional layer of the U-Net model with a residual neural network module and inserting a jump connection part into the residual neural network module; The second neural network model includes a second Res-UNet Plus model and a third Res-UNet Plus model connected by jump connections, and some feature graphs of the first Res-UNet Plus model are jump-connected to the second Res-UNet Plus model and the third Res-UNet Plus model; Among them, the second Res-UNet Plus model and the third Res-UNet Plus model are models formed by replacing the convolutional layer of the U-Net model with a residual neural network module and inserting the jump connection part into the residual neural network module; the second Res-UNet Plus model and the third Res-UNet Plus model have less than 4 pooling layers.

7. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 5 is implemented.

8. An electronic device, characterized in that: The method comprises a memory and one or more processors, wherein a computer program is stored in the memory, and when the computer program is executed by the one or more processors, the method according to any one of claims 1 to 5 is implemented.

Images